MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models
Authors: Jeffrey A Chan Santiago
praveen tirupattur
Gaurav Kumar Nayak
Gaowen Liu
Mubarak Shah
Venue: ICML 2025 oral (2025)
Keywords: Dataset distillation
Dataset Condensation
Diffusion Models
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Dataset distillation has emerged as an effective strategy, significantly reducing training costs and facilitating more efficient model deployment.
Recent advances have leveraged generative models to distill datasets by capturing the underlying data distribution. Unfortunately, existing methods require model fine-tuning with distillation losses to encourage diversity and representativeness. However, these methods do not guarantee sample diversity, limiting their performance.
We propose a mode-guided diffusion model leveraging a pre-trained diffusion model without the need to fine-tune with distillation losses. Our approach addresses dataset diversity in three stages: Mode Discovery to identify distinct data modes, Mode Guidance to enhance intra-class diversity, and Stop Guidance to mitigate artifacts in synthetic samples that affect performance.
We evaluate our approach on ImageNette, ImageIDC, ImageNet-100, and ImageNet-1K, achieving accuracy improvements of 4.4%, 2.9%, 1.6%, and 1.6%, respectively, over state-of-the-art methods. Our method eliminates the need for fine-tuning diffusion models with distillation losses, significantly reducing computational costs.

MGD Mode-Guided Dataset Distillation using Diffusion Models
Jeffrey A. Chan-Santiago' Praveen Tirupattur'
Gauray Kumar Nayak Gaowen Liu
Mubarak Shah
Abstract
Optimization basell Dataset Distillation
Dataset distillation has emerged as an effective strategy, significantly reducing training costs and facilitating more efficient model deployment. Re- cent advances have leveraged generative mod- els to distill datasets by capturing the underly- ing data distribution. Unfortunately, existing methods require model fine-tuning with distil- lation losses to encourage diversity and repre- sentativeness. However, these methods do not guarantee sample diversity, limiting their perfor- mance. We propose a mode-guided diffusion model leveraging a pre-trained diffusion model without the need to fine-tune with distillation losses. Our approach addresses dataset diver- sity in three stages: Mode Discovery to iden- tify distinet data modes, Mode Guidance en- hance intra-class diversity, and Stop Guidance mitigate artifacts in synthetic samples that affect performance. Our approach outperforms state- oh-the-art methods, achieving accuracy gains of 4.4%, 2.9%, 1.6%, and 1.6% on ImageNette, ImagelDC, ImageNet-100 and ImageNet- 1K, respectively. Our method eliminates the need for fine-tuning diffusion models with distilla- tion losses, significantly reducing computational costs. Our code is available on the project webpage: http://ccccannniiggggiuu.b..bb.b didtedddistillation/
T
Matching
Dataset
S
Disilldd
Original Dutaset
Gradiem How
Generative Dataset Distillation
T
Cirmenative
Generative
Original Dataset
Model
Model
Dataset Distribution Learning
Dataset synthests
Figure I. Optimization-basee Dataset Distillation: Optimizes the distilled dataset to match the statistics of gradient/features of the Original Dataset. Generative Dataset Distillation: First, it learns the dataset distribution of the original dataset and then sam- ple a dataset that approximates the original dataset distribution.
presents significant challenges for researchers constrained by limited computation and storage resources. In response, the research community started to focus on developing tech- niques to address these limitations. While model pruning (Liu et al., 2017; He et al., 2019; Ding et al., 2019; Sharma & Foroosh, 2022) and quantization (Wu et al., 2016; Chen et al., 2021; Chauhan al., 2023; Xu al., 2023) have been introduced to improve model efficiency, core selection and dataset distillation (Wang et al., 2018; Liu al., 2022) have emerged prominent techniques for reducing the size of training datasets to accelerate model training.
The process of reducing the training dataset involves remov- ing redundant information while retaining essential data. Core set selection (Welling, 2009; Chen et al., 2010; Rebuffi et al., 2017; Castro et al., 2018) based approaches were initially introduced for building condensed datasets, which involves selecting few prototypical examples from the original dataset to build the smaller dataset. However, these approaches are limited to choosing the samples from the original dataset, which considerably restricts the expressive- ness of the condensed dataset. The task of dataset distillation is distill information from large training dataset into smaller dataset with few synthetic samples such that model trained on the smaller dataset achieves performance comparable to the model trained on the original dataset.
1. Introduction
The rapid advancements in machine learning are marked by a trend towards increasingly large datasets and models to achieve state-of-the-arr performance. However, this trend
CCenter for Research Computer Vision, University of Central Florida, Orlando, Florida, United States MMeha Family School of Data Science and Artificial Intelligence, Indian Institute of Tech- nology Roorkee, Roorkee, Utarrkkann. India *Cisco Research, San Jose, California, United States. Correspondence to: Jeffrey A. Chan-Saaniann <jeffrey chansantiago@ ucf.edu>.
Optimization- prasa dataset distillation methods follow the data matching framework (Cazenavette et al., 2022; 2023; Zhao Bilen, 2023), where the distilled dataset updated
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mode-Guided Dataset Distillation using Diffusion Models
to mimic the influence of the original dataset when train- ing (see the top of Fig. 1). These methods minimize the distribution gap between the original and distilled datasets by considering different aspects, such as model parame- ters, long-range training trajectories, or feature distribution. However, these methods are far from optimal, as they need to repeat the execution of their method to synthesize dis- tilled datasets of different sizes. In addition, they tend to generate out-of-distribution samples.
2. Related Work
Dataset distillation has received increased interest in recent years due to its applications in continual learning (Zhao et al., 2021; Zhao & Bilen, 2021; 2023), privacy-preserving datasets (Li et al., 2020; Sucholutsky & Schonlau, 2021), neural architecture search (Zhao et al., 2021; Zhao & Bilen, 2021), and model explainability (Loo et al., 2022). Prior works have explored the problem of dataset distillation and shown how challenging it is to encapsulate datasets in a limited set of examples. Initially, this task was approached using non-generative models, then with generative priors, and more recently with generative models and with decou- pled dataset distillation. Below, we discuss works belonging to these categories in detail.
To address these challenges, generative dataset distillation methods (Wang et al., 2023: Zhang et al., 2023; Su et al., 2024) propose storing the knowledge of the dataset into the parameters of a generative model instead of directly con- densing it into a smaller synthetic set (see the bottom of Fig. 1). Once trained, the same generative model can generate synthetic datasets of varied sizes. This typically, is achieved by training the generative model with representative and diversity losses.
Non-generative Dataset Distillation Methods. Dataset dis- tillation condenses information from a large dataset into a smaller one with synthetic images, enabling model training on the smaller dataset with performance comparable the full dataset. Initially, Zhao et al. (2021) proposed gradient matching to align the model's gradient trained synthetic data with that on the original dataset. However, this bi-level optimization approach was time-consuming and unscalable. Further advances included feature matching (Zhao & Bilen, 2023), which improved efficiency by removing dependence on bi-level optimization. Later, Cazenavette et al. (2022) proposed long-range matching by matching training trajec- tories (MTT), optimizing network parameters over multiple training iterations to better synthesize relevant features for updates.
Among the generative models, diffusion models (Ho et al., 2020) are known for their impressive capabilities in image synthesis. These models achieve perceptual quality compa- rable to GANs while offering higher distribution coverage, as evidenced by (Dhariwal & Nichol, 2021b). However, they tend to concentrate on denser regions (modes) of the data distribution, resulting in a synthetic dataset that, while representative, often lacks the full diversity of the original data (Gu et al., 2024) (refer to Fig. 2a). Previous works (Gu et al., 2024) address this by explicitly fine-tuning the model with representative and diversity losses to generate representative and diverse samples. With this fine-tuning, the samples are more likely to be generated from different modes of a class (See Fig. 2b). However, this approach requires additional training, which can be computationally expensive and limit its practicality in resource-constrained settings.
Dataset Distillation with Generative Priors. Recent ad- vancements have introduced generative priors into the op- timization process. GAN-IT (Zhao & Bilen, 2022) shifted the focus from the pixel space to latent codes of pre-trained GANs, optimizing these codes rather than working directly in image space. GLaD (Cazenavette et al., 2023) built on this by incorporating generative priors with StyleGAN for high-resolution datasets, yielding images that more closely match the dataset distribution and improve performance. H-GLaD (Zhong et al., 2024) further enhanced this by fo- cusing deeper feature layers for hierarchical optimization. Additionally, LD3M (Moser et al., 2024) utilized latent diffusion model to optimize synthetic datasets directly in the model's latent space, improving performance by refin- ing latent codes through the denoising and diffusion pro- cesses. Despite their success on small-resolution datasets, these methods struggle with high- nigh-resolution datasets (e.g.. 256 256, 20 images per class), often being computation- ally expensive and less efficient, leading to the emergence of more effective distillation methods from generative models.
We propose novel approach that extracts diverse and repre- sentative samples from a pre-trained diffusion model trained on the target dataset, without additional training or fine- tuning. Our method first estimates prevalent data modes in the Mode Discovery stage. Then, diversity is ensured by guiding each sample to a different mode with Mode Guidance. However, guiding samples to modes may re- duce quality, so we introduce Stop Guidance to preserve synthetic data quality (see Fig. 2c). In summary, the key contributions are:
A novel dataset distillation approach leveraging pre- trained diffusion model without retraining or fine-tuning,
Improved diversity and representativeness compared to previous diffusion-basse methods.
Generative Dataset Distillation Methods. Recent works (Zhang et al., 2023; Gu et al., 2024; Su et al., 2024) have explored dataset distillation via generative models, moving
Matching or surpassing state-of-te ooate----theeate art results on multiple benchmarks while reducing computational cost.
2

---

Mode-Guided Dataset Distillation using Diffusion Models
Original Data
.VO,11 1)
Generated Image
Denoising
Mode
Mode Guided Denoising
(a) DIT No Fine-tuning Low Diversity
(b) MinMax Diffusion Requires Fine-tuning High Diversity
(c) MGD (Ours) No Fine-tuning High Diversity
Figure 2. Overview of the gradient field (score function) during the denoising process in latent diffusion for specific class The original data distribution, marked by blue dots, shows denser regions (orange shadow) in the gradient field. To generate an image X,, noise xT² ~ (0, I) is sampled. In (a), pre-trained diffusion model demonstrates imbalanced mode likelihood, leading to limited sample diversity and repeated modes. (b) shows MinMax Diffusion, which fine-tunes the model to enhance diversity by balancing mode likelihoods. but still faces redundancies based on initial noise conditions. (c), the proposed method introduces mode guidance in the denoising process (green and red traces), directing samples towards distinct modes (stars). After it steps of guidance, transitions to unguided denoising (black trace), achieving high diversity and consistency without the need for fine-tuning.
beyond using generative priors merely to optimize latent codes. Instead, generative dataset distillation trains models to synthesize entire distilled datasets. Zhang al. (2023) introduced a class-condittonal GAN with learnable code- book per image, optimized using multiple losses for realism, representativeness, and diversity. Gu et al. (2024) extended this to diffusion models by fine-tuning a pretrained model with representative and diversity losses. Su et al. (2024) proposed D D*M, M. which uses Stable Diffusion and replaces random noise with noisy modes during sampling; however, early denoising noise often leads to limited diversity and rep- resentativeness. While prior methods rely on complex loss designs and additional training, we propose a training-free approach that achieves both representativeness and diversity without such overhead.
diversity- driven method, distilling ImageNet- 1K into 10 im- ages per class within minutes. Shao et al. (2024b) further explored the design space, introducing soft category-aware matching and optimization strategies such as small batches and adaptive learning rates. In contrast to methods that use discriminative models and image optimization producing artifacts and poorly aligned samples- we train generative model to encode data distribution and re- cover samples via guided sampling, yielding results more consistent with the original dataset.
3. Preliminaries
Dataset Distillation: Given a large-scale dataset with the training set T = {(X,, w)) the goal of dataset distillation to build a smaller synthetic dataset S {(X,, 9:) ]:) where Ns << N7 and X;, X; are the origi- nal and synthetic images with the corresponding class labels Vi Wt- In addition, the model dt trained the original training set should achieve similar test performance as the model Φs trained on the smaller synthetic dataset; i.e. A is the accuracy of a model on the test set (Te), then A(ØT) A(ds). During the evaluation, the size of the distilled dataset Ng. is set based on the distillation budget, denoted by IPC, the number of images allocated per class.
Decoupled Dataset Distillation. Recent advances dataset distillation have introduced decoupled formulations that scale to ImageNet. Yin et al. (2023) proposed SR Squeeze-Recover-ee Relabel framework that: (1) squeezes dataset statistics into model through training, (2) recovers information by optimizing synthetic data to match batch- norm statistics, and (3) boosts performance via soft labels from a pretrained model. Extending this, Shao et al. (2024a) introduced G-VBSM, applying statistical matching con- volutional layers with multi-backbone support, achieving state-of ohethe--rt art results from CIFAR-100 to ImageNet- 1 To improve sample fidelity, Sun et al. (2024) proposed a fast,
Our approach builds the foundations of prior generative models, such as Gu et al. (2024); Su et al. (2024); Zhang
3

---

Mode-Guided Dataset Distillation using Diffusion Models
al. (2023), which address dataset distillation by approx- imating the dataset distribution through sampling diverse and representative instances. This line of work can be char- acterized as dataset distillation through dataset matching. Where the objective of the data distillation is defined as
represents the drift of the diffusion process towards the dis- tribution of the class c. Dhariwal & Nichol (2021a) a) employ classifier to estimate the class-conditional log- likeeiid and used it as a guidance signal to direct the diffusion pro- cess towards the desired class. Later, Ho & Salimans (2021) suggested using a combination of unconditional generation and conditional diffusion (eq. 4) to remove the dependency on the classifier and demonstrated improved results and called this classifier-free guidance. Classifier- free guidance is defined as
[|Be-PDD) [((TT())) (1),, x)] PE- [((0s(1)))) w)] || <
where P(D) denotes the real data distribution, and / is a loss function. Note that this formulation is similar to the coreset methods. However, the use of generative models is more flexible because it's not limited to only choosing original samples.
t, (() co(os,t, t. (1__))co(m,ttt.)) éo (o(,,,
w co co( t), (4)
The denoising probabilistic diffusion model (DDPM) is generative model, g. that learns a mapping between Gaussian noise and the data distribu- tion through series of T denoising steps. G assumes Markov chain that gradually adds noise to sample z0 in the data distribution, which is called the forward pro- cess. The forward process of G is defined as (( = NN - f, 3,1), where 3e is the variance schedule for the time step t. In practice, this is done using the reparametrization trick Ia- = vāzo + vI âec, where 4 (O,,I.
Diffusion Model:
where the is the guidance scale that controls how strong the guidance is applied.
4. Method
We propose a method for generating diverse and representa- tive class samples by harnessing diffusion model trained on the target dataset. The core idea is to sample from the denser regions of the data distribution, known as modes, during the reverse process. These modes correspond to clus- ters of images with similar features and are representative of the class. However, diffusion models often oversample the most prominent modes, which creates redundancies in the distilled dataset, especially when the number of domi- nant modes for class is smaller than the desired number of images per class (IPC).
Image generation is done by the reverse process of G. where Eg is the noise prediction network, trained to re- verse the Markov chain po
(z)) (x)))) where 0 corresponds to the parameters of the model and (x( (xx)) are the μ and E predictions of the denoising models. po(xx) is computed as follows:
Our three-stage approach, shown in Fig. 3, eliminates the need for fine-tuning while preserving mode diversity. In the first stage, mode discovery, we estimate a diverse set of modes for each class in the dataset. The second stage leverages our proposed mode guidance to control the re- verse process and enable sampling from the estimated mode distribution. During sampling, the guidance is applied until the stop guidance-the third stage-is triggered, ensuring control over the quality of the generated samples.
1
1
(o(x1))
v1 3, (xt
VI VI-a eo( + oiz (1)
where (O,, 1) and af is the variance schedule. eo (ate ()
is the output of the noise prediction network that is trained to predict the added noise with the simple loss defined as
Lo | t)- cellz.
After training, G can generate samples by sampling from the noise distribution and running the reverse process. In this work, we use dlass-conditioned diffusion model G., where the output of the noise prediction network conditioned with the class c, is denoted as co(x,,
4.1. Mode Discovery
In the mode discovery stage, the main objective is to iden- tify the N modes of specific class in the original dataset distribution. This discovery is performed using the origi- nal dataset in the latent space of the VAE encoder (Vene). The motivation for this approach is that the generative space captures the overall content of the image rather than discrim- inative features, which can be limited specific textures in the image. Any clustering algorithm can be used to estimate the modes for a particular class. In our experiments, we use K-Means centroids, as they are shown to be effective in our ablations with various mode discovery algorithms (see Appendix Section D). Once the modes are identified, our goal is to sample images from these estimated modes.
The sampling process of DDPM is equivalent to basre- generative models by interpret- ing co (2,, = - Va v, log p(x+), where Vz log p(r.) is an estimation of the score function. For the case of class- conditioned generation, by using Bayes' rule the score func- tion can be derived as:
Diffusion Guidance:
Va logp(</(c) = V logp(z,) + V. logp(c)
(3)
where V. log p(c)x.) is the gradient of the class-conditional log-likelihood. It's important to note that Vz log
4

---

Mode-Guided Dataset Distillation using Diffusion Models
Diffusion Reverse Process (c,, Marget
T r
S
Synthetic Dataset
margett
Training Dataset
VAE Decoder
VAE Encoder
Mode Guided Diffusion
Unguided
gec, M. target)
Stop Guidance
Class
Class
Mode
Unguided X Ex t,
Discovery
Random Noise Xy
Mode Guidance (, (m(argge)))
*
*
Starting Point
Sumpled Image As Previous Sempled Image
Generative Feature Distribution
Modes
Node Guited Denotsing
Unguised Denolsing
Figure 3. Overview of the proposed method for distilled dataset synthesis using diffusion model. Our approach consists of three key stages: Mode Discovery, Mode Guidance, and Stop Guidance. (Lefft) Mode Discovery stage, we estimate the N modes of original dataset within the generative space of the latent diffusion model. (Right) Given mode Maarget and class the Mode-Guidee Diffusion process directs the generation toward the specified mode Miarget. This guidance is applied for tstop steps until the Stop Guidance stage, after which unguided diffusion takes During sampling, mode guidance ensures that images from the desired mode my are generated using the pre-trained diffusion model. If no guidance is applied, the generation follows the unguided (grey) path, which lead to redundancies in the dataset.
4.2. Mode Guidance
4.3. Stop Guidance
At the image synthesis stage, our goal is to generate high- quality images belonging to a specific class mode. Given a class e and a set of discovered modes for that class de- noted as M2 = (m1,..., m the mode guidance score computed for particular mode m, using the following equation:
The reverse diffusion process can be divided into three dis- tinct stages: the chaotic stage (first 20%), the semantic stage (20% to 50%). and the refinement stage (final 50%) (Yu et al., 2023). During the refinement stage, mode guidance becomes unnecessary since its primary purpose is guide the synthetic image towards the mode in the high semantic space. Our initial experiments revealed that maintaining strong guidance towards particular mode m, throughout the full reverse process often compromises class fidelity and introduces image artifacts (See Fig. 7b fstop = 0). To address these issues, we introduce the stop guidance mech- anism, which involves setting the guidance parameter À in Equation 6 when the timestep falls below timestep tatop during the reverse process. In the Appendices A and we examine the effects of different stop guidance timesteps (Lstop) on image generation quality.
g: (m, '
(5)
where 26 is the predicted denoised latent vector at timestep t during the reverse process. We apply this guidance signal at the In, timestep as follows:
(x c) ē (0 c)
g: A: Ot.
where a scalar that controls the strength of the guidance signal.
To synthesize an image from a particular mode m, in the diffusion model the mode guidance score is computed at each iteration of the reverse process using Eq. This score represents the direction from the predicted value to the mode m,, The guidance signal is then added to the noise function at the appropriate time step in the diffusion process. By adjusting the strength the guidance signal, we can regulate the impact of the mode on the generated image.
5. Experiments
Datasets and evaluation. To assess our approach's effec- tiveness, thoroughly examine the available benchmarks for distilling high-resolution datasets (256 x 256). The
5

---

Mode-Guided Dataset Distillation using Diffusion Models
Table Comparison of performance between pre-trained diffusion models and state-of-the-art methods ImageNet subsets, evaluated using the hard-label protocol. Results are based on ResNet- 10 with average pooling, with the best performance highlighted in bold. Accuracy is used as the evaluation metric.
Nette
IDC
IPC
10
20
50
10
20
50
Random
1..2+1.6
63. 40.5
11.1
48. +
52.5
68. 18.1
DM (Zhao & Bilen. 2023)
60.8
66.5
76.2
52.810.5
58.5
69.1
MinMaxDiff (Gu et al., 2024)
62.0
66.8
76.610.2
53.1
59.010.4
69.610.2
LDM (Rombach et al., 2022)
60.3
62.042.4
71.0
50.841.2
55.1
63.8+0.4
LDM+ Disentangled Diffusion (D*M (Su et al., 2024))
59.1
64.3
70.2
52.3
55.5
62.7
LDM+ MGD' (Ours)
61.914.11
65.3
74.2
53.2
58.3
67.2
DiT (Peebles & Xie, 2023)
59.1
64.8
73.3
54.110.4
58,910.2
64.3
DiT+ Disentangled Diffusion (D'M M (Su et al., 2024))
60.4
65.5
73.8
51.1
58.0
64.1 12.5
DiT + MGD (Ours)
66.4
71.2
79.5
55.9
61.9
72.1 #0.8
(Tex:-to Texx-to-lmaane Im-leaee)
IDC (Text-to-lmage)
ImageNet1 K (Text-to-lmapg
ImageNet1K
50
Stabicbiflusion
Stablebiffusion
n0
D4M (Su et al., 20241
Stadt ei., 2023;
StaioeCiffusion Ours
StableDiffusion Ours
StatleDiflusion
ROED (Sun et a.. 2024)
70-
79
70
SiableDiflusion Ours
70-
MaxMaa (Gu 2024)
DIT IPoobics S. Kic, 2023)
Accuracy
MGD3 (Dund
50
464
4.5
50
101
40
40
40-
10-
IPC
50
IPC
IPC
10
(a)
(b)
(c)
(d)
Figure 4. Evaluation results across multiple datasets. (a-c) Accuracy of the Text-to-Image model using the soft-label protocol: (a) Nette dataset, (b) IDC dataset, and (c) ImageNct- KK dataset. (d) ImageNet- IK classification accuracy of the DiT + MGD3 model compared other sfate-of-thh-ar (SOTA) methods. All reported values are the mean accuracy over three runs.
datasets we evaluate include ImageNet- IK, ImageNet-1000 100, ImageNetIDC, ImageNette, and ImageNet-A to ImageNet- E. Additionally, we include results from Ioog in the Appendix E. We use two protocols for evaluation: a hard- label protocol and soft-label protocol.
model Øs on the distilled dataset the objective loss is L = Σj Σm Vj.m log (I()).) For ImageNet-1K 1 evaluation, we follow this protocol. Similarly to Sun et al. (2024); Gu et al. (2024), we use ResNet-18 as a teacher and student network architecture for this setup.
The hard-label protocol generates dataset with its corre- sponding class labels, trains network from scratch, and evaluates the network on the original test set. This pro- cess is repeated three times for target architectures, and the accuracy mean and standard deviation are reported. Ran- dom resize-crop and CutMix are applied as augmentation techniques during the target network's 's training. For more detailed technical information about the protocol, please refer to Gu et al. (2024). Similar to the existing literature, we evaluate our model in various IPCs ranging from 10 to 100. This protocol was used to evaluate ImageNet-100, ImageNette, and ImageNeuLDC datasets.
Baselines. We compare several baselines to contextual- ize the performance of our method. First, we include the pre-trained DiT XL/2, which represents diffusion models without mode guidance. Second, we evaluate MinMax dif- fusion with DiT XL/2, where the model is fine-tuned to encourage diversity and representativennss. Additionally, for the ImageNette and IDC datasets, we incorporate class- conditioned Latent Diffusion Model (LDM) (Rombach et al., 2022) trained on ImageNet- k. This allows us to compare the U-Net architecture (used in LDM) with the Transformer- based DiT architecture within the diffusion framework. In our experiments, both DiT and LDM by default use the DDPM sampler. Lastly, to enable a fair comparison with D*M (Su et al., 2024) under our hard label protocol, we apply its disentangled diffusion stage without incorporat- ing the soft labels used in their Training Time Matching procedure on ImageNette and IDC datasets.
In soft-label protocol, region- based labels are generated with pre-trained network as proposed by Sun al. (2024). The region-basee soft-labels Bi.m generated as follows: Di ( where Φr is the pretrained model and Jam is the m-th crop of the i-th image. When training
6

---

Mode-Guided Dataset Distillation using Diffusion Models
Table Performance comparison on ImageNet-100. The best results are marked as bold.
10 (0.8%)
20 (1.6%)
ConvNet-66
ResNetAP-10
ResNet-18 18
ConvNet-6
ResNetAP-10
ResNet-18
Random
07.0
19.1
07.5
0..8+0.2
06.7
0..3
Herding (Welling, 2009)
17.3
19.8
16.1
20.4
27.640.1
24.7
IDC-1 (Kim et al., 2022) MinMaxDiff (Gu et al., 2024)
24.3 10.5
20.
20.1
2..8
29.9
30.2
22.3
24.8+0.2
22.5
29.3
32.3
31.2++0.1
MGD (Ours)
23.4+0..9
25.8
23.64
30.6+0.4
33.9
32.6+0.44
Full
79.9
80.
00.
79.9
80.3
80.8
Text-to-Image Diffusion Model. Our method is adaptable to various diffusion models, with optimal performance ob- served when the model is pre-trained on the target dataset. To assess the generalizability of our approach, we test it on a general-purpose diffusion model, specifically a text-to- image diffusion model. This evaluation poses challenges due to the potential mismatch between the model's training data and the target dataset. For this setup, the baseline is Text-to-Image Stable Diffusion model without mode guid- ance, allowing us to demonstrate the impact of integrating mode guidance in the generated dataset. For sampling, we use the class names as text prompt.
ImageNet-100 and ImageNet-1k. Tab. 2 shows compari- son to SOTA in ImageNet- 100 in IPC 10 and 20 in various target architectures. Our method surpasses the previous SOTA by 1.3%, 1.6%, and 1.4% in IPC 20 for various tar- get architectures. It also outperformes the MinMax diffusion approach IPC 10 and achieves the best performance with the ResNetAP-10 target architecture while delivering the second-best results for ConvNet-6 and ResNet-18 architec- tures. It is important to note that our method is substantially more computationally efficient compared to IDC and Min- Max (see Computational Cost below). We also compare our method with SOTA in ImageNet-IK on the soft-label protocol on IPC 10 and 50 Fig. 4d. Our method achieves SOTA outperforming previous SOTA by 1.3% and 1.6%. While using Text-to- toxt-to-Image diffusion in ImageNet- 1k, our method shows an improvement of 3.4% and 2.3% in IPC 10 and IPC 50 over Stable Diffusion as shown in Fig. 4c.
Implementation details. Our pre-trained model is DiT- XL/2 trained on ImageNet, and the image size is 256 256. We use the sampling strategy described in Peebles & Xie (2023), which uses 50 sampling steps using classifier-free free guidance with a guidance scale of 4.0. For Mode Guid- ance, we set À to 0.1, and in our experiments, we use stop guidance stop = 25. We use K-means to perform mode discovery; we set k = I IPC. We use single NVIDIA RTX A5000 GPU with 24GB VRAM to run our experiments.
Performance on Larger Models. To evaluate the seal- ability of our approach, we assess its performance on larger backbone archiectures-ResNe-aa and ResNet- 101-under the IPC50 setting on ImageNet-Ik: Table compares method against several existing approaches across ResNet-18, 18, ResNet-50, and ResNet-101 101. 1. Our method consistently outperforms prior work on both larger back- bones, demonstrating strong generalization to high-capacity models. Notably, while ResNet-18 18 achieves 69.8% accuracy when trained on the full dataset, our method achieves 86% accuracy using only 3.9% of the data, highlighting both its data efficiency and strong relative performance.
5.1. Comparison with state-of-the-arr methods
We compare our method with current SOTA methods on various image datasets and architectures. Our method sig- nificantly outperforms previous approaches across various benchmark datasets and target architectures.
ImageNette and ImagelDC. On the ImageNette dataset, our method using DiT achieves notable performance gains of 4.4%, 4.4%, and 2.9% for IPC values of 10, 20, and 50, respectively, surpassing previous state-of-the-arr (SOTA) methods (see Tab. 1). Similarly, the ImagelDC dataset, method demonstrates improvements of 2.8%, 2.9%, and 2.5% for IPC 10, 20, and 50, respectively, outperforming prior SOTA results. Tab. highlights that our approach consistently enhances the performance of DiT and LDM. Furthermore, in the Text-to-Iman Image evaluation mode, our method with guidance surpasses Stable Diffusion on both datasets, illustrated in Fig. 4a and 4b.
Computational Cost. Our method achieves state-of fffte-of-ther -rrt performance on all datasets, except ImageNet-1000, where the perf-perfonming method, IDC-1 (Kim et al., 2022), has slightly better results than ours but with much higher compu- tational cost. For example, MinMax (Gu al., 2024) took 10 hours produce distilled dataset for ImageNet-1000 with IPC-10, while IDC-1 (Kim et al., 2022) took over 100 hours for the same. The optimization strategy proposed in IDC-1 (Kim et al., 2022) can not scale up to the ImageNet- IK, and MinMax diffusion requires expensive fine-tuning of the diffusion model, especially for larger datasets like
7

---

Mode-Guided Dataset Distillation using Diffusion Models
Table Comparison of top-1 accuracy across different methods and backbone architectures 1ResNet-18, ResNet-50, ResNet- 101) under the IPC50 setting on ImageNet. A dash (-) indicates that the result was not reported.
Table 4. Comparison of our method with generative prior methods on ImageNet subsets A to E with 10.
Distil Alg.
Mothod
ImNet-A
ImNet-R
ImNer-C
ImNet-D
ImNet-R
Method
Re
ResNet-50
ResNet-101
Pixel
52.3
45.1
40.1
36 36.4
38.1
DC
GLaD
13.1
g0.1
48.
38.911
38.4
Full Dataset
69.8
80.9
81.9
H--LaD
54.1g1a
52.0g1
49.5.oa
39,8
40.1
SR-L et al., 2023)
46.8 1: 0.2
55.6 + 0.3
60.8 = 0.5
LM3D
55.2410
51.8
49.921.
39.5
39.0
G-VBSM (Shao et al., 2024a)
51.8 +.4
58.7 + 0.3
0.1
44,4206
52.6.04
50.6
35.4 47.5
RDED (Sun et al., 2024)
56.5 0.1
61.2 + 0.4
DM
Gl
52.8110
51,3200
49.7.0.4
38. 38.640.7 36.4204
EDC (Shao et al,, 2024b)
58.0 0.2
6L.3
64.9
H-GLaD
55.1
54,2gua
50.8.04
37.6gad
39 20.9
D'M (Su et ul., 2024)
5.1
6.1
6.1
LM3D
57.0
52.3
48.2144
39.5
19.4
Ours
60.2 0.1
64.6 0.4
67.7 + 0.4
MGD)((Ors)
63.4
66.3.11
58.6
46.8
51.1130
ImageNet-Ik. In contrast, we use pre-trained diffusion mod- els to create distilled dataset with no additional computa- tional cost for fine-tuning and minimal overhead for mode discovery. For comparison, our method takes 0.42 hours to generate a synthetic dataset for ImageNet- 100 with 10. This highlights the computational efficiency of our model compared to previous approaches.
Table S. Ablation study on component of our proposed method.
The results are on the ImageNette dataset with IPC 10. Each component contributes to the overall performance.
Test Model
Mode Disc.
Mode Guid. Stop Guid.
Acc.
ConvNet-6 ResNeLAP-10
53.2
17.1
ResNet-18
53.5
ConvNet-6
57.5
Comparison with Generative Prior Methods. We com- pare method against GLaD, H-GLaD, and LM3D in their cross- architecche setup, using AlexNet, VGG11, ResNet18, and ViT for performance evaluation. The evaluation was done by running the evaluation five times per architecture and reporting the mean performance across all the archi- tectures. We evaluate our model in 5 subsets: A, B, C, D, and E of IeageNet. Our method was trained using the hard- label protocol. Tab. 4 shows that our method outperforms previous approaches in this setup. Additionally, these meth- ods face scalability challenges for large datasets such as ImageNet-I1 1K or higher IPC values (>50) due to their high time and space complexity.
ResNetAP-10
63.8
ResNet-18
62.0
ConvNet-6
59.6.2.2
ResNctAP-10
66.4
ResNet-18
64.4
However, the distilled dataset from our method covers a broader area of the data distribution than both methods.
Representativeness and Diversity. While t-SNE provides qualitative visualization of diversity, it does not present the complete picture. We are also interested in represen- tativeness. With this in mind, our goal is to empirically measure diversity and representativeness in the t-SNE space described above. To measure diversity, we calculate the pair- wise distance of all samples within a class for the distilled dataset and report the minimum distance per sample. To measure representativeness, we calculate the mean distance to the 50 closest samples in the original dataset, where greater distance indicates lower representativeness and smaller distance indicates higher representativennss.
5.2. Ablation Experiments
Effect of each component. To assess the impact of each proposed component, we incrementally evaluated the fol- lowing: 1) Mode Discovery, 2) Mode Guidance, and 3) Stop Guidance. Mode Discovery involves performing K-means per class the original dataset and selecting the closest sample to the k-means centroid. We conduct the evalua- tion on the ImageNette dataset with IPC 10, and report the accuracy of ConvNet-6, ResNet10 with average pooling, and ResNet18. Tab. 5 demonstrates that using diffusion with mode guidance enhances mode discovery and that stop guidance is crucial for achieving improved performance.
We compare the diversity and representativeness of each class for DIT, MinMax diffusion, and our method as shown in Fig. 6. For clarity visualization, we plot reprrssentativeness, so that higher values indicate higher representativennss. Our experiment indicates that DiT ex- amples show partial representative and partial diversity. On the other hand, MinMax produces more diverse examples than DIT, although some classes lack diversity. Our method demonstrates that our samples are both diverse and repre- sentative. Furthermore, we provide additional results about representativeness and diversity in the Appendix B.
Visuzalizing t-SNE. age, we visualize a 1-SNE plot of the distilled dataset from the DIT, MinMax Diffusion, and our method. Fig. 5 illus- trates that the DiT distilled dataset is mostly contained in one region of the original dataset distribution, while MinMax Diffusion extends a broader area of the data distribution.
To analyze the distilled dataset's cover-
8

---

Mode-Guided Dataset Distillation using Diffusion Models
RIT Mentmax Ours
Algorithm 1 Mode Guidance with DDIM sampling, given diffusion model (z estimated mode my and mode guidance scale A.
2-
Input: estimated mode my and mode guidance scale À 2T + sample from (0,1))
-10
for all t from T do
gt = (m; - x/)
-30
t cy (2t) ) - v1 - σt A ge
30
2--1 van-
T-
tv1== āt-1
Navirax Ours
Vai
end for return: DD
-30
Table 6, Performance comparison of diffusion models (LDM, DiT) with and without approach, evaluated using DDPM and DDIM sampling methods on the Nette dataset on the IPC-10.
-20-
Figure 5. T-SNE plot showing the original samples and the synthetic samples generated by different diffusion- bascd meth- ods for two classes (English springer and cassette player) from ImageNet-I1 1k. This visualization shows that DiT (Peebles & Xie, 2023) has limited diversity, Minmax (Gu al., 2024) diffusion shows diversity but lacks full coverage, while our approach demon- strates mode diversity, achieving higher coverage.
Method
DDPM
DDIM
LDM
60.3
60.4
LDM + MGD (Ours)
61.924.1
62.3
DiT
58.8
61 61.422.4
DiT + MGD (Ours)
66.4
66.6.0.6
6. Conclusion
Dataset distillation is an important task of condensing in- formation from large training sets. Despite several efforts, the distilled datasets have limited representativeness and diversity in their synthetic samples. Our proposed method, leveraging latent diffusion with mode guidance, addresses this limitation and achieves state-of-the-aan performance in dataset distillation across multiple benchmarks and exper- imental setups. Notably, our approach outperforms previ- ous methods without requiring fine-tuning, demonstrated by our results on ImageNette, ImagelDC, ImageNet- 100, and ImageNet- 1K. We conducted a detailed analysis of our method's key components and demonstrated their utility through rigorous ablation studies. Furthermore, we showed that our approach is compatible with general diffusion mod- els, such as Text-to- Imme Stable Diffusion, even when the training data does not overlap with the target dataset.
Ours
Optimal
DiT
MinMax
Diversity
0.6
0.4
0.0
0.2
Representativeness
Figure 6. Representative score versus Diversity score each class
on Nette for IPC 10 versus various models.
Impact Statement
Mode Guidance with DDIM. Our approach, similar to classifier guidance (Nichol & Dhariwal, 2021), can be in- corporated into DDIM using Algorithm 1. Table 6, we compare the effect of our approach in DDPM and DDIM across LDM and DiT diffusion architectures. Our results demonstrate the effectiveness of our method with denoising samplers in both architectures, showcasing its flexibility with respect to diffusion architecture and sampler choice. This highlights the significant impact of our approach in enhancing the performance while being adaptable with dif- ferent denoising diffusion models.
Efficient dataset distillation reduces storage and compu- tational costs while maintaining high model performance. MGD improves both accuracy and efficiency compared to prior methods, enabling large-scale dataset distillation with minimal performance trade-offs. This has significant implications for deep learning in resource-constrained envi- ronments, such as mobile AI and federated learning. By en- hancing scalability, our work enables more effective model training with limited data while preserving diversity and representativennss..

---

Mode-Guided Dataset Distillation using Diffusion Models
Acknowledgements
He, Y., Liu, P., Wang, Z., Hu, Z., and Yang. Y. Filter prun- ing via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4340-4349 2019.
The authors thank the anonymous reviewers for their valu- able feedback on earlier versions of this manuscript. This work was partially supported by research gift from Cisco.
References
Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Castro, F. M., Marin-Jiménez, M. J., Guil, N.. Schmid, C., and Alahari, K. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Prob- abilistic Models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Cazenavette, G., Wang. T., Torralba, A., Efros, A. A., and Zhu, J.-Y. Dataset distillation by matching training tra- jectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4750- 4759, 2022.
Kim, J.-H., Kim, J., Oh, S. J., Yun, S., Song, H., Jeong, J., Ha, J.-W., and Song, H. O. Dataset condensation via efficient synthetic-date parameterization. In Inter- national Conference on Machine Learning, pp. 11102- 11118, 2022.
Cazenavette, G., Wang, T., Torralba, A., Efros, A. A., and Zhu, J.-Y. Generalizing dataset distillation via deep gener- ative prior. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3739- 3748, 2023.
Li, G., Togo, R., Ogawa, T., and Haseyama, M. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference on Image Processing (ICIP), Pp- IEEE, 2020.
Chauhan, A., Tiwari, U., et al. Post training mixed precision quantization of neural networks using first-order infor- mation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1343-1352, 2023.
Liu, S., Wang, K., Yang, X., Ye, J., and Wang, X. Dataset distillation via factorization. Advances in Neural Infor- mation Processing Systems, 35:1 100-1111,, 2022.
Chen, W., Wang, P., and Cheng, J. Towards mixed peeeddpreciioon quantization of neural networks via constrained optimiza- tion. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 5350-5359, 2021.
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. Learning efficient convolutional networks through net- work slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Chen, Y., Welling, M., and Smola, A. Super-samples from kernel herding. In Proceedings of the Twenty-Sixth Con- ference on Uncertainty in Artificial Intelligence. pp. 109- 116, 2010.
Loo, N., Hasani, R., Amini, A., and Rus, D. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 35: 13877-13891, 2022.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8799 8794, 2021a.
Moser, B. B., Raue, F., Palacio, S., Frolov, S., and Dengel, A. Latent dataset distillation with diffusion models. arXiv preprint 1.XXv:2403.0388// 2024.
Dhariwal, P. and Nichol, A. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 7780-8794, 2021b.
Nichol, A. Q. and Dhariwal, P. Improved Denoising Diffu- sion Probabilistic Models. In International Conference on Machine Learning, pp. 8162-8171, 2021.
Ding, X., Ding, G., Guo, Y., and Han, J. Centripetal sgd for pruning very deep convolutional networks with com- plicated structure. In Proceedings of the IEEE/CVF con- ference on computer vision and paftern recognition, pp- 4943-4953, 2019.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision pp. 4195-4205, 2023.
Gu, J., Vahidian, S., Kungurtsev, V., Wang, H., Jiang, W., You, Y., and Chen, Y. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
10

---

Mode-Guided Dataset Distillation using Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. Righ-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Yin, Z., Xing, E., and Shen, Z. Squeeze, recover and relabel: Dataset condensation at imagenet scale from per- spective. In Advances in Neural Information Processing Systems, 2023.
Yu, J., Wang, Y., Zhao, C., Ghanem, B., and Zhang, J. Free- dom: Training- energy- energy-guided conditional diffusion model. Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), 2023.
Shao, S., Yin, Z., Zhou, M., Zhang. X., and Shen, Z. Gener- alized large-scale data condensation via various backbone and statistical matching. In P Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16709-16718, 2024a.
Zhang, D. J., Wang, H., Xue, C., Yan, R., Zhang, W., Bai, S., and Shou, M. Z. Dataset condensation via generative model. arXiv preprint arXiv:2309.0700 2023.
Shao, S., Zhou, Z. Chen, H., and Shen, Z. Elucidating the design space of dataset condensation. In Advances in Neural Information Processing Systems, volume 37, pp- 99161 -92001, 2024b.
Zhao, and Bilen, Dataset condensation with differen- tiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685. PMLR, 2021.
Sharma, A. and Foroosh, H. Rapid: A single stage pruning framework. In 2022 IEEE International Conference Image Processing (ICIP), pp. 361 1-3615. IEER, 2022.
Zhao, B. and Bilen, H. Synthesizing informative training samples with gan. In NeurIPS 2022 Workshop on Syn- thetic Data for Empowering ML Research, 2022.
Su, D., Hou, J., Gao, W., Tian, Y., and Tang, B. D 4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5809-5818, 2024.
Zhao, B. and Bilen, H. Dataset condensation with distri- bution matching. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pp. 6514 6523, 2023.
Sucholutsky, L. and Schonlau, M. Secdd: Efficient and secure method for remotely training neural networks (stu- dent abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 15897-15898, 2021.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset conden- sation with gradient matching. In Ninth International Conference on Learning Representations 2021, 2021.
Zhong, X., Fang, H., Chen, B., Gu, X., Dai, T., Qiu, M., and Xia, S.-T. Hierarchical features matter: A deep ex- ploration of gan priors for improved dataset distillation. arXiv preprint 24Xxx:2406.05744 2024.
Sun, P., Shi, B., Yu, D., and Lin, T. On the diversity and re- alism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Wang, K., Gu, J., Zhou, D., Zhu, Z., Jiang, W., and You, Y. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.0470 2023.
Wang, T, Zhu, J.-Y., Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint 1811. 10959, 2018.
Welling, M. Herding dynamical weights to learn. In Interna- tional Conference on Machine Learning, pp. 1121-1128, 2009.
Wu, J., Leng, C., Wang, Y., Hu, Q., and Cheng, J. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4820-4828, 2016.
Xu, K., Han, L., Tian, Y., Yang, S., and Zhang, X. Eq-net: Elastic quantization neural networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1505-1514, 2023.

---

Mode-Guided Dataset Distillation using Diffusion Models
A. When should guidance stop?
To determine when to stop the guidance, we assess mode guidance with fstpp ranging from 50 to in increments of 5 steps. stop guidance of Estop = 50 means no guidance, while Esop = 0 means full guidance. Figure 7a shows that the optimal range to stop the guidance between Estop = 30 and faop 10, with the peak at Estep 20. Additionally, Figure 7b illustrates that the guidance introduces more variability in the generation, with more diverse set of backgrounds and poses. However, when the mode guidance is extended (e.g. Sttp 0), it does not guarantee class consistency, as demonstrated in Figure 7b. This makes Estop = 25 a good balance between generating a diverse set of backgrounds and maintaining class consistency.
68
Validation Accuracy (%)
66.3
66.4
86
66
64
62.8
60.7.
00
58
10
29
30
Estop
stop 0
Stop= 25 (b)
tstp 50
(a)
Figure? 7. Ablation of the effect of Estop, where Estop = 0 denotes full guidance and Estop = 50 denotes no guidance. (a) Shows validation accuracy versus Extop on ImageNette dataset. Best performance is achieved when tatop ranges between 20 and 30. (b) Shows generated images for the 'English Springer' class with full guidance (tatop 0), with carly-stop guidance tatop = 25 and no guidance (fatop 50). With early-stop guidance. the generated samples have more diversity w.r.t to the pose and background.
B. Class wise Diversity and Representaativenss
Figure shows the diversity and representativeness of each distilled sample for ten classes in the ImageNet- k dataset for DIT, MinMax, and ours. This Figure shows that our method is consistently have higher representativeness across all the classes in comparison to the previous methods. Overall, our method maintains high diversity across most of the samples within a class. We observe that both MinMax and DiT consistently have a few samples with very low diversity.
tench
Eng hish springer
cassette player
chain saw
church
&
HerMa
Moriee
He-tfax uure
cart
Lurt
Drsmity
Re cemmreetattieeeees
Incrrnentativernt sex
Reprosental venees
French horn
garbage truck
gas pump
golf ball
parachure
Molk
Mo
Out
Ovanty
Clearsty
Oum
Ro Ieresentttterees
Mecresentatveesss
Recrrsorbal wesess
Figure 8. Representativeness versus Diversity by class for the distilled datasct from diffusion- methods on 10 IPC of ImageNet- Ik. Each point represents image of the distilled dataset. DiT shows high representativeness but lacks diversity; MinMax shows diversity but lacks representativenes;; Ours method shows both diversity and representativeness.
12

---

Mode-Guided Dataset Distillation using Diffusion Models
C. Effect of Stop Guidance in Diversity and Representativeness
In order to understand how the stop guidance affects the diversity and representation of the distilled dataset, we perform an evaluation of these metrics on the ImageNette dataset for IPC 10 for various tsopp ranging from 50 to O: our results are shown in Figure 9. Our results show that applying mode guidance at any of the evaluated fstp values increases diversity, with the gains beginning to saturate beyond koup 30. Surprisingly, delaying the stop guidance further into the reverse process (e.g stop 25) leads to noticeable increase in representativeness, while maintaining high diversity.
Diversity
0.50
0.45
2r
0.40
8
Gra 25
0.35
*
Crtpp
40
0.30
0.25
55
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
Representativeness
Figure 9. Representativeness versus Diversity versus tatp, Each point represents distilled dataset. Diversity and representativeness are obtained by computing the mean across the samples in the distilled dataset. Stopping the mode guidance early reverse process (Estop 45 to kstop 35) promotes diversity. While prolonging the mode guidance between ttop 35 and stop 0 increases
representativeness.
D. Effect of mode discovery algorithm
To investigate the impact of the mode discovery algorithm, we assess several strategies: random selection from the original dataset, k-means centroids, closest sample to k-means centroid, DBSCAN, spectral clustering, and Gaussian Mixture Models (GMM). The evaluation is conducted on ImageNette with IPC 10. For DBSCAN and spectral clustering, we compute the mean of each discovered cluster to represent a mode. For GMM, we use the mean of each Gaussian component. Mode guidance was applied with sstop 25 using the estimated modes from each method. The results, summarized in Table 7, show that GMM achieved the highest accuracy, slightly outperforming k-means centroids and other mode discovery techniques.
Table 7. Mode discovery algorithm versus Accuracy on ImageNette with IPC-10.
Mode Discovery method
Accuracy
Random
59.641.8
DBSCAN
61.3+1.9
Spectral Clustering
64.5
GMM
66.9
k-Means (closest sample)
64.640.4
M-Mens (centroid)
66.442.4
E. Evaluation on ImageWoof
Image Woof. We compare our method with SOTA in ImageWoot on IPC 10, 20, 50, 70, and 100 on various target architectures, as shown Table worth noticing that this dataset is fine-grained dataset where all classes belong to
13

---

Mode-Guided Dataset Distillation using Diffusion Models
Table Performance comparison with pre-trained diffusion models and other state-of-the-an methods on Image Woof. All the results reproduced by us for the 256 resolution. The missing results are due to out-of of-memory. The best results are marked as bold. Higher better.Resule shown the previous works are from (Gu al., 2024).
IPC (Rutio)
Test Model
Random
Harding (Welling,
DM Bilen,
IDC1 (Cronnveete al., 2023)
GLaD
MinMaxDDift
MGD! (Ours)
Pell
(Pechles 2023)
ComoNas-6
26.7
2
26.92.
338400
37.0gx.
34.73
86449
(0.8%)
ResNecAP-10 10
29.428
32.0
34.72n.
30.3gt.2
39.1
32,9209
39, 39,2
40.4:19
87.5
R8
27.7.0.
30.2
34.746.
33/4,0.7
37.3
31.7a0a
37.6
38.5.2.
8
ComoNes-6
29.1
29.5.6.3
36. lans
29.
35.5ap.
37,6
39.0
86440
20 (1.61)
ResNecAP-10
14
35.2an
43.4gn
45.8
ResNo-18
29.7an
42.5
6
40.3
40.3.0.7
16.4.1.
43.9
a3.9.a.a
54.5:14
86.4
(3.8%%
ResNecAP-11 10
47.211
48. 18.1
49.3
47.
48.3a1.0
56.3
$6.5.19
87.5
ResNer-18
47.9
48.3
50.1.6.5
46.2,0.4
48.3
57.1
58.5
89.3
Com-Nes-
4
46.2.0
50. 50.1
47.5
48 48.9u5.7
55.7
55.1
86.409
70 (5.45)
ResNetAP-10
80...0
43
54 3gn
$1.720
$2.821
58, 28.32nm
60.2
87.5gn
ResNer 18
52.1:1.
49.7:0.
51,9.0.a
5t.11..7
58.8.0.7
59.7:27
89.31.
6
12.
54.4.1.
53.410.5
35.0.1.
53.2a0.9
61.1
60.1:12
86.410
(7.755)
RP-16
4..
61.7
56 56.400
64.5
66.5.1
87.5
RecNec-18 18
61.5
59.3gn7
58.921.8
60.2g1.0
$8.3g1.2
65, 65.7
68.8.0.
89.3
dog breeds. Due its granularity of features, trained DiT XL/2 on the ImageWoof dataset with just the simple loss mentioned in Eq. 2 following the same training epochs as (Gu et al., 2024). Our method outperforms the previous SOTA across various IPC values for different target architectures. Notably, our method demonstrates superior performance in all IPC values for the ResNet-18 architecture, achieves SOTA in IPC 10, 50, 70, and 100 with the ResNetAP- architecture, and deliveres the best performance in IPC 20 and 50 with the ConyNet-6 architecture.
F. Effect of mode guidance scale À
To study how the mode guidance scale À affects performance, evaluate the various values for À on ImageNette with IPC 10 with ResNetAP-10 Our results show that when the mode guidance is too high, it's catastrophic for the distilled data, dropping the performance significantly; however, the best parameter is achieved by X = 0.1.
70
Validation Accuracy (%)
60
40
30
10
0.01
0.05
0.1
0.5
Mode Guidance Scale (A)
Figure 10. Effect of guidance scale on performance.
G. Diversity Class-Wise diversity score
We calculate the diversity score for each class by averaging the diversity score across all the samples. Table 9 shows the diversity score for each class for DiT, MinMax, and Mode Guidance. Our method consistently generates a more diverse set for each class for ImageNette than the other methods.
14

---

Mode-Guided Dataset Distillation using Diffusion Models
Table 9. Results: Comparison of per-class diversity scores ImageNette with IPC-10
class
DiT
MinMax
Ours
tench
0.35
0.18
0.82
English springer
0.65
0.33
0.62
cassette player
0.55
0.52
1.00
chain saw
0.00
0.37
0.55
church
0.54
0,41
0.77
French horn
0.21
0.13
0.54
garbage truck
0.44
0.38
0.76
gas pump
0.50
0.24
0.67
golf ball
0.20
0.33
0.78
parachute
0.08
0.48
0.79
Average
0.35
0.34
0.73
H. Hard-Label versus Soft-label Protocols
We conduct further analysis on ImageNet-100, where we test our approach from IPC-10 up to IPC-100. As illustrated in Table 10, our performance steadily improves, reaching 57 .7.8 with the hard-label protocol. Additionally, we compare the performance of ImageNet- using soft-label training on IPC-10, 20, 50, and 100. The results underscore a substantial performance boost when employing soft-labels.
Table 10. Evaluation of training with hard-labels versus soft labels in ImageNct- 100 training with ResNet18.
Method
Labels
IPC10
IPC20
IPC50
IPC 100
Hard-Label
23.620.4
32.640.4
51.8
57.8
MGD (Ours)
Soft-lahel
34.0g1.
50.240.7
69.2
75.8
Evaluation Technical Details
For the hard-label protocol, we follow the evaluation method described in (Gu et al., 2024). We train our model on synthetic dataset for 1500 epochs for IPC values of 20, 50, and 100, and extend the training to 2000 epochs for an IPC value of 10. We use Stochastic Gradient Descent (SGD) as the optimizer, setting the learning rate at 0.01. We use a learning rate decay scheduler at the 2/3 and 5/6 points of the training process, with the decay factor (gamma) set 0.2. Cross-entropy was used as the Loss objective.
For the soft-label protocol, we follow the evaluation used by (Gu et al., 2024; Sun et al., 2024) for ImageNet- lk evaluation. We evaluate the model by training network for 300 epochs with Resnet- 18 architecture both teacher and student. We use the AdamW V optimizer, with a learning rate set at 0.001, a weight decay of 0.01, and the parameters 31 = 0.9 and
32 = 0.999.
J. Visualization of Denoising Trajectories with Mode Guidance for Different tstop
Figure 11 illustrates the effect of Stop Guidance (tstop) the generated image. Stopping early (e.g.. tstop 45) can introduce features unrelated to the target class, such as the baby face in the top row. Conversely, extending guidance too long (e.g., tstop 0) degrades image quality.
15

---

Mode-Guided Dataset Distillation using Diffusion Models
20
Stop Guidance
Tma-maa 30
25
10
0
(-50
1-45
[-40
I-35
(-30
(-25
(-20
1-15
1-0
Denaising Timestep
Figure 11. Generated images through the denoising process different values Extap-
16


Reviews:
[{"review_id": "1pPeMs5WP7", "reviewer": "Reviewer_rCkP", "summary": "This paper proposes a mode-guided diffusion model that leverages a pre-trained diffusion model for dataset distillation, eliminating the need for fine-tuning with distillation losses. The proposed approach enhances dataset diversity through three key stages:\n\n- Mode Discovery – Identifies distinct data modes to ensure comprehensive dataset coverage. \n- Mode Guidance – Encourages intra-class diversity by guiding samples to different modes.\n- Stop Guidance – Mitigates artifacts in synthetic samples that could negatively impact model performance.\n\nThe method is evaluated on multiple datasets, including ImageNette, ImageIDC, ImageNet100, and ImageNet-1K, demonstrating accuracy improvements of 4.4%, 2.9%, 1.6%, and 1.6%, respectively, over state-of-the-art techniques. The approach significantly reduces computational costs by eliminating the need for fine-tuning diffusion models with distillation losses while maintaining or improving dataset quality and representativeness.\n\n## update after rebuttal \nI think the author's experiments addressed my concern, so I decided to raise my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of our work as novel and acknowledging our work to address the key limitations of existing techniques without requiring any fine-tuning. We are also grateful that the reviewer highlighted the significance and impact of our approach, with mention of our work reducing the computational costs, thereby making the dataset distillation more accessible to researchers with limited resources. Additionally, we are pleased that the reviewer found our experimental validation thorough highlighting method's robustness and scalability, and found our method to be logically and clearly presented. Below, we provide answers to the reviewer's comments. \n\n### Model Scaling Experiments\n\nIn the main paper, we showed that our method achieved SOTA on ImageNet-1K (IPC-50) with ResNet18. As requested by the reviewer, we extended our experiments to larger models (ResNet-50 and ResNet-101) and showed consistent performance improvements across models of different scales.\n\n| Method         | ResNet-18        | ResNet-50         | ResNet-101         |\n| -------------- | ---------------- | ----------------- | ------------------ |\n| _Full Dataset_ | _69.8_           | _80.9_            | _81.9_             |\n| SR2$^2$L       | 46.8±0.2     | 55.6±0.3      | 60.8±0.5       |\n| G-VBSM         | 51.8±0.4    | 58.7± 0.3     | 61.0±0.4      |\n| RDED           | 56.5±0.1     | -                 | 61.2±0.4     |\n| EDC            | 58.0±0.2     | 64.3±0.2      | 64.9±0.2       |\n| D$^4$M         | 55.2±0.1     | 62.4±0.1      | 63.4±0.1       |\n| **Ours**       | **60.2±0.1** | **64.6±0.4** | **67.7±0.4** |\n\n**Table 1: Results on ImageNet-1K with IPC 50. '-' means not reported.**\n\n### Clarification about ImageNet-1k Baselines\nAlthough the baseline accuracy may appear low in isolation, our method surpasses previous works on ImageNet. To provide proper context, we compare against the accuracy achieved when training the same network on the full dataset. For example, ResNet-18 reaches 69.8% accuracy with the full dataset, whereas our approach attains 86% of this performance while using only 3.9% of the data, demonstrating its efficiency and strong relative performance.", "claims_and_evidence": "see strength & weakness", "methods_and_evaluation_criteria": "see strength & weakness", "theoretical_claims": "see strength & weakness", "experimental_designs_or_analyses": "see strength & weakness", "relation_to_broader_scientific_literature": "no significant contributions to the broader scientific literature", "essential_references_not_discussed": "no essential references need to be further discussed", "other_strengths_and_weaknesses": "## Strengths:\n- Novelty and Contribution: The paper presents a novel approach to dataset distillation by leveraging pre-trained diffusion models without requiring fine-tuning or distillation losses. The proposed method introduces three stages (Mode Discovery, Mode Guidance, and Stop Guidance) to enhance sample diversity and quality, addressing key limitations in existing techniques.\n\n- Significance and Impact: By eliminating the need for fine-tuning, the approach significantly reduces computational costs, making dataset distillation more accessible to researchers with limited resources. The method’s applicability to large-scale datasets (ImageNette, ImageIDC, ImageNet100, and ImageNet-1K) demonstrates its robustness and scalability.\n- Empirical Validation: The reported improvements (e.g., 4.4%, 2.9%, 1.6%, and 1.6% accuracy gains over state-of-the-art methods) suggest the effectiveness of the approach. The evaluation of diverse datasets strengthens the claim that the method is generalized well.\n- Clarity of Key Ideas: The paper provides a structured breakdown of the proposed method, distinguishing it from existing approaches in dataset distillation. The introduction of Mode Discovery, Mode Guidance, and Stop Guidance is well-motivated and logically presented.\n\n## Weakness\n- Small model: Experiments mainly on small size models and baseline for ImageNet are relatively low.", "other_comments_or_suggestions": "Can the author provide experiments on the larger size of the model to show that the methods can be scaled up?", "questions_for_authors": "see suggestion", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tdt8PbOP3A", "reviewer": "Reviewer_24Ct", "summary": "This paper presents Mode-Guided Dataset Distillation using Diffusion Models (MGD³), a novel dataset distillation method that improves diversity and representativeness in distilled datasets without requiring additional fine-tuning of diffusion models. The approach comprises three key components: Mode Discovery to identify distinct data modes within a class to enhance sample diversity. Mode Guidance to guide the diffusion process toward specific modes to avoid redundant samples. Stop Guidance to halt guidance at a certain timestep to maintain sample fidelity.\n\n## update after rebuttal\nThe authors adequately addressed my concerns, and I have raised my score to Accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thoughtful feedback and recognition of our contributions. Specifically, we are pleased that the reviewer highlights the novelty of our Mode-Guided Dataset Distillation using Diffusion Models (MGD³), its computational efficiency, the well-structured methodology, and the extensive experimental evaluation, including ablation studies and comparisons with strong baselines.\n\n## Clarifying Performance and Efficiency Trade-off\nWe acknowledge that while MGD³ achieves state-of-the-art performance across multiple benchmarks, IDC-1 slightly outperforms it in specific cases on ImageNet-100 (e.g., ResNet-18 with 10 samples). However, MGD³ consistently provides strong accuracy while significantly improving computational efficiency, making it more scalable than IDC-1. MGD³ achieves the best performance in several configurations (e.g., ConvNet-6 and ResNetAP-10 with 20 samples). We'll revise our statement to explicitly highlight this trade-off between accuracy and efficiency.\n\n## References\nAlthough we cited [1] and [3] in the main paper, we will expand our discussion to clarify how MGD³ relates to works [1-4], particularly regarding efficiency and scalability, and include comparisons in Table 1(reviewer rCkP).\n\n## Impact Statement\nEfficient dataset distillation reduces storage and computational costs while maintaining high model performance. MGD³ improves both accuracy and efficiency compared to prior methods, enabling large-scale dataset distillation with minimal performance trade-offs. This has significant implications for deep learning in resource-constrained environments, such as mobile AI and federated learning. By enhancing scalability, our work enables more effective model training with limited data while preserving diversity and representativeness.\n\nThe Impact Statement will be included in the final revision.\n\n## Hyperparameter Experiments\nWhile our main experiments used fixed hyperparameters for consistency (still achieving SOTA results), we conducted additional experiments with varying hyperparameters for each dataset as requested. Results for different values of λ and t_stop on Nette, IDC, and ImageNet-100 with IPC 10 are shown below:\n\n| Dataset      | $\\lambda$ = 0.01 | $\\lambda$ = 0.05 | $\\lambda$ = 0.07 | $\\lambda$ = 0.1 | $\\lambda$ = 0.3 | $\\lambda$ = 0.5 |\n| ------------ | ---------------- | ---------------- | ---------------- | --------------- | --------------- | --------------- |\n| Nette        | 62.5±1.81        | 65.5±1.7         | 65.5±1.0         | **67.1±0.6**    | 43.4±2.1        | 12.6±0.4        |\n| IDC          | 48.4±1.31        | 54.9±1.1         | 53.3±0.7         | **55.9±2.1**    | 32.9±1.1        | 16.7±2.6        |\n| ImageNet-100 | 25.0±0.38        | 27.1±0.4         | **27.2±0.4**     | 25.77±0.5       | 12.0±0.4        | 2.3±0.1         |\n| Mean         | 45.3             | 49.2             | 48.7             | **49.6**        | 29.4            | 9.2             |\n\n**Table 2: Results with various mode guidance scale parameters (λ)**\n\n| Dataset      | $t_{stop}$ = 0 | $t_{stop}$ =10 | $t_{stop}$ = 20 | $t_{stop}$ = 25 | $t_{stop}$ = 30 | $t_{stop}$ = 40 | $t_{stop}$ = 50 |\n| ------------ | -------------- | -------------- | --------------- | --------------- | --------------- | --------------- | --------------- |\n| Nette | 62.8±1.6 | 66.3±1.5 | **67.1±0.6**| 66.4±2.4| 65.4±1.6 | 60.0±0.8| 60.7±2.3|\n| IDC | 52.8±4.2 | 53.5±2.0| 52.1±0.8| **55.9±2.1**| 53.2±2.6| 47.8±3.5| 42.5±1.3|\n| ImageNet-100 | 23.1±1.0| 25.1±0.4| **26.2±0.5**| 25.8±0.5| 24.1±0.5| 21.2±0.3| 23.3±0.3|\n| Mean         | 46.2           | 48.3           | 48.5            | **49.4**        | 47.6            | 43.0            | 42.2            |\n\n**Table 3: Results with various values for $t_{stop}$**\n\nWhile performance can be improved for specific datasets by adjusting parameters, our default parameters result in higher average scores across datasets.\n\n## Discussion on ImageWoof\nWe included a discussion of ImageWoof results in the appendix (line 677-679).\n\n## Experiments with Other Mode Discovery Techniques\nPlease refer to the Response of reviewer y4kY.\n\n## Addressing Ultra-Low IPC Performance: MGD³'s Scaling Advantages\nThe results from existing works on dataset distillation suggest that generative methods for dataset distillation underperform in low-IPC setups (e.g., IPC = 1) compared to optimization-based methods, which explicitly optimize the small dataset but at a higher computational cost. However, generative models scale better to higher IPC (e.g., IPC = 50) while maintaining lower computational cost, i.e., within a given time frame,  generative models can efficiently generate higher-IPC datasets compared to the optimization-based methods, making them more practical for scaling to larger datasets like ImageNet.", "claims_and_evidence": "1. The proposed method improves dataset diversity and representativeness without additional fine-tuning. This is demonstrated through quantitative comparisons with MinMax Diffusion and other SOTA methods across multiple datasets (Tables 1-3). The t-SNE visualizations (Figure 6) illustrate better mode coverage than previous methods. The diversity vs. representativeness plots (Figures 5, 8) further confirm these properties. \n\n2. Mode Guidance effectively improves sample diversity while maintaining class consistency. Ablation studies (Table 4) indicate that each component (Mode Discovery, Mode Guidance, Stop Guidance) contributes incrementally to performance gains. Guidance stopping strategies (Figure 7) provide insights into optimal stopping points for balancing diversity and fidelity.\n\n3. The proposed method is computationally efficient. Compared to MinMax Diffusion (10+ hours) and IDC-1 (100+ hours), MGD³ generates a dataset in 0.42 hours for ImageNet-100 (IPC-10), showcasing significant efficiency improvements.\n\n4. The claim of \"MGD³ consistently outperforms all existing methods.\" is problematic. While MGD³ achieves SOTA on ImageNette, ImageIDC, and ImageNet-1K, it does not fully surpass IDC-1 on ImageNet-100. The paper should clarify that MGD³ provides a trade-off between performance and efficiency, rather than always achieving the highest accuracy.", "methods_and_evaluation_criteria": "The proposed methodology is well-structured and logically motivated. The benchmark datasets (ImageNette, ImageNet-100, ImageNet-1K) and evaluation metrics (classification accuracy under hard-label and soft-label protocols) align with the dataset distillation task. The use of Mode Discovery in the VAE latent space is novel and helps mitigate mode collapse. Comparisons with both diffusion-based and optimization-based methods strengthen the evaluation. Ablation studies systematically justify the effectiveness of each component.", "theoretical_claims": "The paper builds on diffusion model guidance principles (e.g., classifier-free guidance from Ho & Salimans, 2022). This derivation aligns with score-based diffusion models but lacks a formal proof of why this improves diversity without fine-tuning. A theoretical justification (e.g., a distributional analysis) could further support why Stop Guidance prevents over-conditioning.", "experimental_designs_or_analyses": "The experiments are well-designed, comparing MGD³ with strong SOTA baselines across multiple IPC values. The ablation study effectively isolates the impact of Mode Discovery, Mode Guidance, and Stop Guidance. The computational efficiency analysis strengthens the argument for real-world applicability. Sensitivity analysis of hyperparameters (λ, t_stop, number of modes) is only partially addressed (Table 5, Figure 10). Are these values consistent across datasets? The ImageWoof evaluation is informative but lacks discussion.", "relation_to_broader_scientific_literature": "Unlike optimization-based distillation, which requires repeated executions to synthesize datasets of different sizes, MGD³ leverages diffusion models to directly sample from a learned data distribution, making it more scalable.", "essential_references_not_discussed": "The paper lacks discussion of several key recent works on decoupled dataset distillation methods [1-4].\n\n[1] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective\n\n[2] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching\n\n[3] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm\n\n[4] Elucidating the Design Space of Dataset Condensation", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "This paper should include an impact statement.", "questions_for_authors": "1. Do the optimal values of λ and t_stop remain consistent across datasets (e.g., ImageNette vs. ImageNet-100 vs. ImageNet-1K)? Or do they need dataset-specific tuning?\n\n2. Did you experiment with other mode discovery techniques (e.g., Gaussian Mixture Models (GMM), Spectral Clustering, or Density-Based Approaches) for estimating modes?\n\n3. Testing on few-shot learning scenarios: Can MGD³ help in ultra-low IPC regimes (e.g., IPC = 1 or 5)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9qnWA9D3QS", "reviewer": "Reviewer_Zv9A", "summary": "This paper focuses on the dataset distillation task based on generative models. Specifically, the authors argue that the current methods cannot guarantee the sample diversity, which is essential for model training. Based on this observation, the authors propose a mode-guided diffusion model. Distinct data modes are first discovered from the original data distribution. Then the modes are used to guide the generation process to enhance intra-class diversity. The proposed method surpasses previous state-of-the-art methods by at least 1.6%, with much less computational cost required.\n## update after rebuttal\nThanks for the author's reply. I would keep my recommendation of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive assessment of our work, particularly the recognition of our method’s state-of-the-art performance, its effectiveness in improving sample diversity, and its relevance to generative model-based dataset distillation. We also sincerely appreciate the reviewer’s positive evaluation of our claims and supporting evidence, which reinforces the validity and impact of our contributions. We address the reviewer’s insightful questions regarding the positive impact of the diffusion process on data quality and the question about mode selection in our response below.\n\n## Clarifying Mode Sampling During Inference:\n\nGiven a IPC = N, our method find N modes then we sample 1 image per mode.\n\n## Experiment using Latent Modes as Samples\n\nAs suggested by the reviewer, we tested using mode images directly as samples on ImageNette (IPC 10). Results in Table 1 show this performs poorly—worse than using the closest real sample to the mode. This occurs because the latent mode is an average of the cluster, creating a blurry image. MGD³ improves performance by using diffusion to generate real images near the mode rather than using the mode directly.\n\n| Method | IPC 10 |\n| -------------- | -------------- |\n| Closest Sample | 57.1 ± 1.3 |\n| Latent Mode | 37.7 ± 1.1 |\n| MGD$^3$ | **66.4 ± 2.4** |\n\nTable 1: Results for suggested experiment with latent modes.\n\n## Clarification about comparisons\nOur method was implemented independently, not built upon baselines, and evaluated under identical conditions as existing approaches. We compared our approach with pretrained diffusion models (LDM, DiT) and incorporated these alongside Disentangled Diffusion for comprehensive evaluation. In text-to-image setups, we benchmarked against Stable Diffusion and D$^4$M. Notably, our method outperformed our closest competitor MinMaxDiffusion even without requiring the finetuning that MinMaxDiffusion needs for dataset distillation.\n\n## Addressing Mode Limitation Concerns in High-IPC Settings\nWhen the selected IPC is larger than the true number of modes in the data for a class, our method captures fine-grained variations within each mode without affecting the diversity of the generated samples. This is controlled by the mode discovery algorithm, and we observe that our default k-means approach works well in practice.", "claims_and_evidence": "The claimed contributions are well supported by the numerical results. The proposed method achieves state-of-the-art validation performance on the full ImageNet-1K. And the improved diversity and representativeness are supported by designed metrics.", "methods_and_evaluation_criteria": "1. The discovered modes well cover the original data distribution. However, is there the possibility that the diversity of the generated samples is in turn restricted by the modes as there are only a limited number of them?", "theoretical_claims": "There is no explicit theoretical analysis presented.", "experimental_designs_or_analyses": "1. The proposed method guides the image generation with discovered modes. And the modes are actually latent embeddings. It would be interesting to provide an experimental comparison with directly using these samples for training under the same IPC. I am curious how the diffusion model helps with the generation quality if there already exist several selected modes. \n2. It is not sufficiently clear how the experiments are carried out and how the comparisons are conducted with the baseline methods. Are the authors building their method on top of the baselines? Or, are they simply methods that the authors used to compare with?", "relation_to_broader_scientific_literature": "Methods based on generative models are a new trend in the field of dataset distillation. The proposed method is a useful tool at inference time to generate diverse samples covering the original distribution. The method can be easily plugged into any diffusion models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n**Weaknesses:**\n1. It is not clear for each sample during generation, which mode is selected at the inference stage.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have two main questions:\n1. How does the diffusion process help improve the data quality compared with the selected modes? \n2. Is there a possibility that the discovered modes restrict the diversity of the generated data with a large IPC? How do the authors deal with this condition?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsS2a8oDzk", "reviewer": "Reviewer_y4kY", "summary": "The paper proposes a mode-guided diffusion model for dataset distillation. It aims to address the limitations of existing dataset distillation methods, such as insufficient sample diversity and high computational costs. The key idea is to leverage a pre-trained diffusion model without fine-tuning using distillation losses. It has three stages: Mode Discovery to identify data modes, Mode Guidance to enhance intra-class diversity, and Stop Guidance to improve the quality of synthetic samples. Experimental results on multiple datasets show that the method outperforms existing approaches.\n\n## update after rebuttal\nI agree to accept the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive assessment of our work, particularly our novel three-stage mode-guided approach, clear presentation, and contribution as a training-free dataset distillation method.\n\n## Addressing Concerns About Our Novelty\nIn dataset distillation, modes have been used to enhance diversity or sample from prototypes of the dataset distribution. Given these dataset prototypes, the goal is to generate a small dataset that maintains a similar distribution as the original dataset. Prior work [1] introduced noisy modes as $x_0$ instead of random noise; however, the high noise levels in the initial denoising steps often lead to outputs that are neither diverse nor fully representative. This explains why Disentangled Diffusion performs similarly to, or even worse than, simply using a pretrained diffusion model. The novelty of our approach lies in its efficient technical implementation, which requires no additional training and generalizes across various datasets and diffusion architectures, including Latent Diffusion (Class-to-Image), DiT, and Stable Diffusion (Text-to-Image), which is not the case of D$^4$M as shown in our experiments (Table 1 and Figure 4c).\n\n[1] Su, Duo, et al. \"D^ 4: Dataset Distillation via Disentangled Diffusion Model.\" CVPR . 2024.\n\n## Mode Guidance Theoretical Support\n\nWe aim to generate samples close to a given mode $m_i$ using mode-classifier guidance:\n\n$$\\nabla_{x_t} \\log p_t(x_t | m_i) = \\nabla_{x_t} \\log p_t(x_t) + \\nabla_{x_t} \\log p_t(m_i | x_t)$$\n\nDefining mode-classifier probability (how far is the current noise from the mode $m_i$) as:\n\n$$p(m_i | x_t) \\propto \\exp\\left(-\\frac{1}{2} |x_t - m_i|^2\\right)$$\n\nTaking the gradient and introducing guidance scale $\\lambda$:\n\n$$\\nabla_{x_t} \\log p_t(m_i | x_t) \\approx -\\lambda (x_t - m_i)$$\n\nSubstituting into the classifier guidance equation:\n\n$$\\nabla_{x_t} \\log p_t(x_t | m_i) = \\nabla_{x_t} \\log p_t(x_t) - \\lambda (x_t - m_i)$$\n\nWith the score function related to noise prediction:\n\n$$\\nabla_{x_t} \\log p_t(x_t) \\approx -\\frac{\\tilde{\\boldsymbol{\\epsilon}}_\\theta(x_t, t)}{\\sigma_t}$$\n\nLeading to modified noise prediction:\n\n$$\\epsilon_\\theta(x_t, t) = \\tilde{\\boldsymbol{\\epsilon}}_\\theta(x_t, t) + \\lambda \\sigma_t (m_i - x_t)$$\n\nSimilar to Classifier Guidance, the model pushes toward images near mode $m_i$, with $\\lambda$ controlling guidance strength.\n\n## Experiments with Other Mode Discovery Techniques\nWe tested more advanced mode discovery algorithms as suggested. GMM and Spectral Clustering improved our method in some settings:\n\n|Method|IPC 10|IPC 20|IPC 50|\n|---|---|---|---|\n|GMM|66.9±0.6|70.8±2.9|79.7±1.7|\n|Spectral Clustering|64.5±2.1|70.5±2.0|79.7±2.0|\n|$k$-means|66.4±2.4|71.2±0.5|79.5±1.3|\n\nTable 1: Experiments with suggested mode discovery methods on ImageNette.\n\n## Experiment with Complex distributions (imbalanced dataset)\nWe tested our method on class-imbalanced ImageNet-100 subsets. We created imbalanced datasets with \"low sample\" classes with 10-20 images and \"normal\" classes with  800 images. We vary low sample classes from 10 to 100 to generate a dataset with different degrees of imbalance and evaluate our method on IPC 10. The original test dataset is used to compute the accuracy. We reported the performance choosing 10 samples per class randomly of the imbalanced dataset,\n\n| Low samples classes | Random     | Ours       |\n| ------------ | ---------- | ---------- |\n| 0| 19.1 ± 0.4 | 25.8 ± 0.5 |\n| 10| 19.3± 0.2  | 25.1 ± 1.0 |\n| 20 | 18.2± 0.7  | 24.4 ± 0.2 |\n| 30| 18.8 ± 0.3 | 24.0 ± 0.2 |\n| 40 | 17.6 ± 0.1 | 23.9 ± 0.5 |\n| 50 | 17.7± 0.4  | 23.8 ± 0.1 |\n| 60  | 18.9± 0.5  | 23.4 ± 0.5 |\n| 70| 19.2± 0.2  | 23.0 ± 0.8 |\n| 80 | 19.0± 0.1  | 23.0 ± 1.0 |\n| 90 | 19.3± 0.2  | 22.2 ± 0.3 |\n| 100 | 18.3± 0.5  | 20.8 ± 0.2 |\n\nEven when estimating modes from complex distributions with limited samples, our method maintains strong performance, consistently outperforming random selection. Notably, using a DiT trained on a balanced dataset achieves an accuracy of 23.3 ± 0.4, which our method surpasses even when 60% of the classes have low sample sizes.\n\n## Measuring Diversity and Representativeness Across MGD³ Components\nOur three-stage process optimizes diversity and representativeness while maintaining realism. We include classification accuracy from an ImageNet-trained model to assess realism:\n\n| Method         | Diversity | Representativeness | Classification Acc |\n| -------------- | --------- | ------------------ | ------------------ |\n| DiT| 0.28| 0.55| 0.99|\n| Mode Discovery |0.65| 0.24| 0.91|\n| MG| 0.71| 0.82| 0.83|\n| MG+StopG | 0.72| 0.80| 0.95|\n\nMode Guidance improves diversity (0.71) and representativeness (0.82) but lowers accuracy (0.83), indicating reduced realism. Stop Guidance mitigates this, restoring accuracy (0.95) while preserving diversity (0.72) and representativeness (0.80). This achieves better trade-off than DiT (0.99 accuracy but significantly lower diversity and representativeness).", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods are well-suited for the problem of dataset distillation. And the evaluation criteria, including accuracy and diversity metrics, are appropriate for assessing the performance of dataset distillation methods.", "theoretical_claims": "The paper does not have complex theoretical proofs. But the concepts and operations in the method, such as mode discovery using clustering algorithms and mode guidance based on the diffusion process, are based on established theories in machine learning and diffusion models. There are no obvious theoretical flaws in the presented ideas.", "experimental_designs_or_analyses": "The experimental designs are relatively sound. The authors use appropriate baselines, different evaluation protocols, and conduct ablation studies. But the number of runs in the comparison with sota methods could be increased to obtain more stable results.", "relation_to_broader_scientific_literature": "Compared with prior works that rely on fine-tuning with distillation losses, this paper's training-free approach is a new contribution in this field.", "essential_references_not_discussed": "No obvious essential missing reference to my best knowledge.", "other_strengths_and_weaknesses": "Strengths: \nThe proposed MGD3 method is novel in its approach to dataset distillation. The three-stage process of Mode Discovery, Mode Guidance, and Stop Guidance is a creative combination of existing concepts in diffusion models and dataset distillation, providing a unique solution to the problem of sample diversity and representativeness. \nAnd the paper is generally in good clarity. For instance, Figure 1 clearly differentiates optimization-based and generative-based dataset distillation.\n\nWeaknesses:\nSome aspects of the method, like using clustering for mode discovery, may not be entirely new in the context of generative models. The novelty might be diluted when compared to truly revolutionary ideas in the field. \nAnd the theoretical depth of the method could be further strengthened. There is few original formula to explain why the three-stage process works optimally in terms of diversity and representativeness.", "other_comments_or_suggestions": "Typos: Unified 'dataset' and 'data set' in the Introduction section.", "questions_for_authors": "Question 1: In the mode discovery stage, the authors use K-Means centroids. Have you considered other more advanced clustering algorithms, and what are the potential advantages or disadvantages? If other algorithms can improve the performance, it might further strengthen your method.\nQuestion 2: The performance improvement of your method is demonstrated on several Image datasets. But how about the performance on datasets with more complex distributions or smaller sample sizes? A response showing good performance on such datasets would enhance the generality of your method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission has been reviewed by 4 reviewers.

After the rebuttal, all reviewers acknowledged the merits of the manuscript and agreed that it meets the bar of ICML.

Essentially, the reviewers found the proposed method well-motivated, well-designed, and yields good results.

There is no basis to overturn the consensus. As such, the AC recommends the acceptance. Congrats!

Please, however, do account for the suggestions in the final version.
Decision: Accept (oral)