Taming Diffusion for Dataset Distillation with High Representativeness
Authors: Lin Zhao
Yushu Wu
Xinru Jiang
Jianyang Gu
Yanzhi Wang
Xiaolin Xu
Pu Zhao
Xue Lin
Venue: ICML 2025 poster (2025)
Keywords: dataset distillation
Primary area: deep_learning
TL;DR: A new paradigm for diffusion-based dataset distillation with high representativeness.

Abstract: Recent deep learning models demand larger datasets, driving the need for dataset distillation to create compact, cost-efficient datasets while maintaining performance. Due to the powerful image generation capability of diffusion, it has been introduced to this field for generating distilled images. In this paper, we systematically investigate issues present in current diffusion-based dataset distillation methods, including inaccurate distribution matching, distribution deviation with random noise, and separate sampling. Building on this, we propose D$^3$HR, a novel diffusion-based framework to generate distilled datasets with high representativeness. Specifically, we adopt DDIM inversion to map the latents of the full dataset from a low-normality latent domain to a high-normality Gaussian domain, preserving information and ensuring structural consistency to generate representative latents for the distilled dataset. Furthermore, we propose an efficient sampling scheme to better align the representative latents with the high-normality Gaussian distribution. Our comprehensive experiments demonstrate that D$^3$HR can achieve higher accuracy across different model architectures compared with state-of-the-art baselines in dataset distillation. Source code: https://github.com/lin-zhao-resoLve/D3HR.

(Deng al., Taming Diffusion for Dataset Distillation with High Representativeness
Lin Zhao' Yushu Wu
Xinru Jiang'
Jianyang Gu
Yanzhi Wang' Xiaolin Xu' Pu Zhao Xue Lin
Abstract
storage efficiency, dataset distillation generates a small dis- tilled dataset to replace the full dataset for training (Wang et al., 2018; 8; Loo et al.; Lee & Chung, 2024), while target- ing comparable performance as the full dataset. In contrast to dataset pruning methods (selecting a subset from the full dataset) (Zhang et al., 2024; Sorscher et al., 2022; Tan et al., 2024), dataset distillation treats the distilled dataset as continuous parameters that are optimized to integrate infor- mation from the full dataset. This approach achieves better performance, especially with relatively high compression rates. (Du et al., 2024b; Sun et al., 2024; Yin et al., 2023).
Recent deep learning models demand larger datasets, driving the need for dataset distillation to create compact, cost-efficient datasets while maintaining performance. Due to the powerful image generation capability of diffusion, has been introduced this field for generating dis- tilled images. In this paper, we systematically in- vestigate issues present in current diffusion- based dataset distillation methods, including inaccurate distribution matching, distribution deviation with random noise, and separate sampling. Building on this, we propose DdHR, a novel diffusion- based framework generate distilled datasets with high representativennss.. Specifically, we adopt DDIM inversion to map the latents of the full dataset from low-normality latent domain to a high-normality Gaussian domain, preserv- ing information and ensuring structural consis- tency to generate representative latents for the distilled dataset. Furthermore, we propose an efficient sampling scheme to better align the rep- resentative latents with the high-normality Gaus- sian distribution. Our comprehensive experiments demonstrate that DHHR can achieve higher ac- curacy across different model architectures com- pared with state-of-the-art baselines in dataset distillation. Source code: https: //gi thub. com/ hhwplliinnhh--eeo resoorresvee/
Recent data distillation methods primarily focus on improv- ing the efficiency of the distillation algorithm (Yin et al., 2023; Sun et al., 2024: Su et al., 2024; Gu et al., 2024). Among them, the diffusion-basee methods achieve strong performance due their remarkable generative capabilities. They capture the most informative features of the full dataset by extracting representative latents from the pre-trained Vari- ational Autoencoder (VAE) latent space (Su et al., 2024; Gu et al., 2024). Thus, the architectural dependency during distillation is eliminated, enabling a one-time generation cost for training various model architectures. This strat- egy enhances the efficiency of distillation and significantly improves cross-architecture generalization.
However, the above diffusion-based methods may struggle to guarantee the representativeness of the distilled dataset. (i) Inaccurate distribution matching. This challenge arises from leveraging diffusion models to generate distilled datasets, relying on accurately modeling the VAE feature distributions for proper decoding. However, as shown in Figure 1a, the low-degree of normality! for the distribu- tion in VAE latent space results in difficulties in effectively matching the distribution. (ii) Distribution deviation with random noise. Besides, these methods generate distilled images from the initial noise, which inject unpredictable ran- domness latents, potentially violating the structural and representative information gathered in the VAE space with potential distribution deviation. (iii) Separate sampling. Moreover, the distilled data are individually matched to parts of the entire VAE distribution without considering the
1. Introduction
Driven by the scaling law, recent deep learning models have expanded in scale, demanding exponentially larger datasets for optimal training performance, which makes dataset maintenance costly and 2009; Alexey, 2020). To achieve both computational and
NNotthesttrrn University "The Ohio State University. Corre- spondence to: Pu Zhao <p.zhao @ northeasterm.edus edu>, Xue Lin <xue.lin&& northeasterneedu..
Normality refers to the degree of the latent space data con- forms to normal distribution. A higher degree of normality indicates closer conformity.
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Taming Diffusion for Dataset Distillation with High Representativeness
dataset distillation methods, with detailed ablation stud- ies and analysis.
2. Related Work
Dataset Distillation. Dataset distillation (Wang et al., 2018) is designed to improve training efficiency by generat- ing a small dataset to replace the original large dataset for training. Previous methods (Zhao et al., 2020; Cazenavette et al., 2022) employ bi-level optimization process, which updates the model parameters and the distilled dataset simul- taneously. According to the different matching strategies in the distillation, the methods can be classified into: gradient matching (Zhao et al., 2020; Zhao & Bilen, 2021; Lee et al., 2022), distribution matching (Wang et al., 2022; Zhao Bilen, 2023; Zhao et al., 2023; Zhang et al., 2023; Deng et al., 2024), and trajectory matching (Cazenavette et al., 2022; Cui et al., 2023; Du et al., 2023; 2024a).
(a) Low- Low-normaiity (VAE)
(b) (oah-oommaity(mapee))
Figure/ 1. I-SNE visualization of the low-normality VAE space and norm-normality noise space for class "Goldfish". The blue contour lines are the probability density curves of the distribution using kemee density estimation, highlighting the structure and concentration of the latents (blue dots). * in (b) marks the 10 representative latents generated in the noise space, corresponding * in (a) after DDIM sampling, which preserves the structure of VAF space and concentrates in high-density regions.
overall distribution of the distilled dataset, which may lead to an incomplete representation of the desired distribution.
Recent methods demonstrate that the bilevel optimization approach is cone-consuming and does not scale well for large datasets (Loo et al.; Sun et al., 2024; Yin et al., 2023; Shao al.. 2024; Yu et al., 2024). To address this, more efficient dataset distillation methods are developed. SRe2L. (Yin et al., 2023) and DWA (Du et al., 2024b) use the BN layers of the pre-trained teacher model as supervisory infor- mation and synthesize each distilled image individually to accelerate the generation. RDED (Sun et al., 2024) empha- sizes the importance of image realism and generates images by stitching patches selected by the teacher model. However, these methods rely BN layers or the data distribution of other layers from the teacher, which limits their applicability to different model architectures.
Inspired by these insights, we propose a novel frame- work, Taming Diffusion for Dataset Distillation with High Representativeness (D²H)). We reveal an optimal strategy for modeling the VAE latent domain by mapping it to simpler distribution domain while preserving fundamental information and ensuring structural consistency. Toward this end, we adopt the deterministic DDIM inversion to map latents from the VAE space to the noise space with higher normality, so that a Gaussian distribution can accurately match the noised latent distribution, as illustrated in Fig- ure lb. Then we sample a set of latents from the estimiated Gaussian distribution and convert the latents back to VAE space by DDIM sampling for generating the distilled dataset. To ensure the distribution matching between the distilled dataset and the full dataset, we incorporate a new constraint with our group sampling method, which tights the similarity between the distribution of the generated latents and that of the original latents.
Unlike the above methods, the diffusion- methods leverage the powerful generative capabilities of diffusion models, eliminating the dependency on teacher models. Specifically, D*M (Su et al., 2024) generates the distilled dataset by clustering the latents of the full dataset in the VAE latent space. Gu et al. (2024) reconstruct the training loss of diffusion model in the VAE space, aiming to generate images representing the full dataset. However, it requires mie-ttnnng multiple models for large datasets. The above methods struggle to accurately capture representative latents that match the full dataset distribution.
Our contributions can be summarized as follows:
We propose DRHR, dataset distillation framework leveraging DDIM inversion to map the original data distribution to a Gaussian distribution, which makes it casier for distribution matching (better normality) with less randomness (bijective DDIM mapping to preserve structural information), effectively addressing inaccu- rate distribution matching and distribution deviation in previous diffusion- based methods.
Diffusion Models. Diffusion models target approximat- ing the data distribution qe (0(x)) with learned model dis- tribution (zo).. Denoising diffusion probabilistic mod- els (DDPMs) (Ho et al., 2020) optimizes a variational lower bound based on the Markov Chain, allowing models to gen- crate high-quality samples. DDIM (Song et al., 2020) built on DDPMs provides more efficient and deterministic sam- pling method by removing the randomness in each reverse step, thus offering a faster and more controlled generation.
We propose a novel sampling scheme to efficiently align the distributions of generated distilled latents and original latents from the full dataset, overcoming the shortcoming of individual sampling.
We conduct extensive experiments and demonstrate that DRHR significantly outperforms state-of-the-art
2

---

Taming Diffusion for Dataset Distillation with High Representativeness
Due to the deterministic approach, DDIM sampling can be inverted, allowing users to map their generated or real im- age back to its corresponding noise representation. Previous works (Rombach et al.. 2022) rely on the UNet architecture, and recent works (Peebles & Xie, 2023; Esser et al., 2024; Shen et al., 2025) demonstrate the superior performance of Diffusion Transformer (DiT) in image generation by intro- ducing Transformer architecture to diffusion models.
3.3. Motivation
Existing dataset distillation works have certain limitations as discussed below, which motivate us to develop our D²HR.
Scalability to multiple model architectures. In some dataset distillation works (Yin et al., 2023; Loo et al.), a teacher model T trained by F is used to guide the distilla- tion process: S = A(F,T)) T)) It relies on a specific model architecture. new model architecture requires distilling an- other dataset with the new teacher architecture expensively trained on the full dataset, making it hard to scale.
3. Background, Formulation, and Motivation
3.1. Preliminaries on Diffusion Models
To address this problem, recent works (Gu et al., 2024; Su et al., 2024) leverage the powerful generative capabilities of diffusion models G (Azizi et al., 2023) to produce S without the guidance from a specific teacher model T: S = A(F, G). In this setup, once S is generated with just a one- time cost, it can be applied various model architectures (e.g., ResNet, EfficientNet, and VGG), instead of generating multiple S for each model architecture.
Diffusson Models (Ho et al., 2020) are proposed to generate high-quality images by transforming the random Gaussian noise Fy into the image Zo through a discrete Markov chain. Among them, Latent Diffusion Model (Rombach et al., 2022) enhances efficiency by leveraging VAE to compress the pixel space X, into the latent space Zg= 2g E(zo), and decoding the latents back to images at the end of diffu- sion backward process: To- = Dzz)). During training, the forward process adds random noise to the initial latent 20:
Inaccurate distribution matching in the latent space. There are still some limitations for the current diffusion- based data distillation methods. Specifically, the origi- nal images of the class C are converted to latents Zo,c
Zt = 20 + 11-1aaa ate, with e~, I), (1)
{ &b c] i=1 through the VAE, and the methods propose to find and describe the data distributions in the latent space, for further synthesis of n latents for C (i.e., n images-per- class (IPC)), hoping that n latents can match the real data distributions in the latent space. For example, Su et al. (2024) apply K-means to cluster Zo,e into n groups to get the centroids for better synthesis of n latents. Gu al. (2024) minimize the cosine similarity between each synthe- sized latent and random subset Z,, where Z, C Zo,c.
where at is a hyper-parameter and 2 represents the noise at timestep t. During inference, the backward process itera- tively removes noise in 2, through schedulers to get 20-
Denoising Diffusioo Implicit Models (DDIM) (Song al., 2020) introduces a deterministic sampling method by config- uring the variance of the distribution at each step, enabling a one-to-one mapping from 2t to 2p, as expressed below:
However, the distribution of the original dataset in the latent space is too complex to accurately describe or match. is a multi-component Gaussian mixture distribution as below.
0t-1
1t-1
+
Ctt
Lemma 3.1. Each latent in VAE latent space is randomly sampled from distinct component of multi-component Gaussian mixture distribution.
Va--
V
1
t.C.
at-1
ax
(2)
where EZ( 1, C) is function with trainable parameters 0, C denotes the class condition.
The proof is shown in Appendix A.I. Zo,c is a multi- component Gaussian mixture distribution and hard-to-fir with examples shown in Figure la.
3.2. Problem Formulation for Dataset Distillation
Therefore, it is challenging to generate representative la- tents the complex Zo,c- The K-means algorithm (Su et al., 2024) assumes that each cluster is spherical for casier clustering Zo,c, which does not align with the more com- plex distribution of Zo,c. The cosine similarity (Gu et al., 2024) focuses on the direction to subset of the distribution, and ignores the distance, making it less effective to capture the differences in probability density within Zu,c-
Dataset distillation aims to synthesis a small distilled dataset Ny
S = ($t, N to replace the full dataset F {xx,9) ) for training, where Ns < NN. &, & x; are images, and D: & y: are labels. To maintain high performance with smaller S, Algorithm A is proposed to address the problem of generating S from F: S = A(F) Ns < NN. It is expected that training a model on S achieves a comparable performance to training on F. assuming that S encapsulates substantial information from the original F.
Distribution deviation with random noise. Current meth-
3

---

Taming Diffusion for Dataset Distillation with High Representaativeness
DDIM Inversion
DDIM Sampling
n
Gaussian Sampling
C/.c
Distribution Matching
Cre
IPC
Zo,c
ZT,C
BT.C
arg (
1<k<m
Figure 2. Illustration of proposed DH framework. With the latents from the VAE, DDIM inversion is applied to map the latent embeddings to a Gaussian domain with better normality, which can further be matched to Gaussian distribution. Then, we follow Sec- tion 4.4 to sample representative latents based on different IPC requirements, and generate true images through DDIM sampling.
ods generate images from the initial noise. They obtain the noise either by adding random noise to the representative latents in the VAE space via Equation (1) (Su et al., 2024), or by generating it randomly, with the representative latents obtained by multiple fine-tuned diffusion models (Gu et al., 2024). However, due to the inherent randomness of initial noise, the distribution of denoised latents may deviate or shift from that of the sampled latents before adding noise, shown in Appendix B- B-Figure A1, leading to potential accuracy loss after training.
4.2. Domain Mapping
It is essential that the small distilled dataset can sufficiently represent the original large data. Previous works (Gu et al., 2024; Su et al., 2024) use the distribution in VAE latent space Zu,c
{ & c) to distill the information of class C from the original dataset. As discussed in Sec- tion 3.3, it is challenging to generate latents to match the VAE latent space zo.c. To make it easier for distribu- tion matching, we further map the distribution in Zo,c to a high-normality Gaussian space Zr.e by DDIM inversion (Dhariwal & Nichol, 2021) for each class as illustrated in Figure 2. Specifically, each latent 2g class C, we perform few DDIM inversion steps follows,
Separate sampling. Additionally, current diffusion-bbsed works generate representative latents without considering their overall distribution. Due to the limited number of samples, although each latent may be matched to parts of the desired distribution Zo,c, the overall distribution of the sampled latents may not closely align with Zo,e, resulting in an under-represented distilled dataset.
a1+1
Ze+1
-V
2,+
Oz
a+++1
V
V
a141
4. Methodology for Dataset Distillation
(3)
Motivated by the limitations, we present DRRR, a diffusion- based dataset distillation framework via domain mapping.
DDIM inversion is based on the assumption that the ODE process of DDIM sampling can inverted few steps. There are other choices for domain mapping such as di- rectly adding random Gaussian noise through the forward of Denoising Diffusion Probabilistic Models (DDPM) in Equation (1). However, DDPM inherently introduces ran- domness in the mapping, leading to difficulties for accu- rately describing the distribution with potential distribution shift or information loss, as presented in Figure 3a. Differ- ent from DDPM. DDIM inversion with Equation (3) offers the key prerequisites for obtaining representative subset in Zr,e: (i) Information Preservation: The mapping be- tween two domains is bijective because of the deterministic process, where each element in Zr.c directly corresponds to an element in Z and vice versa, which avoids the loss of key features during the mapping process. (ii) Structural Consistency: The latents in 3 can retain the structural
4.1. Framework Overview
Our framework has three stages: domain mapping, distribu- tion matching, and group sampling. As shown Figure 2, first convert images to latents through VAE and map the VAE latent domain to the noise domain via DDIM inversion. Next, with distribution matching, we use Gaussian distri- bution to match the mapped noised latent distribution from the previous stage. Finally, in the group sampling stage, we sample a number of latents from the Gaussian distribution and convert the latents back to images to generate the dis- tilled dataset. The algorithm is shown in Algorithm with details introduced in the following subsections.

---

Taming Diffusion for Dataset Distillation with High Representatiiveess
Algorithm 1 D'HHR Algorithm
function (PDF) of žτ,e can be expressed as the following,
Input: Full dataset F (X, Y), VAE encoder-decoder E, D, Inverse steps T. IPC number n
- -
f(br.c) = II i=1 2
exp
(4)
Output: Distilled dataset S
2(0
Initialize S ==0 0
for each class do
where prr and o'e.c denote the mean and standard devia- tion for each dimension of pr,c and Ør,c. respectively.
We + label of class C Z = C(X Initialize ZT,c 0 for each =o Zo.c do
4.4. Group Sampling
Update 2o to 2T via Equation (3) with 7 steps b DDIM Inversion
With the Gaussian distribution 2τ.c for the class C, next we generate the distilled dataset. Specifically, we gener- ate n representative latents for each class C. by capturing the statistic characteristics of 2T,C. Using the Ziggurat algorithm (Marsaglia & Tsang, 2000), we can randomly sample i.i.d. latents probabilistically following &r.c with f2r,,c), which together form subset. However, although each latent is sampled from Sr,c, due to the limited number of samples (n1) for distillation, the overall distribution of the subset with n samples may still deviate from the desired 2T,C. leading to certain performance degradation.
Zr.c* + Zr.c U {zr}
end for
MT.C t- mean of Zr.c
0i.e t varionce of Zr.c
Zr.e + -NWwr,c,
Initialize Lr. =0 for k 1 to do
> Group Sampling
Gaussian Sample n latents to get R² c
Calculate Ch via Equation (5) for R* Lr.C = Lr, U (C+,c)
end for
j arg moggin(Ci.. 1gkgm
To address this problem and further improve the quality of the distilled dataset, we propose to sample numerous ran- dom subsets (each subset consists of n i.i.d. latents) and efficiently search for the most representative subset as the final subset. Specifically, by repeating the above sampling with the Ziggurat algorithm multiple times, random sub- sets are available with varying statistical distributions. To search for the most representative subset, we further propose an efficient algorithm that selects the subset statistically clos- est to ÂT,C among m random subsets, thereby improving the matching effectiveness and stability.
for each êT E R² do Update 2T to 2o via Equation (2) in T steps b DDIM Sampling
S S+ (D(o))
end for end for return S
information of Zo ensuring the structural consistency and distribution alignment.
To select the final subset, we design statistic evaluation metric LT.C to measure the distance between the subset distribution and 2r,c, and select the subset with the minimal value. Specifically, Lr,, computes the differences in the key statistics between two data distributions, including the mean, standard deviation, and skewness. LT,C can be expressed as the weighted sum of the three.
4.3. Distribution Matching
With DDIM inversion, we can obtain a discrete distribution Σ₁,- = { C We have the following lemma.
Lemma 4.1. For DDIM inversion with T steps, with suffi- ciently large T, Zr.c can be approximated as a Gaussian distribution.
Cr. = Ap Cy.T.C + da C.TT. + Aa Ln,T.e (5)
The proof is shown in Appendix A.2. Since the latents in ZT can be interpreted independently and identically Gaussian distributed (i.i.d.) latents, we approximate Z with Gaussian distribution LT,c based on the law of large numbers (Hsu & Robbins, 1947). To obtain the statistical properties of Zre, we compute the the mean frrc and variance cf from Zr.c as the mean and variance of 2r.c.
where Cu.T.e. Lo,T, and L,,,T. denotes the evaluation of pr.e, Ør,, and n,r, respectively, with the formulation below,
Cu = GuTC=((TT.- pT,c)= = Ei= 21e - mTe)), (6)
Lore = (V 1E (8 -rrcc)² on.c) )²
Thus, we can obtain the Gaussian distribution ŁT, A N(PT,e,of 2f,c) in the latent space to represent the class C. Since the dimensions in the noise space of DDIM inversion are independent (Song et al., 2020), the probability density
Cuy (n-1)in-2) Ei
(the-mne) -n
(8)
on.
where PreeTTe and n.re are the mean, standard devia- tion and skewness of 2r, $t. indicates the 7th element
5

---

Taming Diffusion for Dataset Distillation with High Representaativenss
ResNet-18
101
Datasel
IPC
SR
DWA
D'M
RDED
Ours
SRe2L
DWA
D'M
RDED
Ours
10
29.3 + 0.5
32.6 + 0.4
33.5
37.1 = 0.3
41.3 = 0.1
24.3 0.6
25.2 + 0.2
29.4
33.7 0.3
35,8 0.6
CIFAR-10
50
45.0 +: 0.7
53.1 +: 0.3
62.9
62.1 + 0.1
70.8 0.5
34.9 0.1
48.2 +: 0.4
54.4
51.6 6 0.4
63.9 0.4
10
31.6 + 0.5
39.6 0.6
38.1
42.6
49.4 0.2
30.7 0.3
35.9 + 0.5
33.3
41.1 +
46.0 =
CIFAR-100
50
52.2 + 0.3
60.9 + 0.5
63.2
62.6 0.1
65.7 0.3
56.9 0.1
58.9 + 0.6
64.7
63.4 +
66.6 0.2
10
16.1 + 0.2
40.1 + 0.3
34.1
41.9 0.2
44.4 = 0.1
14.6 1.1
38.5 =: 0.7
33.4
22.9 +: 3.3
43,2 = 0.5
Tiny-ImageNet
50
41.4 0.4
52.8 1: 0.2
46.2
58.2
0.1
56.9 :: 0.2
42.5 0.2
54.7 1: 0.3
51.0
41.2 1: 0.4
59.4 0.1
100
49.7
1 0.3
56.0 0.2
51.4
59.3 0.1
51.5 0.3
57.4 0.3
55.3
61.4 0.1
10
21.3 + 0.6
37.9 0.2
27.9
42.0 0.1
44.3 0.3
30.9 = 0.1
46.9 + 0.4
34.2
48.3 1.0
52.1 0.4
ImageNet-1K
50
46.8
+: 0.2
55.2 0.2
55.2
56.5 0.1
:9.4
60.8 = 0.5
63.3 0.7
63.4
0.4
66.1 + 0.1
100
52.8
+: 0.3
59.2 2: 0.3
59.3
62.5 : 0.0
62.8 = 0.2
66.7 + 0.2
66.5
68.1 - 0.0
Table Comparison of Top-1 accuracy across four datasets with other methods. Following SR (Yin al., 2023), all methods use ResNet-18 18 as teacher model, and train separately on ResNct- 18 and 10s with the soft label. As D'M (Su et al., 2024) and our D'HR do not need teacher models, we just use the label as supervision. For IPC 100, some results are unavailable with 5." their inappropriate default parameter settings.
is both time- and space-efficient. The sampling is per- formed based the statistic parameters and we sample different subsets in parallel on the GPU. Our sampling im- plementation only takes 2.6 seconds per class on a single RTX A6000 with m = 1e6 for ImageNet-IK, IPC 10. Additional runtime results under different settings are pro- vided in Appendix B-Tables A2 to A4. Moreover, instead of storing representative latents, tilled image directly from statistical parameters, which are space-efficient as shown in Appendix B- B-Figure A2.
(a) Base-DDPM
(b) D'HR
generate the dis-
Figure 3. t-SNE visualization of the representative latents in VAE space generated by Equation (1) and D'HR for class "Goldfish". It can be observed that the latents accurately represent the VAE distribution in DRH
5. Main Results
of the random subset. As Gaussian distribution is perfectly symmetric, 71 = 0.
5.1. Experimental Details
Datasets. Experiments are conducted on both small-scale and large-scale datasets. For small-scale datasets, we use CIFAR-10 and CIFAR-100 (Krizhevsky al., 2009) with 32 x 32 resolution. For large-scale datasets, we use Tiny- ImageNet (Le & Yang. 2015) with 200 classes (500 images per class, 64 x 64 size) and ImageNet- 1 K (Deng et al., 2009) with 1, 000 classes (1M images, 224 x 224 resolution).
With the above evaluation metric function, for the ktth subset, can compute its metric C And we select the jth subset with the smallest Cr. as the most representative subset:
j = arg min( min(Cr.e)))) c). 15kšm
(9)
After we select the most representative subset, they are converted to real images through Equation (2) with DDIM and VAE decoder.
Network architectures. To validate the upplicability of our method across different architectures, we adopt ResNet- 18/ResNet-10 101 (He al., 2016), MobileNet- V2 (Sandler et al., 2018) and VGG-11 (Simonyan & Zisserman, 2014) as backbones, following prior works (Du et al., 2024b; Yin et al., 2023; Sun et al., 2024).
4.5. Advantages DRR
Compared to other methods, we highlight our approach has the following notable advantages: (1) We present a domain mapping method map VAE latent domain into Gaussian domain with better normality, leading more accurate and efficient distribution matching, as illustrated in Fig- ure lb. (2) We ensure that the entire subset is represen- tative closely aligned with the desired distribution, rather than just sampling individual elements without consider- ing their overall distribution. (3) Our sampling algorithm
Baselines. We compare D²HR with four state-of-the methods: S (Yin al., 2023), DWA (Du al., 2024b). D'M (Su al., 2024), RDED (Sun et al., 2024), following the same evaluation configuration. We exclude compari- son with Minimax (Gu et al., 2024) in the main Table 1, since their method focuses on handling small subsets of ImageNet- K and requires training multiple diffusion mod-
6

---

Taming Diffusson for Dataset Distillation with High Representaativenss
Student Teacher
ResNet-188
MobileNet-V2
VGG-11
DM Sampling Design
Acc. (%)
SRe"L
21.3 0.6
15.4 0.2
Base-DDPM
37.3 0.7
ResNer-18
RDED
42.3 + 0.6
40.4 + 0.1
36.6 0.1
Base-RS
41.6 0.2
Ours
44.3 = 0.3
42.3 = 0.7
38.3 0.2
42.6 02.6000 0.1
SRe"L
19.7 1 0.1
10.2 2.6
Lo
42.3 12.300. 0.2
MohileNet-V2
RDED
34.4 + 0.2
33.8 + 0.6
28.7 + 0.2
42.4 0.1
Ours
43.4 0.3
46,4 +: 0.2
37.8 0.4
C + Laz
43.3 0.2
SRe"L
16.5 0.1
10.6 L 0.1
Cy. + Lyy
43.0 + 0.1
VGG-I1
RDED
22.7 0.1
21.6 = 0.2
23.5 + 0.3 28.1 + 0.1
L. + fyn
42.5 + 0.2
Ours
25.7 + 0.4
24.8 + 0.4
Ln + Lo Ly (Ours)
44.3 : 0.3
Table 2. Evaluating Top-1 accuracy for cross-architecture gen-
Table 3, Ablations of DoHR on 18 for ImageNet-IK with IPC = 10. 'DM' indicates domain mapping, and 'RS' indicates random sampling.
eralization on ImageNet-IK, IPC 10. As VGG-11 lacks BN layers, the results of SRe²L L are not available with '-'
els with expensive computation for large-scale scenarios. Instead, we report results under their setup in Section 6.4. Implementation details. We adopt the pre-trained Diffu- sion Transformer (DiT) and VAE from Peebles & Xie (2023) in our framework, originally trained on ImageNet-1K 1. We further adjust the conditioning labels of DiT, and fine-tune the pre-trained model with 400 epochs for each dataset ImageNe,,,,N,, CIFAR-10 10 and CIFAR-100) to adapt its generative capacity to the specific data distributions. For distillation, we employ 31 steps for DDIM inversion and sampling. During the validation, we follow other works (Yin et al., 2023; Du et al., 2024b; Su et al., 2024; Sun et al., 2024) to use lobt-label of the teacher model as supervision for training. All experiments are conducted on Nvidia RTX A6000 GPUs or Nvidia A100 40GB GPUs.
5.3. Cross-architecture Generalization
To further evaluate cross- moss- performance, on ImageNet- IK, we compare D²HR with the recent baseline SRe'L and RDED (SOTA performance on ImageNet-1K). For DR, the teacher model refers to using its soft label for validation. As depicted in Table 2, we achieve significant accuracy im- provement across all cross-model evaluations. We highlight that with D D'HR, a one-time cost is sufficient achieve sat- isfactory results across various models. Other baselines need to run their algorithms multiple times to generate mul- tiple distilled datasets when the model architecture changes. The comparison results for more architectures are provided in Appendix B- B-Table A8.
6. Analysis
5.2. Comparison with State-of-the-art Methods
6.1. Ablation Studies
As shown in Table 1, DHHR demonstrates superior per- formance across all IPCs compared with baselines.
Effectiveness of Domain Mapping by DDIM Inver- sion. To demonstrate the effectiveness of domain mapping with DDIM inversion, we experiment with two configura- tions: (i) domain mapping with Equation (1) (denoted as DDDM), and (ii) domain mapping with Equation (3) (denoted as Base-RS). Note that for both configurations, we perform distribution matching and individual Gaussian sampling (i.e., random sampling (RS)), while our proposed group sampling method in Section 4.4 is not applied.
Large-scale datasets. We first validate the practicality of DAHR on Tiny-ImageNet and ImageNet-1K K at various IPCs. For Imag-em, while RDED performs well on ResNet- 18, its reliance on the teacher model leads to a significant drop on cross ResNet-101. contrast, HR consistently delivers SOTA performance on both ResNet- 18 and ResNet- 101. Similarly, for ImageNet-IK, D²HR achieves higher performance across all IPCs compared with all baselines.
As shown in Table 3, by comparing the results of Base- DDPM and Base-RS, we can observe that domain mapping with DDIM inversion outperforms that of DDPM with an accuracy improvement of 1 11.5%. As we discussed in Sec- tion 4.2, the added noise through DDPM causes structural information loss and domain shift for the mapping from Zo,c to Zr,c. leading to difficulties to obtain a representa- tive subset for Zo.c- By contrast, the determinism of DDIM inversion resolves the issue with its information preservation and structural consistency, as illustrated in Figure 3b.
Small-scale datasets. For CIFAR-10 and CIFAR-100, DOHR surpasses all baselines. Notably, compared with the best-performing baseline RDED, on CIFAR-10, our method achieves significant improvements of 12.5% for ResNet-18 and 17.4% for ResNet-101 101 at 50 IPC.
In addition, we provide comparison with other state-of- the-art methods that use validation settings different from ours. To ensure fairness, we evaluate our method under their settings, as shown in Tables A6 and A7, to demonstrate our superiority.
Effectiveness of Group Sampling. As shown in Table 3,
7

---

Taming Diffusion for Dataset Distillation with High Representaativeess
34.5
45.5
54.5
Ours
33.5
44.5
53.5
44.0
Accuracy (%)
43.5
52.5
43.0
31.5
42.5
51.5
30.5
41.5
50.5
42.0
40.5
49.5
Base
Ours
Base
Ours
Base
Ours
41.0
(a) 5 IPC
(b) 10 IPC
(c) 20 IPC
24
28
30
32
36
38
Number of Inversion Timesteps
Figure 4. Validation of the effectiveness and stability of group sampling on ImageNet-IK. For our D'HR and Base-RS, we generate distilled datasets each IPC. Each datasct undergoes rounds of validation, resulting in 9 data points per hox plot.
Figure 5. The accuracy variation under different inversion timesteps for ImageNet-IK, IPC 10.
D4M
by comparing results with Base-RS. RS, proposed group sampling method can significantly outperform individual Gaussian sampling, demonstrating its effectiveness.
Base DDPM
Furthermore, we conduct ablation studies with different combinations in Equation (5) to verify the contribution of each metric in the sampling schedule. As demonstrated in Table 3, each metric individually increases the accuracy, indicating that the subset selection effectively brings the dis- tribution of the sampled representative subset closer to ZT.c- Combining all the metrics can lead to the best performance with the most representative subset.
Ours
Figure 6. Visualization the distilled images for the class "gold- fish" on ImageNet-IK, IPC
5.
Besides, we highlight that our group sampling method en- hances the stability of the distilled dataset. As presented in Figure generate distilled datasets for the same dataset multiple times with both our D'HR and Base-RS. The re- sults demonstrate that D²HR not only improves the accuracy but also enhances stability with smaller variance.
mance loss. When f becomes large (e.g., t 40), although our distribution matching can accurately represent the real distributions which becomes more normal, the real distribu- tions suffer from more significantly structural information loss due adding more noise, which in turn degrades the performance of DDIM inversion.
6.2. Analysis of Different Inversion Steps
In Lemma 4.1, as the number of inversion steps increase, 2,c gradually transits from a complex Gaussian mixture distribution to a standard Gaussian distribution, leading to a easier distribution matching. We provide feature visual- increases Appendix B-Figure A3, validating Lemma 4.1. However, L increase, the added noise also increases, gradually diminishing the retention of original image structural information and reduc- ing the reconstruction quality of DDIM inversion.
6.3. Image Visualization
We present the visualization results of distilled images gener- ated by D4M, Base-DDPM, and our D2HR. Figure 6 shows that the images generated by D (Su et al., 2024) are sim- plistic and unrepresentative due to the incorrect retrieval of representative latents. Base-DDPM with Equation (1), the noise space fails to generate representative latents due randomness, resulting in outputs with overly simplistic struc- tures, corresponding to Figure 3a. In contrast, the images generated by D²HR are both representative and diverse.
izations illustrate the changes
Specifically, there is a trade-off between maintaining the Gaussian assumption and preserving image structural in- formation across different steps t. To assess the impact of different steps, we conduct experiments with varying As shown in Figure 5, the accuracy initially improves with increasing I, reaches peak at 31, and then starts to decline. When is small (c.g., t = 20), the distribution is a mixture of Gaussians, and our distribution matching with a single Gaussian in Lemma 4.1 not able to accurately describe the Gaussian mixtures, leading certain perfor-
6.4. HR Outperforms Under Hard Labels
We give the comparison results with Minimax (Gu et al., 2024), which is the state-of-the-arr method with hard labels under their main evaluation setting on Image (Howard, 2019). As demonstrated Table 5, DHHR with 224 x
224 resolution outperforms Minimax (Gu et al., 2024) with
256 256 resolution.
8

---

Taming Diffusion for Dataset Distillation with High Representaativenss
R8sNet-18
ResNet- 101
45.0
Dataset
IPC
D)38
Ours
D38
Ours
42.5
10
39.1
0.3
44.8
42.1
52.8
ImageNet- IK
50
60.2
+ 0.1
60.2
0.0
65.3 0.5
66.8 0.1
Accuracy
40.0
Ours
D4M
Table 4. Comparison with D3S (Loo et al.) on ImageNet-1K. Following the default setup of D3S with pre-trained teacher models, our D'RR apply soft labels from these 5 teacher models.
37.5
35.0
3
6
IPC
Model
Minimax
Ours
Experiment Index
ResNet-18
37.6 + 0.9
39.6 + 1.0
10
Figure 7. The robustness DDHR across different seeds on Tiny ImageNet. IPC 10. The "Experiment Index" represents ten different seeds selected at random.
ResNetAPP11
39.2 + 1.3
40.7 1.0
ResNet- 18
57.1 + 0.6
57.6 0.4
50
ResNetAP-1D
56.3 + 1.0
59.3 0.4
R8
65.7 + 0.4
66.8 0.6
100
RcsNetAP-10
64.5 = 0.2
64.7 = 0.3
mately 27.5% across various seeds.
Table Comparison with Minimax (Gu et al., 2024) under hard labels across different models and IPCs.
6.7. Storage Requirements Smaller than D
For D²HR, all the data information is encompassed in žr. Therefore, we highlight that by storing only the statistical pa- rameters (mean and variance) of 2r and the DiT pretrained weights of the diffusion model, we can preserve all the de- tails of the distilled datasets under any IPCs. As shown in Appendix B-Figure A2, it further reduces the required storage space compared D. This approach is particu- larly effective for large datasets, resulting in substantial reduction in storage requirements.
6.5. Improving Results with More Soft Labels
D3S (Loo et al.) proposes to simultaneously map the distri- bution of multiple teacher models. During evaluation, the soft labels from multiple teacher models are averaged to supervise the training of the student model. The back the increased costs train multiple different teachers.
Although HR does not rely teacher models during the distillation, we find that using multiple soft labels during evaluation can further improve the performance. We present DeHR and D3S with teacher models following their de- fault setting on ImageNet- 1 K in Table 4. Since HR with one soft label outperforms D3S with five on TinylmageNet across various IPCs, we do not experiment with Tiny geNet. As shown in Table 1 and Table 4, leveraging soft labels yields slight improvements over 1 soft labels across different IPCs and ours consistently outperforms D3S. For 10 IPC, D/HR with soft label outperforms D3S with 5 soft labels on ImageNet-IK.
7. Conclusion
In this work, we thoroughly examine the challenges in ex- isting diffusion-based dataset distillation methods, such as inaccurate distribution matching, distribution deviation with random noise, and the reliance on separate sampling. Based on this, we introduce novel diffusion-based framework, DRHR, to generate highly representative distilled dataset. Our method achieves state-of-the-ar validation performance on various datasets for different models.
Impact Statement
6.6. Robustness of D'HRR
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
As shown in Algorithm 1, DdHR involves inherent random- ness: initializing different seeds for the random sampling of multiple subsets. For the diffusion-baat based D (Su et al., 2024), the randomness arises from two main sources: (i) Dif- ferent seeds result in varying initial clusters in the K-means algorithm, and (ii) Different seeds generate varying initial noise by Equation (1) for the distilled dataset generation.
A Acknowledgments
This work is partially supported by the National Science Foundation under Award 10S-231 We would like to express our sincere gratitude to the reviewers for their in- valuable feedback and constructive comments to improve the paper.
To validate the consistent superiority across various seeds of HRR we conduct both our DHH and D'M (Su et al., 2024) ten times with different seeds. As shown in Figure 7, our HRR outperforms D'M (Su et al., 2024) by approxi-

---

Taming Diffusion for Dataset Distillation with High Representativeness
References
Gu, J., Vahidian, S., Kungurtsev, V., Wang, H., Jiang, W., You, Y., and Chen, Y. Efficient Dataset Distillation via Minimax Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Alexey, D. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020.
Azizi, S., Komblith, S., Saharia, C., Norouzi, M., and Fleet, D. J. Synthetic data from diffussion models improves imagenet classification. arXiv preprint (rXxi:2334.08466 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Cazenavette, G., Wang. T., Torralba, A., Efros, A. A., and Zhu, J.-Y. Dataset distillation by matching training tra- jectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750- 4759, 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840 33:6800-6851, 2020.
Howard, J.
A smaller subset of 10 easily classified classes from imagenet, and a little more french. URL hutps://github comffassaal/maggeettte 4. 2019.
Cui, J., Wang, R., Si, S., and Hsieh, C.-J. Scaling up dataset distillation to imagenet- Ik with constant memory. In International Conference on Machine Learning, pp. 6565- 6590. PMLR, 2023.
Hsu, P.-L. and Robbins, H. Complete convergence and the law of large numbers. Proceedings of the national academy of sciences, 33(2):25-31, 1947.
Deng, J., Dong, W., Socher, R., Li, L.-J,, I, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255.. leee, 2009.
Kingma, D. P., Welling, M., et al. An introduction to vari- ational autoencoders. Foundations and Trends@ in Ma- chine Learning. 12(4):307-392, 2019.
Deng, W., Li, W., Ding, T., Wang. L., Zhang, H., Huang, K., Huo, J., and Gao, Y. Exploiting saterssample and inter- feature relations in dataset distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pultern Recognition, pp. 17057-17066, 2024.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Le, Y. and Yang, Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Lee, S., Chun, S., Jung, S., Yun, S., and Yoon, S. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pp. 12352-12364. PMLR, 2022.
Du, J., Jiang, Y., Tan, Y., Zhou, J. T., and Li, H. Minimiz- ing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3749- 3758, 2023.
Lee, Y. and Chung, H. W. Selmatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching. In Forty-first International Conference on Machine Learning. 2024.
Du, J., Shi. Q. and Zhou, J. T. Sequential subset matching for dataset distillation. Advances in Neural Information Processing Systems, 36, 2024a.
Loo, N., Maalouf, A.. Hasani, R., Lechner, M., Amini, A., and Rus, D. Large scale dataset distillation with domain shift. In Forty-first International Conference on Machine Learning.
Du, J., Zhang, X., Hu, J., Huang, W., and Zhou, T. Diversity- driven synthesis: Enhancing dataset distilla- tion through directed weight adjustment. arXiv preprint arXiv:2409 17612, 2024b.
Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient enn architecture design. Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-fire International Conference on Machine Learning, 2024.
Marsaglia, G. and Tsang, W. W. The ziggurat method for generating random variables. Journal of statistical software, 5:1-7, 2000.
10

---

Taming Diffusion for Dataset Distillation with High Representativeness
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
conference on machine learning, pp. 6105-6114, PMLR, 2019.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers & distillation through attention. In International con- ference on machine learning, pp. 10347-10357. PMIR, 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent difffusin models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Wang. K., Zhao, B., Peng, X., Zhu, Z.,, Yang. S., Wang, S., Huang, G., Bilen, H., Wang, X., and You, Y. Cafe: Learning condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12196 12205, 2022.
Sandler, M., Howard, A., Zhu, M., Zhmoginov. A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Wang. T., Zhu, J.-Y., Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint
2018.
Shao, S., Yin, Z., Zhou, M., Zhang, X., and Shen, Z. Gener- alized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16709-16718, 2024.
Yin, Z., Xing, E., and Shen, Z. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Shen, X., Song, Z., Zhou, Y., et al. Lazydit: Lazy learning for the acceleration of diffusion transformers. In AAAI, 2025.
Yu, R., Liu, S., Ye, J., and Wang, X. Teddy: Efficient large-scale dataset distillation via taylor-approximated matching. In European Conference on Computer Vision, pp. 1-17. Springer, 2024.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: arXiv:1409. 1556, 2014.
Zhang, H., Li, Wang, P., Zeng, D., and Ge, S. Echo: Ef- ficient dataset condensation by higher-order distribution alignment. arXiv preprint arXiv:2312.1592. 2023.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Zhang, X., Du, Li, Y., Xie, W., and Zhou, J. T. Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26223-26232, 2024.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Mor- cOs, A. Beyond neural scaling laws: beating power law scaling via data pruming. Advances in Neural Information Processing Systems, 15:19523-19536 2022.
Zhao, B. and Bilen, H. Dataset condensation with differen- tiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685. PMLR, 2021.
Su, D., Hou, J., Gao, W., Tian, Y., and Tang, B. D'4M: Dataset Distillation via Disentangled Diffusion Model. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5809-5818, June 2024.
Zhao, B. and Bilen, H. Dataset condensation with distri- bution matching. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pp. 65233, 2023.
Sun, P.. Shi, B., Yu, D., and Lin, T. On the diversity and re- alism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9390- 9399, 2024.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensation with gradient matching. arXiv preprint arXiv:2000.05599, 2020.
Zhao, G., Li. G.. Qin, Y., and Yu, Y. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023.
Tan, H., Wu, S., Du, F., Chen, Y., Wang, Z.,, Wang, F., and Qi, X. Data pruning via moving-ons gooving-one-sampee sae-sample-out. Advances in Neural Information Processing Systems, 36, 2024.
Tan, M. and Le, Q. Efficientnet: Rethinking model scal- ing for convolutional neural networks. In International

---

Taming Diffusion for Dataset Distillation with High Representatiivness
Appendix
In the appendix, we include additional details that were omitted in the main text:
Section A: Theoretical Analysis.
Section B: More Implementation Details.
Section C: Additional Experiments.
Section D: Image Visualization.
A. Theoretical Analysis
Lemma A.1. Each latent in VAE latent space is randomly sampled from distinct component of multi-compponnt Gaussian mixture distribution; hence, the VAE latent is hard-to-fit.
Proof. The Variational Autoencoder (VAE) (Kingma et al., 2019) learns the latent distribution ( M.....M} of input images = {z.li = l.., 3 through probabilistic modeling and is capable of sampling from this distribution to generate new images. In training, for each image 2i4, VAE aims to maximize the marginal likelihood p(r;). The loss function can be defined by minimizing the negative evidence lower bound:
L (.. [o:)(())))) (L_(o(=\/=:)lb(()))
(10)
The first term represents the reconstruction loss of m; The second term is the KL divergence loss, which is used to measure the difference between the latent distribution ((=])rr) produced by the encoder and the prior distribution p(z) ~ (0, Z). The output of the VAE encoder for 2; is defined as Gaussian distribution q(2 (z,)) ~ NN,,, aî), where μ4 and of are the mean and variance learned by the encoder. The latent 2; is then obtained by randomly sampling from this Gaussian distribution. Therefore, for each class C containing m images, Ze is modeled as a discrete distribution, where the m latents are independently sampled from m distinet Gaussian distributions. The probability density function of the multi-component Gaussian mixture distribution can be expressed as:
p(=) = Σ oz N(x IR, Ex),
(11)
[=1
where, ((x Pe Σ,) denotes the i-th Gaussian distribution, with μr and Σ, representing its mean and covariance, respectively. The weights o satisfy the following condition: Σ a Since the position of each Gaussian component of this Gaussian mixture distribution random, it results in complex and hard-to-fit distribution.
Lemma A.2. For DDIM inversion with T' steps, when (0 S t T') is sufficiently large, E can be approximated as a Gaussian distribution.
Proof. In DDIM training, the forward process also follows Equation (1), which implies that p(zT)) (), During inference, the process from ør to æo is deterministic, following Equation (2). Since as a pre-defined hyperparameter for each I, the process from to 2 can be viewed as a linear transformation:
Rt-1 U4
+ Wei
where U1
V
01-1
We
vo--1
1
ax -
C0(2,,CC))
(12)
04
For different sampled latent at, of remains the same, while wi changes because the network Eq produces different outputs for each z/.
We approach from the DDIM sampling, which is the reverse of the DDIM inversion, proceeding from 7" to 0. Specifically, consider the deterministic transformation from the set Zrv = {2tr li 1, 1...., m,) to 27 (each noise Zr, e corresponds one-to-one with each latent in Zo,c). Each latent 2 1.C is a latent sampled from the Gaussian distribution:
str-1 ~ NWtr, (VT:)2).
(13)
12

---

Taming Diffusion for Dataset Distillation with High Representativveess
According the law of large numbers (Hsu & Robbins, 1947), the latents in the discrete distribution 11e can be interpreted as i..d. latents from continuous distribution -1c- Thus, the - is a Gaussian mixture distribution. Next, consider the transformation from -1c -2c. Each latent -2c is sampled from:
2 -1 w + wt. -1: (1) (2r²)
(14)
Based on above iteration, the transformation from 1,- 2, can he generalized as follows:
T
T
T
=/ ~ W
Σ k=t+1
II
+T++--+1
w WT+--4+11
II
(u;)²
(15)
jok+1
j=t-1
Therefore, E, (0 s - 1) is a Gaussian mixture distribution.
Now, in the DDIM sampling process, we have the recurrence relation for ag:
04
II o where df 1&ta
(16)
this implies the following formulas:
aa 01-1 <e, where is small positive constant,
(17)
Or as l L
(18)
When / is relatively large, on one hand, indicated by Equations (12), (17) and (18), the effect of (= L, on wy becomes minimal; on the other hand, since the number of iterations is small, the change in the statistical properties of the original standard Gaussian distribution not particularly noticeable. Hence, the distributions the Gaussian components in 2 are not significantly different, and &,, can be well-approximated by Gaussian distribution.
B. More Implementation Details
For the group sampling, we set m 100, 000 for the ImageNet- m 1, 000, 000 for the TinylmageNet, and m 5, 000, 000 for CIFAR-10 10, CIFAR-100. The hyper-paramette Aμo λo, Ay are set to 0.5, respectively. Regarding the computational cost of DDIM inversion, our method only requires approximately 4.5 hours single node with 8 A A100 40G GPUs on ImageNet-IK.
For validation, the parameter settings vary slightly across methods. We adhere to the configurations in (Sun et al., 2024), as detailed in Table A1. For other methods, we primarily use the results reported in their paper. If a relevant experiment unavailable, we generate the distilled dataset using their code and validate under the same settings as ours.
Parameter
CIFAR-10
CIFAR-100
Imng-Im
ImageNet-1K
Optimizer
AdamW
Learning Rate
0.01
Weight Decay
0.01
Batch Size
128
Augmentation
RandomResizedCrop + Horizontal Flip Cosene
LR Scheduler
Tempreture
20
Epochs
400
400
300
300
Table A.. Hyper-parameter setttings used for our validation.
13

---

Taming Diffusion for Dataset Distillation with High Representatiivness
C. Additional Experiments
C.1. Validate the Limitations of D'M
We use D'M as an example here to validate the validity of our analysis in Section 3.3 by visualizing the latents in the VAE space. As illustrated in Figure A1, we visualize the cluster centers in the VAE encoding space, which are treated as representative latents of the method, and marked x. It is evident that there are many outliers in the VAE space, and the K-means algorithm treats these outliers separate cluster. Consequently, the latents generated by D become concentrated around these outliers, leading to inaccurate representation of the VAE distribution. Besides, each individual X aims to map single cluster, rather than considering the entire subset of representative latents as whole. Moreover, * demonstrates that after adding noise and removing noise, the latents undergo a shift from x. Therefore, these results align well with our previous discussion.
Tench
Goldfish
Hammerhead
Black Swan
Figure AI. Illustration of representation-shift in D'M for different classes, ImageNet- IK, IPC 10. The blue dots indicate the latents in the VAR space. X represent the cluster centers after the VAF encoding. * denote the latents after adding noises and removing noises for decoding the distilled images.
C.2. More Analysis about Group Sampling
Computational efficiency. We report the runtime of the group sampling process across different m and different GPUs in the table, demonstrating its simplicity and efficiency. As discussed in Lines 302-306, left column, this is because multiple subsets can be sampled in parallel on the GPU instead of sequential sampling, and the operations such as random sampling and mean/variance computations are very lightweight and efficient in current computation frameworks, making the entire process highly efficient.
14

---

Taming Diffusion for Dataset Distillation with High Representattivensss
IPC
5
10
20
50
Time (8)
0.3901 +0.0057
1.0288 -0.009
1.8352++0.0012
3.506+0.0006
8.5252+00.005
Table A2. Runtime A100 40G with 1c6.
IPC
10
20
50
Time (s)
0.1442+:00013 0.1135+0.0016
0.2001=000003
0.3806+0000066
0.9013=0.0060
Table A3. Runtime on A100 40G with m = 1e5.
IPC
5
10
20
50
Time (s)
0.0902110.0088
0.167010.0042
0.27831100018
0.522710.0073
1.2873=0.0023
Table A4. Runtime A6000 with 1c5.
Increasing/decreasingg Variance. As discussed in Section 4.4, we generate synthetic samples following the real data distribution, to ensure the training performance on synthetic samples. However, fow low IPCs, is not feasible to fully match the original distribution. An intuitive approach distribution. We perform the experiment verify the influence of variance. As shown in Table A5, we adjust the variance of the distribution by 50% in sampling. The results show that neither increasing nor decreasing the variance leads to higher accuracy. Therefore, changing the variance may lead to a distribution deviation with samples not similar to real ones and degraded training performance.
incrassig//decresiing the variance yields a better target
50%
30%
-10%
O
+10%
+30%
+50%
39.8+00.5
42.2+0.4
43.1+0.4
44.1-0.3
41.4+0.44
37.4+0.5
34.0=t0.6
Table A5. Impact of increasing/deerresiing variance group sampling.
C.3. Visualization of Storage requirements
We present the storage requirements of different settings as discussed in Section 6.7 here. It can be seen in Figure A2, the statistical parameters are quite small, requiring only about 0.016 GB, which includes 320 MB from the VAB weights. The storage space needed for the distilled dataset scales linearly with the increase in IPCs. When the IPC is large, the storage demand becomes significantly high. However, the combined storage requirement for the DiT weights and statistical parameters remains smaller than that of distilled dataset with 50 IPC, yet it encapsulates all the essential information from the distillation process, regardless of IPC size. Thus, this approach offers a highly efficient way to reduce storage consumption, especially for large IPCs. Moreover, it can even encapsulate more images than the original dataset for improved training performance.
15

---

Taming Diffusion for Dataset Distillation with High Representaativenss
17.0 GB
17.5
15.0
12.5
Memory (GB)
10.0
8.3 GB
5.0
4.1 GB
DIT weight
3.2 GB
0.8398 GB
Statistics 0.01573 GB
0.0
10IPC
50IPC
100IPC
200IPC Ours storage
The Scale of Distilled Dataset
Figure A2. The storage requirements of different settings on ImageNet- IK.
C.4. Evolution of the Latent Space Across Timesteps
We analyze the impact of different inversion timesteps on accuracy Section 6.2. Here, we provide the visualization the latent space transformation for different classes as inversion timestep increase Figure A3. As demonstrated in Appendix A.2, the latent space progressively transforms into high-normality Gaussian space with the increasing timesteps.
16

---

Taming Diffusion for Dataset Distillation with High Representatiiveess
Tench
-2
VAE Space
Inversion timesteps = 10
Inversion timestape 20
Inversion timestepe 30
Inversion timestaps = 40
Goldfish
-08-
VAE Space
Inversion timestops - 10
Inversion timestops
Inversion timesteps
Inversion timestops
Hammerhead
-08-
26
VAE Space
Inversion timestops - 10
Inversion timestops
Inversion timesteps
Inversion timestops
Black swan
VAE Space
Inversion timesteps - 10
Inversion timestaps 20
Inversion timesteps 30
Inversion timesteps 40
Figure A3. T-SNE visualization of latent space at different inversion timesteps.
C.5. More Comparison of Other SOTA Methods under Different Setting
Some SOTA methods adopt different validation settings. Specifically, they train their validation models for 1000 epochs on CIFAR-10 10 and 100, whereas our main comparison uses 400 training epochs. For completeness, we provide a comparison under their settings on both CIFAR and ImageNet- IK,, as shown Tables A6 and A7. Our method consistently outperforms these approaches under their setup.
17

---

Taming Diffusion for Dataset Distillation with High Representattiveess
Model
Dataset
IPC
G-VBSM
ESFR
HMDC
Ours
CIFARIO
53.5:0.6
56.911.55
69.8+0..4
69.8
ResNet-18
CIFARIO
50
59.2:10.4
68.3
75.8
85.2:10.4
CIFARTON
50
65.0+0..
67.3+0.4
CIFARIO
46.5+0.77
05.2
ConvNetW 128
CIFARIO
54.3:10.3
6..8:10.4
CIFARI00
50
45.7+0..4
0..1
CIFARIO
10
59.910.2
47.510.7
57.3
ConvNet
CIFARIO
50
69.0:10.2
52.420.1
69.3+0.44
CIFARI00
50
51.340.4
54.6110.4
Table A6. Comparison with SOTA methods CIFAR-10 and CIFAR-100 (1000-epoch validation).
Model
IPC
G-VBSM
Teddy(post)
Teddy(prior)
Ours
10
31.4
32.7=0.2
34.1:10.1
44.3:10.3
ResNet-18 18
50
51.8+0.4
52.5+0.1
52.5#
59,4:10.1
100
55.740.4
56.2+..2
56.5+0011
6..5
10
38.240..4
40.0+
40.3++0.1
52.1++0.4
ResNet-101
50
61.0+:0.4
66.1110.1
100
63.7.L0.2
68.1
Table A7, Comparison with more SOTA methods on ImageNet-1k.
C.6. Cross-architecture Generalization
We broaden our experiments in Section 5.3 by integrating diverse neural network architectures, including EfficientNet-B0 (Tan & Le. 2019), ShuffleNet-V2 (Ma et al., 2018), DeiT-Tiny (Touvron et al., 2021). It can be seen in Table A8, D'HR exhibits exceptional performance across substantial different architectures, incurring only a one-time generation cost.
18

---

Taming Diffusion for Dataset Distillation with High Representatiiveess
Student Teacher
ResNet-18
EfficientNet-BO
MobileNet-V2 V2
ShuffleNet V2
VGG-11
DeiT-tiny
RDED
42.3 + 0.6
31.0 + 0.1
40.4 + 0.1
43.3 + 0.3
36.6 0.1
+ 25.7
ResNet-18
Ours
44.2 + 0.3
30.6 = 0.2
42.3 = 0.7
44.7 = 0.1
38.3 = 0.2
28.0 0.4
RDED
42.8 + 0.5
33.3 + 0.9
43.6 = 0.2
50.5 + 0.6
35.8 = 0.5
32.9 0.4
EfficientNet-BO
Ours
50.3 + 0.2
42.0 + 0.6
54.0 + 0.1
56.3 + 0.1
43.3 + 0.1
36.8 + 0.4
RDED
34.4 0.2
24.1 0.8
33.8 = 0.6
48.2 + 0.5
28.7 0.2
24.9 0.6
MobileNet-Ve
Ours
43.4 0.3
31.8 = 0.8
46.4 = 0.2
50.0 + 0.2
37.8 = 0.4
26.5 = 0.8
RDED
37.0
23.7 = 1.0
35.6 = 0.2
40.5
29.4 0.3
21.8 0.6
ShufleNet-V2
Ours
37.2 = 0.0
25.8 0.3
38.5 = 0.4
44.1 - 0.3
32.6 0.6
23.1 = 0.3
RDED
22.7 0.1
16.5 + 0.8
21.6 + 0.2
25.7 + 0.4
23.5 0.3
17.6 + 0.4
VGG-11
Ours
25.7 + 0.4
20.2 + 0.2
24.8 + 0.4
29.1 + 0.4
28.1 + 0.1
15.1 + 0.2
RDED
13.2 + 0.3
12.6 + 0.5
13.6 + 0.4
16.1 + 0.3
11.4 0.1
15.4 + 0.3
DeiT-tiny
Ours
17.1 0.3
14.8 + 0.2
17.9 = 0.7
21.4 + 0.8
14.6 0.5
15.0 = 0.3
Table A8. Comparison of Top-1 accuracy for cross-architecture sentrrccittttuuurreeeenrrritttoo ImageNet-IK, MingeNet-IK,IPC. IPC 10.
C.7. More Ablation Studies the Hyper-parameters
Choice of m across Different Datasets. As shown in Tables A9 and A10, present the performance of varying m datasets of varying scales. As expected, increasing improves accuracy, larger candidate pool offers greater diversity and higher chance of including representative sets. However, when m becomes sufficiently large, the performance gains plateaus cattcaus-iniicating indicating that the marginal benefit of adding more candidates diminishes, as the top- top-performing candidates become increasingly similar. We choose m at the saturation point, typically between 1e5 and 1e7, which can work well across different datasets.
m
1e3
1e4
1e5
5e5
1e6
5e6
Accuracy (%)
40.
40.910.4
41.7=00.5
42.4+0.4
43.8=0.3
44.2=0.2
44.1++0.3
Table A9. The results across different m under TPC
10 on Tiny-ImageNet.
m
1e3
1e5
1e6
5e6
1e7
5e7
Acc (%)
38.940.3
39.6:10.2
40.4
41.010.1
41.810.1
41.9:0.2
42.1
Table A10. The results across different under IPC 10 CIFAR-10.
Different Combination of Aua Ao, A For Au. Às and X,y we set them to make the corresponding metries on the same scale. We provide results for different in Appendix C.7, which shows that performance on LT,C is relatively robust to X.
Setting
1:1:0.5
1:1:1
1:1:2
1:0.5:0.5
Acc (%)
44.240.1
44.010.4
43.610.3
43.4+0.2
Table AII. The results of different combination of Ays Aos A71
D. Image Visualization
We present more visualization results of the distilled images in this section. As shown in Figure A4 and Figure A5, D²HR generates diversity, high-quality images for each class, effectively representing the full dataset.
19

---

Taming Diffusion for Dataset Distillation with High Representativeness
Agaric
Alp
American black bear
Bald eagle
Beacon
Bell pepper
Boathouse
Border terrier
Broocoll
Cook
Figure A4. Visualization of the distilled images for different classes on ImageNet-1K, /PC = 10, and the resolution is 224 x 224.
20

---

Taming Diffusion for Dataset Distillation with High Representativeness
Fountain
Giant parda
Goldfish
Hamster
Jellyfish
Mearkat
Monarch
Orange
Pirate
Waffle iron
Figure A5. Visualization of the distilled images for different classes on ImageNet-IK, /PC 10, and the resolution is 224 x 224.
21


Reviews:
[{"review_id": "jYwhMCo9PM", "reviewer": "Reviewer_VDRK", "summary": "This paper proposes a diffusion-based dataset distillation method. The paper claims that the VAE space of the diffusion model is more difficult for distribution matching. To tackle this challenge, the core idea of the proposed method is to apply DDIM inversion to each sample in the original dataset and then model the distribution of the inverted samples. To obtain distilled samples, the method performs latent-space sampling for multiple times and select the sample group that yields the lowest loss. Experiments on some datasets validate the effectiveness of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: no clear definition of normality**\n\nThanks for pointing this out. In our context, normality refers to the degree of the latent space data conforms to a normal distribution. A higher level of normality indicates that the latent distribution more closely resembles a normal distribution. This usage is consistent with that in statistical normality test, which assess whether sample data is drawn from a normal distribution. We will add the clarification in the updated version.\n\n**Q2: not informative Figure 1**\n\nPlease refer to Q1 of the response for phyT.\n\n**Q3: Why not use the embedding space of a pre-trained model**\n\nFirst, the scalability to multiple architectures in Lines 149-155 left column are the motivation why we use diffusion models. We agree with prior works (RDED, D4M, Minimax) that realism and cross-model generalization are key qualities of a high-quality distilled dataset, which can be effectively achieved with the help of diffusion models.\n\nBased on this, as shown in Lines 110–164 right column, our main motivation is that we identify three key limitations in current diffusion-based methods, primarily because of conducting optimization within the VAE latent space. This motivates us to explore a more effective optimization space, aiming to provide a better paradigm for diffusion-based methods. \n\nCertain previous approaches that rely on the embedding space of a pretrained model or a small set of pretrained models, can introduce architecture bias, which may limit their generalization capabilities in large scale datasets or different independent architectures. In contrast, we identify a better optimization space that is agnostic to any specific architecture, allowing a single distilled dataset to be directly used across a wide range of models for users. As shown in the Tabel A1 in the appendix, our method can achieve SOTA performance across various models just with one-time generation cost.\n\n**Q4: Sampling-based method instead of gradient-based optimization for group sampling**\n\nWe propose an efficient and effective sampling design based on the mapped high-normality property. As discussed in Lines 260-266 left column, the n latents (corresponding to n IPC) probabilistically follows the distribution of the whole class C. Building on this, our group sampling method is simple and efficient to search the most representative subset. For computation overhead analysis, please refer to Q3-(1) of the response for xaqL. In contrast, the gradient-based optimization is computationally expensive and may compromise the probabilistic nature of the n latents.\n\n**Q5: The missing references**\n\nRegarding [c], we have already included the comparison in our paper under the name DWA in Table 1.\nAs for [a] and [b], we provide the comparison on ImageNet-1K below, and will include the corresponding reference in the next version.\n\nResnet 18:\n|IPC|G-VBSM|Teddy(post)|Teddy(prior)|Ours|\n|-|-|-|-|-|\n|10|31.4±0.5|32.7±0.2|34.1±0.1|44.3±0.3|\n|50|51.8±0.4|52.5±0.1|52.5±0.1|59.4±0.1|\n|100|55.7±0.4|56.2±0.2|56.5±0.1|62.5±0.0|\n\nCross Resnet101:\n|IPC|G-VBSM|Teddy(post)|Teddy(prior)|Ours|\n|-|-|-|-|-|\n|10| 38.2±0.4|40.0±0.1|40.3±0.1| 52.1±0.4|\n|50|61.0±0.4|-|-|66.1±0.1|\n|100|63.7±0.2|-|-|68.1±0.0|\n\n**Q6: Time cost of a pre-trained diffusion model and DDIM inversion**\n\nIt is common to use pre-trained models in dataset distillation. Diffusion-based methods (D4M, Minimax) rely on a pre-trained diffusion model, while other methods often use pre-trained teacher models to guide the distillation process. Thus, we do not count the training of pre-trained models towards our time cost.\n\nThe DDIM inversion is effective without high computation cost. (i) As discussed in Sec. 5.3, we only need to perform DDIM inversion once with a one-time cost, to generate multiple distilled datasets for different model architectures under various IPC settings. This is different from other methods which need to run their whole algorithms one more time if the architecture or IPC setting changes. (ii) The results of DDIM inversion are easy to store with little storage requirement, as shown in Appendix-Figure A3. (iii) Moreover, the computation cost of DDIM inversion is affordable. Even for ImageNet-1K, our method only requires approximately 4.5 hours on a single node with 8 A100 GPUs. In comparison to the SOTA diffusion-based method Minimax, which requires fine-tuning 50 separate pre-trained DiTs for ImageNet-1K, our approach is significantly more efficient. In addition, if more efficient denoisers (e.g., DPM-Solver families) are used, our paradigm can be further accelerated. This may be an orthogonal improvement direction and not the focus of our work.\n\n**Q7: Comparison with hard labels**\n\nWe give the comparison results with Minimax diffusion, which is the SOTA method with hard labels under their main evaluation setting on ImageWoof. Our method with 224×224 resolution outperforms Minimax Diffusion with 256×256 resolution. Please refer to Q1 of the response for xaqL.", "claims_and_evidence": "The claims are not well supported. The authors claim that \"distribution in the VAE space has low-degree of normality\". However, there is no clear definition of \"normality\" or any reference. The visualization in Fig. 1 is not informative, either. One can hardly understand the difference between two plots. The only difference in the current version is that there are some offsets between the points.", "methods_and_evaluation_criteria": "I am not sufficiently sure that the proposed method makes sense. If the target is to find a space that can effectively matching the distribution, why not consider the embedding space before the final linear layer of a pre-trained classifier, where samples in various class are almost separable. Indeed converting to the feature space of inverted samples can reduce the difficulty of distribution matching, but I am not sure whether this strategy is optimal. Although I can find some motivation about architectural scalability between Line 149 and 155, this issue can be potentially addressed by including more structures during training [a].\n\nBesides, why use sampling-based method to select sample groups instead of conducting gradient-based optimization ?\n\n[a] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching, Shao et al., CVPR 2024.", "theoretical_claims": "I quickly go over the introduced lemma and the proof, and I find no issue within these parts.", "experimental_designs_or_analyses": "The experiments are clear to validate the effectiveness of the proposed method over some baselines. But I am not sure if it is optimal.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "Missing Reference:\n\n[a] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching, Shao et al., CVPR 2024.\n\n[b] Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matching, Yu et al., ECCV 2024.\n\n[c] Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment, Du et al., NeurIPS 2024.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The experimental results are good and the method surpasses the diffusion-based methods.\n2. The writing is generally clear.\n\nOther Weaknesses:\n\n1. I am afraid that the time efficiency may not be superior as claimed by the authors. First, we need to train a diffusion model on the original dataset, which can take multiple days especially on large-scale ones. Second, the DDIM inversion process for each sample is obviously non-trivial.", "other_comments_or_suggestions": "There are no blue lines in Fig. 1.", "questions_for_authors": "I am also curious about the results of using hard labels, because this case can reflect the capacity of distilled samples best.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZRauCbBIQR", "reviewer": "Reviewer_xaqL", "summary": "This work proposes a novel diffusion-based dataset distillation solution. Based on the fact that previous methods suffer from inaccurate distribution matching, the authors propose to convert the images to latents with DDIM inversion and model it as Gaussian. Then, sample multiple subsets from the Gaussian and select the subset with the most similar distribution statistics. These latents are sent to DDIM for image generation. The proposed method achieves leading performance on regular DD benchmarks.\n\n\n\n**Update after rebuttal**: I appreciate the authors' feedback and some concerns are addressed. So I still leans toward positive and would keep my initial positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Comparison to MiniMax with hard labels**\n\nAs noted in Lines 297–300 right column, we did not include Minimax in the main table as it focuses on small subsets of ImageNet-1K. For large datasets, it requires extra training of multiple diffusion models with high computational cost. They only report results for ResNet-18 on ImageNet-1K, and do not support the other three datasets and other architectures used in our table. For a fair comparison with their results, we give the comparison with Minimax using their main setting on ImageWoof. Minimax is the SOTA method with hard labels on ImageWoof. Our method with 224×224 resolution outperforms Minimax Diffusion with 256×256 resolution.\n\n|IPC|model|Minimax|Ours|\n|-|-|-|-|\n|10|resnet18|37.6±0.9|39.6±1|\n|10|resnetAP-10|39.2±1.3|40.73±1|\n|50|resnet18|57.1±0.6|57.6±0.4|\n|50|resnetAP-10|56.3±1.0|59.3±0.4|\n|100|resnet18|65.7±0.4| 66.8±0.6|\n|100|resnetAP-10|64.5±0.2|64.7±0.3|\n\n**Q2: Why is Gaussian mixture \"hard-to-fit\"**\n\n\"hard-to-fit\" means that a Gaussian mixture is much more difficult to fit than a single Gaussian, due to its uncertain number of components and unknown structure. In contrast, a single Gaussian is a simpler and well-defined parametric form. In addition, in Fig. A1, we present the visualization of fitting the Gaussian mixture using K-means clustering under the D4M setting. It can be observed that some outliers near the edges of the distribution are selected, indicating that the Gaussian mixture is not well captured by the clustering. This further supports our point that the Gaussian mixture is relatively hard to fit.\n\n**Q3: About group sampling**\n\nThanks for the valuable feedback. We explain the group samping more clearly, and then address your comments. \n**Rationality of group sampling**: We propose an efficient sampling design based on the mapped high-normality property. As discussed in Line 266-270 left column, the distribution of n latents (for n IPC) may still deviate from the desired distribution due to limited size n. To address this, we sample multiple subsets in parallel and choose the most representative subset with the closest distribution to the desired distribution, as detailed below. \n\n**(1) Computational efficiency**: We report the runtime of the group sampling process in the table, demonstrating its simplicity and efficiency. As discussed in Lines 302-306 left column, this is because multiple subsets can be sampled in parallel on the GPU instead of sequential sampling, and the operations such as random sampling and mean/variance computations are very lightweight and efficient in current computation frameworks, making the entire process highly efficient.\n\nm = 1e5, A100 40G:\n|IPC|1|5|10|20|50|\n|-|-|-|-|-|-|\n|time(s)|0.3901±0.0057|1.0288±0.0092|1.8352±0.0012|3.506±0.0046| 8.5252±0.0055|\n\n**(2) About analytical solutions**: Group sampling is straightforward and efficient, taking just a few seconds to sample millions of subsets and select the best one. The mentioned analytical method to sample N-3 and solve the rest for accurate distribution matching may require additional complex solvers with high computation cost. Furthermore, with the solved examples, although the statistics of the distribution may match perfectly, the overall distribution of all samples may not be a Gaussian distribution. It is possible that the designed examples are far away from the sampled examples to match the desired distribution which is in the middle between them.\n\n**(3) Increasing/decreasing variance**: Our domain mapping (Sec. 4.2) and distribution matching (Sec. 4.3) try our best to build an easy-to-fit distribution of real data. Then we generate synthetic samples following the real data distribution, to ensure the training performance on synthetic samples. Changing the variance may lead to a distribution deviation with samples not similar to real ones and degraded training performance.   We further perform the experiment to verify the influence of variance. As shown in the table, we adjust the variance of the distribution by ±50% in sampling. The results show that neither increasing nor decreasing the variance leads to higher accuracy. \n\n|-50%|-30%|-10%|0|+10%|+30%|+50%|\n|-|-|-|-|-|-|-|\n|39.8±0.5|42.2±0.4|43.1±0.4|44.1±0.3|41.4±0.4|37.4±0.5|34.0±0.6|\n\n**(4) Statistics may be biased for low IPCs**: As the law of large numbers suggests, smaller sample sizes indeed leads to higher statistical bias. This also provides a reasonable explanation for the observed drop in accuracy at low IPC. This is a general issue for dataset distillation works. To mitigate this, our group sampling generates m subsets and selects the most representative one with the closest distribution. Thus, although one subset may suffer from large statistical bias, it is possible to find another subset with smaller bias. The results in Table 3 show our best performance compared with baselines in low IPC, showing the effectiveness of our method in addressing the bias issue.", "claims_and_evidence": "The authors made three main claims (summarized in lines 21-44), which are reasonable and supported in the paper.", "methods_and_evaluation_criteria": "The method is constructed following the previous observations, which is sound.\n\nThe authors adopt regular benchmarks for dataset distillation, including small and large-scale image datasets, and also cross architecture protocol (appendix). The only concern is that they did not compare with MiniMax diffusion (Gu et al., CVPR 2024) which is also diffusion-based and may yield competitive performance (58.6\\% on ImageNet-1K with IPC 50).", "theoretical_claims": "The proofs are checked with no issue found.", "experimental_designs_or_analyses": "The experiment design including the selection of teacher models, IPCs, and ablation study is sound.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No missing references.", "other_strengths_and_weaknesses": "Other weaknesses:\n\n1. The \"group sampling\" use sampling-reject paradigm to find the best latent subset, which seems be computationally heavy.", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "I summarize my concerns as follows, including these in the previous questions for the authors convienience:\n\n1. The experimental comparison to MiniMax (Gu et al., CVPR'24) is missing, which might be a competitive baseline.\n2. For Lemma 3.1: why the Gaussian mixture is \"hard-to-fit\"? This is concluded from which perspective?\n3. For group sampling:\n    (1) It seems to be computationally heavy. Is the \"2.6s per class\" on 306 the time for group sampling?\n    (2) Why not sample $IPC-3$ latents and \"solve\" the other two latent based on the three constraints to avoid repeated sampling?\n    (3) This sampling method implies that the distribution of real samples is exactly the target of synthetic samples. However, this is not supported. It is possible that increasing/decreasing the variance yields a better target distribution.\n    (4) For low sample capacity (IPC), the distribution statistics may be biased.\n\nI would tune my rating if my questions are well addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DW5FfoIvMy", "reviewer": "Reviewer_xB8L", "summary": "This article introduces the D³HR framework (Taming Diffusion for Dataset Distillation with High Representativeness) to tackle the issue of inaccurate distribution matching in existing diffusion-based dataset distillation methods. D³HR enhances distribution matching accuracy by utilizing DDIM inversion to transform the VAE latent space into a high-normality Gaussian domain. Additionally, it ensures thorough alignment of dataset distributions through a group sampling strategy that incorporates statistical constraints. Extensive benchmark experiments reveal significant performance improvements compared to previous approaches.\n\n## update after rebuttal\nThis paper raises the issue of inaccurate distribution matching in existing diffusion-based dataset distillation methods and proposes a solid solution to it. The response answered my concerns on experiments. Thus, I keep my original rating of weak acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: primarily focuses on optimal subset selection**\n\nIn line 027-044 right column, we identify three key limitations in diffusion-based methods due to their reliance on optimization in the VAE latent space. This motivates us to identify a more effective space by DDIM inversion, aiming to provide a better paradigm for diffusion-based methods. Our contribution lies in **the entire pipeline designed for dataset distillation, including DDIM inversion, distribution matching, group sampling, and generation**—not merely the sampling. Each component is carefully designed to cohesively operate in the pipeline to ensure that the distilled subset is both compact and highly representative.\n\n**The process in the inversion space is not input selection. It is a principled generative procedure.** As discussed in Sec. 4.3, we first map the original latent distribution to a Gaussian distribution due to Lemma 4.1, enabling efficient sampling via its high-normality. We first generate latents that **probabilistically follow the distribution of the entire class**, as shown in line 260–266 left column. Building on this, group sampling then supports parallel generation of candidate subsets (each maintaining distributional alignment), from which we identify the most representative one. The whole process tightly integrates sampling and optimization, enabling effective and distribution-aware subset generation tailored for diffusion-based synthesis, and goes beyond simple input selection. \n\n**Q2: The computation overhead for DDIM inversion**\n\nPlease refer to Q6 of the response for Reviewer VDRK.\n\n**Q3: Extreme high T incur accuracy degradation**\n\nAs discussed in Line 378–415 left column, there is a trade-off between maintaining the Gaussian assumption and preserving image structural information across different steps t. When t is small (such as 20), the distribution is a mixture of Gaussians as shown in Figure A4, and our distribution matching with a single Gaussian (Sec. 4.3) is not able to accurately describe the Gaussian mixtures, leading to certain performance loss. When t becomes large such as 40, although our distribution matching can accurately represent the real distributions which becomes more normal (Figure A4), the real distributions suffer from more significantly structural information loss due to adding more noise, which in turn degrades the performance of DDIM inversion. Thus, it is a trade-off between maintaining the Gaussian assumption and preserving image structural information. \n\n**Q4: Using cross-selection in group sampling**\n\nTo validate cross-selection effect, we experiment on Tiny-ImageNet using 10 representative subsets with 10 IPC (performing group sampling 10 times; average accuracy: 44.07; individual results shown in Figure A2). We randomly combine the subsets, as shown in the table below, all three combined results yield lower accuracy than individual subsets. As discussed in Line 240–244 right column, this is because our $L_{T,C}$ is designed to ensure that the entire subset aligns well with the desired distribution, rather than fitting individual latents. Cross-combining latents from different subsets breaks this design principle without any distribution alignments.\n\n|Setting|Combined set1|Combined set2|Combined set3|\n|-|-|-|-|\n|Acc|42.3±0.4|42.0±0.4|41.9±0.5|\n\n**Q5: The role of $L_{T, C}$, is $L_{T,C}$ complexity for usage and reproducibilit**\n\nIndeed, $L_{T,C}$ ​ is a static evaluation metric, which is easy to compute with basic mathematical operations, and the sampling process is efficient (Please refer to Q3-(1) of the response for xaqL). For $\\lambda_\\mu$, $\\lambda_\\delta$ and $\\lambda_\\gamma$, we set them to make the corresponding metrics on the same scale. We provide results for different $\\lambda$ in Table below, which shows that the performance on $L_{T,C}$ is relatively robust to $\\lambda$.\n\n|Setting|1:1:0.5|1:1:1|1:1:2|1:0.5:0.5|\n|-|-|-|-|-|\n|Acc|44.2±0.1|44.0±0.4|43.6±0.3|43.4±0.2|\n\n**Q6: Choice of m across the datasets with different sizes**\n\nWe empirically set the value of m. The table illustrates how accuracy varies with different m across datasets of different scales. As expected, increasing m improves accuracy, as a larger candidate pool offers greater diversity and a higher chance of including representative sets. However, when m becomes sufficiently large, the performance gains plateaus—indicating that the marginal benefit of adding more candidates diminishes, as the top-performing candidates become increasingly similar. We choose m at the saturation point, typically between 1e5 and 1e7, which can work well across different datasets.\n\nTiny-Imagenet, 10IPC:\n|m|1|1e3|1e4|1e5|5e5|1e6|5e6|\n|-|-|-|-|-|-|-|-|\n|Acc|40.2±0.4|40.9±0.4|41.7±0.5|42.4±0.4|43.8±0.3|44.2±0.2|44.1±0.3|\n\nCIFAR10, 10IPC:\n|m|1|1e3|1e5|1e6|5e6|1e7|5e7|\n|-|-|-|-|-|-|-|-|\n|Acc|38.9±0.3|39.6±0.2|40.4±0.3|41.0±0.1|41.8±0.1|41.9±0.2|42.1±0.2|\n\n**Q7: Comparison with Minimax**\n\nPlease refer to Q1 of the response for xaqL.", "claims_and_evidence": "Yes. The authors claim the efficient alignment of distribution of latent vectors, which is supported by results and visualization.", "methods_and_evaluation_criteria": "Yes. The proposed method is reasonable and the evaluation follows the common practice. Detailed ablation study is implemented to verify the effectiveness of the method.", "theoretical_claims": "Yes, the authors provided the theoretical analysis about the VAE latent fitting and the latent distribution in DDIM inversion in the supplementary.", "experimental_designs_or_analyses": "Yes. The experimental design follows the common practice. More diffusion-based DD methods should be included, for example minimax diffusion.", "relation_to_broader_scientific_literature": "D^3HR considers the overall distribution when generating distillation datasets, rather than sampling them individually, thus improving the representativeness of distillation datasets.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1.The proposed method effectively resolves the issue of overall distribution deviation caused by individual sampling in previous diffusion-based dataset distillation approaches.\n\n2.The experimental results demonstrate that the method is both effective in improving distillation quality and efficient in terms of computational performance.\n\nWeakness:\n\n1.The paper seems to focus more on selecting the optimal subset, i.e., choosing a subset from the original dataset that best matches the complete distribution. It seems that the method added an optimal input selection step to the diffusion-based image synthesis process.", "other_comments_or_suggestions": "Please address the weaknesses.", "questions_for_authors": "1. Does the DDIM inversion process introduce significant additional computational costs when mapping the latent space of the entire dataset to a Gaussian noise domain?\n\n2. Section 6.2 highlights that performance degrades sharply when the DDIM inversion step count (T) is too high (e.g., T > 31). What is the root cause of this phenomenon?\n\n3. In practice, is there potential for cross-selection between different candidate subsets in group sampling? The paper appears to select only one subset—could this lead to suboptimal global distribution alignment?\n\n4. Does the loss function $\\mathcal{L}_{T,\\mathcal{C}}$ function more as a static evaluation metric rather than an optimizable loss?\n\n5. How is the optimal number of candidate subsets (m) determined theoretically? For datasets of varying sizes, how should m be chosen?\n\n6. The $\\mathcal{L}_{T,\\mathcal{C}}$ includes multiple hyperparameters (e.g., $\\lambda_\\mu$, $\\lambda_\\sigma$, $\\lambda_{\\gamma_1}$). Does this complexity complicate practical usage and reproducibility?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DxdP1bQOo5", "reviewer": "Reviewer_phyT", "summary": "This paper addresses limitations in diffusion-based dataset distillation methods and introduces D3HR, a novel framework that enhances the representativeness of distilled datasets. This paper reveal that current methods suffer from issues like inaccurate distribution matching, distribution deviation due to random noise, and separate sampling, leading to suboptimal performance. The proposed D3HR framework outperforms existing state-of-the-art dataset distillation methods across multiple datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K. It demonstrates better generalization across architectures like ResNet, MobileNet, and VGG.\n\n## update after rebuttal\nThe authors' response has addressed my concerns, so I maintain my overall positive assessment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Figure 1 and 3 less informative**\n\nA1: Thanks for the suggestions. The “blue lines” are the “blue contour lines” which indicate the probability density of the distribution. We will revise the caption to clarify this.\nFigure 1 is intended to convey two key messages: (1) The VAE latent space exhibits significantly lower normality compared to our mapped (inversion) space. The “blue contour lines” allow us to clearly observe the structure and concentration of the latents (blue dots). Denser and more centralized contours reflect regions with higher latent density, which are the areas we should pay more attention to. (2) We visualize representative latent points generated by our method. In Figure 1(b), we show the latents sampled in the mapped (high-normality) space, demonstrating that our sampling process closely matches the desired noise distribution. To further validate the fidelity of the mapping, Figure 1(a) presents the corresponding latents in the original VAE space after DDIM sampling. The blue contour lines help illustrate that our sampling process successfully captures the structure of the VAE latent space, and effectively concentrates in high-density regions.\nSimilarly, Figure 3 visualizes n latents in the VAE space—comparing those generated by the default DiT model and our proposed method. It demonstrates that our approach yields more representative and diverse latents.\nWe will provide more detailed explanations and clearer visual indicators to improve interpretability. \n\n**Q2: Deep discussion or empirical validation on why better alignment can be achieved in the inversion space**\n\nA2: The reason behind this is that a Gaussian mixture in the VAE space is more difficult to fit than a single Gaussian distribution in the inversion space. This is because compared to a Gaussian mixture with an uncertain number of components and unknown structure, a single Gaussian is a simpler and well-defined parametric form. \n\nTo prove our method achieves better distribution alignment, we visualize 10 generated latents in the VAE space for the same class (\"Goldfish\") using both D4M and ours, as shown in Appendix Figure A1 and Figure 1 in the main paper. The results show that D4M, which performs optimization directly in the VAE space, tends to select latents near the edges of the distribution. In contrast, our method produces a more representative and diverse set of latents, better capturing the overall structure of the distribution.\nFor the quantitative comparison of accuracy, we refer to three key results: the D4M in Table 1, the default DiT generation in Table 3 (row 1), and our method with domain mapping but without group sampling in Table 3 (row 2). Together, the last has the highest accuracy, which demonstrates the effectiveness and necessity of our domain mapping strategy. \nFinally, to further support our claims, we also include a visual comparison of generated images for D4M, the default DiT, and ours. These qualitative results clearly illustrate the advantages of our approach in producing more representative samples.\n\n**Q3: Ablation study on domain mapping and group sampling**\n\nA3: We have the ablation studies in Section 6.1.\n\nFor the group sampling, we present the results in Table 3 (rows 2–9). These include **our method without group sampling (row 2)**, as well as ablations that include only partial components of $L_{T, C}$. This allows us to verify the importance of group sampling and to assess the individual contributions of each component in $L_{T, C}$. \n\nFor the domain mapping, as discussed in lines 366–374, left column, we use the results in Table 3 (rows 1–2) to verify its importance. We compare **the default DiT generation with DDPM (row 1)** and **domain mapping with DDIM (row 2)**, with the same configuration for all other steps. The results clearly demonstrate the advantages of domain mapping with DDIM (Line 375-384 left column).\nWe will give a more clear description and annotation.\n\n**Q4: Performance on hard labels**\n\nA4: Please refer to Q1 of the response for xaqL.\n\n**Q5: Clarifying C for Equation 2**\n\nA5: C denotes the class condition, which means the same as other equations. We will provide a clearer explanation of this in the updated version.\n\n**Q6: Discussion of different Inversion Timesteps**\n\nA6: As discussed in Lines 378–415, left column, there is a trade-off between maintaining the Gaussian assumption and preserving image structural information across different time steps t. Please refer to Q3 of the response for xB8L with more details. \nThe choice of inversion steps is guided by empirical observations. As shown in Figure 5, using 24–31 inversion steps consistently achieves SOTA accuracy, making it a reasonable and effective choice. \n\n**Q7: Storage of decoder**\n\nA7: In Appendix Figure A3,  the decoder size has already been accounted for in our computation under “DiT weight.” The storage size of the VAE weights is 320MB. We will include this detail more clearly.", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have checked the correctness of the proofs for theoretical claims.", "experimental_designs_or_analyses": "The experimental designs are reasonable.", "relation_to_broader_scientific_literature": "D3HR advances the field by improving distribution alignment, reducing noise-induced artifacts, and enhancing dataset compression efficiency, making it a significant step forward in scalable dataset distillation methods.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n-\tThe proposed method improves distribution matching. Uses DDIM inversion to transform latents into a more Gaussian-like distribution, enhancing representativeness.\n\n-\tThe proposed method achieves state-of-the-art performance on different neural architectures.\n\n-\tThis work propose an efficient sampling strategy to reduce randomness in dataset generation and ensures that distilled samples align better with the original distribution.\n\nWeaknesses:\n\n-\tFigures 1 and 3 appear to be less informative. Even after carefully reading the paper, their meaning remains unclear. Additionally, in the caption of Figure 1, the \"blue lines\" mentioned cannot be found, which may cause confusion. Providing more detailed explanations or clearer visual indicators would improve their interpretability.\n\n-\tThis work highlights the issue of inaccurate distribution matching in the latent space found in previous methods. However, it is not clearly explained why the proposed method achieves better distribution alignment. A more in-depth discussion or empirical validation of how the mapping improves distribution matching would strengthen the argument.\n\n-\tThe paper lacks an ablation study on domain mapping and group sampling. Since these are key components of the proposed method, conducting and presenting ablation experiments would help validate their individual contributions to performance improvements.\n\n-\tSRe2L relies on a large amount of soft labels to enhance dataset distillation performance. It would be valuable to evaluate the effectiveness of the proposed method using hard labels. This would provide a better measure of the quality of the generated images, independent of external supervisory signals.", "other_comments_or_suggestions": "In Equation (2), the variable C is not explicitly defined, which may cause confusion for the reader. Clarifying its meaning within the equation or referring to its definition elsewhere in the text would improve clarity.", "questions_for_authors": "In Figure 5, the performance appears to be significantly influenced by the number of inversion timesteps. It would be helpful to discuss why this dependency occurs and whether there is an optimal range of timesteps that balances performance and computational efficiency.\n\nRegarding storage in Section 6.6, does the proposed method also require storing the decoder D? If so, how does this affect the overall storage efficiency? Clarifying whether the decoder needs to be retained separately or if it can be reconstructed from stored parameters would provide a more complete picture of the method’s storage requirements.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qiICMdQLd8", "reviewer": "Reviewer_YwBU", "summary": "This paper utilizes DDIM inversion to map the VAE latent space into a high-normality Gaussian space. Numerous subsets are then sampled from this Gaussian distribution. The final selection is determined by identifying the subset whose distribution has the smallest loss compared to the Gaussian distribution.\nThe results improve across different datasets.\n\n**## update after rebuttal:**\nThe author has addressed my concerns regarding the theoretical aspects and the results on CIFAR-10/100 are also greater than the state-of-the-art. I will increase my current score to Accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The concern about the multi-component Gaussian mixture distribution assumption.**\n\nA1: Thank you for highlighting this. To clarify the assumption, we offer a more comprehensive explanation here. \nFor current diffusion-based methods and ours, the pre-trained VAE are not conditional VAE, so both the training and inference processes of the VAE are not conditioned on the class label C. In theory, each latent variable is encoded by independent statistical parameters (mean and variance), resulting in an N-component Gaussian mixture distribution for N images. In practice, as shown in Equation (10) (in Appendix), the VAE loss includes a KL divergence term, which pulls each latent distribution moderately toward N(0, I). This results in a Gaussian mixture with overlapping components, indicating that the overall mixture distribution can be well-approximated by M (< N) effective components.\n\nWe give the visualization of the VAE space in the first column in Appendix B.4-Figure A4. Although the number of components is smaller than N, it remains challenging to fit the distribution, which is not in conflict with our  method. We will add the above analysis in the updated version.\n\n**Q2: More comparison with other SOTA methods on CIFAR-10, CIFAR-100.**\n\nA2: We did not include comparisons with more SOTA results on CIFAR because the methods adopt varying validation settings, making us unable to perform direct comparisons. **Since our method is geared towards more practical scenarios involving large-scale datasets**, we follow the validation setting commonly adopted by these methods that support large datasets—training for 400 epochs on CIFAR. \n\nFor completeness, we also provide a comparison with [a], which achieves SOTA results on CIFAR and is trained for 1000 epochs in their paper. We directly obtained the 10 and 50 IPC distilled subsets from their official GitHub repository and evaluated them using the same validation code as in our setting to ensure a fair comparison. Our results outperform theirs under the same setup. We will include this comparison in the updated version.\n\n| Dataset  | IPC |\\[a\\] (ResNet-18)  | Ours (ResNet-18) | \\[a\\] (ResNet-101) | Ours (ResNet-101) |\n|-|-|-|-|-|-|\n| CIFAR10  | 10  | 36.4±0.3 | 41.3±0.1  | 30.4±0.6 | 35.8±0.6   |\n| CIFAR10  | 50  | 55.4±0.4 | 70.8±0.5  | 41.9±0.6 | 63.9±0.4   |\n| CIFAR100 | 10  | 46.6±0.1 | 49.4±0.2  | 31.9±0.5 | 46.0±0.5   |\n| CIFAR100 | 50  | 61.0±0.2 | 65.7±0.3  | 57.5±0.5 | 66.6±0.2   |\n\n**Q3: The experiment for IPC=1 on CIFAR-10 and CIFAR-100.**\n\nA3: We provide a comparison with RDED in the table below, which achieves excellent results on CIFAR under the 1 IPC setting.\n\n| Method| RDED (ResNet-18)| Ours (ResNet-18)| RDED (ResNet-101) | Ours (ResNet-101) |\n|-|-|-|-|-|\n| CIFAR-10  |22.9±0.4 | 24.2±0.3 | 18.7±0.1 | 21.6±0.2|\n| CIFAR-100|11.0±0.3 | 11.8±0.1 | 10.8±0.1 | 10.4±0.2|\n\n**Q4: The specific VAE model, and the role of diffusion models.**\n\nA4: Thanks for pointing this out, We adopt the pre-trained DiT and VAE from [b].\nThe role of the VAE encoder in the framework is to encode the image to the latent space, while the DiT model is to inverse the latent for sampling new latent representation and then denoise the new latent representation to the VAE space. The VAE decoder is used to decode the sampled latents to pixels. Besides, as shown in Lines 149-155 left column, the motivation for using diffusion models lies in their ability to enhance realism and cross-model generalization, which are the key qualities of a high-quality distilled dataset. Based on this, as described in Lines 110–164 right column, our main motivation is that we identify three key limitations in current diffusion-based methods, primarily because of conducting optimization within the VAE latent space. This motivates us to explore a more effective optimization space, aiming to provide a better paradigm for diffusion-based methods. \nWe will include further clarification in the updated version.\n\n[a] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" CVPR. 2024.\n\n[b] Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" ICCV. 2023.", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Proposed methods make sense. The domain mapping technique efficiently generates synthetic samples that accurately capture the underlying data distribution.\nAll benchmark datasets are used in the dataset distillation problem.", "theoretical_claims": "About the Lemma A.1 in line 577-578, the author claim: for each class C containing m images, Zc is modeled as a discrete distribution, where the m latents are independently sampled from m distinct Gaussian distributions.\nthis statement is not necessarily true because it assumes independence between latent variables for each image within a class. However, in most cases, VAEs do not enforce such a strict assumption. The reason is latents are not independently sampled per image and their distributions are not necessarily distinct for each image. If the model captures class-level semantics well, the means of similar images may cluster together in the latent space. Also, In a standard VAE, the encoder learns a continuous latent space rather than modeling a discrete distribution for each class.\nThus, the claim about a multi-component Gaussian mixture distribution might be incorrect.", "experimental_designs_or_analyses": "The current experimental designs or analyses are soundness and validity, but there are several issues can be discussed.\n1. In table 1, the results on CIFAR-10 do not appear to be state-of-the-art, as multiple existing works have reported higher performance.\n2. The author does not provide results for IPC = 1 on CIFAR-10 and CIFAR-100 to compare with state-of-the-art methods.", "relation_to_broader_scientific_literature": "The use of DDIM inversion in this work is closely related to recent advancements in generative modeling that focus on improving sampling efficiency and controllability within the latent space.", "essential_references_not_discussed": "The reference of dataset distillation and diffusion models are discussed.\nThe authors incorporate a pre-trained Diffusion Transformer (DiT) in their implementation details section line 302, but do not specify which VAE model is used. The described process appears to involve encoding input data, mapping the latent space, sampling new latent representations, and decoding back to the original data domain. However, the role of diffusion models in this framework remains unclear.", "other_strengths_and_weaknesses": "There are several strengths:\n1. The approach demonstrates improved performance, particularly on large-scale datasets, by leveraging a structured latent space for effective sample generation\n2. The method only needs a one-time generation cost, making it computationally efficient compared to iterative optimization-based distillation techniques.", "other_comments_or_suggestions": "I have no other comments.", "questions_for_authors": "Why the author did not compare the results with other state-of-the-art methods. \nThe reason for ImageNet-1K can be they handle subsets, but most other approaches can be compared for small-scale datasets CIFAR10 and CIFAR100.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission has been reviewed by 5 reviewers. 

After the rebuttal, all reviewers acknowledged the merits of the manuscript and agreed that it meets the bar of ICML.

Essentially, the reviewers found the proposed method makes sense and gives rise to good results, among others.

There is no basis to overturn the consensus. As such, the AC recommends the acceptance. Congrats!

Please, however, do account for the suggestions in the final version.
Decision: Accept (poster)