Influence-Guided Diffusion for Dataset Distillation
Authors: Mingyang Chen
Jiawei Du
Bo Huang
Yi Wang
Xiaobo Zhang
Wei Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset Distillation
Dataset Condensation
Diffusion Model
Guided Diffusion Generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a training-free influence-guided diffusion sampling method as a novel dataset distillation scheme and achieve state-of-the-art performance in distilling full-sized ImageNet datasets.

Abstract: Dataset distillation aims to streamline the training process by creating a compact yet effective dataset for a much larger original dataset. However, existing methods often struggle with distilling large, high-resolution datasets due to prohibitive resource costs and limited performance, primarily stemming from sample-wise optimizations in the pixel space. Motivated by the remarkable capabilities of diffusion generative models in learning target dataset distributions and controllably sampling high-quality data tailored to user needs, we propose framing dataset distillation as a controlled diffusion generation task aimed at generating data specifically tailored for effective training purposes. By establishing a correlation between the overarching objective of dataset distillation and the trajectory influence function, we introduce the Influence-Guided Diffusion (IGD) sampling framework to generate training-effective data without the need to retrain diffusion models. An efficient guided function is designed by leveraging the trajectory influence function as an indicator to steer diffusions to produce data with influence promotion and diversity enhancement. Extensive experiments show that the training performance of distilled datasets generated by diffusions can be significantly improved by integrating with our IGD method and achieving state-of-the-art performance in distilling ImageNet datasets. Particularly, an exceptional result is achieved on the ImageNet-1K, reaching 60.3\% at IPC=50. Our code is available at https://github.com/mchen725/DD_IGD.

Published as a conference paper at ICLR 2025
INFLUENCE-GUIDED DIFFUSION FOR DATASET DISTIL- LATION
Mingyang 2 Jiawei Du', Bo Huang Yi Wang', Xiaobo Zhang', Wei Wang!?
'The Hong Kong University of Science and Technology (Guangzhou)
The Hong Kong University of Science and Technology CFAR,, A STAR, Singapore
*Dongguan University of Technology
Southwest Jiaotong University
mchenbt @connect. ust. hk, wei was vessuust..... hk
ABSTRACT
Dataset distillation aims streamline the training process by creating compact yet effective dataset for much larger original dataset. However, existing methods often struggle with distilling large, high-resolution datasets due to prohibitive resource costs and limited performance, primarily stemming from sample-wise optimizations in the pixel space. Motivated by the remarkable capabilities of diffusion generative models in learning target dataset distributions and controllably sampling high-quality data tailored to user needs, we propose framing dataset distillation controlled diffusion generation task aimed at generating data specifically tailored for effective training purposes. By establishing a correlation between the overarching objective of dataset distillation and the trajectory influence function, we introduce the Influence-Guiie to generate saiiing-effecttvv data without the need retrain diffusion models. An efficient guided function is designed by leveraging the trajectory influence function an indicator steer diffusions to produce data with influence promotion and diversity enhancement. Extensive experiments show that the training performance of distilled datasets generated by diffusions can be significantly improved by integrating with our IGD method and achieving state-of-the-ard performance in distilling ImageNet datasets. Particularly, an exceptional result is achieved on the ImageNet- K,, reaching 60.3% IPC=50. Our code is available at tts //github. http://i....cooocccee2e2255555/0 IGD
Diffusion (IGD) sampling framework
INTRODUCTION
Dataset distillation has gained significant attention due to its ability to balance the conflict demands of maintaining training effectiveness while overwhelming resource overhead. This method involves crafting a compact yet effective surrogate dataset for large-scale original dataset. The surrogate is optimized retain essential information from the cumbersome original. enabling models trained on it to achieve performance comparable to those trained on the complete one.
Early dataset distillation methods have made significant strides in distillation efficacy through various insightful paradigms (Zhao et al. 2021 Kim et al. 2022: Nguyen et al. 2021 Cazenavette et al. 2022 Du al. 2023 Cui et al. (023). However. their success is mainly limited to distilling small
datasets like CIFAR (Krizhevsky & Hinton 2009) or downscaled ImageNet (Russakovsky et al.
2015) with low resolution. Extending these methods higher-resolution datasets (e.g., 2 128 x 128) hindered by treating data entity and refining it at the pixel level. This escalates time and computational costs with data dimensionality and preset compression ratios, typically indicated by Images Per Class (IPC). Moreover, prioritizing pixel-level optimization overlooks distributional shifts from the original dataset. Yet, at higher resolutions. synthetic data retains ineffective high-frequency patterns, leading to performance degradation (Cazenavette al. 2023).
Recognizing the robust capability to capture intricate data distributions, a recent approach (Gu et al. 2024) integrates diffusion models tackle the high-resolution challenges faced by previous

---

Published as a conference paper at ICLR 2025
pixel-oriented methods, achieving cutting-edge performance. This technique entails fine-tuning a latent diffusion model through a minimax criterion, yielding distilled datasets that harmonize representativeness and diversity for better alignment with the authentic data distribution. However, research on core-set selection techniques (Killamsetty et al., 2021a b) [yer et al. 2021) indicates that data sampled directly from the authentic distribution Concerns remain about the effectiveness of the proposed objective in generating distilled datasets that are optimally tailored for highly effective training.
contribute unevenly model training.
In this work, we introduce a new paradigm of using diffusion models in the task of dataset distillation, termed the Influence-Guie pling method. This method is conceptually tailored to directly guide diffusion models in generating data under a generalized qaining-effective condition, with-
0.50
Diffusion (IGD) sam-
Validatton Accuracy
out requiring to retrain diffusion models. We high-
light the challenges inherent in achieving this target,
particularly due to the abstract nature of the tailored condition, in contrast to existing controlled diffusion generation tasks that involve explicit content spec- ifications (Rombach et al.] 2022, Ho & Salimans 2022). To address this challenge, we first establish a correlation between the overarching objective of dataset distillation and the trajectory influence func- tion (Pruthi et al. 2020), which approximates the impact of training on given data in terms of test loss variation. Building on this connection, ance as indicators to steer the diffusion models towards generating data with influence promotion and diversity enhancement, significantly enhances the performance of the vanilla Diffusion Transformer (DiT), outperforming results obtained through the fine-tuning method Minimax. Moreover, IGD complements Minimax achieve even better results, with a simultaneous increase in influence.
0.36
DiT-IGD
Minimax
Minimax-IGD
Fire Turing
Guided Sampling
(Durs)
Figure 1: Enhanced arosstarchitecture perfor- mance with average influence by integrating IGD in distilling ImageNette with IPC=100.
develop diversity
and influence-based guid-
illustrated in Figure 5] As evidenced by Figure I integrating IGD
In summary, contributions are follows:
We propose new scheme for dataset distillation by framing the task as guided-diffusee generation problem.
We establish a novel diffusion sampling framework that pioneers the integration of the influence function guidance for the controlled diffusion generation, with the aim of achieving generalized training-enhancing objectives.
Experimental results illustrate that our method significantly improves the performance of diffusion models across different architectures on two ImageNet subsets. Furthermore, state-of-the-ar result is achieved on the ImageNet-IKe reaching 60.3% at IPC=50.
PRELIMINARIES
2.1
BACKGROUND ON DATASET DISTILLATION
We refer to the target dataset as T (xx,, 1 Each sample x; is drawn i.i.d. from a natural distribution q(x), where æ; E Rd and y; E y = (1, ......C C} refers to the ground-truth label. Dataset Distillation (DD) aims condense this large labelled dataset T into smaller synthetic dataset S {(u; !E with u, E Rd and y, E 3. such that |S| < [T]. The reduced dataset optimized to retain essential information from T to ensure that any model initialized with parameters 0, can be optimized to minimize the validation loss on the target dataset T:
min
(xi, yo 05) (w yes 0o)]
s.l. 0 Alg(S, 0o)
Here, ASg(S, 0o) arg ming E(u.yyes [e(u, 1, yı: 0)] represents the training algorithm that opti- mizes the initialized parameters O over the synthetic data S. and f(x, y; 0) denotes the prediction
2

---

Published as a conference paper at ICLR 2025
loss of model with parameters 0 on data pair (z, y). To prevent unexpected distributional shift, we propose to frame DD as learning a conditional distribution of the authentic distribution, e.g., (ondditii) to sample near-real data under the generalized training-effecttin conditions.
2.2 GUIDED DIFFUSION GENERATION
Given samples from the data distribution q(z), diffusion models are capable of learning a parameter-
ized distribution (a) that approximates ((=) and is easy sample from it (Song et al. 2020b). On a high level, this is implemented through a forward noising process and a reverse denoising process. Concretely, the forward process gradually adds Gaussian noise ~ (N I) of different magnitudes to clean data point 20: 2, = 20 + v1 0gg, where a, controls the noise scale at step t. A diffusion model is denoising function that learns by minimizing the dissimilarity, e.g., mean squared error, between the predicted noise c,(2,, t. c) and e, where e is a conditional input such as labels. The reverse process generates denoised samples by sampling from (x( = 20). which is generally parameterized as Gaussian distribution and varies across studies in its approximation (Ho et al.) 2020). For instance, Denoising Diffusion Implicit Model (DDIM) (Song et al.. 2020a) first predicts the clean data point Zuje based on 2e as:
1 1 (zi / val
201
VI (,c
2t-1 is then sampled from N (VoL-T2ol + -1-aa-1-occol t,)), where or is the predefined noise factor. For notation simplicity, we abstract this process _T-1 s(2, t, c2)- In this work, we adopt the widely utilized latent diffusion (Peebles & Xie 2023) as the backbone. Here, an encoder is employed to transform images latent codes
E x and decoder ((-)
reconstructs latent codes back to the image space to obtain the distilled dataset S ((D(z,), ))) !
Diffusion models typically employ conditioning to tailor outputs specific user inputs, such as labels or text prompts. However, our purpose diverges from explicit content specifications, focusing instead on more abstract requirements. We aim to guide the diffusion model to identify conditional distribu- tions within the learned distribution and selectively sample data to optimize training effectiveness. To this end, we employ more adaptable method of controlling model outputs through guided diffuddnnoosooon
generation (Bansal et al. 2023 Yu et al. 2023 Gopalakrishnan Nair et al., 2023). These methods are largely inspired by the energy- based model (HBM) used for formulating coreebbaed diffusions (Song) al. 2020b 2021). I. Intuitively, any metric function fc(.) that subtly measures the compatibility of the noisy sample 2; to the condition C' valid for providing steering guidance. By this means, the sampling step can generally be implemented as:
- s(z,, t,
P: fo(z)))..
where Pt is defined to align with the denoising scale of the current (z) By introducing meticulously designed guided function that effectively measures the impact of data training efficacy (e.g., as depicted by validation loss), this implementation seamlessly aligns with our objective of framing dataset distillation sampling data from desirable conditional distribution.
3 METHOD
3.1 ESTIMATING DATA INFLUENCE AS DIFFUSION CONDITIONAL GUIDANCE
We identify influence function Koh & Liang 2017) 7) as insightful parallel research that can quantify the impact of specific training data on model validation loss. This highly relevant to the design of metric functions used for steering guidance in diffusion models under our training-effectte
condition. Leveraging the Fundamental Theorem of Calculus, Pruthi et al. (2020) introduced trajectory influence to estimate the cumulative influence of training data pair (æ, y) on validation data pair (æ', y'). This method integrates the stepwise changes in the loss of the validation data throughout the training process. In our case, employing Stochastic Gradient Descent (SGD) the training algorithm Alg. the model update can be expressed as 0,41 0, = - -n Vo (xx y: 0.) where represents the learning rate at timestep I. Utilizing the first-order Taylor expansion, the loss change of (æ', y²)
3

---

Published as a conference paper at ICLR 2025
each timestep can be approximated by:
f( y':: 0(+1) - -(x'' y': ~ Vol(='. 01) (0.+1 -0)
(4)
= -/ Vol(a', y': 0.) Vol(a, y; 0.).
The overall influence of (ar, y) on (ar, y²) throughout the training trajectory is quantified by aggre- gating these stepwise changes across epochs:
I(x, x') = Σ Th Vol (w', 0.) Vot (z 0.) x 2('',y':0 0o) ela', v/: Or),
(5)
where The denotes the learning rate of the e-th epoch, for total of E epochs. By substituting the validation data (a', 3/' as the real data in the original dataset, this formulation is an effective approximation to the general objective of dataset distillation defined in Equation (I). Based on this insight, we define the objective of the guidance for a latent code 2 given a certain class c as:
Tel
E
max
Σ
I(D(2),
= max nee Vole (2c; 03) Vole (D(z):0?),
Σ ==0
(6)
ITc
1=1
Vol (Ii,, c; C.)) based on the Fubini's Theorem and Te is the subset of the given class c, 0 represents checkpoint obtained the decoded data. Intuitively, this objective can be optimized models trained on synthetic data obtain trajectories equivalent to those trained on Te, thereby maximizing the validation loss drop. This essentially shares a similar purpose with the Mradient-Matching (GM) scheme (Zhao et al., 2021]; Zhao & Bilen 202 However, we identify three primary issues with directly adapting this formulation as the metric function for guided diffusion in dataset distillation: (1) prohibitive cost: the necessity of model retraining at each diffusion sampling step is computationally burdensome; (2) accumulated error: akin the limitations of the GM method, the gap between trajectories inevitably accumulates during training on synthetic data, leading to ineffective matching and consequently degraded performance (Cazenavette et al.. 2022); (3) information redundancy: the relatively poor diversity of deffusiion-geeraate data limits its effectiveness for dataset distillation (Du et al. 2023), and matching with the averaged real gradients, as shown in Equation (6, may further exacerbate this issue.
where Volc(X 0.)
1Ta1
if
In the following section, we tackle these challenges by developing functions and detailing our Influence- Guided Diffusion (IGD) sampling framework.
guided
3.2 EFFICIENT INFLUENCE- GUIDEED DIFFUSION SAMPLING WITH DIVERSITY CONSTRAINT
Denote 07==0% 7- (Z c: 0Z=,) as checkpoints trained on the real subset To with SGD and the same learning rate schedule as on the synthetic data. Replacing the checkpoints 0 with TT in Equation (6) is an optimizably equivalent target. This equivalence holds because these two targets converge to the same optimal solution when can provide the same training dynamics as To, i.e. (alo 0%) implementation, we extend this usage to the checkpoints 0² obtained through standard mini-batch updates over the entire dataset T. This adjustment mitigates the mismatch caused by the discrepancy between synthetic and real trajectories (Kim et al. 2022). while also eliminating the time cost associated with retraining models on S at each sampling step. Additionally, we use cosine similarity instead of the dot product to stabilize the magnitude of the guidance signal provided by the influence function. These modifications yield the influence guided loss function as:
Vole (Dz); )!:) Ve E [0. EJ. Building on this insight, in practical
E
Vol. (x 0?) Vol. (D(=):0)) Vol. .x OT) Vo (D(2):
gr(=) =
T/e 1
(1-
Directly computing the influence over an intermediate noisy z, is undesirable, as the checkpoints 0T trained clean data and do not adapt provide meaningful guidance as metric function when the input is noisy (Ho & Salimans| 2022). To mitigate this issue, we utilize the predicted clean data Soft of the current 2,, based on Equation (2) defined by DDIM, as an approximation of the real 20- Subsequently, we compute the influence guidance Gr (Zoj) the predicted clean data and derive the guided gradient Vz, V2,9 Gr((2, vi ar (2,, ())/ Voz) through backpropagation.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: Influence-Guidee Diffusion Sampling
Parameters: Class c, influence factor Pt, deviation factor "/. scales Y==) guided range A,B
Required: Pre-trained diffusion model 6g, list of retained checkpoints R, list of averaged gradients Ge. generated data memory Me. decoder model D
Initialize: Sample initial random noise 2T (0, I);
for ( 7 to do
Obtain the denoised signal EΦ(zt, t, c) from the diffusion model;
if t in [A, B] then
Calculate the influence metric 97 (20j)) as Equation (7 with R. and Ges
Calculate the deviation metric GD (2e) as Equation (8) with Mci
Implement guided sampling zt-1 s (,,,,, cØ) - Ps V= 91 (20je) 7 ~2 ID(i)):
10
else
11
Implement vanilla sampling 21-1 = s(2,,
12 return Decoded synthetic image D(zo):
To ensure diversity and avoid excessive redundancy in the surrogate dataset's training signals, we propose adding a constraint to the generation objective. This constraint ensures that the similarity between generated data within a certain class does not exceed specified threshold: sim(z, 2,) S 5, Vz, 2j E Ze, where * zj. In practice, we incorporate this constraint using Lagrangian multiplier and propose a deviation guidance function to optimize in each guided sampling step:
2' subject
&
(D(2)
arg max E
|=|||||||
(8)
where M represents the set of all previously generated data for certain class c.
Ultimately, we utilize the influence guidance of Gr (Zoje) alongside the deviation guidance of Gp (2:). reformulating the guided sampling step as:
tz,,
2:-1 = s(z,, po V where Pt = k k. at 91
-
(9)
is the scale factor designed to adaptively adjust the magnitude of the influence guidance alongside the dynamics of the denoised signal cg., and 7t is empirically preset for the deviation guidance. Furthermore, we introduce two practical techniques that are essential for enhancing both the efficiency and efficacy of the proposed IGD framework.
Choosing representative checkpoints via gradient similarity. For efficiency, trajectory influence initially suggests saving checkpoints at regular intervals to compute step-wise influence. However, given the non-lineat nature of training dynamics, evenly spaced checkpoints may scatter attention to critical stages. To efficiently calculate the influence guidance, we propose a simple yet effective filtering algorithm. We store 0g the first checkpoint in a list R and compute its averaged gradient E. Vole(X' 0T) as the initial reference. For each subsequent checkpoint, we compute the averaged gradient and calculate its cosine similarity with the reference. If the similarity below given threshold, we store the current checkpoint and update its averaged gradient as the new reference. This process traverses all epochs, and only the retained checkpoints in R used by influence guidance.
Mitigating overfitting and reducing runtime by carly-stage Guidance. Guided diffusion tasks face trade-off between generation quality and the impact of guidance (Lugmayr al. 2022 Bansal et al. 2023). In our problem, we observe that samples generated with a large preset ls in Pe achieve significant influence loss reduction but also exhibit noticeable abnormalities and degraded performance. Detailed evaluations are provided in Section 4.4 Empirical observations in diffusion generation demonstrate that most semantic content is generated during the carly-to-mid stages of sampling (Yu et al. 2023). We adopt guided sampling only in these partial steps, allowing vanilla sampling to refine details in the remaining steps. For example, in DDIM with 50 sampling steps, guided sampling is applied only when t is in [30, 45). This approach allows data generated with strong guidance to maintain comparable influence without noticeable abnormalities or performance degradation. Consequently, this also reduces the runtime associated with guidance calculation.
5

---

Published as a conference paper at ICLR 2025
Table I: ImageNette ImageWoor Woof: Performance comparison with state-of-the-ai pixel-level distillation methods, pretrained DiT and Minimax-tuned DiT models with vanilla generation. DiT- IGD and Minimax-IGD denote utilizing our proposed IGD sampling framework for generation.
Datteeet
Model
IPC
Kandom
DM
IDC-1
DN1
DIEGG
Minimax
Minimax-Gii
Pall
10
66.0200.5
49.8.1.1
48.2:12
56.2113
61.9199
58.2409
58.841.0 0
6 6
50
71
70.3408
7244.07
74.1406
80.910.9
76.9.09
8233:0.8
94.3
100
79.9.0.8
78,550.8
60.6111
78.2:033
84.510.7
8:.1
863:...8
Netie
10
54.2:12
6022007
60440066
62.8408
66.32111
03.2=1.0
63.521.1
ResNarAP-10
50
77
76.7e1.
77.4-0.7
76.9095
81.0-12
7822
S2.3e1.1
94.660.5
100
81.140.
809
81.5112
801el11
85.24005
S1.3
$6.1=9.9
55.8+1.0
60.00077
610000
62.5+09
67.7
64.-00
2-1.2
ResNet-18
50
75.8+1.1
75.0+1.0
77.8+0.7
752200
01.0007
18.1
52.00.3
95.3e0.6
100
82.0604
81.58004
81.720.
773400
84.4
81.3
56.0:0.6
10
25.241.1
27
34.1208
32.3408
35.040.8
33.5+14
36.2+16
ConvNet 6
50
41.9114
45 43
62.6609
48.5113
$1.210.7
50.721.8
55,7
85.920.4
100
52.3a1.5
50.1
51.0a1.1
54.2a155
61.1a1.0
17.1
63.0
Woof
10
$1.620.8
29.21.0
38.5.0.7
39 39.090
41.020.8
69.6
43.3
ResNetAP-10
50
50.121.6
47
69 69.3g09
55 85.1111
62.71122
89.8
65.0:9..
87,220.6
100
59,24099
59.3113
56.400.
62.5
69.7409
66.81122
71.5+0.8
30.9+1.3
60.340.6
36 760.8
40 64066
8
2212.112
47.2+1.6
ResNet-18
50
54.0+0.8
53.9=07
$4.5+1.0
57 4-0.7
62.0+1.1
50.5:0
65.4=1.8
80.0
100
63.e20.
64.960.
57.780.
62.34005
70.6+1.8
67.40077
72.1=9.9
Algorithm [ outlines the detailed process of our influence-guided diffusion sampling framework for generating each synthetic image. Before constructing the surrogate dataset, we first obtain = ) trained on T and apply the proposed filtering algorithm to retain representative checkpoints in the list R. Before initiating generation for a specific class c, we calculate the averaged gradient Vol.( c; O across each retained checkpoint and store them in list G.. Subsequently, we execute the algorithm, storing the generated images in memory Me until the desired number of images reaches the preset target IPC (images per class).
checkpoints
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. As our primary interest lies in large-scale, high-resolution distillation tasks, we assess the performance of our method on the complete ImageNet- 1K dataset (224x224) (Russakovsky et al., 2015). To provide comparable evaluations across varying task difficulties, we conduct comprehensive experiments on two representative subsets, ImageNette and Image (Howard 2019). ImageNette, consisting of 10 classes with less similarity and therefore easier to distinguish between, contrasts with Image Woof, a challenging subset containing 10 classes of dog breeds.
Baselines and evaluation metric. We compare method with several state-of-tte thethe-rrt dataset distillation methods including DM (Zhao& & Bilen 2021b), IDC-1 (Kim et al. 2022). SRe"L (Yin et al. 2024), G-VBSM (Shao al.; 2023), and RDED (Sun al. 2024). Additionally, we regard pretrained DiT (Peebles & Xie| 2023) as a notable baseline because it achieves performance comparable to state-of-the-art methods even without tailored optimizations for dataset distillation. Furthermore, we include Minimax (Gu et al. 2024). recent work refined DiT specifically for dataset distillation through fine-tuning scheme, as perpendicular baseline. Test architectures include ConvNet-6, ResNet- 10 (He et al. 2016) with Average Pooling, ResNet-18, ResNet- 101, MoblieNet-V V2 (Sandler ct al. 2018). EfficientNet-BOO (Tan 2019) and Swin Transformer (Liu al., 2021). The top-1 test accuracies of models trained on distilled datasets with different IPC (Image Per Class) are reported.
Implementation detail. For a fair comparison, we follow the official implementation of Minimax, utilizing a latent DIT model from Pytorch's 's official repository and an open-source VAE model from Stable Diffusion. DDIM (Song et al. 2020a) with 50 denoised steps is used as the vanilla sampling method for generation. For each test dataset, we train 6-layer ConvNet (ConvNet-6) for 50 epochs with the learning rate 1 x 10 to collect the surrogate checkpoints used in Equation similarity threshold for choosing representative checkpoints is set as 0.7. The detailed setup of hyperparameters i and 7, for each datasets is discussed in Appendix AA.1 All the experimental results of our method can be obtained single RTX 4090 GPU.
The

---

Published as a conference paper at ICLR 2025
Table 2: ImageNet-IK: Performance comparison over ResNet-18 with state-of-the-art dataset
distillation methods, pretrained DiT and Minimax-tume DiT models with vanilla DDIM generation.
Dataset
IPC
SRc²L
G-VBSM
RDED
DIT
DIFIG
Minimax
Minimux- Minimax-ICiD IGD
ImageNet-IK
10
21.3+0.6
31 410.5
42,080.1
39.6204
45.5=0.5
44.3+0.3
46.2+0.6
46.8a0.2
51.840.4
56.5a0.1
52.920.6
59.8
58.620.3
60.320.4
Table ImageNet-IK: Cross-acchitecture generalization performance comparison.
ResNet101
MobileNet-V2
EthicientNet-BO
Swin Transtorner
IPCI0
TPC50
IPCI0
TPC50
IPCI0
JPC50
IPC10
IPC50
48.3+1.0
61.2+0.4
40.4:0.1
53.340.2
31.0=0.1
58.540.4
42.340.6
53.240.8
DiT-IGD
52.6++1.2
66 2=0.2
39.2=0.2
57.8+0.2
47.7=0.1
62.0+0.1
44.16
58.6+0.5
Minimax-IGD
53.4#09
66.8:0.2
39.720.4
58.5+0.3
48.5=0.1
62.750.2
44.8+0.8
58.240.5
4.2 COMPARISON WITH STATE-OF-THE-ART METHODS
Evaluation on Woof & Nette. As a training-free sampling framework, our IGD method can be incorporated into the pretrained DiT and Minimax- tuned DiT during the generation process. We designate these two methods as DiT-IGD and Minimax-IGD, respectively. As depicted in T our IGD-based methods demonstrate significant improvement over the original backbone methods, and achieve tate-of-theear performance across both Woof and Nette datasets in all IPC settings. These enhancements are consistently observed across all evaluations conducted on the three tested architectures, highlighting robust cross- architecture generalization ability. Particularly for IPC 250, DiT-IGD notably enhances the performance of DiT by 5.8% on Nette and by 6.6% on Woof, on average. Comparing with Minimax, Minimax-IGD averagely provides 4.7% boost Nette and a 5.1% boost on Woof. Moreover, we observed that DiT-IGD outperforms Minimax in most evaluations. Especially for the easier dataset Nette, despite the class distinctions facilitating knowledge condensation, Minimax only shows marginal average improvement of 2.5% over DiT at IPC=100. In contrast, DiT-IGD achieves an average boost of 6.1%. Compared to diffusion-basee methods, the pixel-level optimization methods DM and IDC-1 achieve moderate performance gains over random original images IPC=10. However, as the IPC increases, the performance gain drastically diminishes or even becomes negative.
Evaluation on ImageNet-1K. Recent approaches proposed for efficiently distilling ImageNet-IK KK data rely on using well-trained models to provide synthetic images with soft labels to acquire richer information. Following the evaluation protocol of the RDED, we employ a ResNet-18 18 model, trained on the original dataset, to generate soft labels for synthetic images. The performances shown in Table are evaluated over the same ResNet-18 architecture. The results demonstrate consistent improvements in integrating our IGD method over the DiT and Minimax methods. In particular, DiT-IGD demonstrates significant improvement to raw DiT, with enhancements of 5.9% at IPC=10 and 6.9% IPC=50. This also positions Minimax-IGD method the forefront of this practical distillation task, surpassing the state-of-the-art image-based method RDED by 4.0%. In the cross- architecture comparison detailed in Table 3 synthetic datasets generated using our IGD methods generally outperform those created by RDED across four different unseen networks. Notably, our DiT-IGD and Minimax-IGH methods surpass RDED by an average margin of 4.6% and 5.0% at IPC=50, respectively. These remarkable performance improvements underline the promising potential of diffusion-based methods in the future of dataset distillation research.
4.3 CROSS-ARCHITECTIE ROBUSTNESS OF INFLUENCE GUIDANCE
In our IGD framework, influence guidance necessitates a surrogate model to be trained on the original dataset, collecting representative checkpoints for calculating guided loss. Here, we test the impact of influence guidance obtained over networks of different architectures, including ConvNet- 6, ResNettA-11,, and ResNet18. We then train these networks on generated surrogate datasets from scratch and evaluate their cross- arohttcrch performance. Table demonstrates that datasets generated based on ConvNet-6 generally exhibit superior performance. In most cross-arce cooss-architecture evaluations involving ResNetAP-10 and ResNet-18 18, they even outperform datasets generated with
7

---

Published as a conference paper at ICLR 2025
Table 4: Cross-architecture performance of DiT-IGD using different surrogate architectures to
calculate influence guidance.
ConvNes-6
ResNerAP-10
ResNes-18
Surnogate
IPC10
IPC50
IPC100
IPC10
TPC50
IPC100
IPC10
IPC50
IPC100
Com-Nee
61.921.9
$0.9+0.9
84,5+0.7
66.8a111
81.0s1.2
85.250.5
67.740.3
$1.0.0.7
$4.1208
Netie
ResNetAP-10
58.910.4
79.520.8
63.740.4
66.240.8
82.350.2
84,410.8
66.750.6
8233,.9
85.40..4
ResNNe18
78.3609
50.120.3
63.321.6
79,560.8
82.12005
63.180.3
$0.300.7
83.111.4
Nos-
35.0e8.8
54.240.7
61.1+1.0
41,040.8
62.7s1.2
69.7410.9
4488008
62.0-1.1
70,6118
ResNerAP-10
33.8+100
39/el
61.5408
Res/Net 18
34,3+0.8
54.360.8
61.0e1.8
39.5+1.1
61.0+14
68.7e0
43.8e1.4
62.9.1.0
60.5e04
Table 5: The ablation study of proposed influence guidance Gr and deviation guidance Gn tested with ResNet- 18 on ImageNette.
Table 6:
Comparison of checkpoint selec- tion strategies for Minimax-IGD: the gradient- similarity-bar based method versus regular interval selection, on ImageNette with ResNet- 18.
DIT-IGD
Minimax-IGD
G2
GD
IPC
TPC100
JPC50
JPC100
Threshold
Checkpoints
Regular
Ours
75.2+0.9
77.840.6
78.140.6
81.3=0.7
0.65
79.5w0.6
80.420.7
76.5+0.6
79. 1+0.4
81 540.4
85. 1=0.4
79.8+1.1
82.0+0.3
78.2+0.4
80.7=0.7
78.5+0.2
82.8=0.3
0.75
80.5+0.4
81 4+0.5
81.020.7
84.420.8
82.010.3
86.020.6
0.80
10
81.140.5
80.840.3
the test architecture. Additionally, due to fewer model parameters compared to the other two, the computational time required for influence loss calculations is reduced. Based on these observations, we choose to utilize ConvNet-6 as the surrogate in our formal implementation. However, we also note that the performance gap between datasets generated with different architectures not significant. Particularly, datasets generated with ResNetAP-11 notably outperform ConvNet-6 in several tests against ResNet- 18. These results further validate the robustness and generalization ability of our proposed IGD sampling framework.
4.4 ABLATION STUDY AND ANALYSIS
Guidance component analysis. Table presents the performance achieved by independently applying influence guidance and deviation guidance to raw DiT and Minimax. The independent utilization of the two proposed guidance mechanisms still enhances the performance of both backbone methods. Specifically, in the case of raw DìT, the incorporation of deviation guidance yields results akin to those obtained with raw Minimax, primarily due to its ability to augment the diversity of generated data. Conversely, for Minimax, sole reliance on influence guidance markedly elevates its performance, achieving parity with the comprehensive framework. Despite Minimax's 's inherent focus on refining sample diversity through fine-tuning, additional gains can be attained through the integration of deviation guidance. Moreover, it is important note that although influence guidance yields moderate improvements for raw DiT, the integration of deviation guidance results in significant enhancements. These observations substantiate our discourse regarding the critical role of data diversity in optimizing influence effectiveness. Conclusively, the synergy between influence guidance and deviation guidance complements each other, facilitating our guided sampling framework harmoniously in aligning with the training-enhancio enhancing objective.
Early stage guidance analysis. We assess the practicability our carly-stage guidance strategy by comparing it with the entire guided sampling approach on Image with variations in the influence guidance scaling factor k. Figure 2b demonstrates that applying the influence guidance throughout the entire generation stage with large preset & can significantly reduce influence loss. However, as illustrated by Figure 2c when A: > 10, despite a reduction loss, validation accuracy notably drops, likely due to overfitting to the surrogate used for influence calculation. Moreover, this also leads to abnormal image generation shown in Figure 2a In contrast, the early-stage guidance strategy allows strong guidance signals to steer the generation process effectively while mitigating the overfitting problem. Consequently, this strategy achieves superior performance in less generation time, thereby enhancing both the efficacy and efficiency of the process.
8

---

Published as a conference paper at ICLR 2025
Nor Hormalbed Influence
Validation Accurac (%6
Entire Guiding
Early Stage Guiding
6.60
0.5
0.50
Larly
Latly
P:
kin
(a)
(b)
(c)
Figure (a) Examples generated using entire and early-stage guidance with varying influence magnitude k on Image Woof; (b) Averaged normalized loss G1 of datasets generated with different values of and IPC=100; (c) Corresponding validation accuracies for varying k.
Wasserstein distance : 6.04
Wasserstein distance to To TT:40 4.61
70.6%
Audiira Sprner
Coglin godkaadd
Goldee
-40
-00
-00
(a) DiT
(b) DIT-IGD
Wasserstein distance to:: 5.78
Wasserstein distance to 4.92
Acc: 67.4%
Acc: 72.1% 1%
43-
23
20
Dnstoiiniggggee Maggeck
Golden oetheer
-0
Minimax
(d) Minimax-IGD
Figure 3: Visualization study for sample distributions of synthetic datasets (IPC=100) generated by four methodologies versus the original ImageWoot dataset. Smaller Wasserstein distances to the original dataset T signify closer alignment with the authentic distribution.
Checkpoints selection strategy analysis. We assess the efficacy of the gradient-similarity-bee checkpoint selection strategy proposed for computing the influence-guided loss (Equation (7)). A predetermined threshold is utilized to determine checkpoint selection based on the similarity of its averaged real gradient the current reference, with an empirically identified suitable range set as [0.6, 0.8). Thresholds beyond this range result in excessive checkpoint selection, leading to diminished efficiency, while overly small thresholds yield minimal selection. The baseline comparison involves the original trajectory influence's strategy, which saves checkpoints at fixed regular intervals. In Table 6, we contrast our strategy's ' results with various thresholds against the original regular strategy. To ensure fairesss, equal number of regularly collected checkpoints is used for guidance calculation at each threshold scenario. Comparative analysis reveals superior performance the regular approach. Notably, threshold of 0.7, strategy with 4 checkpoints outperforms the results of 10 regularly selected checkpoints, demonstrating enhanced efficiency and efficacy. For a case study, checkpoint selection indexes of (0, 4, 11, 40) observed threshold of 0.7, compared
our strategy over

---

Published as a conference paper at ICLR 2025
DIT
DIT-IGD
Minimax-IGD Minimax
Figure 4: Comparison of image generation results from raw DIT, DIT-IGD, Minimax, and Minimax- IGD. Images each column share the same random seed. Integrating IGD directly into the generation process produces high-quality data with varying semantic content and enhanced diversity compared to vanilla generation. Many instances exhibit robust consistency under the guidance IGD.
the regular indexes {0. 16, 38). This adaptive selection indicates better alignment with typical training dynamics, more checkpoints are selected from the early stages of training.
4.5 VISUALIZATION STUDY GENERATED DATA
Data distribution comparison. To clearly investigate the effect of our guided sampling method on diffusion generation, Figure3] t-SNE distribution comparisons among the full ImageWoof training dataset and data produced by two baseline methods, DiT and Minimax, as well our two IGD-based approaches, each set at IPC=100. Additionally, we use the Wasserstein distance to quantitatively evaluate how well the distributions of the generated datasets align with the entire training dataset. Relative to the Minimax method, our IGD approach guides the diffusion process to achieve closer match the original training set's distribution, offering more comprehensive coverage and lower Wasserstein distances. Notably, Minimax-IGI surpasses DiT-IGD in performance, despite higher Wasserstein distance from the original dataset. This finding lends partial support to our hypothesis that pinpointing a pivotal conditional distribution within the authentic distribution can be more beneficial than mere distribution alignment.
Synthetic image comparison. Figure 4 compares images generated by vanilla sampling of raw DiT and Minimax with those from guided sampling methods DiT-IGD and Minimax-IGD. using the same random seeds for each column. While baseline DiT generates high-quality images, they often share similar content, such as poses and structures. Minimax attempts address the diversity issue in the generated data through fine-tuning DiT, but in many cases, the primary content layout of the objects does not significantly change. In contrast, our method introduces additional signals in each guided generation step, achieving significant content variation and enhanced diversity without reducing quality. Furthermore, the guided signal from IGD robust, producing similar content in both Minimax fine-tuned DiT and raw DiT in many cases.
CONCLUSION
In this work, we introduce novel approach to dataset distillation by framing it as a guided diffusion generation problem. We correlate the general objective of dataset distillation with the trajectory influence function, designing an efficient influence-guided function for the diffusion sampling process. Additionally, we implement a deviation guidance function to ensure diversity and prevent training signal redundancy. These innovations enable us to create an efficient influence- guided diffusion sampling framework. Comprehensive experimental results illustrate that our method significantly improves the performance of diffusion models and demonstrate remarkable cross- architecture generalization ability.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Jiawei Du was supported by the A*STAR Career Development Fund (Grant No. C233312004), Yi Wang was supported part by the Guangdong Basic and Applied Basic Research Foundation (Grant No. (02381515120058) Wei Wang was supported by the Guangdong Provincial Key Laboratory of Integrated Communication, Sensing, and Computation for Ubiquitous Internet of Things (Grant No. 2023B 12120200010001 the Guangzhou Municipal Science and Technology Project (Grant Nos. 2023A03J0003, 2023A03J0013, and 2024A03J0621) 1), and the Institute of Education Innovation and Practice Project (Grant Nos. G01RF000012 and GO1RF000117)
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Good-101-mining discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexci A. Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. CVPR, pp. 10708-10717. IEFE, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun- Yan Zhu. General- izing dataset distillation via deep generative prior. In CVPR, pp. 3739-3748. IEEE, 2023.
Mingyang Chen, Bo Huang, Junda Lu, Bing Li, Yi Wang, Minhao Cheng, and Wei Wang. Dataset distillation via adversarial prediction matching, 2023. URL ptttp::/aarxv..or//eea
05122.89122
Hyungjin Chung. Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.1460 14687, 2022.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 6565-6590. PMLR, 2023.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, pp. 8780-8794, 2021.
Jiawei Du, Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error improve dataset distillation. In CVPR, pp. 3749-3758. IEEE, 2023.
Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akina Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug-and and--aay conditional image synthesis. arXiv e-prints, pp. arXiv-2310, 2023.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-plas priors. Advances Neural Information Processing Systems, 35:14715-14728, 2022.
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffuusio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty- trajectory matching. In ICLR. OpenReview.net, 2024.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778. IEEE Computer Society, 2016.
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840- 6851, 2020.
Jeremy Howard. Imagenette: A smaller subset of easily classified classes from imagenet, March 2019. URL butttp::/githulle www.hh...ooooooooomm wtw.asttaiiiiamaieeeeeeee
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 236669,--3360,, 2022.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464 5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian. Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118 2021b.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 11 11102-11118. PMLR, 2022.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. https:
//www. cs www..toooott.ooooommoo edw/ -wria/ cifar..far.. html 2009. Accessed: March 1, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Noel Loo, Ramin M. Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 12649-226747 PMLR, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022. URL https
/ l/a.xii.org/bb//22222
00927
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 11461-11471, 2022.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-leaming from kernel ridge- regression. In ICLR. Openkeview.net, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/VV confer- ence on computer vision and pattern recognition, pp. 10684-10699 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115 2015.
12

---

Published as a conference paper at ICLR 2025
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chighh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520 2018.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. arXiv preprint arXiv:2311.17950 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356 2020b.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5809-5818, 2024.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: /946, 2019.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinehao Wang, and Yang You. CAFE: learning condense dataset by aligning features. In CVPR, pp. 12186 12195. IEEE, 2022a.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv: 1xXx:22212.00490, 2022b.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from new perspective. Advances in Neural Information Processing Systems, 36, 2024.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang Freedom: Training- free energy-guidee conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23174-23184 23184, 2023.
Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. arXiv preprint arXiv:2310.10400 2023.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. ICML, volume 139 of Proceedings of Machine Learning Research, pp. 12674-12685. PMLR, 2021a.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. CoRR, abs/2110.04111 2021b.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, In ICLR. OpenReview.net, 2021.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In NeurIPS, 2022.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Influence guidance -R (Fop)
4
Dencised
5(3,, Gg)
f--0
Diffusion
(v. (ze)
eice
Diversity guidance
High-influence
(
Influence-Guin Guided Diffusion Generation
Vanilla Diffusion Generation
Low-influence region
Denoised Diffusion
7
If =
Figure 5: An intuitive comparison between influence-guinn guided diffusion generation and the vanilla diffusion generation frameworks.
A.1 RELATED WORK
Dataset Distillation. Current dataset distillation methods can be categorized into meta-learning data-matching, and model-inversion approaches. Meta-learning methods (Nguyen et al. 2021 Zhou
et al. 2022, Loo et al. 2023) tackle dataset distillation as a nested optimization problem, aiming minimize generalization errors on original data caused by models trained on distilled data. Data- matching methods involve synthesizing data to replicate specific behaviours from the original dataset, such latent distributions (Zhao & Bilen 2021b Wang et al., 2022a), gradients (Zhao al. 2021
Zhao & Bilen 2021a; Kim et al. 2022), training trajectories (Cazenavette et al.) 20022 Du et al. 2023 Cui et al. 2023) and surrogate predictions (Chen et al. 2023). Model-inversion methods (Yin al. 2024 Shao al. 2023) are established on data-free knowledge distillation (DFKD) techniques
with specific batch normalization statistic alignment. Additionally, recent research (Gu et al. 2024)
has integrated diffusion models into dataset distillation alongside fine-tuning scheme, perpendicular
to our training-free free, campling-orienteee approach.
Guided-Diffusiin Sampling. Works in this category employ pre-trained diffusion model foundation but modify the sampling method to guide generation with feedback from the guidance function (Kawar al. 2022 Chung et al. 2022 Graikos et al. 2022). Early work employed classifiers as guidance, adjusting gradients during sampling (Dhariwal & Nichol 20. However, classifiers for noisy images are domain -omain-specific and often unavailable. (Wang al. (022b) introduced linear operator-based based external guidance, generating images in the null space of these operators. though extending non -inear functions challenging. Several recent works (Gopalakrishnan Nair et al.) 2023; Yu et al. 2023 Bansal et al., (023) explored general guidance functions. modifying the sampling process with gradients of the guidance function on expected denoised images. However, these methods rely on existing metric functions that can concretely measure specific requirements. In contrast, our contribution lies in guiding the model to generate data that meets abstract, training- enchaining criteria.
A.2 LIMITATIONS AND FUTURE WORK
The main limitation of our method the additional time incurred by guidance calculations during the diffusion sampling process. Despite efforts improve efficiency, sampling framework takes 5 to 6x longer than the vanilla method. For example, raw DDIM generates a 256 x 256 image in ~ 1.5 seconds, method takes ~8.2 seconds a RTX 4090 GPU. This is particularly challenging
14

---

Published as a conference paper at ICLR 2025
for distilling extensive datasets in resource- shouurce-oonstraine scenarios. Consequently, improving the generation efficiency of guided diffusion sampling method will be a key focus of our future research.
A.3 MADDIETT-SIMILATTII BASED CHECKPOINT SELECTION ALGORITHM
In Algorithm we present detailed implementation of the cradient-similarity -baeed----dddd-- checkpoint selection algorithm introduced in Section 3.2 This algorithm is designed to select representative checkpoints for calculating the influence guidance 9r. The core intuition behind this algorithm is that if the gradients at checkpoint closely resemble those at the previous one, the previous checkpoint can effectively represent the current one.
Complexity analysis. The computational overhead primarily stems from calculating the averaged gradient gr w.r. the model parameters 0 at each of the checkpoints collected during training. When using the same cross-entropy loss as in model training, due its additíve nature, the computational complexity of calculating I: at a given checkpoint O, is equivalent to the complexity of one epoch of gradient descent, approximately O(10) B A/2 d), where B is the batch size, N is the number of data instances, and is the data dimension. Essentially, without any optimization, the complexity of this algorithm is similar to training model parameterized by 0 for E epochs. In practice, instead of loading the entire dataset into the dataloader to compute the average gradient Vol, for each class, first load all images from class folder into CPU memory and slice them into GPU memory for gradient computation and accumulation. Empirically, this approach further reduces the runtime of the filtering algorithm. Additionally, the cros-architecctuee evaluation discussed in Section 4.3 and Table f demonstrates that using models with simpler architectures (e.g.. ConvNet) as surrogates can provide more effective influence guidance, further reducing the time overhead for selecting representative checkpoints.
Algorithm 2: Filtering Algorithm for Influence Guidance
Input: Original dataset T, Initial checkpoint 07, Threshold 8 Output: Retained checkpoints list R
Initialize: R. +- 0
Compute E2
05) ) as reference gradient Greti
for t to E do
Compute averaged gradient g. E, (o 07 Calculate cosine similarity Tg. s-ell
i < 5 then R + R [
Update reference gradient gref gti
return R;
A.4 ADDITIONAL. PERFORMANCE EVALUATION ON 101 DATASET
We evaluate the performance of our IGD methods on Food-101 (Bossard et al. 2014) dataset to provide further test on distilling other large, high-resolution datasets. Food- 101 is challenging dataset that includes 101 food categories, totaling 101,000 1,0000 images, with each category containing 250 manually reviewed test images and 750 training images. All images are scaled to a maximum side length of 256 pixels. Results detailed in Table 7 show that our IGD methods achieve superior performances over all IPC scenarios. Furthermore, applying our method to baseline methods, including DiT and Minimax, results in noticeable performance enhancements, with average improvements of 3.8% and 3.5%, respectively.
contrast, the Minimax method yields only a marginal average improvement of 0.8% to DIT. These findings align with evaluations conducted on ImageNet, indicating robust scalability across large, high-resolution datasets.
A.5 ADDITIONAL PERFORMANCE EVALUATION ON CIFAR DATASETS
The results presented in Tables D and demonstrate the outstanding performance of our method in distilling targeted high-resolutie datasets. In this section, we further investigate the generalizability of our method to smaller datasets. Specifically, we compare the performance of our framework with
15

---

Published as a conference paper at ICLR 2025
Table Food-101: performance comparison with state-of-the-art pixel-Ievel distillation methods, pretrained DiT and Minimax-tuned DīT models with vanilla generation. The results are obtained on ResNetAP-10 at different IPCs.
IPC
Random
DM
DiT
DiT-IGD
Minimax
Minimax-IGD
Full
10
16.2+0.5
18.5+0.8
23.9#1.0
27.2+0.9
24.8+0.9
28.3=0.9
50
36.940.3
37.8#0.4
40.8+0.7
45.2+0.7
41.6+1.0
44.8+0.7
78.6+0.4
100
46.8+0.3
44.8=0.3
45.9+0.5
49.7=0.3
46.5+0.5
50.3+0.6
Table 8: CIFAR-10 & CIFAR-100: Perfomrance comparison with two low-resolution-oriene
methods DM and DATM and two highhrreoolution-orrinted methods SRe and RDED.
CIFAR-10
CIFAR-100
IPC
50
500
1000
10
50
100
Ratio (%)
1.0
10.0
20.0
2.0
10.0
20.0
DM
63.1#0.4
74.3+2
79.2+0.2
29.7+0.3
43.6+0.4
47.120.4
DATM
76.1+0.3
83.5+0.2
85.5+0.4
47.2+004
55.0+0.2
57.5+0.2
SRe²L
43.2+0.3
55.320.4
57.1+0.4
24.5+0.4
45.2+0.3
46.6#0.5
RDED
68.420.2
78.1
79.8+0.4
46,440.3
51.540.3
52.620.4
DiT-IGD
66.8+0.5
82.620.6
84.6+0.5
45.8+0.5
53.9+0.6
55.9+0.4
Two state-of-the-art highhresollution-orinneed methods, SRe"L Yin et al. (2024) and RDED Sun et al. (2024), well two low-rrsolution-oorieeee methods, DM Zhao & Bilen (2021b) and DATM Guo et al. (2024), on CIFAR- 10 and CIFAR- 100. Notably, DATM, recent strong baseline, has been shown to achieve lossless distillation on small-scale datasets such as CIFAR. Table 8aale@@peee presents the comparison results on ConvNet under varying IPC values. Our experimental findings indicate that our method consistently outperforms both SRe2L. and RDED across most scenarios. Remarkably, approach achieves nearly lossless performance, comparable to that of DATM, even at 20% compression ratio. These results, combined with the exceptional performance observed on larger datasets like ImageNet, suggest that our method is versatile and unified solution that excels in both low-resolution and high-resolution settings.
A.6 COMPARISON WITH OTHER DIFFUSION-BASE METHODS ON IMAGENET-IK
We compare our method with two other synthetic dataset generation methods, namely TDSDM (Yuan al. 2023 and D (Su et al. 2024). which leverage pre-trained diffusion models. Although TDSDM was not initially designed for dataset distillation tasks, its goal is to enhance the training efficacy of synthetic data generated by diffusion models. Along with our baseline method, Minimax, these three approaches utilize Distribution-matciing. matching-like objectives to fine-tune diffusion models and improve the performance of synthetic data in training. The results presented in Table are evaluated on ImageNet- K under the evaluation protocol of RDED. Our findings demonstrate that our guided-diffusion method consistently outperforms the others, reinforcing the importance of introducing informative guidance when applying diffusion models in dataset distillation.
Table 9: Performance comparison over ResNet-18 18 with state-of-the-an diffusion-fiiettning methods
Dataset
IPC
TDSDM
D'M
Minimax
DiT-IGD
Minimax-IGD
10
44.5+0.4
27.9+0.7 55.2+0.3
44.3+0.5 58.640.3
45.5+0.5 59.8+0.3
46.2+0.6 60.3+0.4
ImageNet-IK
50
59.4+0.3
16

---

Published as a conference paper at ICLR 2025
A.7 EVALUATING THE ROBUSTNESS OF INFLUENCE- FILUENCE-GUIDED DIFFUSION WITH DPM SOLVER
In the practical implementation of our IGD methods, we propose using DDIM with 50 denoising steps. To assess the generalizability of our proposed Influence Guidance and Deviation Guidance across different diffusion solvers, we additionally explore the use of the DPM solver Lu et al. (2022 as a replacement for the DDIM solver in outputting 8Zz; L, EΦ) in Equation ( The DPM solver is a fast, high-order solver for diffusion ODEs with convergence order guarantee. We employ the second-order DPM solver with 20 denoising steps by default. Accordingly, we adjust the guided range for diffusion guidance to [12, 18]. This adjustment results in significant 50% reduction in average sampling time, from 8.2 seconds to 4.3 seconds on an RTX 4090. Table IO compares the average performance of DDIM with 50 steps and the DPM solver with 20 steps for distilling ImageNette and ImageWoof with IPC=50. The results indicate that using the DPM solver with fewer denoising steps does not lead significant degradation in performance, and in some scenarios, even yields slight improvements. This further validates the robustness of our influence-guided sampling method.
Table 10: Performance of ResNet-18 using the DDIM-50 and DPM-20 solvers for diffusion genera- tion.
Solver
DiT
DiT-IGD
Minimax
Minimax-IGD
DDIM-50
75.2+0.9
81.020.7
78.1
82.0#0.3
ImageNette
DPM-20
73.5+0.8
82.0+0.5
76.6#0.4
80.6+0.5
DDIM-50 DPM-20
57.4+0.7 57.8+0.6
62.0#1.1
60.5+0.5 60.3+0.6
65.4+1.8
ImageWoot
64.0+1.0
64.5+1.3
A.8 DISTRIBUTION DIVERSITY AND COVERAGE ANALYSIS
In Figure 3 we provided t-SNE visualizations comparing the distributions of data generated by two baseline methods (DiT and Minimax) and our IGD-based methods with IPC=100. The figure shows that integrating IGD enhances diversity and aligmment with the original dataset, supported by lower Wasserstein distances to the original dataset. In this section, we further compare the FID scores and coverage of surrogate datasets (IPC=100) generated for Image Woof by different methods. Here, coverage metric was assessed based on whether each original data point had a nearest neighbor the surrogate dataset within given threshold (e.g.. 300 in the Inception V3 latent space). For fairness, excluded data selected by the Random method from the original dataset during coverage calculation. From the results shown in Table II also demonstrate clear diversity improment achieved by our IGD method to vanilla DiT. However we also observed that although the randomly selected dataset has the lowest FII and highest coverage, its performance was the worst. Similarly, while Minimax-IGD has worse FID and coverage than DiT-IGD, it performed better. These findings suggest that our diversity anftuence-guide objective is more effective measure for DD than relying solely on distribution alignment.
Table I1: FID and distribution coverage comparison among different methods.
Metrics
DiT
DiT-IGD
Minimax
Minimax-IGD
Random
FID
81.1
75.9
80.1
76.4
54.1
Coverage (%)
65.4
68.1
66.5
67.2
72.1
Accuracy (%)
62.3
70.6
67.4
72.1
63.6
A.9 PARAMETER ANALYSIS
In our IGD sampling framework, two critical apper-parameters are k. which controls the magnitude of influence guidance, and yh, which controls the magnitude of deviation guidance. In Figure 6 we examine the impact of these scaling factors on DiT-IGD and Minimax-IGD using the ImageNette dataset an instance. For DiT-IGI, variations in both & and 7. significantly influence performance.
17

---

Published as a conference paper at ICLR 2025
Increasing the values of these parameters enhances performance, highlighting the importance of influence and dataset diversity for model training. However, setting too high results in a notable performance drop. As discussed in Section 3.2 this is likely due excessive overfitting to the surrogate data with distorted content. In contrast, for Minimax-IGID, increasing "/e contributes marginally to performance improvement. This is because Minimax-IGID inherently focuses on increasing diversity as a core aspect of its fine-tuning- guidance by enlarging k significantly improves its results. Despite this improvement, similar performance drop is observed when & becomes excessively large. These findings underscore the necessity of carefully tuning and Y/ optimize the effectiveness of our IGD sampling framework, ensuring balanced influence and diversity without overfitting.
scheme. However, increasing influence
DIT-IGD
DiI-GD
Volldet on Accuracy (%)
IPC10
IPC50
IPC100
Valide Accuracy (5)
IPC1
IPC30
IPC100
of influence Guidance
of Daviation Guidance
(a)
(b)
Minimax-IGD
Minimax-IGn
IPC10
IPCSO
1PC100
PC10
IPC100
Vo lottion Accuracy 196
Ve Saattttoon Accuracy
of influence Guidance
Deviation Guidance
(c)
(d)
Figure 6: Hyper-paramen parameter analysis on (a) & (c) the scaling factor & of influence guidance, and (b) & (d) the scaling factor "/ of deviation guidance for DiT-IGD and Minimax-IGD.
A. HYPERPARAMETER SETUP AND GUIDELINES
In Table 12 provide detailed hyperparameter configuration for k and 7t in Equation replicate the results obtained across ImageNette,, Image Woof, and ImageNet- datasets. Despite incorporating an adaptive scaling factor based on the ratio between the denoised signal magnitude from diffusion and the guided signal from the influence guidance G,, manual specification of the scale factor & remains essential to forestall unexpected overfitting resulting from the influence guidance. Drawing from insights gleaned from our ablation study, as illustrated in Figure 6 we recommend setting the value range within [1, 50) for scaling our method in distillation tasks involving other ImageNet subsets. Similarly, we suggest grid-search range for the scaling factor t of the deviation guidance as [10, 200). Particularly for scenarios with small IPC, we advocate for starting from relatively smaller value of k to hold the representatives of generated data.
A.11 MORE VISUALIZATION COMPARISON OF SYNTHETIC DATA.
Here, we provide an additional visual comparison between images generated by two backbone models with vanilla DDIM sampling: the raw DiT and the Minimax-tuned DiT, and with our (GD-samplin framework: DiT-IGD and Minimax IGD. All synthetic data were generated for the Image Woof and ImageNette datasets.
18

---

Published as a conference paper at ICLR 2025
Table 12: Detailed setup of hyperparameters &t and Y/ Equation for reproducing the results reported in Table &
DIFIGD
Minimux: IGD
Parameter
IPC10
IPC50
IPC100
IPC10
IPC50
IPC100
k
5
5
15
15
15
Nette
7e
50
120
120
10
10
5
5
5
10
10
10
woof
76
50
120
120
100
100
5
IK
120
120
120
100
100
100
DiT
DiT-IGD
Rhodesian
Ridgeback
Border
Terrier
Golden
Retriever
Shih Tzu
English Foxhound
Australian Terrier
Beagle
Dingo
Samoyed
Old English
Sheepdog
Figure 7: Visualizaiton comparison between raw DiT and DIT-IGD on Image Woof.
19

---

Published as a conference paper at ICLR 2025
Minimax
Minimax-IGD
Rhodesian Ridgeback
Border Terrier
Golden Retriever
Shih Tzu
English Foxhound
Australian Terrier
Beagle
Dingo
Samoyed
Old English Sheepdog
Figure 8: Visualizaiton comparison between Minimax and Minimax-IGD on Image Woof.
20

---

Published as a conference paper at ICLR 2025
DiT
DiT-IGD
Parachute
Gas Pump
Tench
Cassette
Player
French
Horn
English Springer
Chain Saw
Church
Golf Ball
Garbage Truck
Figure 9: Visualizaiton comparison between raw DiT and DiT-IGD on ImageNette.
21

---

Published as a conference paper at ICLR 2025
Minimax
Minimax-IGD
Parachute
Gas Pump
Tench
Cassette
Player
French Horn
English
Springer
Chain Saw
Turos
Church
Ball
Garbage
Truck
Figure 10: Visualizaiton comparison between Minimax and Minimax-ION ImageNette.
22


Reviews:
[{"review_id": "iWp8Wkzfob", "reviewer": "Reviewer_rhZM", "summary": "This paper addresses the challenges of dataset distillation, which aims to create compact yet effective datasets for training larger original datasets. Existing methods often face limitations when dealing with large, high-resolution datasets due to high resource costs and suboptimal performance, largely due to sample-wise optimizations in the pixel space. To overcome these challenges, the authors propose framing dataset distillation as a controlled diffusion generation task, leveraging the capabilities of diffusion generative models to learn target dataset distributions and generate high-quality data tailored for training.\n\nThe authors introduce the Influence-Guided Diffusion (IGD) sampling framework, which generates training-effective data without retraining the diffusion models. This is achieved by establishing a connection between the goal of dataset distillation and the trajectory influence function, using this function as an indicator to guide the diffusion process toward promoting data influence and enhancing diversity. The proposed IGD method is shown to significantly improve the training performance of distilled datasets and achieves state-of-the-art results in distilling ImageNet datasets. Notably, the method reaches an impressive performance of 60.3% on ImageNet-1K with IPC (Images Per Class) set to 50.", "questions": "As shown in Table 5, the main contributions of the authors include the proposed influence guidance and deviation guidance. What is the relationship between these contributions and the \"train-free\" concept? Notably, even when these components are excluded from Equation 9, good results are still achieved.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to read and understand.\n2. IGD appears to be superior to existing diffusion model-based approaches.", "weaknesses": "1. In the introduction, the authors introduce the concept of Influence-Guided without clearly explaining what \"Influence\" entails or why it is used for guidance. The motivation is not well established. While Figure 1 effectively shows performance, adding an additional subfigure to illustrate the motivation or highlight differences from previous methods might be more valuable.\n\n2. The primary contribution of the authors is the proposal of a train-free diffusion framework for Dataset Distillation. While train-free approaches are common in the AIGC field, how does the proposed method differ from existing ones?\n\n3. The experiments only report results on ImageNet, without including results on classic datasets such as CIFAR-10 and CIFAR-100.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "amWvJgooJW", "reviewer": "Reviewer_Xr47", "summary": "The work proposes a guidance scheme for dataset distillation with two main contributions. The first is to do gradient matching between sampled data with the training data, and the second is to add diversity constraints among samples inside a class.The experimental results show clear improvement over other baselines", "questions": "1. The performance is still very far from the original datasets. What is the least IPC to achieve similar performance with full data?\n2. Which experiments show an improvement in diversity? The diversity should be measured in terms of FID/Recall values.\n3. The equation (7) uses cosine similarity instead of product; is it purely due to experimental results or based on some other hypothesis?\n4. How will the work be performed on different tasks apart from classification?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is reasonable\n2. The paper is well written\n3. The performance is significant", "weaknesses": "1. The performance is still far from full dataset.\n2. Lack of diversity measurement experiments\n3. The design of equation (7) lacks clarification.\n4. The application of the work seems to be not flexible. From my understanding, according to one architecture, there will be a need for one-time distillation. Is it possible to have one time distillation and use that distilled datasets to validate across models? I can see Table 4 for the robustness between models, yet the performance is not the same for the used models for guidance. This results in the concern in the application in reality due to computational exhibitions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAEfCOaRF3", "reviewer": "Reviewer_brCT", "summary": "This paper proposes Influence-Guided Diffusion (IGD) for dataset distillation. IGD solves the problem of poor performance and high resource costs of existing methods at high resolution. IGD proposes a training-free sampling framework which can be used in pretrained diffusion models to generate training-effective data. Extensive experiments show IGD achieves state-of-the-art performance in distilling ImageNet datasets.", "questions": "Please see weeknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. IGD is a training-free framework that can be easily used in any pretrained diffusion models.\n2. The target this paper hopes to solve is clear, and the proposed methods solve the problem of data influence and diversity constraint theoretically.\n3. The performance improvement of IGD used in DiT and Minimax finetuned DiT is obvious.\n4. The ablation study is adequate, including all proposed methods and hyperparameters.", "weaknesses": "1. In Table.1, the compared methods are missing like latest method RDED [1] mentioned in section 4.1.\n2. Although the proposed method IGD is training-free for diffusion models. It requires training a model to collect the surrogate checkpoints used in Eq. 7. The time consumption should be listed as the paper emphasizes efficiency.\n3. The model used in Eq.7 is ConvNet-6. If we change the model like for a bigger one Swin Transformer, will the performance better? Or this model choice is relatively insensitive?\n4. Is IGD can be used in other efficient diffusion sampling strategy like DPM [2] solvers?\n5. The generation time should be compared between IGD and other methods like current SOTA RDED [1].\n\nReference:\n\n[1]. Sun P, Shi B, Yu D, et al. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9390-9399.\n\n[2]. Lu C, Zhou Y, Bao F, et al. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps[J]. Advances in Neural Information Processing Systems, 2022, 35: 5775-5787.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPTZAWsVof", "reviewer": "Reviewer_hMXA", "summary": "This paper works on dataset distillation by generating the distilled dataset using diffusion models guided by an influence function. In the implementation, two guidance terms are used. One is to increase the similarity between the gradient using the generated sample and the average gradient using the original training samples. The other is to decrease the similarity between generated samples. Experiments are conducted on ImageNette and ImageWoof.", "questions": "Please refer to W1 and W2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using guided diffusion to generate samples for distillation is an interesting application of diffusion models.\n\n2. The proposed two guidance terms, i.e., increasing the gradient similarity and sample diversity, are well-motivated, simple, and intuitive. \n\n3. The paper is well written and presented in general.\n\n4. In the experiments, the proposed method achieves better performance and shows effectiveness. Comprehensive ablation studies and analyses are provided.", "weaknesses": "1. There are a lot of writing issues in the math part:\n- It is unclear how the derivation is transferred from stepwise (Eq4) to epochwise (Eq5).\n- C duplicated defined on L096 and L110.\n- In Sec 2.2, some z is bold, and some are not.\n- In L132, D is not clearly defined.\n- In Eq5, theta_e and theta_E are not clearly defined.\n\n2. The proposed method seems to have a high computational cost. Both computing and storage costs are high for the gradient calculation in L294. The similarity calculation with respect to all generated samples is also high in Eq8.  The computing and storage costs should be clearly provided and analysed in all the experiment sections.\n\n3. I doubt the statement this method is training-free. I agree that it is training-free as commonly understood in the diffusion community. But there are still a lot of training efforts here. It is only training-free given all the checkpoints, stored gradients, and pre-trained diffusion models. I would suggest revising this statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cG4ICRbWb", "reviewer": "Reviewer_Qttw", "summary": "This paper proposed a guided diffusion generation method for dataset distillation problem. The trajectory influence and deviation guidance are introduced to the vanilla diffusion process for generating synthetic samples as efficient training data. The results on ImageNet and its subsets demonstrates the improvements over the baselines.", "questions": "Please address the above weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n\n1.\tThe paper is easy to follow. The motivation of importance guided synthesis is clear and method is well presented. \n\n2.\tThe new idea is reasonable and neat that bridge the diffusion-based generative models and importance-based sample selection.\n\n3.\tThe results are promising. The performance improvements on challenging datasets are remarkable. Extensive ablation studies, cross-architecture validation and visualization are implemented.", "weaknesses": "Weaknesses:\n\n1.\tThere are several hyper-parameters in the algorithm, such as influence factor, deviation factor, scale, guided range etc. The sensitiveness of performance on these hyper-parameters should be studied. How do authors search the hyper-parameters?\n\n2.\tI have a concern whether the new method cause the collapse of the distribution of the generated data, though with the deviation guidance. \n\n3.\tSince DiT and VAE pre-trained on huge dataset are utilized for generating training samples on small datasets. It naturally brings advantages over traditional dataset distillation methods. Hence, more recent methods that also use pre-trained diffusion models should be compared with. \n\n4.\tThere are also some similar works that improve the efficiency of diffusion-model generated training samples, such as [1], which should also be discussed in the paper.\n\n[1] Real-Fake: Effective Training Data Synthesis Through Distribution Matching, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)