Unlocking Dataset Distillation with Diffusion Models
Authors: Brian Bernhard Moser
Federico Raue
Sebastian Palacio
Stanislav Frolov
Andreas Dengel
Venue: NeurIPS 2025 spotlight ()
Keywords: Dataset Distillation
Diffusion Models
Latent Diffusion Models
Vanishing Gradients
Generative Priors
Primary area: optimization
TL;DR: This paper proposes LD3M, a novel framework for latent dataset distillation with generative priors that improves the gradient flow of diffusion models during the distillation process.

Abstract: Dataset distillation seeks to condense datasets into smaller but highly representative synthetic samples.  While diffusion models now lead all generative benchmarks, current distillation methods avoid them and rely instead on GANs or autoencoders, or, at best, sampling from a fixed diffusion prior. This trend arises because naive backpropagation through the long denoising chain leads to vanishing gradients, which prevents effective synthetic sample optimization. To address this limitation, we introduce Latent Dataset Distillation with Diffusion Models (LD3M), the first method to learn gradient-based distilled latents and class embeddings end-to-end through a pre-trained latent diffusion model. A linearly decaying skip connection, injected from the initial noisy state into every reverse step, preserves the gradient signal across dozens of timesteps without requiring diffusion weight fine-tuning. Across multiple ImageNet subsets at $128\times128$ and $256\times256$, LD3M improves downstream accuracy by up to 4.8 percentage points (1 IPC) and 4.2 points (10 IPC) over the prior state-of-the-art. The code for LD3M is provided at https://github.com/Brian-Moser/prune_and_distill.

Unlocking Dataset Distillation with Diffusion Models
Brian B. Moser 1,244
Federico Raue 2.4
Sebastian Palacio
Stanislav Froloy! 1.2
Andreas Dengel* 1.2
University of Kaiserslauter--Landee Kaiserslautern
2 German Research Center for Artificial Intelligence (DFKI), Kaiserslautern
ABB AG, Mannheim
Equal Contribution
First.second@dfff
Abstract
Dataset distillation seeks to condense datasets into smaller but highly representative synthetic samples. While diffusion models now lead all generative benchmarks, current distillation methods avoid them and rely instead on GANs autoencoders, or, at best, sampling from a fixed diffusion prior. This trend arises because naive backpropagation through the long denoising chain leads vanishing gradients, which prevents effective synthetic sample optimization. To address this limitation, we introduce Latent Dataset Distillation with Diffusion Models (LD3M), the first method to Iearn gradient-bart based distilled latents and class embeddings end- to-end through pre-trained latent diffusion model. linearly decaying skip connection, injected from the initial noisy state into every reverse step, preserves the gradient signal across dozens of timesteps without requiring diffusion weight fine-tuning, Across multiple ImageNet subsets 128 x 128 and 256 x 256, LD3M improves downstream accuracy by up to 4.8 percentage points (1 IPC) and 4.2 points (10 IPC) over the prior state-of-the-ar The code for LD3M is provided at attps://githrr www./oooiiiieeeeee Briinn Moser Morerrrrrunrrreeeeeeee.. http:/runeeeeeeeeeee dindiii
1 Introduction
Large-scale datasets fuel the advancements in modern computer vision but demand substantial computational resources and raise scalability concerns [27 14]. Dataset distillation emerges as compelling solution, aiming to synthesize small set of information-rin samples that preserve While early methods operated directly pixel space, promising recent direction involves leveraging powerful generative models as priors [ I By optimizing compact latent codes instead of raw pixels, these approaches, exemplified by GLaD using StyleGAN-XI XL [ can generate higher-resolution synthetic images (128 x 128 and beyond) that generalize better across diverse network architectures.
the essence of the original dataset [35
However, GAN-based priors like in GLaD suffer from complex multi-space latent optimization and require cumbersome inversion processes for initialization 5
21 Diffusion models [18] 28). having surpassed GANs as the state-of-the-art image generators 9 represent natural next step. Yet, inherent vanishing gradients severely hinder their direct application dataset distillation. Optimizing latents through the long denoising chain leads to exponentially decaying gradients, which prevents effective learning of the synthetic data
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mathematically, if Z is the initial latent code to be optimized and Zu the final denoised state after T steps, the gradient 0c/0z a product of Jacobians:
0C
ac
dzz
dzr 0z
II
(1)
0z
Izo
Oz.
If the norm of each Jacobian Oz.-1 1/ is bounded by À < 1, the product term I (dzz 102,)
diminishes towards zero as T increases. This gradient decay is empirically observable and significant: Our analysis confirms that in standard diffusion models, gradient norms for Z decrease nearly tenfold as I increases from 10 to 90 (see Table 1p.
Table Gradient norms for the initial latent code Z across different maximum diffusion steps T for fixed inputs. The decrease in norm as 7" increases empirically demonstrates the vanishing gradient
problem, which severely hinders the optimization of Z through the standard reverse diffusion process.
T
10
20
30
40
50
60
70
80
90
C/02 X 10
58.1
33.5
19.8
15.4
13.9
12.1
10.4
8.7
6,5
This critical bottleneck has forced prior diffusion-based distillation attempts to circumvent end-to-end optimization entirely, resorting instead to sampling selecting fixed representations from pre-trained models 12 34 16. These approaches, while computationally faster, forfeit the fine-grained gradient- matching optimization erucial for potential benefits like enhanced privacy of distilled data their
robustness against adversarial attacks 7 37 44
10].
To unlock diffusion models for true dataset distillation, we introduce Latent Dataset Distillation with Diffusion Models (LD3M). Our core contribution is a tailored modification to the reverse diffusion process (Equation 7) that introduces linearly decaying residual connections, specifically designed to enhance gradient flow for optimizing latent representations Z and conditioning codes c the context of dataset distillation. This mechanism enables, for the first time, effective end-to-end optimization of distilled latent codes Z and class embeddings с through a pre-trained latent diffusion model. LD3M is readily compatible with existing distillation objectives and diffusion model architectures. Experiments across numerous ImageNet subsets demonstrate that LD3M significantly outperforms the state-of-the-art at 128 128 and 256 x 256 resolutions, achieving superior cross-architecture generalization, i.e., 4.8 percentage points (1 IPC) and 4.2 points (10 IPC), and faster distillation times.
2 Preliminaries
Dataset Distillation: Let (Xy, Yr), where X, R-NxHXxxC be real image classification dataset and N its cardinality. The goal is to compress T into a small synthetic set S = (X., Y.), where X, E RM xC where M is the total number of synthetic samples with M C IPC, C the number of classes and IPC the Images Per Class (IPC). We aim to achieve M < N with
S = arg min C(S,T)
(2)
where L is a distillation loss between the distilled set S and the real dataset T. Common choices for C include matching gradients (Dataset Condensation, DC [42)). feature distributions (Distribution Matching, DM [41]), or model parameter trajectories (Matching Training Trajectories, MTT 4)). We refer to the supplementary material for detailed definitions.
Dataset Distillation with Generative Priors:
To improve the quality, resolution, and generalization
of distilled images, recent work incorporates deep generative models as priors [5] Instead of
optimizing pixels directly, they optimize latent codes 2 IR. XMxxxxx with h fed into pre-trained generator D. The optimization objective becomes
2 arg min L(D(Z),
(3)
3 Related Work
GAN Priors (GLaD): GLaD (5) pioneered distillation with generative priors using a pre-trained
StyleGAN-XL [30]. While successful, it inherits GAN-specific drawbacks: (1) Optimizing the

---

condilioning code
denoiser
any distilltion algorithm
latent code
syntheti codes
residuals enhanced gradient flow
optimizes
Figure 1: The LD33 Framework. Learnable latent codes 2 and conditioning codes с are optimized. Z is noised to initialize the reverse diffusion at ZT. A pre-trained LDM denoiser iteratively refines the state (Z; 1r- Key innovation: Residual connections (red arrows) inject zr with linearly decaying weight into each step (Equation 7), enhancing gradient flow. The final latent zo is decoded (D) into images S, which are optimized using a standard distillation algorithm.
complex, multi-level latent space (W+) required for high quality is computationally demanding 443 (2) Initializing latent codes Z from real images x requires solving costly GAN inversion problem
(ming [(x, ))))) 3 36.. hindering standard initialization practices 21
Prior Diffusion-based Distillation Attempts: The inherent vanishing gradient problem [ signifi- cantly challenges using diffusion models for end-to-end distillation. Faced with this gradient barrier, existing diffusion methods for distillation have adopted non-optimization strategies:
Sampling/Selection Methods: Minimax Diffusion [6 and D4M 34 use criteria to select or sample representative latents from diffusion model, avoiding backpropagation to latents entirely. While faster, this hypasses gradient -based optimization that needed for privacy
robustness, also crucial motivations for dataset distillation 7 33 44 [0].
Autoencoder-Only Methods: Duan et al. [12) leverage the pre-trained autoencoder from LDM but do not utilize the diffusion process itself. They optimize latent codes Z that are directly decoded by D, essentially using only the autoencoder, not its core denoising mechanism. This simplifies optimization but fails to exploit the diffusion prior.
In contrast, LD3M is designed to directly optimize latent codes Z by enabling gradient flow through the reverse process.
Residual Connections for Gradient Flow: Enhancing gradient flow in deep networks is commonly addressed using residual connections, famously introduced in ResNets 17 or LSTMs [19. Similar ideas have appeared within diffusion models; for instance, SAGE [2 used residuals connecting to the initial noise to enable adversarial latent search. While conceptually related, our contribution differs significantly: we introduce structured, linearly decaying residuals injected at every step from the initial noisy latent 2r, explicitly designed for end-to-end optimization of distilled latent codes through the diffusion chain for the unique characteristics dataset distillation.
Decoupled Distillation Methods: Distinet from methods optimizing synthetic data via generative priors, another line of work decouples distillation from end-to-end training. Methods like SRe2L. 40 and others 39 32 leverage statistics (e.g., from BatchNorm) of pre-trained networks to recover informative images, offering scalability benefits but following fundamentally different optimization
strategy than mradient-matching approaches like LD3M.
4 Latent Dataset Distillation with Diffusion Models (LD3M)
Our approach enables end-to-end optimization of synthetic data directly through a pre-trained LDM. As shown in Figure LD3M optimizes both initial latent codes Z and conditioning codes c. These are processed by modified reverse diffusion process to generate expressive latent states 2o, which are then decoded into images, S D(zn). This section details the core components: our modified diffusion sampling process designed to boost gradient flow ($4.T). the efficient initialization strategy 4.22.. and the gradient checkpointing used for memory efficiency ($4.3).
3

---

4.1 Sampling Process
We leverage pre-trained LDM involves a reverse diffusion process pe that iteratively denoises a state starting from noise Zy, conditioned on an embedding с (typically derived from class labels) [38 [26.. Each step predicts less noisy state 74-1 based on the current state 2, and condition c. The standard update rule [18 29 calculates the subsequent state 2t-1 using the predicted mean (40 and variance of:
28
without tinettuning its weights. Standard LDM operation
pe (c, Zt. 70)
Zi
1 az
fo (c, Ze. (e)
(4)
Vat
VI
2e-1 + po(c, Zt,7t) 71) + of
(5)
where fo is the LDM's pre-trained noise prediction network (usually U-Net), ar and T relate the noise schedule, and E, ~ (0,1) I) is random noise added at step I.
Algorithm Latent Dataset Distillation with Diffusion Models (LD3M)
Input: randomly selected collection X pre-trained encoder ε, pre-trained decoder D, pre-trained denoiser μe with frozen parameters 0, noise levels dr.
Z 8 (X,)
2T (TT 2)
for 1 - T, T....,I do
Es (0.
2/-1 (11- - +) 2o(c,,4,7))+ 4 ur) + ofet
end for
X.yn <- D (zo) Return: Xsyn
For dataset distillation, our goal is to learn the optimal initial latent representations Z and conditioning codes that minimize the distillation loss L between the synthetic images and the target dataset:
z z².c² = arg min (D(po wr. c)), T), where Zt ~ q
(6)
2,c
Here, po (2/2/2r, 2r, c) denotes the final state Zg resulting from the 7-step reverse process starting from zr. which itself is noised version of the learnable 2 obtained via the forward process q. We want to highlight that the sampling of e introduces stochasticity during inference. We fix, however, for one sampling phase, the residual variable žr be constantly the same.
To effectively minimize distillation losses like gradient or trajectory matching (Equation 6). which rely on fine fineggraiedd alignment between synthetic and real data processing, requires guiding the generative process. While conditioning с provides class guidance, optimizing the initial latent Z offers the necessary degrees of freedom to shape the generated sample 2 (zo) precisely. Relying solely on fixed Z or only optimizing с proved insufficient empirically, yielding suboptimal results resembling simple LAION-5B 31 data retrieval similar to D4M
A4
Vanishing gradients inherent in backpropagating through the T steps Eqqution 5 present the primary obstacle to optimizing Equation 6 [19]. To overcome this, we introduce simple yet effective modification to the reverse step, injecting a residual connection from the initial state Zr:
2 ((1 ) (c, Zz, 71) + ZT +ofEL
Modified Mean
Crucially, while this modification deviates from standard sampling aimed at matching the distribution of real images, its purpose here is to enable gradient flow for distillation optimization, a task where downstream utility, not photorealism [2 is paramount (see supplementary material for detailed discussion). In other words, our method breaks the sampler's fidelity to the original data distribution in order to better achieve the distillation objective matching feature distributions.
In conclusion, the modification replaces the standard predicted mean μá with weighted average of pn and the initial state Zr. The weight of 2y decreases linearly from (at 1 = T)) to nearly 0 (as t + 0). This creates a direct pathway for gradients from the loss L (computed using 2o) back to zr.

---

and thus to the learnable 2, bypassing the long chain of Jacobian products that causes gradients to vanish. The enhanced gradient flow Z can conceptually represented as:
ac 0z
T T
Oc
Oz:
dzo 0z
JC
dz Ozr
Σ t=1
(+)
(8)
iz--1
dzt 02
Original (Docaying) Path
Entanced Path via Skip Connection
comprehensive description can be found in Algorithm
Notes on Markovian Property: 24-1 depends on 24 and 27. but not on any earlier states such 7, 24+2, and on. Therefore, the probability distribution for 24-1 only depends 2 and the fixed initial state Zr, which constant throughout the diffusion process. Thus, we have: p(2,-1 2,, 2441o.... 27) p(2,-1 22, Zr). This confirms that LD3M remains Markovian.
Notes on Generalisability: Our gradient enhancement technique (Equation 7) applies to various diffusion model architectures beyond LDMs, suggesting broader potential for future work. In this study, we focused on LDMs primarily as a proof of concept, leveraging readily available pre-trained models and LDM's foundational role in spact diffusion.
4.2 Efficient Latent Code Initialization
Standard practice in dataset distillation initializes synthetic data using real images from the target classes [2 GAN-based methods like GLaD face a significant challenge here, requiring complex and costly GAN inversion techniques find latent codes that reconstruct target real images x [36].
LD3M benefits immensely from the autoencoder structure inherent in LDMs. We initialize the learnable latent codes Zmi simply by encoding small, randomly selected set of real images X, using the pre-trained image encoder. Similarly, the initial conditioning codes Caut are obtained using the pra-trained class embedding network.
4.3 Memory Efficiency via Gradient Checkpointing
Optimizing through the steps of the reverse diffusion process, even with our modification, can be memory-intensive. Following GLaD [5] we employ gradient checkpointing 6 to manage VRAM usage. The procedure involves:
1. Perform the forward pass = (po kr. c)) without storing intermediate activations.
2. Calculate the distillation loss L(S, T and the gradient with respect to the output, 0L/aS.
3. To compute the gradient OC/0Z (and OL/Oc), recompute the necessary segments of the forward pass through the diffusion process and decoder D, storing only the activations needed for the immediate backward pass segment.
This avoids storing the full computation graph, which GLaD also exploits for a single generator pass.
Experiments
We evaluate LD3M against relevant baselines, primarily the state-of-the-art latent generative prior method GLaD (5), following its experimental setup for fair comparison. We conduct extensive experiments on 10 diverse 10-class subsets of ImageNet-1k k [8] at 128 x 128 (IPC=1, IPC=10) and 256 256 (IPC=1) resolutions, well CIFAR-IO. Key implementation details in $5.1 full hyperparameters and setup details are in the supplementary material.
5.1 Setup Details
Datasets & Evaluation: We use ImageNet subsets (ImNet-A to E, ImNette, ImWof Birds, Fruits,
Cats) from [5] 20, 4 and CIFAR-10. Following standard protocol, we distill datasets using DC 42
DM 41, or MTT 4 and evaluate by training unseen architectures (AlexNet (22), VGG-11 33
ResNet-18 [17, VIT 11 from scratch on the distilled set, reporting mean test accuracy (+ std. dev.)
over 5 runs.
5

---

Table 2: CIFAR-10 comparisons with GLaD on IPC=1 and SRe2L/D4M on IPC=50.
IPC=1
(b) IPC=50
Dist. Method
AlexNet
ResNet18
VGGI
ViT
Average
Method
ConvNet
pixel space
23.9+ua
27.3-us
28.0+as
22.9=ua
26.0+u4
SRe2L 40
60.2
GLaD (rand G)
30.1
27.3=0a
28.0209
21.2200
26. 26.6105
D4M 134
72.8
DC
GLaD (trained
26.0
27.6
28.2+ao
23.4+uu
26.3+uu
L.33M (Ours)
73.2
LD3M (trained G)
27.2+us
26.6=u9
+1.5+aa
29.0=ua
28.6+uu
Reported (1)) IPC=IK:
Decoupled method.
pixel space
22.9+uu
22.2-u7
23.8+a5
21.3+us
22.6+us
Reported [34] Sampling-
GLaD (rand G)
23.7:203
21.7=10
24.3198
21.4200
22.8106
based Diffusion.
DM
GLaD (trained
25.1:005
22.5=07
24.8
23.0=a:
23.8
LD3M (trained G)
27.2
23.0
25.4+
23.8=ua
24.9
Table 3: Crost-architecture performance (%) with IPC on ImageNet subsets (128 x 128). LD3M (bold) generally achieves the best results within each algorithm and overall best (blue) compared to Pixel Space and GL For instance, LD3M improves DC, MTT, and DM by +3.76%, +5.68% and + +16.34%, respectively.
Divill Space
Alp
ImNet-A
ImNe--B
ImNee-C
ImNe--D
ImiNes-B
ImNette
InWoof
ImNes-Binss
ImNet-Frits
ImNet-Caax
25.4
33
34.0:34
31,4334
20.7::277
249:13
241111
160:12
25.1:3
183:23
187115
pisel space
DC
27.9
38.7-1.0
33.54199
26.411.1
27.4209
282111
174411
28.5114
204
193109
DM
19.2-100
232-122
24.--111
23.0-14
1844007
17.+-09
206+0.3
14.5+09
178+08
14.5-11
140+1.
MIT 29.0:12
1..
3..4.1.3
34.9111
30.411.
29001111
304115
17.11111
2821111
21:1112
196212
GLaD
DC
208-1.1
41.8-1.7
42.1
35.8+14
20.0008
20.3-11
31.0116
178+11
201+10
223-16
21
DM
22:41144
31.6.11 1.4
31.3 3.9
26.91112
21.5.1100
20.41..8
21.91111
152.09
182110
20.111
16.1100
MIT
1.3
40.9
1.1
41.6.1.7
34.1
31.5
1:2
30.1.1.
32.011.3
19.9 12
36.4
214-111
22.1110
LD3M
DC
36.9
42.3
42.0
3911
29.7 1.3
31.4
1:1
32.9 112
189106
362114
226.13
21.7108
DM
2.3
35.8:11
34.1=1.0
30.3:1.2
24.7::1.0
24.5:009
26.6::177
181:007
2300::
24.5
17.0:111
Table 4: Cross-architecture performance (%) with 10 IPC on ImageNet A-E (128 128). LD3M (bold, blue) consistently outperforms Pixel Space and GLaD with, for instance, an improvement of +2.52% and +3.46% with DC and DM, respectively.
Distil. Space
Alg.
All
ImNet-A
ImNet-B
ImNet-C
ImNet-D
ImNet-E
DC
42.
5.7
4..1
40.1
36.1
18.14
pixel space
DM
44.4
52.6-10.4
50.6-00.5
47.5
35.4
36.0
DC
45.9
53.1+11.
50.1
48.9+11.1
38.9
38.4
GLaD
DM
45.8
1.0
51.3
49.7-10.4
36.4:10.4
38.6
DC
47.1
55.2
1.0
51.8 1.4
49.9 F13
39.5 110
39.0+11.3
LD3M
DM
E..1
57.0
1.3
52.3
1111
48.2
1..
39.4 E.8
Models: We use ConvNet-5/CConNNet6 15] for distillation. As in GLaD (5), we use AlexNet [22]
VGG-11 (3 ResNet-18 [17] and Vision Transformer (TT) for evaluating the distilled dataset quality for unseen architectures. For LD3M, we use the public ImageNet pre-trained LDM (28
with its 2x compression autoencoder, without fine-tuning. Default diffusion steps T 10 (1282)
or T 20 (256") are used unless specified. For GLaD comparison, the ImageNet pre-trained StyleGAN-XL 30 used.
5.2 Results
Main Results: LD3M Outperforms State-of-the-Art. We first establish LD3M's effectiveness
against the primary baseline GLaD and other tfthearft methods on CIFAR- 10 Table 2). At IPC=1, LD3M matches exceeds GLaD using DC/DM distillation. The sampling-bes based D4M (34] delivered z =10% accuracy for all tested models. At IPC=50, LD3M surpasses reported results from D4M and decoupled 2R 40 with MTT (though optimization and IPC differ, caption). We subsequently focus our main analysis on GLaD [5] as the leading state-of-the. baseline for latent generative prior gradient-matching distillation.
Moving to the more challenging ImageNet subsets, LD3M consistently outperforms GLaD across resolutions and IPC values. For IPC=1 at 128 x 128 Table 3). LD3M achieves the best overall

---

Table 5: Cross crossttrchteeeeree performance (%) for 256 x 256 images (DC, IPC=1) using different generator initializations (ImageNet, FFHQ, Random). LD3M outperforms GLaD across initializations. Best results marked bold, top 3 blue. For example, LD33 improves by roughly +2 p.p. on average, whereas it improves the performance by roughly +7 p.p. compared to pixel space.
Distil. Space
All
ImNet-A
ImNet-B ImNet-C
ImNet-D
ImNet-E
pixel space
29.5+
38.3
32.8+4.11
27.6++3.3
2..
23.5++2.4
GLaD (ImageNet)
34.4
57.4
41.5+112
35.7++4.0
27.9+
1..
GLaD (Random)
34.5+11.6
39.3++2.0
40.3
35.0
17.9
1.4
GLaD (FFHQ)
34.0+t2.1
38.3+.5.2
40.2+t111
34.9+
27.2+0.9
29.4+2.1
LD3M (ImageNet)
36.3+t1.6
42.1
42.1
35.7=11.7
3..5
30.9
LD3M (Random)
3..5+116
42.0=2.0
41,9
37.1
30.5+1..
31.1
LD3M (FFHQ)
36.3
41.9
36.5
30.5 0.9
30.6
Table 6: Ablation: Impact of Diffusion (DC, IPC=1). Comparing LD3M (w/ diffusion) against GLal) and LD3M using only the autoencoder (w/o diffusion). Full LD3M consistently outperforms both. Method
All
ImNet-A ImNet-B
ImNet-C
ImNet-D
ImNet-E
GLaD
1.3
41.8
4..1+112
1..8
28.0
29.3
LD3M (w/o diffusion)
35.
40.6
41.9
35.
29.4
1..
LD3M (w/ diffusion)
1.3
4..
42.0
37.1
1..
31.4
+1.2+0.0
+1.7-0.6
+0.140.0
+1.8+0.8
+0.3-0.1
+1.9-0.2
average accuracy in 9/10 subsets, significantly boosting performance particularly for DM (+16.3% avg.). This advantage persists at IPC=10 (Table 4). where LD3M improves average accuracy over GLaD by +2.5% (DC) and +3.5% (DM), reaching over 47% overall. The performance gains extend to 256 x 256 resolution TTblle 5), where LD3M surpasses GLaD by +6% on average, even when comparing differently pre-traines generators (ImageNet, FFHQ, random), highlighting the robustness of leveraging the LDM prior via LD3M.
Ablation Studies: Why LD3M Works. The diffusion process itself is crucial for LD3M's improved performance over autoencoder-onf, approaches. Table 6 provides empirical evidence: removing the diffusion steps ("wlo diffusion", ", akin to Duan et al. (12)) results in performance comparable to GLaD (35.3% avg.). which is significantly lower than full LD3M utilizing the diffusion process (36.5% avg.). This +1.2 percentage points highlight that simply using the LDM's autoencoder is insufficient; successfully optimizing through the reverse process is necessary to unlock the performance benefits observed, validating our core approach over simpler AE-only methods. Naturally, we can expect similar limitations with few- one-step diffusion models in the context of dataset distillation.
Furthermore, both learnable latents and our proposed gradient enhancement drive LD3M's effective- ness. Table 7 dissects these contributions: optimizing only conditioning yields poor results (15.8% avg.), but making the latent learnable provides substantial boost (22.3% avg.). confirming the necessity of latent optimization motivated in $4.1 Critically, however, this alone does not surpass GLaaD; only by subsequently adding our enhanced gradient flow (Equation 7 does LD3M achieve its SOTA performance (28.1 % avg., vs. 26.6% for GLaD). This clearly demonstrates that while learnable latents offer vital degrees of freedom, they cannot be effectively optimized through standard diffusion backpropagation; the enhanced gradient flow enabled by our residual connection (Equation 7) is the key component that makes end-to-end optimization through diffusion truly effective for this task.
Robustness and Practical Considerations. LD3M exhibits robustness in various aspects. For instance, standard initialization using real images encoded via the LDM's encoder significantly benefits DC and DM performance compared to Gaussian noise initialization (Table 8). while being
vastly simpler than GLaD's GAN inversion ($4.) Visually (Figure 2 andd Figure 3). LD3M generates abstract but class-informative images that appear more consistent across different LDM pre-training datasets (ImageNet, FFHQ, Random) compared to GLaD. Our analysis of diffusion steps T FFiuure 4)
reveals an optimal performance/runtime trade-off around T 10- 40.
7

---

Table 7: Ablation: Different LDM variations (MTT on ImageNette and IPC=1). Tested was the distillation by making only the conditioning learnable (c), one with also learnable latent representation (2), and lastly, one which incorporates our modified reverse process (Equation 7). Method
All
AlexNet
VGG-I1
ResNet- 18
VIT
GLaa
26.6+11.4
28.7++0.3
1.2
30.8
17,8+114
LDM learnable conditioning (c)
1..
14.2:22.6
15.1 =1.6
16.51:49
16.8=14.0
learnable latent code (2)
22.3+-2.0
22.8++2.0
26.3++0.9
23.4
17.5++2.0
enhanced gradient flow (Eq. 7
28.1+-3.3 +33
19.2++1.9
29.2+112
30.6
25.1
Table 8: Ablation: Impact of Initialization (ImageNette, IPC=1, 5K iter.). Compares initializing Z and from Gaussian noise vs. encoded real images across different distillation algorithms. Dist. Method
Dist. Space
All
AlexNet ResNet18 VGG11
VIT
Gauss. noise 1.4
1..7:11.6
3..1=1.5
32.2:0.66 1..1 3.2
MTT
random image 323
30.1#:.44
35.6
0..
Gauss. noise 13.1-2.1
13.1=1155
1..6
13,8
13.7+2.7
DC
random image
32.9--2.1
31.6
0.6
37.7 1.2
13.4 Gauss. noise 1..4
12.4
13.4
14.4
DM
random image
26.8 1.7
31.
23.2+222
25.9
26.1+114
GLaD
LD3M
English Springer
Cassette Player
Chainsaw
French
Garbage Truck
Pump
Golf
Parachute
Horn
Visual comparison of LD3M versus GLaD (MTT, ImageNette, IK iter.). GLaD outputs tend toward smooth, photo-realistir textures but can blur class- details, whereas LD3M produces bolder, higher-contrast shapes that highlight key discriminative features (e.g., wing contours, beak outline). This abstraction trade-off suggests LD33M prioritizes core class signals over pixel-perfect fidelity, which empirically enhances downstream model generalization, contrasting claims made by sampling-bast methods like D4M 34
Figure 2:
LD3M also offers computational flexibility. Our analysis indicates an optimal balance between performance and distillation time occurs with around T = 10 40 diffusion steps (Figure 4). Using T 20, LD3M requires slightly less peak GPU memory on an A A100-40GB compared to GLaD (29.4GB 31.2GB) and completes the distillation process faster (574 min 693 min). This computational efficiency, coupled with the inherent flexibility to adjust the number of diffusion steps (T) based on available resources, enhances LD3M's practical appeal.
6 Conclusion
We addressed the critical challenge preventing end-to-end dataset distillation through powerful diffusion models: vanishing gradients across the long denoising chain. We introduced LD3M, which unlocks optimization through diffusion priors via a simple yet effective modification: injecting linearly decaying residual connections from the initial noisy state into each reverse step. This novel approach enhances gradient flow sufficiently to effectively learn both latent codes (Z) and conditioning (c) without altering pre-trained model weights. This direct optimization contrasts sharply with previous diffusion distillation methods that circumvented the gradient challenge by relying solely on sampling fixed representations [6).
8

---

Reference Image (from orig. data)
GLaD
LD3M
ImageNet
FFHQ
Random
Figure 3: Example 256 x 256 images of distilled class (ImageNet-B. Lorikeet) with differently initialized generators GLaD and LD3M. The various initializations, i.e., which dataset was used for training the generators, denoted at the bottom. We used DC as distillation algorithm.
38
140
GLaD
Cottllato S Time [s]
37
130
Accuracy [%]
LD3M
36
120
LD3M
35
110
GLaD
34
100
33
90
32
2
10
20
30
40
50
60
70
80
90
100
Max. Time Steps
Figure 4: Accuracy vs. Distillation Time Trade-off with Diffusion Steps T' (ImageNet A-E avg.. MTT, IPC=1). LD3M performance (blue line, mean + std) peaks around T 35. GLaD baseline (dashed lines) and optimal trade-off (X) shown for reference.
Our experiments (55) demonstrate LD3M's clear advantages. It significantly outperforms the state- of-the-art GAN-prior method, GLaD, on diverse ImageNet subsets 128 x 128 and 256 256 resolutions, improving cross-architecture generalization by up to 4.8 percentage points (IPC=1) and 4.2 points (IPC=10). Ablation studies confirmed that both leveraging the diffusion process and our specific gradient enhancement crucial for this success. Furthermore, LD3M offers practical benefits: vastly simpler initialization than GAN inversion ($4.2) and faster overall distillation times compared to GLaD
Future Work
By successfully enabling gradient- based optimization through diffusion models, LD3M paves the way for leveraging state-of-the-ar generative diffusion models creating highly effective and compact distilled datasets. Future work should explore alternative residual formulations and integration with fast samplers like DPM-Solver (2). Scaling and evaluation on larger benchmarks like ImageNet-1K also remain important next steps, contingent computational resources.

---

Acknowledgments
This work was supported by the EU project SustainML. (Grant 101070408) and by the BMBF Project
Albatross (Grant 01/W24002). All compute was done thanks to the Pegasus cluster at DFKI.
References
[1] O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
[2] S. Bengio, K. Dembezynski, T. Joachims, M. Kloft, and M. Varma. Extreme classification (dagstuhl seminar 18291). In Dagstuhl Reports. Schloss Dagstuhl-Leibnniz-Zenttuu fuer Infor- matik, 2019.
[3] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Neural photo editing with introspective adversarial networks. ICLR, 2017.
[4] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu. Dataset distillation by matching training trajectories. CVPR, 2022.
[5] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu, Generalizing dataset distillation via deep generative prior. In CVPR, pages 3739-3748, 2023.
[6] Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint, 2016.
[7] Y. Chen, W. Huang, and Weng. Provable and efficient dataset distillation for kernel ridge regression. NeurlPS, 37:88739-887777 88771, 2024.
[8] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[9] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
[10] T. Dong, B. Zhao, and L. Lyu. Privacy for free: How does dataset condensation help privacy? In ICML, pages 5378-5396. PMLR, 2022.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2020.
[12] Y. Duan, Zhang, and L. Zhang. Dataset distillation in latent space. arXiv preprint, 2023.
[13] S. Frolov, B. B. Moser, and A. Dengel. Spotdiffusion: A fast approach for seamless panorama generation time. arXiv preprint, 2024.
[14] D. Ganguli, Hernandez, Lovitt, A. Askell, Bai, A. Chen, Conerly, N. Dassarma, D. Drain, N. Elhage, et al. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022.
[15] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
[16] J. Gu, S. Vahidian, V. Kungurtsev, H. Wang, W. Jiang, Y. You, and Y. Chen. Efficient dataset distillation via minimax diffusion. arXiv preprint, 2023.
[17] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
[19] Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 1998.
10

---

[20] J. Howard. A smaller subset of 10 easily classified classes from imagenet, and a little more french. URL
2019.
[21] J.-H. Kim, J. Kim, S. J. Oh, S. Yun, H. Song, J. Jeong, J.-W. Ha, and O. Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference Machine Learning. PMLR, 2022.
[22] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 25, 2012.
[23] Q. Liu, A. Kortylewski, Y. Bai, S. Bai, and A. Yuille. Discovering failure modes of text-guided diffusion models via adversarial search. arXiv preprint, 2023.
[24] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurlPS, 2022.
[25] B. Moser, F. Raue, J. Hees, and A. Dengel. Less is more: Proxy datasets in nas approaches. In CVPR, 2022.
[26] B. Moser, A. Shanbhag, F. Raue, S. Frolov, S. Palacio, and A. Dengel. Diffusion models, image super-resolution and everything: A survey, arXiv preprint. 2024.
[27] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint, 2022.
[28] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[29] C. Saharia, Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE TPAMI, 2022.
[30] A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[31] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes,
A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 15:25278-25299 2022.
[32] S. Shao, Z. Yin, M. Zhou, X. Zhang, and Z. Shen. Generalized large-scale data condensation via various backbone and statistical matching. In CVPR, 2024.
[33] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2014.
[34] D. Su, J. Hou, W. Gao, Y. Tian, and B. Tang. D* Dataset distillation via disentangled diffusion model. In CVPR, pages 5809-5818, 2024.
[35] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint, 2018.
[36] W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou, and M.-H. Yang. Gan inversion: A survey, IEEE TPAMI, 2022.
[37] E. Xue, Li, H. Liu, P. Wang, Y. Shen, and Wang. Towards adversarially robust dataset distillation by curvature regularization. In AAAI, volume 39, pages 9041-9049, 2025.
[38] L. Yang, Z. Zhang, Y. Song. S. Hong. R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang. Diffusson models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[39] Z. Yin and Z. Shen. Dataset distillation in large data era. arXiv preprint, 2023.
[40] Z. Yin, E. Xing, and Z. Shen. Squeeze, recover and relabel: Dataset condensation imagenet scale from new perspective. NeurlPS, 2023.
[41] B. Zhao and H. Bilen. Dataset condensation with distribution matching. In WACV, 2023,

---

[42] B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. ICLR, 2020.
[43] X. Zhong, H. Fang, B. Chen, X. Gu, T. Dai, M. Qiu, and S.-T. Xia. Hierarchical features matter:
A deep exploration of gan priors for improved dataset distillation. arXiv preprint, 2024.
[44] Z. Zhou, W. Feng, S. Lyu, G. Cheng, X. Huang, and Q. Zhao. Beard: Benchmarking the adversarial robustness for dataset distillation, arXiv preprint ar 4.X9v:241 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes]
Justification: The abstract and the Introduction section described the task of distilled dataset and our presented solution based on pre-trained Diffusion Models. Our model evaluated on several standard datasets for Image Classification.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification:
We include discussion on the limitations of our method in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide derive theoretical motivation for the sampling process in section 4. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-
referenced.
13

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All hyperparameters are described in our paper. The presented model is clearly described for reproducibility and the algorithm is summarized in Algorithm 1.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets are standard and well-known benchmarks for Image Classification. The code will be also released after the publication of our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm public/guided httpg/uddde//eeeuubibiiaa Soooodeeumisssoooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
14

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CodeSuomissss wwwiebbmisssiiioi ssionooiccy for more details. httppccccpbplccccccc wewci//giisseeee /ode:////
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our experimental section described all required information such as models and datasets. We follow the same experimental setting as previous work in Distilled Dataset (GLaD). Also, the code repository will contain the training scripts for reproducing the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them. The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes]
Justification: our experiments were run five times, and it is clearly mentioned that we are reporting mean test accuracy and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
15

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the supplementary materials, we include information on the required compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper). 9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics pictps://neur//
Answer: [Yes]
Justification: We comply with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide societal impact in the sections and 3.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
16

---

Justification: Our method does not produce new risks of misuse, because we re-use existing models and datasets in compressed manner. Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit all owners of the assets that we use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pattaasts..ppersstttt paperswi thcode. cow datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper did not require any input for crowdsourcing experiments. We
evaluated our presented model using standard benchmarks for Image Classification..
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
17

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: We conduct no user studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable). such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We do
L.MMs part of our method.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy httt:p/:/mnaurrppeecc///offreeee for what should or should not be described.
18

---

Supplementary Material
A Definitions of DC, DM, and MTT
Dataset Condensation (DC) ensures alignment by deriving the gradients via classification error [42] It calculates the loss on real (ET) and the respective synthetic data (ES). Next, it minimizes the distance between the gradients of both network instances. More concretely.
V Val (0) V (0)
DD =
(9)
V LET (0)l
Distribution Matching (DM) obtains gradients by minimizing the logits on the real and synthetic datasets. It enforces the feature extractor (ConvNet) to produce similar features for real and synthetic images [1]. The distribution matching loss is
CDM =
Σ
Σ XET.
(x)
Σ xES.
(ss)
(10)
To
SS
where Te, S. are the real and synthetic images for a class c.
Matching Training Trajectories (MTT) concentrates on the trajectory of network parameters 44 In more detail, MTT exploits several trained instances of model, called experts, and stores the training trajectory of parameters {07 ) at predetermined intervals, called expert trajectories. For dataset distillation, MTT samples a random
of parameters 0; from the trajectory at a given timestamp. Next, it trains a new network, 0,+N, initialized with the parameters on the respective synthetic images (for N iterations). Finally, the distance between the trajectory on the real dataset, 0f with M steps,
and the trajectory on the synthetic one, de+N, is minimized. As a result, MTT tries to mimic the original dataset's training path (trajectory of parameters) with the synthetic images:
|1@+.N +f+ ml1
LMTT-
(11)
100 0++M
B Justification for Modified Reverse Process in Distillation
Our core modification to the reverse diffusion step is crucial for improving the gradient flow back to the initial latent 2, allowing end-to-end optimization. A natural question arises regarding the validity of this modified process, as intentionally deviates from standard diffusion sampling procedures designed for high-fidelit, image generation that perfectly match the learned data distribution.
Distillation vs. Faithful Sampling. The key insight that the objective of dataset distillation differs fundamentally from standard image generation. Distillation aims to synthesize a small set of maximally informative samples that enable efficient training of downstream features over photorealism [35] Pixel-pen perffett adherence to the original data distribution not necessarily required or even optimal; abstract or stylized images often yield excellent distillation performance if they capture core class characteristies effectively [5).
Different Objectives:
models, prioritizing the encoding of essential flass-discriminatii
Impact of Modification. Our modification introduces a direct dependency on the initial noisy state zr throughout the reverse process. While preserving the Markov property (as shown in the main paper) and the representational power of the pre-trained denoiser fe. this change means the resulting process pa (Zy 2z c) longer guarantees sampling exactly from the original distribution μ learned by the LDM. If applied without the corrective feedback loop of distillation optimization (e.g., for unconditional generation), this modification can lead to more abstract outputs that deviate from the expected style, as illustrated with an "FHQ-trainee model in |Figure 5] This deviation is expected, as the process is no longer constrained solely by the standard denoising objective. We also observe slight reduction in sample diversity (measured by average LPIPS between generated samples: 0.386 with modification vs. 0.420 without, on ImageNette samples), likely due to the persistent influence of the fixed ZT.
19

---

Figure 5: Influence of our modified reverse process in a classical image generation setting (uncondi- tional FFHQ). It shows that the residual connections alter the generation process significantly, leading to abstract artifacts and the loss of coherence expected in facial dataset: (top) with modification and (bottom) without modification.
20
Modifiationn
Without Mocification
16
14
12
SWR
10
6-
2-
0
10
12
14
16
18
20
Timesteps
Figure 6: Gradient flow analysis comparing the Signal-to-Noise Ratio (SNR) of gradient norms for LD3M with and without our modification. Diffusion demonstrates more stable gradient flow, indicating enhanced optimization dynamics. Dashed lines show a polyfit plot to highlight the trends.
Suitability for Distillation. Crucially, within the dataset distillation framework, the latent codes Z (and conditioning are continuously optimized to minimize the distillation loss L. This opti- mization process actively counteracts potential adverse effects the modified sampling path by guiding the generation towards producing images (even abstract) that are highly effective for the downstream task defined by C. The essential properties needed are: (I) sufficient generative capacity create diverse, class-relevant features, which the LDM provides; and (2) strong gradient flow for optimization, which our modification enables (as also shown in Figure 6). The empirical success of LD3M significantly outperforming GLaD and AE-only baselines, and performing robustly even with randomly initialized LDMs demonstrates that this trade-off (sacrificing perfect distribution matching for tractable optimization) highly beneficial for the specific goal of dataset distillation. The resulting "abstract" representations effectively encode class information for robust generalization. Therefore, while distinet from standard sampling, our modified reverse process is well-justified and necessary component for unlocking diffusion models for effective, end-to-end dataset distillation.
20

---

workstation equipped with NVIDIA RTX A6000 GPU (48 GB Table Common hyperparameters for training the distillation algorithms used in this work. Parameter
Value
DSA Augmentations
Color Crop / Cutout Flip / Scale / Rotate 5,000 (128 x 128) / 10,000 (256 256)
Iteration (Distillation)
Momentum
0.5
Batch Real
256
Batch Train
256
Batch Test
128
C Hyper-Parameters for Distillation Algorithms
LDM. For all our LDM experiments, we set the unconditional guidance scale to the default value of For 128 x 128 images, we used max. time steps of 10, and for 256 256 images, we used 20.
DC. We utilize a learning rate of 10-3 throughout our DC experiments to update the latent code representation and the conditioning information.
DM. In every DM experiment, we adopt a learning rate of 10 10-2, applying it to updates of the latent code representation alongside the conditioning information.
MTT. For MTT experiments, uniform learning rate of 10 is applied update the latent code representation and the conditioning information. We buffered 100 trajectories for expert training, each with 15 training epochs. We used ConvNet-5 and InstanceNorm. During dataset distillation, we used three expert epochs, max. start epoch of 5 and 20 synthetic steps.
D Large Scale Datasets
Although LD3M is compatible with various distillation algorithms -including DC, DM, and MTT current experiments focus baseline variants that do not leverage inter-class relationships during optimization. This is an essential avenue for further improvement: incorporating inter- class information (e.g., through contrastive losses hierarchical label structures) may enhance the discrininative quality of the synthetic data. Future work will explore how LD3M's expressive latent trajectories be used facilitate such structured, cross-class- distillation.
E
Limitations
While LD3M improves dataset distillation compared to GLaD, it is essential to acknowledge certain limitations. primary concern arises from the linear addition in the diffusion process, which may not sufficiently combat the vanishing gradient problem for larger time steps, as observed in our experiments [19. Further alternative strategies for integrating the initial state Zr the diffusion process should be evaluated to address this issue, e.g., non-linear progress towards 0 as approaches 0. These alternative approaches could offer more nuanced and dynamic ways to manage the influence of zr across different stages of the diffusion, potentially mitigating the problem of vanishing gradients and enhancing the overall efficacy the distillation process.
Hardware and Software
All experiments
VRAM). Our implementation uses PyTorch 1.10.1 with torchvision 0. 1.2, and we build upon the GLaD library for dataset distillation with generative prior.
21

---

Table 10: Class listings for our ImageNet subsets.
Datract
InsgsNNes-A
Loanborg
Poriisis
Monkey
Repesoed
Thaep- loal
Sleth
Cliif Dwolling
Lady's s Slipper
Vellow
Hamster
Condela
Orea
Limpkin
ImapeNet-B
Spoonhill
Website
Lonikeet
Hlena
Carthetar
Trillyuus
Echidha
Pomeratian
Odometer
Tumssone
Ruilly
ImageNet
Freight
Hummingbird
Beaury
Callinule
Cabbage Butertly
ImageNer-D
Ostrich
Samoyed
Snowhird
Gaiffon
Chickadee
Sorrel
Admiral
Gray Owl
Cheat
Hombill
Kinglet
ImageNer E
Spindle
Toucan
Black Swan
Penguin
King
Poter's Wheel
Photocopter
Screw
Tazanuiaa
Sscilloscope
Lycaenid
ImageNeue
Tench
Faglish Springer
Caoaciti
Player
Chaissow
Church
Freach Hom
Carlage
Truck
Glold
Parachure
Samoyed
Beagle
English
Ridzeback
Dingo
Sheepdog
lnggish
ImageNes Birds
Pewsock
Flamingo
Masao
Pelican
Panguin
King
Bald Engle
Toucas
Osurch
Black Swan
Cookatoo
InnueeattFriix
Pissaple
Banaa
Stawbeny
Orange
Lemon
Pomegrmmat
Fig
Bell Popper
Cuzamber
Granay Smath
Apple
InagoNet-Ces
Tabby
Cal
Dengal
Pension
Siumese Cut
Laypean
Snow
Tyex
22

---

ImNet-A
ImHet-8
ImWoof
Immoet-Biee -Brrd
InNNet-Cato ImNet-Frrits
Figure 7: Images distilled by MTT in LD3M for IPC=1.
23

---

ImNet-A
ImNet-B
ImNettCC
ImNet-D
ImNette
ImWooo
ImnNet-Cate ImNiet-Fruits ImNott-Birde
Figure 8: Images distilled by DC in LD3M for IPC=1.
24

---

ImNer-B
ImNett-C
ImNet-E
InNette
ImNiet-Fruits ImNet-BBirds
InnNet-Cate
Figure 9: Images distilled by DM in LD3M for IPC=1.
25

---

Figure 10: Images distilled by DC in LD3M for IPC=10 and ImageNet-A.
26

---

Figure 11: Images distilled by DC in LD3M for IPC=10 and ImageNet-BB BB
27

---

Figure 12: Images distilled by DC in LD3M for IPC=10 and ImageNet-C.
28

---

Figure 13: Images distilled by DC in LD3M for IPC=10 and ImageNet-DD D.
29

---

Figure 14: Images distilled by DC in LD3M for IPC=10 and ImageNet-E.
30

---

Figure 15: Images distilled by DM in LD3M for IPC=10 and ImageNet-A.
31

---

Figure 16: Images distilled by DM in LD3M for IPC=10 and ImageNet-B B.
32

---

Figure 17: Images distilled by DM in LD3M for IPC=10 and ImageNet-C.
33

---

Figure 18; Images distilled by DM in LD3M for IPC=10 and ImageNet-D.
34


Reviews:
[{"review_id": "ZoXqNeui3b", "reviewer": "Reviewer_qEHp", "summary": "This paper proposes LD3M (Latent Dataset Distillation with Diffusion Models) for end-to-end dataset distillation to prevent vanishing gradients. The authors introduce linearly decaying residual connections from the initial noisy latent state into each reverse diffusion step without requiring diffusion model fine-tuning. Evaluated across multiple ImageNet subsets and CIFAR-10, LD3M outperforms GAN-based methods (GLaD), while demonstrating faster distillation times and simpler initialization compared to GLaD.", "questions": "(1) As mentioned above, it is unknown how LD3M scale to distillation of full ImageNet-1K at 224 × 224 resolution and IPC = {10, 50, 100}. These experiments have been done in SRe$^2$L [1], DWA [2], D$^4$M [3], and RDED [4]. Could LD3M still outperform these SOTA methods at the lower cost?\n\n(2) Why is linear decay (in Eq. 6) used instead of exponential or learned schedules? Is this choice empirically or theoretically motivated?\n\n(3) The base model is sometimes from ‘Random’, which is confusing for its definition. Moreover, if the base network is Stable Diffusion trained on LAION (not trained for one specific dataset, such as ImageNet and FFHQ), how does LD3M perform, compared with the current network?\n\n(4) Although the paper makes some claims on privacy and robustness applications, there are no experiments and theoretical analysis to contribute to such statements. Therefore, beyond accuracy, how do LD3M’s distilled samples compare to GLaD in metrics like adversarial robustness or privacy leakage?\n\n(5) The paper optimizes class embeddings but does not explore text prompts as conditioning. Could text-guided distillation improve cross-class discriminability?\n\n(6) Is it possible that an efficient sampling algorithm or few-step model (e.g., DPM-solver and Consistency Models) can achieve better sample-quality in fewer steps?\n\n**Reference:**\n\n[1] Yin, Z., Xing, E., and Shen, Z. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. In NeurIPS, 2023.\n\n[2] Du, J., Zhang, X., Hu, J., Huang, W., and Zhou, J. T. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. In NeurIPS, 2024.\n\n[3] Su, D., Hou, J., Gao, W., Tian, Y., and Tang, B. D$^4$M: Dataset Distillation via Disentangled Diffusion Model. In CVPR, 2024.\n\n[4] Sun, P., Shi, B., Yu, D., and Lin, T. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In CVPR, 2024.", "limitations": "Yes, the authors adequately discuss technical limitations, including dependence on LDM’s autoencoder structure and trade-offs between diffusion steps and performance.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n(1) The proposed method seems technically sound with various experiments across ImageNet subsets, multiple resolutions and IPC settings. Ablation studies convincingly isolate contributions of key components.\n\n(2) The paper is overall easy to follow and well-structured with clear problem formulation and gradient decay analysis. It solves a critical bottleneck (vanishing gradients) to prevent diffusion models from being used in dataset distillation.\n\nWeaknesses:\n\n(1) Experiments are limited to ImageNet subsets and CIFAR-10. It is necessary to show validations on larger-scale datasets (e.g., full ImageNet-1K).\n\n(2) Some of stated advantages remain unsupported by experiments. For example, it lacks evaluations to show maintained or even improved robustness against adversarial attacks or membership inference, which could differ LD3M from other baselines.", "was_revised": true, "final_justification": "The authors have already addressed my concerns. Thus, I consider increasing my final score to 'Borderline accept'.", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for your detailed and insightful review. We are grateful that you recognized the technical soundness of our proposed LD3M, the clarity of the paper, and that our work solves the critical bottleneck of vanishing gradients for dataset distillation with diffusion models.\n\nWe appreciate the constructive weaknesses and questions you raised. We believe they can be fully addressed. We provide point-by-point responses below.\n\n> **(1) As mentioned above, it is unknown how LD3M scale to distillation of full ImageNet-1K at 224 × 224 resolution and IPC = {10, 50, 100}. These experiments have been done in SRe$^2$L [1], DWA [2], D$^4$M [3], and RDED [4]. Could LD3M still outperform these SOTA methods at the lower cost?**\n\nWe agree that evaluation on the full ImageNet-1K dataset is an important long-term goal for any distillation method. However, we respectfully argue that our paper's primary contribution is foundational and, therefore, compared with GLaD, the state-of-the-art in generative prior-based distillation. The methods you cited (SRe2L, DWA, RDED) represent a different, parallel line of research that often relies on statistics matching (to be precise: BN alignment as prior) or other decoupled strategies  and does not use a generative prior for end-to-end optimization. While their large-scale results are impressive (and fast), our work opens a fundamentally new pathway for distillation.\n\nAs you correctly noted, scaling is a crucial next step. We explicitly acknowledge this in our paper, stating that evaluation on larger benchmarks like the full ImageNet-1K remains an important direction for future work. Since our work does not rely on dataset statistics (like in SRe2L with BN alignment), we expect that our results (final performance) translate to other classes in ImageNet as every class is distilled in isolation of every other class (great for multi-gpu distillation). By exploiting knowledge distillation and soft-labeling [5], we would expect even greater performance similar to the aforementioned methods. However, we want to note that the cited works are explicitly designed to be fast, which we believe is hard to beat with respect to distillation speed (e.g., RDED is essentially a concatenation of real-images, which is incredible fast in comparison). \n\nYet, despite their fastness, they are still comparable with more straightforward methods like MTT/TESLA. See the RDED paper as example: MTT/TESLA reaches 7.7±0.2 and 17.8±1.3, whereas SRe2L reaches 0.1±0.1 and 21.3±0.6 on IPC=1 and 10, respectively on ImageNet-1K. MTT/TESLA can be used on top of our generative prior and improved upon with that strategy.\n\n[5] Tian Qin, Zhiwei Deng, David Alvarez-Melis: A Label is Worth a Thousand Images in Dataset Distillation, In NeurIPS, 2024.\n\n> **(2) Why is linear decay (in Eq. 6) used instead of exponential or learned schedules? Is this choice empirically or theoretically motivated?**\n\nA linear schedule provides a simple, parameter-free, and interpretable way to bridge the gradient flow. It ensures that the influence of the initial noisy state is strongest at the beginning of the reverse process (when the denoiser's prediction is less reliable) and smoothly fades as the denoised, final state is formed. Thus, it was chosen (theoretically motivated) for its simplicity. We agree that other schedules could be explored. We explicitly mention in our \"Future Work\" section that investigating \"alternative residual formulations\"  is a promising direction. \n\n> **(3) The base model is sometimes from ‘Random’, which is confusing for its definition. Moreover, if the base network is Stable Diffusion trained on LAION (not trained for one specific dataset, such as ImageNet and FFHQ), how does LD3M perform, compared with the current network?**\n\nWe apologize for the confusion. \"Random\" in Table 5 refers to a generator (in this case, StyleGAN-XL for GLaD and the LDM for our method) whose weights are randomly initialized instead of being pre-trained on a specific dataset like ImageNet or FFHQ. This is a standard ablation to test the importance of the pre-trained prior. Our results in Table 5 show that LD3M is robust and performs well even with a random prior. So, we expect only marginal improvements with a DM trained on LAION. Nevertheless, using a general text-to-image model like a newer Stable Diffusion model would be a very exciting future direction, which leads directly to your next question.\n\n> **(5) The paper optimizes class embeddings but does not explore text prompts as conditioning. Could text-guided distillation improve cross-class discriminability?**\n\nThis is a great point. The extension of LD3M to use text prompts is a natural and powerful next step. We focused on learnable class embeddings in this work to maintain a controlled comparison against GLaD. This allowed us to isolate the specific benefits of using a diffusion prior over a GAN prior. However, our framework is indeed general. Replacing the learnable class embeddings with text embeddings from a model like CLIP is a straightforward extension that could enable distillation of datasets with more rich textual descriptions, potentially improving cross-class discriminability as you suggest. We believe this is a key avenue for future research spawned by our work. A great starting point could be the following work: [6]\n\n[6] Stanislav Fort, Jonathan Whitaker: Direct Ascent Synthesis: Revealing Hidden Generative Capabilities in Discriminative Models, In arXiv, 2025.\n\n> **(4) Although the paper makes some claims on privacy and robustness applications, there are no experiments and theoretical analysis to contribute to such statements. Therefore, beyond accuracy, how do LD3M’s distilled samples compare to GLaD in metrics like adversarial robustness or privacy leakage?**\n\nWe acknowledge that the phrasing in the introduction could be clearer. Our intention was not to claim that LD3M achieves superior robustness or privacy, but rather that it enables the gradient-based optimization necessary to even explore these properties.\nThe core motivation for pursuing gradient-matching optimization is its potential for fine-grained control, which is crucial for applications like enhancing adversarial robustness or privacy. For instance, see [7] as an example, where a modified MTT for improved robustness is proposed, which can be applied on top of our generative prior. Previous diffusion-based methods like D4M, which rely on sampling, cannot perform these types of optimization. LD3M is the first diffusion-based method that would make this possible.\n\nThe scope of our paper is to solve the foundational vanishing gradient problem. A full evaluation of all potential downstream benefits (like adversarial robustness or privacy leakage) is substantial enough for follow-up work and beyond the scope of this initial paper.\n\n[7] Lai et al., Robust Dataset Distillation by Matching Adversarial Trajectories. In arXiv, 2025.\n\n> **(6) Is it possible that an efficient sampling algorithm or few-step model (e.g., DPM-solver and Consistency Models) can achieve better sample-quality in fewer steps?**\n\nThis is another excellent and forward-looking question. Our work strongly suggests that fewer steps are indeed better for the distillation task.\nOur analysis on the number of diffusion steps T (Figure 5) clearly shows that LD3M's performance peaks around T=10-40 and does not improve (and can even slightly decrease) with more steps. This indicates that a long, costly diffusion chain is not required.\n\nWe explicitly identify this as a key area for future work. In our conclusion, we state that future work should explore \"integration with fast samplers like DPM-Solver\". Such an integration would further improve LD3M's efficiency, reducing the distillation time and memory usage reported in our paper and, ultimately, enhancing its practical appeal."}, {"review_id": "ZlpAvuSo6P", "reviewer": "Reviewer_2fpv", "summary": "- This paper looks at using pre-trained latent diffusion models for dataset distillation\n- The method is compatible with any dataset distillation loss\n- The authors argue that directly optimizing latent variables $z_T$ is difficult to vanishing gradients in the diffusion process\n- To address this, they modify the standard diffusion backwards process with one which has an additional skip connection\n- The method has improved performance compared to existing baselines (GLaD) for different types of dataset distillation losses", "questions": "- Are there baseline methods or papers which use guided diffusion rather than directly optimizing z_T?\n- How does the randomness of the denoising process affect LD3M? From Algorithm 1, it seems that because of the sampling of $\\epsilon$, you will get an different image every time? Are these samples fixed for each example (making the process deterministic)?\n- What is the impact of the choice of residual connection in equation (6)? Have the authors experimented with different types of flows?\n- Does the existing backwards diffusion process not already have some aspect of the skip connection? I.e. in equation 3 we have the $z_t/{\\alpha}$ in the RHS of the equation. Could the authors comment on the relative magnitude of that skip connection compared to the one which they introduce?\n- Why does it make sense to use the autoencoder as initialization for $z_T$? Isn't the autoencoder supposed to produce values for $z_0$?", "limitations": "See weaknesses section/limitation for more.\n\n- Some more motivation for why optimizing $z_T$ instead of $z_0$ directly would be nice. While readers might suspect it's due to that the distribution of  $z_T$ might have more favourable structure (i.e. not having distinct clusters like $z_0$, it would be nice to see this explained mored, or explored.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Very good presentation, reading this was very easy and the background goes over prior work with a good amount of detail, and the method is clearly described\n- Method is direct and not overly engineered\n- Consistent performance improves over baselines\n- Good set of ablations, with each change good improvements\n\nWeaknesses\n- Should have comparisons to other diffusion based methods such as Minimax diffusion or D4M", "was_revised": true, "final_justification": "The paper is a solid contribution to field of dataset distillation, and will likely be used in future work (i.e. any method that uses a pre-trained generator like GLaD).", "rebuttal": "Dear Reviewer,\n\nThank you for your very positive and encouraging review. We are thrilled that you found the paper to have excellent presentation and clarity, and that you appreciated the directness of our method and the strong empirical results. Your high rating and confidence in our work are greatly appreciated.\nWe are happy to answer your questions and address the points you raised to further improve the paper.\n\n> **Are there baseline methods or papers which use guided diffusion rather than directly optimizing z_T?**\n\nOur main comparison is with GLaD because it is the state-of-the-art for our specific paradigm: end-to-end, gradient-based distillation using a generative prior. That said, we do include a direct comparison with D4M on CIFAR-10 (Table 2b), where LD3M shows superior performance, and D4M could be seen as guided diffusion rather than directly optimizing Z. \n\nIn general, it would be interesting to see guided diffusion to directly infer a trainable dataset, but our preliminary experiments suggest that this line of research needs more work to be usable in that way.\nBelow, we provide our findings for ImageNet-1K (IPC roughly 1200) with ResNet-18 as a classifier (90 Epochs), and it can be seen that, on average, the performance is worse compared to any distillation approach. Interestingly, with more modern LDMs, the performance gets even worse (probably due to recent fine-tuning approaches towards better aesthetics).\n\n|Variant|Final val. acc. (%)|\n|-|:-:|\n|Baseline (real data)\t| 70.0 |\n| SD 1.0 | 17.1 |\n| SD 2.1 | 14.5 |\n| SDXL | 9.7|\n\n> **How does the randomness of the denoising process affect LD3M? From Algorithm 1, it seems that because of the sampling of $\\epsilon$, you will get a different image every time? Are these samples fixed for each example (making the process deterministic)?**\n\nYou are correct that the sampling of $\\epsilon$ introduces stochasticity. We fix, however, for one sampling phase, the residual variable $z_T$ to be constantly the same. Thus, we stochastically sample $z_T$ in the first step but keep it constant for all remaining time steps. Therefore, it is not a deterministic process due to its stochastic sampling in the first step.\n\n> **What is the impact of the choice of residual connection in equation (6)? Have the authors experimented with different types of flows?**\n\nThis is an important design choice. We found that the simple, parameter-free linear decay was highly effective empirically and robust across different datasets and settings. We chose it for its simplicity and to avoid introducing additional learnable parameters. As we note in our future work section, exploring alternative residual formulations (e.g., exponential or learned schedules) is a promising avenue for future research, which we found orthogonal to the current state and goal of the paper.\n\n> **Does the existing backwards diffusion process not already have some aspect of the skip connection? ...**\n\nThis is a very sharp observation. You are right that the standard update rule (Equation 3) contains $z_t$ on the right-hand side, which acts as a \"local\" skip connection from step $t$ to step $t-1$. However, the vanishing gradient problem arises from the cumulative product of Jacobians across the entire chain of T steps. Our proposed modification is a \"global\" skip connection that creates a direct path from the final generated latent $z_0$ all the way back to the initial learnable codes, bypassing the long chain. This is what fundamentally preserves the gradient signal and is empirically shown in Table 1 of the paper.\n\n> **Why does it make sense to use the autoencoder as initialization for $z_T$? Isn't the autoencoder supposed to produce values for $z_0$?**\n\nIt is both true since the diffusion model operates in the latent space of the autoencoder, so between the encoder (which maps the pixel space to the latent space, i.e., $z_T$) and the decoder (which maps $z_0$ back to pixel space). Thus, in order to have a good initialization of the latent codes, it makes sense to use the encoder to derive the respective latent representations prior to any distillation."}, {"review_id": "w19IpbQzuo", "reviewer": "Reviewer_2brr", "summary": "The goal of the paper is to adjust the sampling process used in diffusion, using a different mean formulation in order to allow for backpropagation through time. Backpropagation enables distilling datasets into a (handful) of (latent) examples on which a novel architecture could be trained. Experimentally the method is evaluated on different datasets, varying number of image per class (used for distillation) and mainly compared to GLaD [5], generally leading to (small) performance improvements.", "questions": "I'm rating now borderline reject because of the missing details and missing connections. Hopefully the following can be addressed in the rebuttal:\n\n1) [**M1**] How does L3DM addresses the complex multi-space optimization and the cumbersome inversion process?\n3) [**M5**] How would L3DM compare to / can make use from PaGoDA-style of 1-step diffusion models?\n2) [**S1**] Please make the connection of the proposed L3DM model with existing samplers and flow matching\n4) [**S1**] What is the influence of the new mean function on the photorealism of the generated examples?", "limitations": "The limitations are not discussed in the main paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n**S1**: A modification of the mean estimation used in diffusion models to allow for gradient back propagation.\nThe paper would be much stronger, if\n-   it is clarified in the main body of the paper why downstream utility is preferred over photorealism;\n-   the influence of the proposed modification on photorealism is investigated/shown;\n-   the relation is made to existing samplers (like overshooting in DPM), distillation methods (like consistency methods), and flow based methods (where also a linear interpolation between two estimates is used).\n\n\n## Missing details\nI'm missing the following details in the main manuscript to make a full assessment:\n\n**M1**: It should be clarified how the proposed L3DM is not a 'complex multi-space latent optimization' and 'does not require cumbersome inversion processes'. These are the two drawbacks of the GLaD method mentioned in the 2nd paragraph of the introduction;\n\n**M2**: It should be clarified why distribution matching is not a goal here (just below Eq 6), and hence why the proposed modification is sound. This should not be deferred to the supplementary, as this is key for the core contribution of the paper. Please consider that distribution matching is both not the goal (just below Eq 6) and the goal of the paper: it is one of the distillation losses used in the paper (DM [41]).\n\n**M3**: How many latent codes are learned? Are there as many latent codes as IPCs? Or is there a single latent code per class, and then IPC images sampled from this latent code? [Related comparison is discussed below].\n\n**M4**: If the base LDM would be a few/one-step LDM - eg by using a distilled version (eg Pagoda [*Kim et al., 2024*]), then the difference in the mean estimation might be unnecessary. How does L3DM works in such a scenario? And what are the expected results?\n\n\n## Comparisons\n**C1**: What is the quality when class conditional sampling is used from the base LDM (or text based)? These are obtained only from a prompt and the sampled images could also be used to train any network. See also the paper of [*Bulent et al, 2023*].\n\n**C2**: I might mis understand the pre-training of the LDM, but how are the Random / FFHQ LDM trained? How come the resulting image is almost identical? What information of the pre-trained LDM is then used if these are near identical for so different pre-training tasks?\n\n\n## Minor Remarks:\n1) L3DM is not defined in the abstract, nor the title. It is used as an abstract name for a method in the introduction. Please add the full acronym in the abstract.\n2) Figure 2 does not illustrate the efficiency of L3DM, please make the difference in optimization between GLaD and LD3M more clear - or remove the image.\n3) Next to SAGE [23] also Simpler Diffusion [*Hoogeboom et al, 2025*] uses identity preserving connections to speed up learning. Please add that reference.\n4) The pixel-space model is not explained, is that just a random subset of the dataset images?\n5) The ablation results (Table 7, Table 5) are different from the results in the main tables (3 and 4). Please explain the differences.\n6) In the captions (eg Table 5) it is stated: \"improves by roughly +6.03%\" and \"roughly +23.28%\", that are extremely detailed / significant numbers for a `roughly`, please change to 6 and 23. Moreover, these seem not percentpoint improvements, these are 7 (29.5 vs 36.5) and 2 (34.5 vs 36.5). Be humble!\n7) Table 6: What does the \"+1.7-0.6\" mean? It is maximum of 1.7 improvement?\n8) Conceptually the following is unclear: the aim is to reduce a dataset to a few synthetic samples, such that training a novel architecture on this synthetic dataset yields strong classifiers. But in the current setup ImageNet is used to train an auto-encoder/decoder, followed by a Latent Diffusion Model, and then subsets of ImageNet are reduced to just a few samples to train a new classifier on a weaker (Convent/ResNet) architecture. The rational (in the introduction) is to reduce the computational cost. But the cost is already put into training the auto-encoder/decoder, the LDM and the synthetic dataset. This seems mostly relevant if the new dataset is vastly different from ImageNet. Or to manage storage in an online out-of-distribution (with respect to ImageNet) system, where new data flows in and only a small fraction can be stored.\n\n\n### References\n- [*Bulent et al, 2023*] Fake it till you make it: Learning transferable representations from synthetic ImageNet clones, Bulent et al., CVPR 2023\n- [*Hoogeboom et al, 2025*] Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion, Hoogeboom et al, CVPR 2025\n-  [*Kim et al., 2024*] PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher, Kim et al., NeurIPS 2024.", "was_revised": true, "final_justification": "The authors addressed my concerns in their rebuttal. When these are incorporated into the final version (and the comments of the other reviewers) I believe this paper is of interest for NeurIPS, hence I've increased my score to accept.", "rebuttal": "Dear Reviewer,\n\nWe thank you for your incredibly thorough and thoughtful review. Your detailed feedback is invaluable for improving our paper. We are glad you recognized the novelty of our core contribution. You have raised several important points regarding missing details and connections to related work, which we are happy to clarify. We will address your main concerns first, followed by the other specific points you raised.\n\n> **[M1]** It should be clarified how the proposed L3DM is not a 'complex multi-space latent optimization' and 'does not require cumbersome inversion processes'. These are the two drawbacks of the GLaD method mentioned in the 2nd paragraph of the introduction;\n\nWe will clarify this in the introduction. LD3M is significantly simpler than GLaD in two key ways:\n- **No Cumbersome Inversion:** GLaD requires a costly, iterative optimization process known as GAN inversion to find a latent code that reconstructs a given real image for initialization. LD3M leverages the LDM's inherent autoencoder structure. We initialize our learnable latents with a simple, single forward pass through the pre-trained encoder. This is vastly more efficient.\n- **Simpler Latent Space:** GLaD requires optimizing in the complex, multi-layered $W^+$ space of StyleGAN to generate high-quality images. LD3M optimizes a single latent code Z and a conditioning vector c, which is a much lower-dimensional and less complex optimization problem.\n\n> **[M2]** It should be clarified why distribution matching is not a goal here (just below Eq 6), and hence why the proposed modification is sound. This should not be deferred to the supplementary, as this is key for the core contribution of the paper. Please consider that distribution matching is both not the goal (just below Eq 6) and the goal of the paper: it is one of the distillation losses used in the paper (DM [41]).\n\nWe revised the main manuscript accordingly for improved clarity.\nIn short, sampler-level (Below Eq. 6): Here, \"distribution matching\" refers to the sampler's goal of generating images that match the data distribution the LDM was trained on. We explicitly state that our modified sampler deviates from this goal.\nClarification: Our method breaks the sampler's fidelity to the original data distribution in order to better achieve the distillation objective of matching feature distributions. \n\n> **[M3]** How many latent codes are learned? Are there as many latent codes as IPCs? Or is there a single latent code per class, and then IPC images sampled from this latent code? [Related comparison is discussed below].\n\nWe sincerely apologize for the confusion! We learn one unique latent code and one conditioning code per synthetic image. Therefore, for a dataset with C classes and IPC (Images Per Class), we learn C×IPC latent codes and C×IPC conditioning codes in total.\n\n> **[M4]** If the base LDM would be a few/one-step LDM - eg by using a distilled version (eg Pagoda [Kim et al., 2024]), then the difference in the mean estimation might be unnecessary. How does L3DM works in such a scenario? And what are the expected results?\n\nIf one were to use a true one-step diffusion model, the long denoising chain would not exist, and the vanishing gradient problem would be moot. In that specific scenario, our proposed modification in Equation 6 would indeed be unnecessary.\n\nHowever, our work is designed to unlock the potential of existing, powerful, off-the-shelf multi-step diffusion models (like the LDM pre-trained on ImageNet) for distillation without needing to retrain or distill the generator itself. This is a significant advantage, as it allows practitioners to leverage the best available generative priors directly. This opens an interesting new research question: Is it better to distill the dataset for a powerful multi-step model (our work) or to first distill the model into a one-step version and then distill the dataset? We believe in similar results, yet slightly worse, somewhere between LD3M and GLaD (one-step generator, although adversarially trained). We will add this point to our discussion.\n\n> **[C1]** What is the quality when class conditional sampling is used from the base LDM (or text based)? These are obtained only from a prompt and the sampled images could also be used to train any network.\n\nWe hope we understand the comparison request correctly: What happens if we just use the pre-trained LDM to generate class-conditioned images and then train a classifier on top? Below, we provide our findings for ImageNet-1K (IPC roughly 1200) with ResNet-18 as classifier (90 Epochs), and it can be seen that, on average, the performance is worse compared to any distillation approach. Interestingly, with more modern LDMs, the performance gets even worse (probably due to recent fine-tuning approaches towards better aesthetics).\n\n|Variant|Final val. acc. (%)|\n|-|:-:|\n|Baseline (real data)\t| 70.0 |\n| SD 1.0 | 17.1 |\n| SD 2.1 | 14.5 |\n| SDXL | 9.7|\n\n> **[C2]** I might mis understand the pre-training of the LDM, but how are the Random / FFHQ LDM trained? How come the resulting image is almost identical? What information of the pre-trained LDM is then used if these are near identical for so different pre-training tasks?\n\nThe random LDM is just a random initialization of the LDM. For the FFHQ LDM, it was trained on FFHQ based on the setup proposed by Rombach et al.. These identical results actually highlight the strength of our optimization framework. Figure 4 shows that even when the generator is pre-trained on faces (FFHQ), our distillation process forces the latents into a region of the LDM's space that produces the key discriminative features of the target class (\"Lorikeet\"). The pre-training provides a powerful prior for creating coherent textures and shapes (in other words, the images do not look like adversarial images), which our optimization then steers towards the desired class.\n\n> **[S1]** Please make the connection of the proposed L3DM model with existing samplers and flow matching\n\nWe will add this discussion to the suppl. materials. In short, we will state:\n- **DPM-Solvers/Flow Matching:** These methods use interpolation during the training of a generative model (or for the solver) to define a continuous path between noise and data. Our approach is fundamentally different: we use interpolation during the inference/sampling process of a pre-trained, fixed model. Our goal is not to train the generator but to create a gradient shortcut that allows us to optimize the initial latent codes passed into it.\n- **Consistency Models/PaGoDA:** These methods aim to distill a multi-step diffusion model into a one- or few-step generator. Our work addresses an orthogonal problem: distilling a large dataset using a fixed, pre-trained generator.\n\n> **[S1]** What is the influence of the new mean function on the photorealism of the generated examples?\n\nIts influence is significant and should not be used in the context of photorealistic image generation (see Figure 1 in the supplementary material). The primary goal of dataset distillation is not to create photorealistic images, but to synthesize a small set of samples that are maximally effective for training a downstream model. Our method deliberately trades photorealism for improved gradient flow, which is essential for achieving this higher downstream utility.\n\n# Minor Remarks\n\n> **L3DM is not defined in the abstract, nor the title. It is used as an abstract name for a method in the introduction. Please add the full acronym in the abstract.**\n\nThank you for pointing this out! We revised the paper accordingly!\n\n> **Figure 2 does not illustrate the efficiency of L3DM, please make the difference in optimization between GLaD and LD3M more clear - or remove the image.**\n\nWe removed the image from the manuscript.\n\n> **Next to SAGE [23] also Simpler Diffusion [Hoogeboom et al, 2025] uses identity preserving connections to speed up learning. Please add that reference.**\n\nDone.\n\n> **The pixel-space model is not explained, is that just a random subset of the dataset images?**\n\nThe pixel-space is not a model per se, but distillation in the pixel-domain (so no latent space or generator between the distilled entity and the respective distillation algorithms).\n\n> **The ablation results (Table 7, Table 5) are different from the results in the main tables (3 and 4). Please explain the differences.**\n\nTable 7 is the ablation run with fewer distillation steps compared to the main tables in 3 and 4. In Table 5, we analyzed our performance on $256\\times256$, which is different to Table 3 and 4 (128$\\times$128).\n\n> **In the captions (eg Table 5) it is stated: \"improves by roughly +6.03%\" and \"roughly +23.28%\", that are extremely detailed / significant numbers for a roughly, please change to 6 and 23. Moreover, these seem not percentpoint improvements, these are 7 (29.5 vs 36.5) and 2 (34.5 vs 36.5). Be humble!**\n\nWe apologize! We corrected the cited values to reflect percentage points!\n\n> **Table 6: What does the \"+1.7-0.6\" mean? It is maximum of 1.7 improvement?**\n\nWe apologize for the confusion! We meant that it improves the mean by 1.7 and reduces the std by -0.6.\n\n> **Conceptually the following is unclear: the aim is to reduce a dataset to a few synthetic samples, such that training a novel architecture on this synthetic dataset yields strong classifiers...**\n\nWe want to clarify that we do not train the autoencoder or the diffusion model itself at any point in the paper. We agree that, for a completely new dataset domain (e.g., medical domain), it might be necessary to train LDM on that new dataset, which questions the approach of dataset distillation with generative priors. However, in the current setting, we leverage off-the-shelf pre-trained models. Yet, we believe that even in those extreme orthogonal domains, LD3M will at least reach decent results since a randomly initialized LDM already gives a strong prior."}, {"review_id": "Z7Lg3LZ2mB", "reviewer": "Reviewer_DhwT", "summary": "This work introduces LD3M, a new method of dataset distillation that parameterizes the distilled data in the initial latent space of a pre-trained diffusion model. This method improves upon previous similar methods that optimize in the latent space of GANs like GLaD. To rectify the vanishing gradient problem that comes with the many passes through the diffusion model, this work introduces skip connections between passes that facilitate proper gradient flow through the back-propagation process. The authors show the effectiveness of their method through extensive experiments and ablations.", "questions": "I would appreciate it if the authors could answer my questions about the optimizer and initialization outlined in the previous section.\n\nIf the authors can address these questions as well as the minor issues I mentioned, I will happily raise my score.\n\nNote: rather than saying [Yes] on the checklist when asked if the code is included (when it actually isn't), you should simply say [No] and then explain, as you did, that the code will be released upon publication. There are not penalties for saying [No], but saying [Yes] without actually including the code is misleading and can waste reviewers' time.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nThe method itself is intuitive and easy to follow. The additions of skip connections seem like a sensible improvement to the naive method. The method seems to show decent improvement over the GLaD baseline in most settings.\n\n# Weaknesses\n\nI do not see any major problems with the paper, but there are a few things I would like to see cleared up. \n\nWhich optimizer was used to learn the distilled data? I’m curious if simply using Adam instead of SGD would alleviate the performance drop-off seen in Table 7 when the skip connections are removed. This would also keep the method more intuitive, since, as seen in Figure 1 of the supplementary material, the addition of the skip connections destroys the learned generative capabilities of the model, and I suspect the model is mostly just acting as a frequency regularizer at that point. \n\nWhen the authors mention initializing with a real image, shouldn’t the input to the diffusion model at t=T be closer to random noise? I’m not sure if it would affect the results dramatically, but I think it would make more sense to initialize the latent code with the DDIM inversion of the real image. \n\n## Minor Issues\n\nIn the equation below line 30 (which should be numbered like the other equations) and Equation 7, it is unclear what should be part of the product/sum operations. I recommend placing brackets around the intended operands so that it is immediately clear to the reader. Based on the text following the first equation, it seems that $\\frac{\\partial \\mathbf{z}_T}{\\partial\\mathcal{Z}}$ should be excluded from the product, but in Equation 7, the entire line is included in the sum.\n\nFurthermore, aren’t $\\mathbf{z}_T$ and $\\mathcal{Z}$ the same quantity? I think it should be made clear either way.\n\nIn lines 223 and 225, the same sentence is repeated almost word for word.\n\nIn Figure 4, which distillation method are we seeing?\n\nIn Figure 2 of the supplementary material, what exactly is the SNR of the gradient norms? \n\nIn line 61 of the appendix, are you saying that you trained 5,000 trajectories for each model and dataset? This seems quite excessive; GLaD only used 50 if I remember correctly.", "was_revised": true, "final_justification": "The reviewers have addressed all the questions and concerns I raised in my initial review. \n\nThis work will be a value addition to the dataset distillation literature.\n\nI recommend this paper for acceptance.", "rebuttal": "Dear Reviewer,\n\nThank you for your positive and constructive review. We are delighted that you found our method intuitive, the skip connections to be a sensible improvement, and that you recognized the performance gains over the GLaD baseline.\nWe provide our responses below and are confident they will address your points.\n\n> **Which optimizer was used to learn the distilled data?**\n\nWe used a standard SGD as in GLaD.\n\n> **I’m curious if simply using Adam instead of SGD would alleviate the performance drop-off seen in Table 7 when the skip connections are removed.**\n\nThank you so much for this interesting idea! We tested your suggestion on ImageNette (IPC=1) but the results are within std, so the optimizer has little to no influence, unfortunately.\n\n> **... the addition of the skip connections destroys the learned generative capabilities of the model, and I suspect the model is mostly just acting as a frequency regularizer at that point.**\n\nWe believe so too... You are correct that the addition of our skip connection alters the standard generative process, trading photorealism for improved downstream utility. This is a deliberate and necessary trade-off inherent to dataset distillation, where the goal is to create maximally informative, not necessarily visually pleasing, samples. As such, the role of the pre-trained LDM in our setup is indeed to steer the distilled images as a prior to \"somewhat natural\" looking images (away from adversarially looking images).\n\n> **When the authors mention initializing with a real image, shouldn’t the input to the diffusion model at t=T be closer to random noise? I’m not sure if it would affect the results dramatically, but I think it would make more sense to initialize the latent code with the DDIM inversion of the real image.**\n\nIntuitively, we believe you are right! We find this alternative initialization an interesting avenue for future work!\n\n\n# Minor Issues \n\n> **... (which should be numbered like the other equations) ...**\n\nThank you for pointing this out! We corrected this in our revised manuscript!\n\n> **I recommend placing brackets around the intended operands so that it is immediately clear to the reader.**\n\nThanks for this very clarifying recommendation! We corrected accordingly!\n\n> **Furthermore, aren’t $z_T$ and $Z$ the same quantity? I think it should be made clear either way.**\n\nWe apologize for the confusion and enhanced the clarity in the manuscript: \n$z_T$ is sampled from $q(z_T | Z)$ , which means that $z_T$ is the noised version of the learnable latent codes $Z$.\n\n> **In lines 223 and 225, the same sentence is repeated almost word for word.**\n\nWe removed the second sentence!\n\n> **In Figure 4, which distillation method are we seeing?**\n\nWe are sorry for the missing clarity! We used DC for this $256\\times256$ visualization and added this detail to the image caption.\n\n> **In Figure 2 of the supplementary material, what exactly is the SNR of the gradient norms?**\n\nThe \"Signal-to-Noise Ratio (SNR) of Gradient Norms\" refers to the ratio of the L2 norm of the gradient with respect to the learnable latent codes versus the L2 norm of the gradient with respect to our modification (initial state added at each step).\n\n> **In line 61 of the appendix, are you saying that you trained 5,000 trajectories for each model and dataset? This seems quite excessive; GLaD only used 50 if I remember correctly.**\n\nWe sincerely apologize for the wrong value, we used the same script as GLaD and their setting (100 trajectories). We confused the value with the number of iterations (5,000)."}]

Author rebuttal:

Meta-review: (a) The paper introduces LD3M, an end-to-end dataset distillation method in the latent space of a pretrained LDM to prevent vanishing gradients. The key idea is to add linearly decaying residual (skip) connections from the initial noisy latent z_T into each reverse diffusion step. LD3M can be used with existing distillation objectives, outperforms GLaD on several datasets, and is efficient.

(b) The method is simple and relatively general (adding skip connections), it addresses the problem of vanishing gradients and enables the use of diffusion models for database distillation, it performs better than GLaD on several datasets. The reviewers also found the paper to be well-written.

(c) The reviewers raised a number of concerns: 1) The evaluation scope is found limited and thus the scalability is unclear; 2) No experiments to support robustness claims; 3) Some technical details are not clearly described in the paper; 4) Missing evaluation of the computational load.

(d) All reviewers agree that this paper should be accepted and find the contribution solid. The proposed method is indeed simple but effective and would be of interest to the general conference audience. It is not recommended as an oral because the paper needs some more work to integrate more extensive comparisons and robustness evaluations.

(e) The reviewers raised several concerns but these were mostly addressed by the authors and the reviewers found them satisfactory. The concerns of qEHp, 2fpv, 2brr were on missing experiments on the full ImageNet-1K and recent diffusion-based distillation comparisons. There were several concerns on the technical details of the method by all reviewers (eg, linear decay, relation to existing methods,  stochasticity/determinism, and optimizer and initialization choices). There were concerns on the presentation (eg, equations and tables).
The authors addressed all these concerns. Moreover, based on the fact that reviewers had no follow up concerns and were satisfied with the answers, I decided that the paper should be accepted (and the authors should incorporate their answers into the final version of the paper).
Decision: Accept (spotlight)