Title: Unlocking Dataset Distillation with Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dataset Distillation

Diffusion Models

Latent Diffusion Models

Vanishing Gradients

Generative Priors
Abstract: Dataset distillation seeks to condense datasets into smaller but highly representative synthetic samples.  While diffusion models now lead all generative benchmarks, current distillation methods avoid them and rely instead on GANs or autoencoders, or, at best, sampling from a fixed diffusion prior. This trend arises because naive backpropagation through the long denoising chain leads to vanishing gradients, which prevents effective synthetic sample optimization. To address this limitation, we introduce Latent Dataset Distillation with Diffusion Models (LD3M), the first method to learn gradient-based distilled latents and class embeddings end-to-end through a pre-trained latent diffusion model. A linearly decaying skip connection, injected from the initial noisy state into every reverse step, preserves the gradient signal across dozens of timesteps without requiring diffusion weight fine-tuning. Across multiple ImageNet subsets at $128\times128$ and $256\times256$, LD3M improves downstream accuracy by up to 4.8 percentage points (1 IPC) and 4.2 points (10 IPC) over the prior state-of-the-art. The code for LD3M is provided at https://github.com/Brian-Moser/prune_and_distill.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces LD3M, a new method of dataset distillation that parameterizes the distilled data in the initial latent space of a pre-trained diffusion model. This method improves upon previous similar methods that optimize in the latent space of GANs like GLaD. To rectify the vanishing gradient problem that comes with the many passes through the diffusion model, this work introduces skip connections between passes that facilitate proper gradient flow through the back-propagation process. The authors show the effectiveness of their method through extensive experiments and ablations.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nThe method itself is intuitive and easy to follow. The additions of skip connections seem like a sensible improvement to the naive method. The method seems to show decent improvement over the GLaD baseline in most settings.\n\n# Weaknesses\n\nI do not see any major problems with the paper, but there are a few things I would like to see cleared up. \n\nWhich optimizer was used to learn the distilled data? I’m curious if simply using Adam instead of SGD would alleviate the performance drop-off seen in Table 7 when the skip connections are removed. This would also keep the method more intuitive, since, as seen in Figure 1 of the supplementary material, the addition of the skip connections destroys the learned generative capabilities of the model, and I suspect the model is mostly just acting as a frequency regularizer at that point. \n\nWhen the authors mention initializing with a real image, shouldn’t the input to the diffusion model at t=T be closer to random noise? I’m not sure if it would affect the results dramatically, but I think it would make more sense to initialize the latent code with the DDIM inversion of the real image. \n\n## Minor Issues\n\nIn the equation below line 30 (which should be numbered like the other equations) and Equation 7, it is unclear what should be part of the product/sum operations. I recommend placing brackets around the intended operands so that it is immediately clear to the reader. Based on the text following the first equation, it seems that $\\frac{\\partial \\mathbf{z}_T}{\\partial\\mathcal{Z}}$ should be excluded from the product, but in Equation 7, the entire line is included in the sum.\n\nFurthermore, aren’t $\\mathbf{z}_T$ and $\\mathcal{Z}$ the same quantity? I think it should be made clear either way.\n\nIn lines 223 and 225, the same sentence is repeated almost word for word.\n\nIn Figure 4, which distillation method are we seeing?\n\nIn Figure 2 of the supplementary material, what exactly is the SNR of the gradient norms? \n\nIn line 61 of the appendix, are you saying that you trained 5,000 trajectories for each model and dataset? This seems quite excessive; GLaD only used 50 if I remember correctly.\n\nQUESTIONS:\nI would appreciate it if the authors could answer my questions about the optimizer and initialization outlined in the previous section.\n\nIf the authors can address these questions as well as the minor issues I mentioned, I will happily raise my score.\n\nNote: rather than saying [Yes] on the checklist when asked if the code is included (when it actually isn't), you should simply say [No] and then explain, as you did, that the code will be released upon publication. There are not penalties for saying [No], but saying [Yes] without actually including the code is misleading and can waste reviewers' time.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of the paper is to adjust the sampling process used in diffusion, using a different mean formulation in order to allow for backpropagation through time. Backpropagation enables distilling datasets into a (handful) of (latent) examples on which a novel architecture could be trained. Experimentally the method is evaluated on different datasets, varying number of image per class (used for distillation) and mainly compared to GLaD [5], generally leading to (small) performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n**S1**: A modification of the mean estimation used in diffusion models to allow for gradient back propagation.\nThe paper would be much stronger, if\n-   it is clarified in the main body of the paper why downstream utility is preferred over photorealism;\n-   the influence of the proposed modification on photorealism is investigated/shown;\n-   the relation is made to existing samplers (like overshooting in DPM), distillation methods (like consistency methods), and flow based methods (where also a linear interpolation between two estimates is used).\n\n\n## Missing details\nI'm missing the following details in the main manuscript to make a full assessment:\n\n**M1**: It should be clarified how the proposed L3DM is not a 'complex multi-space latent optimization' and 'does not require cumbersome inversion processes'. These are the two drawbacks of the GLaD method mentioned in the 2nd paragraph of the introduction;\n\n**M2**: It should be clarified why distribution matching is not a goal here (just below Eq 6), and hence why the proposed modification is sound. This should not be deferred to the supplementary, as this is key for the core contribution of the paper. Please consider that distribution matching is both not the goal (just below Eq 6) and the goal of the paper: it is one of the distillation losses used in the paper (DM [41]).\n\n**M3**: How many latent codes are learned? Are there as many latent codes as IPCs? Or is there a single latent code per class, and then IPC images sampled from this latent code? [Related comparison is discussed below].\n\n**M4**: If the base LDM would be a few/one-step LDM - eg by using a distilled version (eg Pagoda [*Kim et al., 2024*]), then the difference in the mean estimation might be unnecessary. How does L3DM works in such a scenario? And what are the expected results?\n\n\n## Comparisons\n**C1**: What is the quality when class conditional sampling is used from the base LDM (or text based)? These are obtained only from a prompt and the sampled images could also be used to train any network. See also the paper of [*Bulent et al, 2023*].\n\n**C2**: I might mis understand the pre-training of the LDM, but how are the Random / FFHQ LDM trained? How come the resulting image is almost identical? What information of the pre-trained LDM is then used if these are near identical for so different pre-training tasks?\n\n\n## Minor Remarks:\n1) L3DM is not defined in the abstract, nor the title. It is used as an abstract name for a method in the introduction. Please add the full acronym in the abstract.\n2) Figure 2 does not illustrate the efficiency of L3DM, please make the difference in optimization between GLaD and LD3M more clear - or remove the image.\n3) Next to SAGE [23] also Simpler Diffusion [*Hoogeboom et al, 2025*] uses identity preserving connections to speed up learning. Please add that reference.\n4) The pixel-space model is not explained, is that just a random subset of the dataset images?\n5) The ablation results (Table 7, Table 5) are different from the results in the main tables (3 and 4). Please explain the differences.\n6) In the captions (eg Table 5) it is stated: \"improves by roughly +6.03%\" and \"roughly +23.28%\", that are extremely detailed / significant numbers for a `roughly`, please change to 6 and 23. Moreover, these seem not percentpoint improvements, these are 7 (29.5 vs 36.5) and 2 (34.5 vs 36.5). Be humble!\n7) Table 6: What does the \"+1.7-0.6\" mean? It is maximum of 1.7 improvement?\n8) Conceptually the following is unclear: the aim is to reduce a dataset to a few synthetic samples, such that training a novel architecture on this synthetic dataset yields strong classifiers. But in the current setup ImageNet is used to train an auto-encoder/decoder, followed by a Latent Diffusion Model, and then subsets of ImageNet are reduced to just a few samples to train a new classifier on a weaker (Convent/ResNet) architecture. The rational (in the introduction) is to reduce the computational cost. But the cost is already put into training the auto-encoder/decoder, the LDM and the synthetic dataset. This seems mostly relevant if the new dataset is vastly different from ImageNet. Or to manage storage in an online out-of-distribution (with respect to ImageNet) system, where new data flows in and only a small fraction can be stored.\n\n\n### References\n- [*Bulent et al, 2023*] Fake it till you make it: Learning transferable representations from synthetic ImageNet clones, Bulent et al., CVPR 2023\n- [*Hoogeboom et al, 2025*] Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion, Hoogeboom et al, CVPR 2025\n-  [*Kim et al., 2024*] PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher, Kim et al., NeurIPS 2024.\n\nQUESTIONS:\nI'm rating now borderline reject because of the missing details and missing connections. Hopefully the following can be addressed in the rebuttal:\n\n1) [**M1**] How does L3DM addresses the complex multi-space optimization and the cumbersome inversion process?\n3) [**M5**] How would L3DM compare to / can make use from PaGoDA-style of 1-step diffusion models?\n2) [**S1**] Please make the connection of the proposed L3DM model with existing samplers and flow matching\n4) [**S1**] What is the influence of the new mean function on the photorealism of the generated examples?\n\nLIMITATIONS:\nThe limitations are not discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper looks at using pre-trained latent diffusion models for dataset distillation\n- The method is compatible with any dataset distillation loss\n- The authors argue that directly optimizing latent variables $z_T$ is difficult to vanishing gradients in the diffusion process\n- To address this, they modify the standard diffusion backwards process with one which has an additional skip connection\n- The method has improved performance compared to existing baselines (GLaD) for different types of dataset distillation losses\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Very good presentation, reading this was very easy and the background goes over prior work with a good amount of detail, and the method is clearly described\n- Method is direct and not overly engineered\n- Consistent performance improves over baselines\n- Good set of ablations, with each change good improvements\n\nWeaknesses\n- Should have comparisons to other diffusion based methods such as Minimax diffusion or D4M\n\nQUESTIONS:\n- Are there baseline methods or papers which use guided diffusion rather than directly optimizing z_T?\n- How does the randomness of the denoising process affect LD3M? From Algorithm 1, it seems that because of the sampling of $\\epsilon$, you will get an different image every time? Are these samples fixed for each example (making the process deterministic)?\n- What is the impact of the choice of residual connection in equation (6)? Have the authors experimented with different types of flows?\n- Does the existing backwards diffusion process not already have some aspect of the skip connection? I.e. in equation 3 we have the $z_t/{\\alpha}$ in the RHS of the equation. Could the authors comment on the relative magnitude of that skip connection compared to the one which they introduce?\n- Why does it make sense to use the autoencoder as initialization for $z_T$? Isn't the autoencoder supposed to produce values for $z_0$?\n\nLIMITATIONS:\nSee weaknesses section/limitation for more.\n\n- Some more motivation for why optimizing $z_T$ instead of $z_0$ directly would be nice. While readers might suspect it's due to that the distribution of  $z_T$ might have more favourable structure (i.e. not having distinct clusters like $z_0$, it would be nice to see this explained mored, or explored.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LD3M (Latent Dataset Distillation with Diffusion Models) for end-to-end dataset distillation to prevent vanishing gradients. The authors introduce linearly decaying residual connections from the initial noisy latent state into each reverse diffusion step without requiring diffusion model fine-tuning. Evaluated across multiple ImageNet subsets and CIFAR-10, LD3M outperforms GAN-based methods (GLaD), while demonstrating faster distillation times and simpler initialization compared to GLaD.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) The proposed method seems technically sound with various experiments across ImageNet subsets, multiple resolutions and IPC settings. Ablation studies convincingly isolate contributions of key components.\n\n(2) The paper is overall easy to follow and well-structured with clear problem formulation and gradient decay analysis. It solves a critical bottleneck (vanishing gradients) to prevent diffusion models from being used in dataset distillation.\n\nWeaknesses:\n\n(1) Experiments are limited to ImageNet subsets and CIFAR-10. It is necessary to show validations on larger-scale datasets (e.g., full ImageNet-1K).\n\n(2) Some of stated advantages remain unsupported by experiments. For example, it lacks evaluations to show maintained or even improved robustness against adversarial attacks or membership inference, which could differ LD3M from other baselines.\n\nQUESTIONS:\n(1) As mentioned above, it is unknown how LD3M scale to distillation of full ImageNet-1K at 224 × 224 resolution and IPC = {10, 50, 100}. These experiments have been done in SRe$^2$L [1], DWA [2], D$^4$M [3], and RDED [4]. Could LD3M still outperform these SOTA methods at the lower cost?\n\n(2) Why is linear decay (in Eq. 6) used instead of exponential or learned schedules? Is this choice empirically or theoretically motivated?\n\n(3) The base model is sometimes from ‘Random’, which is confusing for its definition. Moreover, if the base network is Stable Diffusion trained on LAION (not trained for one specific dataset, such as ImageNet and FFHQ), how does LD3M perform, compared with the current network?\n\n(4) Although the paper makes some claims on privacy and robustness applications, there are no experiments and theoretical analysis to contribute to such statements. Therefore, beyond accuracy, how do LD3M’s distilled samples compare to GLaD in metrics like adversarial robustness or privacy leakage?\n\n(5) The paper optimizes class embeddings but does not explore text prompts as conditioning. Could text-guided distillation improve cross-class discriminability?\n\n(6) Is it possible that an efficient sampling algorithm or few-step model (e.g., DPM-solver and Consistency Models) can achieve better sample-quality in fewer steps?\n\n**Reference:**\n\n[1] Yin, Z., Xing, E., and Shen, Z. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. In NeurIPS, 2023.\n\n[2] Du, J., Zhang, X., Hu, J., Huang, W., and Zhou, J. T. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. In NeurIPS, 2024.\n\n[3] Su, D., Hou, J., Gao, W., Tian, Y., and Tang, B. D$^4$M: Dataset Distillation via Disentangled Diffusion Model. In CVPR, 2024.\n\n[4] Sun, P., Shi, B., Yu, D., and Lin, T. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In CVPR, 2024.\n\nLIMITATIONS:\nYes, the authors adequately discuss technical limitations, including dependence on LDM’s autoencoder structure and trade-offs between diffusion steps and performance.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-10"}