Diffusion Models as Dataset Distillation Priors
Authors: Duo Su
Huyu Wu
Huanran Chen
Yiming Shi
Yuzhu Wang
Xi Ye
Jun Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion mdoels
dataset distillation
diffusion priors
kernel method
Primary area: generative models

Abstract: Dataset distillation aims to synthesize compact yet informative datasets from large ones. A significant challenge in this field is achieving a trifecta of diversity, generalization, and representativeness in a single distilled dataset. Although recent generative dataset distillation methods adopt powerful diffusion models as their foundation models, the inherent representativeness prior in diffusion models is overlooked. Consequently, these approaches often necessitate the integration of external constraints to enhance data quality. To address this, we propose Diffusion As Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space using a Mercer kernel. We then introduce this prior as guidance to steer the reverse diffusion process, enhancing the representativeness of distilled samples without any retraining. Extensive experiments on large-scale datasets, such as ImageNet-1K and its subsets, demonstrate that DAP outperforms state-of-the-art methods in generating high-fidelity datasets while achieving superior cross-architecture generalization. Our work not only establishes a theoretical connection between diffusion priors and the objectives of dataset distillation but also provides a practical, training-free framework for improving the quality of the distilled dataset.

Published as a conference paper at ICLR 2026
DIFFUSION MODELS AS DATASET DISTILLATION PRI- ORS
Duo Su', Huyu Wu²,
Huanran Chen', Yiming Shi
Yuzhu Wang',
Xi Ye', Jun Zhu!
'Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University. Institute of Computing Technology, CAS. University of Electronic Science and Technology of China.
*South China University of Technology.
https: //sduusuuu/444// github..hh..gg 1oDD-ffussoontt--Prio
ABSTRACT
Dataset distillation aims to synthesize compact informative datasets from large ones. A significant challenge in this field is achieving a trifecta of diversity, gen- eralization, and representativeness in single distilled dataset. Although recent generative dataset distillation methods adopt powerful diffusion models as their foundation models, the inherent representativeness prior in diffusion models is overlooked. Consequently, these approaches often necessitate the integration external constraints to enhance data quality. To address this, we propose Dif- fusion As Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space using Mercer kernel. We then introduce this prior as guidance to steer the reverse diffusion process, enhancing the representativeness of distilled samples without any retraining. Ex- tensive experiments on large-scale datasets, such as ImageNet-1 and its subsets, demonstrate that DAP outperforms state-of-the-art methods in generating high- fidelity datasets while achieving superior cross-architecture generalization. Our work not only establishes a theoretical connection between diffusion priors and the objectives of dataset distillation but also provides practical, training-free framework for improving the quality of the distilled dataset.
INTRODUCTION
Data undeniably functions as the "primordial fuel" that drives modern AI systems. This critical source provides large models with foundational knowledge, spatiotemporal comprehension, visual awareness, and pattern recognition capabilities (Brown al., 2020)] Qin al.] 2025). Despite this, data faces depletion exponentially scaling models rapidly consume finite human-generated data, persisting as a bottleneck in advancing next-generation large models (Muennighoff al. [023 Vil-
lalobos et al. 2024). Current industry practices suffer dual burdens: insufficient data and expensive human annotation costs. Fortunately, synthetic data emerges as a renewable alternative capable of powering AI development scale (Jordon et al., 2022 Liu et al. 2024). While large models can generate samples in arbitrary categories and sizes, unfiltered synthetic data poses two critical risks: 1) Data Quality Limitations encompassing distribution drift and semantic mismatch (Alaa et al.) 2022 Yang et al. (2024). 2) Training Hazards, where flawed data patterns propagate through er- amplification, triggering failures like model collapse (Shumailov al., 20224] Dohmatob al.)
2024). Therefore, generating high-quality synthetic data remains challenging task.
Recent advances in dataset distillation (DD) offer a promising solution to the above challenges by generating highly compact datasets while preserving critical features often obscured in rerl-world data (Wang et al. 2018). In parallel, diffusion models (DMs) have emerged as state-of-the-art gen- erative methods due to their ability to accurately model the entire dataset distribution through score function estimation (Song et al. 202T). As a result, DMs have been adopted as foundation models
"Corresponding Author

---

Published as a conference paper at ICLR 2026
for DD, giving rise to generative DD Gu al. 2024 Su et al.,
2024 Leveraging priors acquired
from well-trained DMs, distilled samples maintain diversity and fidelity, achieving competitive ac- curacy with up to 10x x reduction in training size (Chen et al. 2025). Although encouraging, a theoretical analysis remains underdeveloped, which raises the following questions about the diffu- sion priors in generative DD methods.
Do the priors in vanilla DMs satisfy the requirements for DD? To answer this, we align the desired properties of distilled datasets with the priors captured by DMs via the original score func- tion. From the perspective of log-like oggliikelihood estimation and evaluation metrics (e.g., FID, IS), we observe that the inherent diversity and generalization priors in vanilla DMs can yield higher-quality synthetic data. Naturally, the main challenge shifts to enhancing the representativeness of synthetic data, which still not embodied in vanilla sampling pipelines. Previous approaches attempt to ad- dress this by imposing external representativeness constraints (Santiago et al. [025 Chen et al.) 2025). However, we argue that such constraints are unnecessary and introduce additional complex- ity. Thus, we raise the next question.
Are there unused priors in DMs that could benefit DD? Inspired by the diffusion classifiers (Chen et al. 2024a|b) b). we posit that the feature extraction capability inherent in well-trained diffusion model itself constitutes a representativeness prior highly relevant to DD. We hypothesize that high representativeness corresponds to high similarity between synthetic and original data in the repre- sentation space. To formalize this, we employ the Mercer kernel, a specific type of kernel func- tion (Zaanen| [964),, quantify the similarity within feature spaces. The Mercer kernel provides us with mathematical guarantees of convexity and tractability in optimization, ensuring that the representativeness prior is computationally feasible. Empirically, we define the representativeness score function as an energy function based on Mercer kernel, which allows us to inject the unused representativeness prior into the distilled data through guided sampling.
We propose Diffusion As Priors (DAP) and apply to datasets of varying scales, including large-scale ImageNet-IK IK (Deng et al., 2009) and its small subsets. Both quantitative and qualitative results show that DAP significantly enhances the quality of distilled datasets. It validates the theoretical connections between diffusion priors and DD task, while achieving competitive perfor- mance compared to other methods (see fig. I each di- mension is normalized independently for clear visualiza- tion). We further show that by introducing the desired priors, the distilled datasets have the same generalization and transferability as the original ones. Our contributions can be summarized as follows: in the well-trained DMs meet the diversity and general- ization requirements of DD. We derive the overlooked representativeness prior from DMs and formalize it into a kernel-induced distance, which guides the sampling dy- namic and improves the quality of distilled datasets.
Perfermance
63
We prove the priors
Figure 1: Our diffusion as priors (DAP)
method is beneficial for the DD task. Diversity: 1+FID FID. Represen- tativeness: (()()) Performance: classification results ImageNet-IK.
2
PRELIMINARIES
2.1 DATASET DISTILLATION
Given a labeled training dataset Terain (x. C N y where E R.N i.i.d. drawn from Paata. and E y y= (1, (1,...,,] C} drawn from the label space. The objective of DD is to synthesize compact dataset Sayn uy y} S aM x y (M < N) that encapsulates the knowledge of the original data. Consequently, the model trained with small Sayn can achieve considerable generalization performance (measured by loss L) to the large training dataset Terasn:
E B. [c (Jale( Tima, orony pron (x)) v)] E 00000 [ (fatetssw oi0 (.)
(1)
The algorithm alg(; ((0)) is determined by training set T S and the initialized parameters ((().
2

---

Published as a conference paper at ICLR 2026
2.2 DIFFUSION MODELS
Given a dataset 2n E RN i.i.d. drawn from an unknown distribution (o(a))) a diffusion model parameterized by 0 tries to learn distribution (2 that approximates (o(x))) Specifically, the diffusion model places a reversible process that gradually adds Gaussian noise from Bo to ær at time T and then maps them back. The forward diffusion process is defined by the Itō Stochastic Differential Equation (SDE) Song et al. (202T):
dæ, f (æt,, t) dt g(t)du);
where f (x,,,) - - 3, is the drift term and g(t) = v denotes the diffusion coefficient that controls the noise strength at each timestep. 3t (0. 1) is sequence of pre-defined time-dependent noise scales. Meanwhile, wy is the Brownian motion. And the reverse diffusion process is given by the time-reverse SDE: dæ = [f(x,,t) V px:)) dt + g(t)dw.
(3)
where T represents the rime-reversed Brownian motion. The only unknown term in eq. (3) is the
score function Va, log (-) of distribution pt at each time (we use p for simplicity). A neural network (a t) is trained to estimate the score function ~n, log(( Finally, we can sample
æo by solving the reverse diffusion SDE (Lu et al. 2022).
2.3 INHERENT PRIORS IN DIFFUSION MODELS
A key question in evaluating generative models is whether they capture the full variability of the dataset (Alaa et al. 2022). DMs inherently encode diversity and generalization priors through es- timating Vz x log pl p(z), which compels the model capture global manifold geometry rather than memorizing individual samples, thereby avoiding mode collapse (Thanh-Tune & Tran 2020). More- over, the stochastic perturbations in the forward process act as implicit regularizers, enforeing Lips- chitz continuity and improving robustness to distributional shifts (Chen et al. 2024a|b).
In addition, let H denote entropy, and &p is an inception classifier. High Inception Score (IS) indi- cates uniform class coverage (high (P.((y) and discriminative sample quality (low H (P (vl=))) While low Fréchet Inception Distance (FID) certifies alignment between generated and real distri- butions (Payn = Pdata). Empirical results (Dhariwal & Nichol (02T) demonstrate that the structure- induced priors within DMs produce sufficient diversity and generalization.
DIFFUSION As PRIORS
3.1 MOTIVATION
An ideal distilled dataset should satisfy (Gu et al. 2024: Su et al. 2024):
Distilled Dataset s.t. Diversity +
Generalization
Representatiieness.
These attributes enable the distilled dataset to be effectively applied across a variety of tasks, yielding competitive performance. Diversity ensures that synthetic data captures the full variability present in the original data, while Gemeralization prevents overfitting to the architecture of distillation models. Most importantly, Representativeness guarantees that the synthetic data retains the most critical information from the raw dataset. Consequently, we seek to study: how to align the priors of DMs with these attributes and make the distilled dataset desirable?
Formally. the objective of DMs that estimates the score function V log p(æ) provides synthetic dataset with inherent diversity and generalization priors (discussed in section (.2). In terms of the representativeness prior R, we consider introducing it into the score function as condition. According to Bayes' theorem, the conditional score function can be decomposed as:
V log [og)(=)))= = Va logp(())
V log ((R|2))
(4)
Diversity Generalization
Representativeness
Given well-trained diffusion model, the first term in eq. 4 same the original score function, already estimated by en. Thus, we focus on the second term to fulfill the representativennss requirement during sampling (discussed in section 3.3

---

Published as a conference paper at ICLR 2026
3.2 DIFFUSION AS DIVERSITY AND GENERALIZATION PRIORS
In the field of DD, diversity is characterized by the breadth of feature distribution and comprehen- sive coverage of categorical information. Meanwhile, generalization refers the ability prevent overfitting to the training data and enable datasets with cross-architecture adaptation. These proper- ties enable the distilled dataset reflect the information and knowledge of the original dataset like mirror. In this section, we argue that the pre-trained diffusion model provides inherent diversity and generalization priors for dataset distillation.
3.2.1 INHERENT DIVERSITY AND GENERALIZATION PRIORS
As mentioned in section 2.3 diffusion models pro- vide a principled foundation for DD, since effec- tive DD requires distilled data that both cover di- verse modes (diversity) and faithfully approximate the original dataset distribution (generalization), We quantify these properties with likelihood- based evaluations. The negative log-likelihood (NI.L) is Ee--pans [log pe (z)]. Identi-
Table I: NLLs on different datasets. The results are computed by vanilla diffusion model (Ho et al.
2020 trained on ImageNet.
Dataset
Training Set
Test Set
ImageNette
2.4452
2.6327 +1008 3.083810.86
ImageWoof
2.585610.88
defined as LNLL cal and low NLL values on training and testing sets indicate that po(x)) converges to Pdata instead
of overfitting (see table
3.2.2 BEYOND PRIOR: CROSS-ARCHITECTIO GENERALIZATION
Unlike conventional DD methods that match training dynamics (e.g.. Gradients, Parameters, and features) of specific downstream classifiers, DMs distill datasets without pixel-level optimization. The distilled dataset captures data-relevant rather than architecture-rrlevant knowledge, eliminating dependence on predefined classifier architectures. This architceture-agnosti DD paradigm produces distilled datasets with arshtteccie generalization, enhancing their versatility.
3.3 DIFFUSION AS REPRESSNTTATIVENESS PRIOR
Representative samples refer to a subset of data that accurately reflects the characteristics of the larger population from which is drawn (Gabbay et al. 201T). Generating a more representative dataset leads to better dataset distillation performance. In this section, we argue that a well-trained diffusion model itself can serve as a representativeness prior
3.3.1 REPRESENTATIVENESS PRIOR IN DMS
To capture the representativeness prior hidden in the DMs backbone network, we require similarity measure that quantifies how closely synthetic sample reflects the characteristics of the real sample. Kernel function is simple yet effective tool for defining similarity, allowing us to a) express rep- resentativeness through an induced distance and b) inject this representativeness energy term into the sampling process. Formally, let kernel function K(æ,, y) X' x X - R be the smooth and differentiable similarity measurement which characterizes the similarity between synthetic sample x -= and a single training sample x frain We argue that the larger the similarity between the synthetic samples and the entire training set Egtroin [K frain))) )]. the better repre- sentativeness of "mm to the raw dataset. Suppose that Dx(I, y) is a distance measure induced by the kernel function I.. Typically, we expect Dxc satisfy the fundamental properties of the distance measures. The following theorem demonstrates that, as long as the kernel function K. is positive semi-definite (PSD), the induced distance Dx is well-defined distance measure.
differentiable
Theorem 3.1. Let K. X x X + R. be PSD kernel. Then the K-induced distance measure
Dx(0, v) [K(,, xt) + K(y. v) 2K(x,, w)) 1))
(5)
satisfies:
1. Non-negativity: Dz(a,y) 2 0, and (x v) = 0 if and only if x y.
2. Symmetry: Dx((, Dx(
4

---

Published as a conference paper at ICLR 2026
3. Triangle inequality: For any y, X, Di((2, + (x(x,)))
Proof. (Sketch, details in section A.2.1) According to Mercer's theorem (Mereer 1909), the distance metric induced by the PSD kernel can be expressed the Hilbert norm in reproducing kernel Hilbert space (RKHS), which satisfies the property of norms.
Therefore, DE is valid distance metric. The Mercer kernel KM is family of PSD kernels that guarantees the existence a spectral expansion under continuity and compact conditions. Thanks to these desirable properties, we adopt Mercer kernel as the representativeness measure in our method.
Theorem 3.2. Let K. X x X R be Mercer kernel, then the K-induced distance Dic can be factorized as (x( v) = d (ø (I y). where & is a feature mapping and d is simple norm in Hilbert space.
According to the reproducing property of the kernel func- tion, there exists mapping and a feature space H. that allows the kernel I. be factorized into XC = ((.)).(()))n) (())) The distance formalized by the linear combination of kernel functions can then factorized into a combination of the complex and simple norm ||
Proof. (Sketch, details in section A.2.2)
Mercer kernel allows us to quantify representativeness in RKHS, and the associated kernel-induced measure ensures the underlying optimization problem remains convex and tractable. Hence, the task reduces to identifying suitable feature extractor that maps inputs into feature space, where the distance metric d (Ø (z) @ (v)) x k(z.y) We posit that the diffusion model itself is good feature extractor, supported by two observations: its strong image-text alignment reflects a comprehensive understanding of visual content (Yang & Wang, 2023), and its performance as a discriminative classifier exhibits high accuracy, robustness, and certified robustness (Chen et al. |222aab] b).
We propose Diffusion As Priors (DAP), which utilizes the diversity, generalization, and represen- tativeness priors contained in the well-trained diffusion models to distill datasets. Specifically, the backbone networks (e.g.. U-Net or Transformer) are viewed as mapping function Φ X + R", transforming an image or latent code into an n--dimensional feature vector. During the pre- training phase, the backbones are endowed with the representativennss prior, which enables them to capture meaningful and high-level features.
3.3.2 GUIDANCE OF REPRESENTATIVVNNSS PRIOR
We formalize the conditional probability of representativeness term in eq. (4) Boltzmann distri- bution w.r.t. Dx:
[exp
N Σn Dx um p train) }
p( [R| syn)
(6)
Z
where > 0 denotes the normalizing constant, and y >> controls the scale of representativeness prior. According to theorem 3.2 the conditional score function of representativeness term is:
(exp
1/4 EN DR vw .rann))
Varyn log uV) = Var log
Z
(exp - N En d ( di (rttaan))) )
nson log
2
avnn d (0(x d(pttrann
N
which is referred to energy- based guidance (Dhariwal & Nichol 2021).
Practically, the
classifier guidance method, which employs the pre-trained diffusion itself as a training-free time- dependent classifier ((xx)) such that d(x, t) Rs dbx0) (Shen al. 2024). Therefore, the reverse diffusion process with guidance is defined as:
dæ [f( .t) - (Va;un log (( + v Vagun (()2( [")) dt + g(t)du
< [f ,t) =()) (-co (I:2m,, ,1) V and (a: (2 (mffrin ))}] di + g())dw) (8)
5

---

Published as a conference paper at ICLR 2026
diesto
Cho11
dete
clastlo
daisi
cland
doss1
clarsto
Coss]
0053
Segat
cap3
Seeps
cmoe
3 Caae
claSs
claSS
21865
68598
63590
58.2% (y-3) 10
(ym 1)
62.5% (y- 0)
[y- d)
Acc: 63.4% 1y 5)
Acc: 39, 3% (y= 5) 39.8%
Acc: 06.4% ty= 0.5)
Acci 40 3% ty 0.5)
20)
67.8% (y-
(a) SD: ImageNette
(b) Image
(c) DiT: ImageNette
(d) DIT: ImageWoof
Figure 2:
Visualization of average representativensss (x (()) ) of distilled samples (IPC10).
As y increases, the representativeness (sector area) gets larger, yielding better DD performance.
where V Vagyn log x;)")) is treated as an auxiliary score derived from the representativeness prior. 2 synn and arrrann are the noised sy and trriin at timestep
Empirically, we compare the salient features across samples using the linear kernel (Mercer kernel g((,,)) = x y) due to its tractability. As indicated by eq. (6). the representativeness of * in- creases as the energy Dx decreases. Figuree]vssualizes the representativenes: of class-wise samples under different setups. The distillation performance improves on synthetic samples with higher rep- resentativeness, as reflected by the area of the sector. It is worth noting that according to eq. (4), the gradient field of diversity and generalization (V log p( x)) is determined and fixed by pre-trained DMs. Therefore, the gradient field of representativenes. cannot be increased indefinitely, otherwise the other priors will lose their effectiveness (see section 4.4 and section A.4.1).
Hereto, we successfully distilled the priors within DMs into the synthetic dataset. Specifically, Diversity prior arises from the stochasticity of diffusion trajectories where different noise initializa- tions lead to distinct denoising paths. Generalization prior stems from the original score function V. log p(ær) estimated by vanilla DM. Representativeness prior is directly implemented by the guid- ance term in eq. (8), which guides each denosing trajectory toward gradient regions that are well represented by real data. We implement the guided sampling process using VP-SDE and summa- rize the procedure in algorithm | The extensive experimental results in section demonstrate the validity our "Diffusion As Priors (DAP)" method.
Algorithm DAP Sampling (VP-SDE)
Require:
Noisy data samples trainle within class c, pre-trained diffusion model cø, layer output Φ from diffusion backbone network, a Mereer Kernel induced distance measurement d, energy- based guidance scale y, pre-defined noise scales 3r.
1: 2T () T)
2: for = .. do
3: é [0, if else 0
4:
ē = (2 v1 3e) + BrED (xi, 1) + VBrE trainle)
5:
2: (xx)), 2,
trainle d(x
Diffusion as representativeness priors
6: 9: = Va, ( Zz, 21
trainle)
7: It-1 ::-1 + 7gt
Guided sampling
8: end for
Output: Do
# The distilled sample of class c.
EXPERIMENTS
In this section, we conduct extensive experiments to validate the effectiveness DAP. Our evalua- tion aims explore the following questions:

---

Published as a conference paper at ICLR 2026
Does DAP achieve state-of-the-an performance large-scale DD benchmarks?
How do the three priors: diversity, generalization, and representativeness contribute to the effectiveness of DAP?
Can DAP generalize across network architectures and datasets?
We evaluate DAP ImageNet-IK IK and its widely used subsets (ImageNette, ImageWoof, and ImagelDC), comparing against advanced DD methods, including Minimax, D*M, IGD, MGD", D²HR and VLCP. We employ two diffusion architectures, U-Net-based Stable Diffusion (SD) and Transformer-bant based DiT, for distillation. We also use them as baselines to demonstrate the advantage of the diffusion priors. All results are reported under either hard-label soft-label evaluation pro- tocols, as specified by the benchmarks. Further experimental details are provided in the section 4.1
4.1 EXPERIMENTAL SETUP
4.1.1 DATASETS AND BENCHMARKS
We evaluate DAP on range benchmarks that vary in scale, resolution, and task difficulty. Our primary evaluation is conducted on large-scale ImageNet-1K (224 x 224) (Deng et al., (009). To study the effect of inter-class similarity, we further consider two 10-class subsets of ImageNet- IK: ImageNette (Howard 2019a). which consists of visually distinct categories and represents relatively simple task, and Image Woof (Howard 2019b), which contains visually similar dog breeds and thus poses fine tine-grained classification challenge. Additionally, incorporate ImagelDC (Kim et al. 2022) to evaluate performance.
4.1.2 MODELS AND EVALUATION PROTOCOLS
For each dataset, we distill subsets of 10. 50, and 100 images per class (IPC) and assess their utility on downstream classification tasks. Two evaluation protocols are adopted:
Following Chen al. 2025 direetly train classifiers from scratch using the distilled images with ground-truth labels (one-hot labels). We evaluate on three commonly used architectures: ConvNet-6, ResNetAP-10, and ResNet-18.
Hard-label protocol:
Following Sun et al. 2024), we provide soft labels via pre-trained clas- sifiers (e.g., ResNet-18). This protocol is crucial for challenging datasets such as ImageNet- IK, where training from scratch on few synthetic images is relatively difficult.
Soft-label protocol:
To demonstrate the compatibility of DAP, conduct experiments on a) Stable Diffusion V1.5 with the U-Net backbone, and b) DiT-XL/2-2256 with the transformer.
4.1.3 OTHER DETAILS
All experiments were implemented in PyTorch and conducted with NVIDIA A40 GPUs. For fair comparison, we reproduce baseline methods under the same setup. The reported results follow these conventions: a) For DAP and reproduced baselines, we report the mean stanaal deviation over three runs. b) For other methods, we report results from the original papers. c) In tables, the best result is highlighted in bold, while the second best is underlined. Practically, DAP does not require selecting any specific trainle samples before sampling.
4.2
COMPARISON WITH SEATE-OF-THE-AAE METHODS
4.2. RESULTS ON DIT
We begin with ImageNet- the most widely adopted benchmark for generative dataset distillation. Across both IPC (Images Per Class) settings, DAP consistently achieves the best results, demonstrat- ing its superiority in large-scale DD tasks. As listed in table DAP achieves 49.1% Top-1 accuracy at IPC10, exceeding the strongest baseline IGD and MGD* by 3.5%. With more distilled samples, DAP further improves to 62.7%, establishing superior results this challenging benchmark.
To examine robustness across different scales and architectures, we also evaluate on ImageNet sub- sets, including ImageNette and ImageWord (table 3). DAP again outperforms almost all competing

---

Published as a conference paper at ICLR 2026
Table 2: Top-1 Accuracy on ImageNet-IK. The results are evaluated with soft-label protocol based on ResNet- 18.
Datasct
IPC
SRc"L
G-VBSM
RDED
Minimax
DIT
IGD
MGD*
D'HR
VLCP
DAP
21.3g0.
31.420.5
22.0go.1
44.320.5
39.6g9.4
45.5
45.6
44.3
46.Z.04
49.1g1
ImageNet. IK
50
16.8_0.
51.810.s
36.5_0.1
58.610.3
52.9_a.
59.81e.3
00.2
59.410.
00.5
62.7
methods. An exception occurs with ResNet-18 at IPCI0, IGD slightly surpasses DAP. This deviation is attributed to the fact that IGD explicitly incorporates ResNet-18 as the surrogate network for its influence-guider sampling. thereby introducing an inductive bias favoring the specific architecture. While this bias yields localized gains, it also risks overfitting (see table (5). In contrast, DAP does not rely on architecture-specffic heuristics and remains effective across multiple backbones.
Table 3: Top-1 Accuracy ImageNette and ImageWoof. The results evaluated with hard- label protocol.
Dataset
Model
IPC
Random
DM
DIT
Minimax
IGD
MGD*
DAP
Full
10
46.0
49.8
56. 56.221.3
58.2
61.9
56.2
64.8
ConvNet-6
50
71.8
70.320.
74.1
76.9209
80.9.0.9
79.0
82.2
94.3go.3
100
79.9
78.520s
78.210.3
81.1
84.5.0.7
84.4
85.7
24.2 a
60.2
62. 20.8
63.2
66.5g1.1
66.4gz.4
67.8
Nette
ResNetAP-10
50
7..
76.7=L.
76.9
78.2
81.0.1.
79.5
82.3.0.7
94 91.6.
100
81.1
80.9207
80.1
81.3
85.2.00.5
85.0g0.4
86.0g2.1
10
15.8
60.920.7
62. 62.5g0.9
64.92..a
67.7
61.2+1.4
66.440.5
ResNet-18 18
50
75.8
75.0
75.220.9
78.1
81.0
80.82aa
82.8.
95.3go.6
100
82.0
81.5+0.4
77.8+0.6
81.3g0.7
84.4.0.a
83.7
85.5
25.
27.6
32.320.
33.5
35.0gn.s
34.7.1.1
37.6.0.9
ConvNet-6
50
41.9 1..
43.8+1.1
48.5+1.3
50.7=1.8
54 64.2.0.7
54.5
55.8.:0.4
85.9
100
1..3
50.1
54.2
57.1
61.
60.1
62.4;:12
10
31. 6...
29.8
30 00..9
39.6
41 41.0.o.s
40.4
41.8.o.7
Woof
ResNetAP-10
50
50.1 1..6
47.8
55.811.1
59.8
62.7.1.2
56.5
63.320.5
87.210.
59.2.0.9
59.8
62.5.n.9
66.821.2
69.7.0.9
66.5.1.
70.8
30.9
30.220.
40.62n.
42.2
44.8
38.511.
43.9
ResNet-18 18
50
54.0
53.910.7
07.4
60.5
62.0
58.3
63.2.0.7
89.0g0.s
100
63.610.
64.920.7
62.310.5
67.4
70.6.1.8
68.8g0.7
71.6
4.2.2 RESULTS ON STABLE DIFFUSION
We next apply DAP to Stable Diffusson (SD) as the generative backbone. As shown in fig. 3 DAP consistently surpasses the baseline MGD and vanilla Stable Diffusion across all datasets and IPC settings. For instance, DAP reaches 81.4% accuracy on ImageNette with IPC50, approaching the accuracy of training the full dataset while using only fraction of the data size.
50
50
50
DANDUU
CANOUSI
140
10312
PCSS
(a) ImageNet-
(b) ImageNette
(c) ImageWoof
(d) ImageNet-IDC
Figure 3: The comparison results on Stable Diffusion. The results are evaluated with both hard- label (HL) and soft-label (SL) protocols based on ResNet-18. The results of SL protocol are marked with a light blue background, while those without background color are from HL protocol.
A surprising finding arises when comparing hard-label and soft-label protocols. Most previous meth- ods achieve competitive results only under soft-label supervision, whereas DAP already matches or
8

---

Published as a conference paper at ICLR 2026
surpasses them under the stricter hard-label supervision. This demonstrates that the representative- ness prior substantially improves the quality of distilled datasets, even without auxiliary supervi- sion. Moreover, DAP maintains robustness under domain shifts between the SD pre- pre-training dataset (LAION (Schuhmann et al. 2022)) and the distilled dataset (ImageNet), further highlighting its abil- ity to leverage diffusion priors to bridge domain gaps, which is not observed in existing approaches.
4.3
ANALYSIS OF DIFFUSION PRIORS
4.3.1
GENERALIZATION PRIOR
Many existing methods overfit to the distillation settings and suffer performance degradation when the dataset scale reduced or the evaluation architecture changed. As listed table we obtain IPC50 and IPCI0 datasets by subsampling them from IPC100 datasets rather than generating them specially. IGD and MGD² suffer degradation under this reduction, whereas DAP preserves accuracy across scales without noticeable performance loss. This generalization indicates that DAP captures sufficient transferable knowledge rather than memorizing samples at a fixed scale.
Table 4: A study dataset scale reduction. The results Top-1 Accuracy evaluated with hard- label protocol. The failure cases (degradation > 5% compared to table are marked in blue.
ImageNette
ImageWoof
Model
IPC
IGD
MGD
DAP
Full
IGD
MGD*
DAP
Full
10
59.8 12.3
54.211.9
64.5
32.6
27.0 27.011.2
36,5
ConvNet-6
50
79.8
77.0+1.3
80.1
94.320.4
53.4
51.40s
53.1
85.9
100
82.810.6
83.7+...
85.7
60.2
58.810.8
62.4
63.2
50. 50.2+1.6
66.1+0.4
35.641.7
31.8+144
37.9
ResNetAP-10
73.4
79.0+
79.8
94.6.0.5
60.4
58 18.661..
62.6:0.6
87.2
82.5g1.2
83.0
86.0
66.8+0.9
64.920.4
70.8
10
62. 62.612.1
56.011.8
63.7
35.2
29 29.812.3
39.4
ResNet-18
50
78.4
78.8
80.4
95.310.6
19.3
59.4
59.7 19.7
89.01a.6
100
83.611.1
84.2
85.5
68.8 68.810.s
67 67.811.1
71.610.9
We further evaluate cross-architecture generalization in table 5 The distilled datasets are trained with soft-labels provided by ResNet-18 and tested on other architectures, including ResNet- 101, MobileNet-V2. EfficientNet-BO, and Swin Transformer. While baselines show performance drops due to inductive bias on the architectures, DAP consistently achieves the highest accuracy across all cases. These findings confirm that representativeness prior enables architecture-agnssee DD.
Table 5: A study on cross- arestteccie generalization. The results are Top-1 Accuracy on ImageNet-
IK evaluated with soft-label protocol.
ResNet-10 101
MobileNet- V2
EfficientNet-BB
Swin Transformer
Method
IPC10
IPCS0
IPC10
IPC50
IPC10
IPC50
IPC10
IPC50
RDED
48.341.0
61.2 6l.22n.4
40.4+0.1
53.3
31.0_a.1
58.5
42.3
53.240.8
IGD
52.6
66.2
39.2+0.
57.8
47.7
62.0
44.1
58.6
DAP
54.9 10.9
68. 68.1
43.1
61.4
49.7
65.2
48.3
61.7 +0.4
4.3.2 DIVERSITY AND REPRESENTATIVENESS PRIORS
To investigate whether DAP enforces diversity and representativeness priors the distilled datasets, visualize the data distribution using t-SNE alongside both the training and test sets. Figure 4 reveals that the synthetic data aligns well with the training set while generalizing to the test set, demonstrating that the DAP accurately match the underlying data manifold. Moreover, the em- beddings show intra-class diversity and inter-class separability, indicating that the distilled datasets capture meaningful variability without sacrificing discriminability.
Across all benchmarks and analyses, DAP achieves competitive performance and surpasses existing DD methods. The improvements arise from the combined effect of diffusion priors: diversity and

---

Published as a conference paper at ICLR 2026
Trining v.s. Synthetic
Test vs. Synthetic
Talling v.s. Synthetic
Text Synthetic
(a) ImageNette- Training
(b) ImageNette-Test Test
(c) TrageWoof-Trrime
(d) ImageWoof- Woof-Test
Figure 4: Visualization results of t-SNE. We compare the feature distribution of real (training and test set) versus synthetic data under IPC50. Dark/Light points: Synthetic/Real samples.
generalization priors contribute to broad coverage and cross architerche transfer. Meanwhile, the representativeness prior enforces information alignment with the real dataset. Moreover, DAP in- troduces no extra training cost, which makes the approach both efficient and scalable in scenarios where deployment architectures agnostic. We also discuss the sampling costs in section A.3.7
4.4 ABLATION EXPERIMENTS
We conduct ablation studies to investigate the influence of feature layer selection and guidance scale 7 in representativeness guidance. We observe from fig. thathat that the "Mid" layer of the U-Net yields the strongest results. For DiT, the most effective features originate from the early transformer blocks (e.g., the 12th layers shown in fig. 5bb. which outperform those in later layers. Despite this difference, both cases consistently reveal that the final output layers are suboptimal for repre- sentativeness guidance, as they prioritize distribution alignment over representativeness. Regarding y, we find that increasing its value generally enhances representativenesss, as reflected by improved downstream accuracy in figs. [c|] and 5d and the sector areas in fig. 2 but excessive scales distort the gradient field of the sampling process and bias the generation trajectory, thereby diminishing the contributions of diversity and generalization priors and leading to performance degradation.
CCC
BC56
9929
PC50
Imapehieite
Imragaliette
Imapetintte
ACCUIDA yo
Aecursce
Image
Imanellest
Imageittof
Imaneleast
Layer
Layer
SD: U-Net
(b) DIT: Transformer
(c) SD: Guidance scale
(d) DiT: Guidance scale
Figure 5: Ablation studies under ResNet-18, (a-b) Top-1 I Accuracy under different backbone layer selection. (c-d) Top-1 Accuracy under varied guidance scale 7
CONCLUSION
This paper introduces Diffusion as Priors, a framework for dataset distillation that leverages the in- herent priors of diffusion models. We iddeniffy diversity, generalization, and representativeness priors in diffusion models, and demonstrate how they can be integrated to guide the generation process. Representativeness prior is further formulated through kernel- energy guidance, enabling the sampling process to align more information with real data. Extensive experiments on ImageNet- IK and its subsets demonstrated that DAP achieves state-of-the-arr results, preserves generalization under scale reduction, transfers effectively across architectures, and remains robust under domain shifts, making the approach both efficient and scalable. Future work may fall in extending diffu- sion priors to other powerful models (e.g., FLUX, Stable Diffusion 3.5) and exploring applications beyond vision, including language, video, and multimodal datasets.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work uses only publicly available datasets, including ImageNet- IK and its subsets (ImageNette. ImageWoof, and ImagelDC). No human subjects, private data, or sensitive information are involved.
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure reproducibility. All theoretical results are presented with complete proofs in section A.2 Details of datasets, backbone architectures, hyper-parameters, and evaluation protocols are provided in section [.1 while algorithms and [2 specify the guided sam- pling procedure. Additional visualizations and ablation results are included in section A.4 to further support empirical findings. These resources ensure that theoretical and experimental results can be independently verified.
ACKNOWLEDGMENTS
This work was supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM101), CPSF (No. 2025M771546) and NSFC (No. 625B200186).
REFERENCES
Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela Van Der Schaar. How faithful is your synthetic data? sample-level metries for evaluating and auditing generative models. In International conference on machine learning, pp. 290-306. PMLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Gener- alizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3739-3748, 2023.
Huanran Chen, Yinpeng Dong, Shitong Shao, Hao Zhongkai, Xiao Yang, Hang Su, and Jun Zhu. Diffusion models certifiably robust classifiers. Advances Neural Information Processing Systems, 37:50062-50097, 2024a.
Huanran Chen, Yinpeng Dong, Zhengyi Wang. Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. In Proceedings of the 41 International Con- ference on Machine Learning, pp. 6643-6665, 2024b.
Mingyang Chen, Jiawei Du Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence-guided diffusion for dataset distillation. In The Thirteenth International Conference Learning Repre- sentations, 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 17057-17066, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-877 8894,, 2021.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case regres- sion. In The 38th Annual Conference on Neural Information Processing Systems, 2024.
Dov M Gabbay, Paul Thagard, John Woods, Prasanta Bandyopadhyay, and Malcolm Forster. Philosophy of Statistics, volume 7. Elsevier, 2011.

---

Published as a conference paper at ICLR 2026
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, pp. 15793-15803, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jeremy Howard. Imagenette: A smaller subset of easily classified classes from imagenet, March 2019a. htRt:ttt:://gtt http:///wi..ccemeaaaaaaaaaaaaaee
Jeremy Howard. Imagewoof: a subset of 10 classes from imagenet that aren't so easy to classify,
March 2019b. |tt_a//eee//gg.aaaaaccaaaaaaaaaa
James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. Synthetic data-what, why and how? arXiv preprint arXiv:2205.03257, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11102-11113 PMLR, 2022.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7,7)33,000 2015.
Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(1):17-32, 2023.
Ping Liu and Jiawei Du The evolution of dataset distillation: Toward scalable and generalizable solutions. arXiv preprint M.kiv:2502.06675 2025.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data. In First Conference on Language Modeling, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-577 5787, 2022.
James Mercer. Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philosophical transactions the royal society of London. Series A, containing 1909.
papers of mathematical or physical character, 209(441-453::41554444
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language 00300358-03766 2023.
models. Advances in Neural Information Processing Systems,
Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, YiR R Fung, al. Scaling laws of synthetic data for language models. arXiv preprint arXiv:2503. 19551, 2025.
Jeffrey Chan Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, and Mubarak Shah. Mgd3: Mode-guided dataset distillation using diffusion models. In International Confer- ence on Machine Learning, pp. 52861-52876 PMLR, 2025.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278 25294,, 2022.
Yifei Shen, Xinyang Jiang, Yifan Yang, Yezhen Wang, Dongqi Han, and Dongsheng Li. Understand- ing and improving training- free loss-based diffusion guidance. Advances in Neural Information Processing Systems, 37:108974-100002 2024.
12

---

Published as a conference paper at ICLR 2026
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Paperot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631:755-759, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D'4m: Dataset distillation via disentangled diffusion model. In IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 5809-5818. IEEE, 2024.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9390-9399, 2024.
Hoang Thanh-Tung and Truyen Tran. Catastrophic forgetting and mode collapse in gans. 2020 international joint conference neural networks (ijenn), pp. 1-10. IEEE, 2020.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of Ilm scaling based on numan-generann data. In Proceedings of the s1st International Conference Machine Learning, 2024.
Haoxuan Wang. Zhenghao Zhao, Junyi Wu, Yuzhang Shang, Gaowen Liu, and Yan Yan. Cao .2: Rectifying inconsistencies in diffusion-based dataset distillation. arXiv preprint WrXiv:2506.22637, 2025.
Kai Wang. Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang. Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12196-12205, 2022.
Kai Wang, Jianyang Gu. Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.04700 2023.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 1.159,,.1.99,,, 2018
William Yang, Ye Zhu, Zhiwei Deng, and Olga Russakovsky. What is dataset distillation learning? In Corty-First International Conference on Machine Learning, 2024.
Xingyi Yang and Xinchao Wang. Diffusion model as representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18938-18949, 2023.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE transactions on pattern analysis and machine intelligence, 46(1):150-170, 2023.
A.C. Zaanen. Linear Analysis. Bibliotheca Mathematica. N-H Publishing Company, 1964.
Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching. Proceedings of the International Conference Learning Representations (ICLR), 2021.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7856-7865, 2023.
Lin Zhao, Yushu Wu, Xinru Jiang, Jianyang Gu, Yanzhi Wang, Xiaolin Xu, Pu Zhao, and Xue Lin. Taming diffusion for dataset distillation with high representativeness. arXiv preprint mrXiv:2505.18399 2025.
Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Tao Dai, Meikang Qiu, and Shu-Tao Xia. Hierar- chical features matter: A deep exploration of gan priors for improved dataset distillation. arXiv preprint arXiv:2406.05700 2024.
Yawen Zou, Guang Li, Duo Su, Zi Wang, Jun Yu, and Chao Zhang. Dataset distillation via vision- language category prototype. arXiv preprint 25Xi::2506.238800 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
Appendix organization:
Section A.1
Background
A
Dataset distillation
A.1.2
Generative dataset distillation
Section A.
Proofs
A.2.1
Validity of kernel-induced distance
A.2.2
Distance factorization
Section A.3
Discussions
A.3.1
Compatibility of DAP
A.3.2
Guidance on noisy latent
A.3.3
Kernel selection
A.3.4
Comparison with recent methods
A.3.5
Cross-datasets generalization
A.3.6
Early Stop strategy
A.3.7
Sampling- scaling
Section A
Visualizations
A.4.1
Representativeness guidance scale
A.4.2
Representativeness comparison
Section A.5
Disclosure of the use of Large Language Models
A.1
BACKGROUND
A.1.1 DATASET DISTILLATION
Systematic analysis research in dataset distillation reveals two paradigms: a) traditional matching- based approaches focused on level-level optimization, and b) modern generative frameworks em- phasizing distribution learning (Yu et al. 2023: Lei & Tao 2023 Liu Du (025). Traditional methods adopt an "imitation" philosophy, involving continuous pixel optimization align model behavior, such as gradients, feature distributions, or checkpoints between synthetic and original data (Zhao & Bilen 202I; Waan al.) 2202) Zhao et al., 2023 Deng et al. 2024). In contrast, generative frameworks prioritize improving dataset quality through fidelity and diversity metrics. These approaches extract key informational patterns from source data, enhancing the realism and generalization of distilled datasets. We will examine the related work in the following subsection.
A.1.2 GENERATIVE DATASET DISTILLATION
Generative dataset distillation utilizes models such as Generative Adversarial Networks (GANs) and Diffusion models (DMs) to synthesize compact and informative datasets. Unlike pixel optimization methods, which are limited to small-scale, low-resolution data due to computational costs, genera- tive techniques support large-scale, high-resolution applications. This flexibility promotes sample diversity and better generalization across model architectures. This section reviews the two primary categories of generative dataset distillation methods: CANNbased and Diffusion-bann based approaches.
GAN-based approaches. GANs serve as foundation models for dataset distillation In early re- search. DiM (Wang et al., 2023) condenses dataset information into a conditional GAN, enabling sample synthesis from random noise during deployment. GLaD (Cazenavette al. 2023) enhances cross-architecture generalization by distilling data into the latent space of pre-trained models like StyleGAN (Karras al. 2019). H-PD (Zhong et al. 2024) introduces hierarchical parameterization distillation, optimizing across latent spaces in GANs to capture hierarchical features from the initial latent space the pixel space.
14

---

Published as a conference paper at ICLR 2026
Diffusion-based approaches. Diffusion- based methods leverage diffusion models to improve dataset distillation. For example, Minimax diffusion (Gu et al. 2024) fine-tunes a diffusion model with minimax criteria boost representativeness and diversity. VLCP [Zo et al. 2025 constructs text prototypes to enrich the labels with semantic information and then fine-tunes the DMs with image-text pairs. D'M (Su et al. 2024) disentangles feature extraction and generation via Training- Time Matching (TTM) with category prototypes. IGD (Chen et al. (025) guides the sampling process of pre-trained diffusion models using function combining trajectory influence and diver- sity constraints, generating synthetic data without retraining. Additionally, MGD3 (Santiago et al., 2025) enhances diversity by identifying latent space modes and directing data toward them during sampling. In order to enhance the objective and conditional consistency of the distillation process, CaO2 (Wang et al., 2025) ) employs target-guided sample selection to optimize the latent condition- ally. D*HR (Zhao et al. 2025) utilizes DDIM inversion to map the image latents to the Gaussian domain, then aligns the representative latents with the high-normality Gaussian distribution with their proposed sampling scheme.
A.2 PROOFS
A.2.1 VALIDITY OF KERNEL- DISTANCE
Theorem A.I. Ket x X R be PSD kernel. Then the induced distance measure Dx(r, v) = [K(,, 2) + K(y.v) - 2K((x, w))
satisfies:
Non-negativity: Dx( v) 2 0, and (2, y) if and only if x y.
2. Symmetry: Dx(a,y) 2x)
3. Triangle inequality: For any x, y, E X. Dx(x,2) Dx(z,0)) 2 Dx(x,v).
Proof. Since IC is positive semi-definite, by Mercer's 's theorem (Mereer 1909) there exists a repro- ducing kernel Hilbert space H. and feature map Φ X' + H. such that K(x,,) (0(z), (v))..
(10)
Therefore,
Dx(æ, v)2 = K(x, + K(y,8) 2K(x,
(11)
(((x), (Ø(v), p(y)) 2(0(x), 6(1))x
(12)
(1((x) (1)
(13)
Thus,
Dx(x,0) = (1x(x) - bl
(14)
Since the norm in Hilbert space || 2e is valid metric, it satisfies:
Non Non-negativity and identity of indiscernibles: [10(x) - Φ(v))l 2 0. and [1((x) - Φ(v) iff 6(x) = o(2). which implies = y.
Symmetry: []ő(z) <(v) (1ő(w) 6(x)]l
Triangle inequality: [|d(()) p(v))l S [|x(z) - d())ll + [[((2) <(y))l for any z.
Therefore, Dx is valid metric induced by the kernel IC.
A.2.2 DISTANCE FACTORIZATION
Theorem A.2. Let X X R be Mercer kernel, then the induced chordal distance Dic can be factorized as (x(x,)) y) = do (0 x (x, y). where ó is a feature mapping and is a simple norm in Hilbert space.
15

---

Published as a conference paper at ICLR 2026
Proof. By the reproducing property of the reproducing kernel Hilbert space Hi. we have K(x,, v) (f(x)), H(v))
(15)
Therefore,
Dx(2, ()( K(æ, 2) K(y.y) 2K(x, y) (D(z), ((z)) + (1(v), ((y)) 2(0((x), m(
(16)
(17)
(1(()) (yy)) li
(18)
Taking square roots yields
Dx (z, y) = [1 (zvll
(19)
we set f = and d((u, v) = 1|u - v/ Hi. then clearly
Dx(x,y) d d(f(z), d (f f)
(20)
A.3 DISCUSSIONS
A.3.1 COMPATIBILITY OF DAP
Compatibility with codebase. DAP is fully implemented using the native components of the dif- fusion model itself, without relying on any additional modules or external dependencies. This care- fully designed approach not only preserves complete compatibility with existing diffusion architec- tures but also ensures that the method can be readily adopted across diverse codebases. As a result, DAP can be seamlessly incorporated into widely used diffusion libraries, such as the Di f fusers library in Hugging Face, thereby promoting both reproducibility and broad applicability in contem- porary research and practical deployment scenarios.
Compatibility with other methods. Since the DAP pipeline works orthogonally with the existing approaches, it exhibits strong compatibility, allowing complement them without interference. Before empirical results, we analyze the sources of representativeness priors employed by different methods in table Minimax introduces rep- resentativeness via fine-tuning under the supervision of the proposed training loss, D'M and MGD3 capture rep- resentativeness through clustering algorithms, IGD and CaO2 uses pre-trained classifier, D²HR calculates the key statisties including mean, standard deviation and skew- while DAP exploits the representativeness priors embedded in diffusion models. To validate the compati- bility of DAP, incorporate into Minimax for instance (see table 7). The addition DAP consistently enhances the performance of distilled samples. These results indicate that DAP can serve versatile and modular enhancement, improving the performance of DD approaches while preserving its intrinsic advantages.
Table 6: The source of representative- ness knowledge from different genera- tive DD methods.
Method
Representativeness
Minimax
Training Loss
D'M
Clustering
MGD
Clustering
IGD
Pre-trained Classifier
CaO₂
Pre-trained Classifier
D'HR
Statistic Metric
DAP
Diffusion Prior
Table 7: Top-1 Accuracy on ImageNette and Image Woof. Evaluated with hard-label protocol.
ImageNette
ImageWoof
Model
IPC
Minimax
Minimax-IGD
Minimax-DA
Minimax
Minimax-IGD
Minimax DAP
10
58 2
58.8-1.
64.2+1.4
33.5+14
36.2
38.2+0.8
ConvNet-6
50
76.9
82.3
83.5
50.7
55.7
55.9
10
63.2+11
63.5
66.1
39.621.2
43.3+o.3
43.5
ResNetAP-10
50
78. 78.2+0.7
82.3 1.1
83.7+1.a
59.8
65.0+0.8
66.4
64..940.6
66.2
66.9
42.2
47.2+1.6
45.4
R8
78.1
82.0
82.5
60.5go.5
65.4+1.
65.8
16

---

Published as a conference paper at ICLR 2026
A.3.2 GUIDANCE ON NOISY LATENT
We adopt the VP-SDE combined with the DDIM sampling algorithm, which enables a deterministic and efficient approximation of the reverse diffusion trajectory. A detail in this setting is the choice of the feature representation for prior guidance. Under DDIM dynamics, the conditional estimate of Zoje can be expressed an affine transformation of the current noisy state: Eglt = Ox Been St, !), where ot, St are deterministic coefficients and eg is the score predictor (Song et al. 2021 From the perspective of reverse dynamics, this relation holds tion, implying that the gradient fields induced by guiding 2t and guiding 2ojz are approximately equivalent (see fig. (). Hence, instead of explicitly computing the denoised estimation Eojt, di- rectly apply guidance on the noisy latent 24, while avoiding the computational overhead of explicit decoding process each timestep.
first-order approximation under lineariza-
Guided Sampling Trajectory
Output Sample
ZT
Z
Initial Noise
Guided Denoising Trajectory
Original Sampling Trajectory
ZajT
Denoising Output
Noise Distribution
Original Denoising Trajectory
Data Distribution
Figure 6: A sketch map of the relationship between 20je and z4.
A.3.3 KERNEL SELECTION
Besides the linear kernel, we also install our DAP with other Mercer kernels, such as the Radial Basis Function kernel (RBF, also known the Gaussian kernel):
|| y y13
K(x, v) = exp
(21)
202
The bandwidth a controls the sensitivity: small a emphasizes fine groo-graned local features, whereas large approaches the behavior of the linear kernel. Based on eq. (21). the induced distance becomes
[)(z)) - (v) K(z, x) K(y. v) 2K((x 9) 2exp (- || v112
(22)
20
which is a non-linear function of the Euclidean distance.
Table 8: Top-1 Accuracy with different Mercer kernels. The results are evaluated on ResNet-18 with hard-label protocol.
RBF kornel (p-0.5)
00
RBF kernel (o=1.0)
RBF kernel (p-2.0)
Kemelinduced
15
Kernel
ImageNette
Image Woof
10
IPC10
IPC50
IPC10
IPC50
Linear RBF(o 0.5)
66.440.5
82.8
43.940.9
63.2 63.2
65.7
82.5
44.6
63.8
RBF(a 1)
64.1
83.1
44.1
60.940.3
Euclidran distance ||-l||
RBF(a 2)
66.0
82.2+1.0
43.7
62.642.
Figure 7: Curves of different Mer- kernel-induced distances.
17

---

Published as a conference paper at ICLR 2026
As drawn in fig. 7 the RBF-induced distance grows quickly for small differences and saturates for large differences, effectively compressing large deviations while being sensitive to local differences. reeeeals that the distillation performance of RBF kernel is comparable to that of linear kernel. To avoid introducing additional hyperparameters, we recommend using the linear kernel due to its simplicity and tractability.
A.3.4 COMPARISON WITH RECENT METHODS
the distillation results for ImageNet subsets on DiT. CaO2 surpasses DAP in several cases, but this advantage arises from differences evaluation protocols. The CaO:2 paper reports the best accuracy across soft-label and hard-label protocols, selecting the protocol that yields higher performance. In contrast, D'HR, VLCP, and DAP consistently adopt the hard-label protocol in this experiment. Since the soft-label protocol typically leads to higher accuracy. the results of CaO₂ likely benefit from this more permissive approach. Under a consistent hard-label evaluation setting, DAP remains competitive and performs better in most cases compared to these recent methods.
Table Top-1 Accuracy on ImageNette and ImageWoof. f: The results are evaluated with hard-
label protocol except for CaO2.
Dataset
Model
IPC
Random
DM
DIT
CaO
D²HR
VLCP
DAP
Full
10
46.0go.5
49.8
56.221.3
64.8
ConvNet-6
50
71.8
70.3
74.1
82.2
94.3
79.9
78.5
78.22m.
85.7 11.7
54.2
60.2g0.7
62.82n.8
64.8
67.8
Nette
ResNerAP-10
50
77.311.
76.711.0
76.910.
81.220.
82.3
94.610.5
100
81.1g0.6
80.920.7
80.1g1.1
86.0
10
55. 8g1o
60.9 20.7
62. 02.5+0.9
65.0ga.
66.420.5
R8sNet-18
75.8
75.0 1..00
75.2+0.9
84.5gaa
82.8.1.1
95.3
100
02.0
81.5gu.4
77.820.6
85.5
2
27. G21.2
32.3
34.8-22
37.6.0.9
ConvNet-6
50
41. 41.9
43.8 43.8
48.5
544.520.8
55.8 $5.8
85.9
100
52. 1.5
50. 50.1
54.2+1.5
62.7
62.4g1.2
10
31 6.0.a
29.8 121.
39.0+n.9
40.7
39.5g1.5
41.8
Woof
ResNetAP-10
1+14
47 8+1.2
55.8+1.1
59.3
57.3
63.3g0.5
87. 27.2
100
59.2
59.8
62.540.9
64.7.0.3
65.720.5
70.8
10
30.9
30.2
40.6+0.6
45.6
39.6
39.9+2.
43.9
R8sNet-18
50
54.0.0.s
53.940.7
57.420.7
68.9
57.6
58.921.5
63.2
89.010.6
100
63 63.6.0.5
64.920.7
62.3+n.5
66.8
68.3g0.4
71.6
A.3.5 CROSS-DATASET GENERALIZATION
We posit that the cross-datasets evalua- tion is essential to measure the general- ization and versatility
Table 10: Top-1 Accuracy on Tiny ImageNet (IPC50).
The results are evaluated with soft-label protocol.
a DD method, which is overlooked by most DD meth- ods. According to Su et al. (2024), we ex- tract 200 categories, which are predefined in Le & Yang (2015), from the ImageNet- IK dataset distilled by DAP as the distilled Tiny-ImageNet dataset. Table IO shows that the extracted subsets (end with "-G") maintain strong validation performance on the target set, thereby confirming that our distilled data not only preserves the utility of the original dataset but also supports effective reuse across datasets. The results highlight the advantage of DAP: the distilled dataset is not tied to single dataset domain but can flexibly transferred and reused.
Method
ResNet-18
ResNet-50
ResNet-101
Full
61.9
62.0
62.3
S
44.0
47.7
49.1
D
46.2
51.8
51.0
D'M-G
46.8
51.9
53.2
DAP-G
50.3
53.6+1.0
54.7+
A.3.6 EARLY STOP STRATEGY
In the guided sampling process, employ the early stop guidance mechanism, which enhances sampling quality by only guiding earlier diffusion timesteps than the entire timesteps. thereby pro- viding better trade-off between sample diversity and fidelity (Chen et al. 2025 (Santiago et al.
18

---

Published as a conference paper at ICLR 2026
2023).
Besides, applying representativeness prior guidance in the early stage of the sampling trajec- tory also reduces the sampling cost (refer to section|A...7) We summarize the sampling process with conducted experiments with
early stop strategy in algorithm To evaluate its effectiveness, different stopping parameters L stop. The mechanism deactivates guidance for timesteps & < 1sstpp in the reverse process, ksop 0 means complete guidance, while ssop 50 represents guidance. The qualitative and quantitative results, as illustrated in fig. g indicate that Etop = 25 yields the best performance.
Algorithm 2 DAP Sampling with Early Stop(VP-SDE)
trainie within class c, pre-trained diffusion model Es, layer output
Require:
Noisy data samples zi @ from diffusion backbone network, a Mercer Kernel induced distance measurement d, energy- based guidance scale y, pre-defined noise scales St, early stop parameter ttop
1: 2r ~ I)
2: for T. ... do
3:
6 <~~ <0, if 1, clse =0 i = (2 VI 3. BrEg (sr; 1) + VBc
4:
5:
if't S tsop then
6:
1t-1 = .t-1
Stop Guidance
7:
else
8:
21 (xx), 2z train
trainle ó(z; trainley
Diffusion as representativennss priors
9:
9t Vay dl 2z
10:
Ft-1 = 8t-1 79t
# Guided sampling
end if
12: end for
Output: To
# The distilled sample of class c.
65
Accuracy
60-
Tog-1
50
Stable Diffusion
tstop = 0
tstop 25
tstop 50
(a) Top-1 Accuracy on ImageNette under different tstop. The results are evaluated on ResNet-18 with hard- label protocol.
(b) Visualizations of the distilled samples with different fstop-
Figure 8: Ablation study on tstp selection.
A.3.7 SAMPLING-TIME SCALING
DAP does not introduce additional training costs, since no external pre-training or fine-tuning is required. The representativennss prior is directly derived from the pre-trained diffusion backbone. However, to inject this prior during sampling and improve data quality, we must extract features from the noisy training data æ:frain using the backbone network d. This step inevitably brings additional sampling time overhead, which must be acknowledged.
To quantify this overhead, report the GPU memory and the sampling speed for different data sizes in table II While sampling-time scaling introduces overhead, the cost remains manageable and predictable single GPU card.
19

---

Published as a conference paper at ICLR 2026
Table 11: The overhead of sampling-time scaling (tstop 25). The Top-1 Accuracy is evaluated on ImageNet-1K with hard-label protocol (IPC10). The memory and speed are reported on x A40.
Stable Diffusion
DIT
Data Size
GPU Mem.(GB) Speed(s/ite)
Acc(%)
GPU MGm.(GB)
Speed(s/iter)
Acc(%)
500
35.9
32.1_0.6
15.3
44.6+2.4
1000
23.1
39.9
1.3
10.6
24.0
48.8
1500
47.1
=0.7=15
32.3
49.1
A.4 VISUALIZATIONS
A.4.1 REPRESENTATIVENESS GUIDANCE SCALE
As suggested by our ablation results (see figs. and 5d in section increasing within a moder- ate scale effectively boosts representativeness prior, leading improved downstream performance. However, excessive introduces adverse effects. Over-amplifying the representativeness prior dis- torts the sampling trajectory, resulting in over-constrained generations that sacrifice diversity and generalization. Since the gradients of the other two priors are fixed, an imbalanced emphasis on representativeness suppresses their contribution, yielding biased and less informative images. This trade-off is clearly visualized in fig. g
DiT
SD
guidance scale 2=20
guidance scale 7= 1000
guidance scale 1=1.5
guidance scale ny=1
(a) ImageWoof: Rhodesian ridgeback (n02087394)
guidance scale 2==20
guidance scale >=1000
guidance scale 1=1.5
guidance scale 1
(b) ImageWoof: English foxhound (a02089973)
Figure 9: Samples distilled by DiT (left three columns) and SD (right three columns). The excessive representativeness guidance scale will generate representativeness bias in the sampling trajectory, affecting the diversity and fidelity of the synthetic images.
20

---

Published as a conference paper at ICLR 2026
A.4.2 REPRESENTATIVENES COMPARISON
To provide intuitive comparison, visualize the distilled datasets obtained from different meth- ods, as shown in fig. TO For each group, we compute the distance measure defined in eq. ( and report its representativeness (xx d(())):o(p)) The results demonstrate that while all methods can preserve semantic information thanks to the powerful DMs, the images distilled by DAP consis- tently achieve the highest representativeness. This highlights the advantage of DAP in generating distilled datasets that are not only semantically valid but also representative.
DIT
DiT
representaatvenesss-...
represeetaativvnesss1..83 1.80
IGD
IGD
representaatvvenesss 1.93
representaativeness=2.0000
MGD
MGD
representatvenesssI1y 1.97
representtativneess1199
DAP(OUrS)
DAP(OuiS)
reprexentativeness-2e
representattivnesss=2
(a) ImageNette: Gas pump (n03425413)
(b) ImageNette: Chain saw (n03000684)
DIT
DiT
representativeneexs... ..87
representativeness:: 1.93
IGD
IGD
reprexentaativnnes=22..9
representativenesss-
MGD
MGD*
reprexxentativeness=2.33
reprexenttativenexs=2.111
DAP(Ours)
DAP(Ours)
representativeness=..66
represeettativeness=221111
(c) ImageWoof: Rhodesian ridgeback (n02087394)
(d) ImageWoof: Samoyed (n0211188)
Figure 10: Visualization results of different DD methods. At the bottom of each group, the
pre-trained DiT to calculate the average representativeness values (x10-
A.5 DISCLOSURE OF THE USE OF LARGE LANGUAGE MODELS
Given that the use of large language models (LLMs) is allowed as a general- purpose assist tool, this work utilizes L.MSs polish the sentences of the article. There is no significant role in research ideation and writing to the extent that they cannot be regarded as contributors.
21


Reviews:
[{"review_id": "v5MzWU0EAk", "reviewer": "Reviewer_2soE", "summary": "Inspired by the diffusion classifiers, it posits that the feature extraction capability inherent in a well-trained diffusion model itself constitutes a representativeness prior highly relevant to DD. It hypothesizes that high representativeness corresponds to high similarity between synthetic and original data in the representation space. To formalize this, it employs the Mercer kernel, a specific type of kernel function, to quantify the similarity within feature spaces. The Mercer kernel provides mathematical guarantees of convexity and tractability in optimization, ensuring that the representativeness prior is computationally feasible. Empirically, it defines the representativeness score function as an energy function based on Mercer kernel, which allows to inject the unused representativeness prior into the distilled data through guided sampling.", "questions": "see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It proposes Diffusion As Priors (DAP) and applies it to datasets of varying scales, including large-scale ImageNet-1K and its small subsets.\nBoth quantitative and qualitative results show that DAP significantly enhances the quality of distilled datasets. It validates the theoretical connections between diffusion priors and DD task, while achieving competitive performance compared to other methods.\n\nIt prove the priors in the well-trained DMs meet the diversity and generalization requirements of DD. It derives the overlooked representativeness prior from DMs and formalize it into a kernel-induced distance, which guides the sampling dynamic and improves the quality of distilled datasets. It further shows that by introducing the desired priors, the distilled datasets have the same generalization and transferability as the original ones.\n\nTo investigate whether DAP enforces diversity and representativeness priors in the distilled datasets, it visualizes the data distribution using t-SNE alongside both the training and test sets. It reveals that the synthetic data aligns well with the training set while generalizing to the test set, demonstrating that the DAP can accurately match the underlying data manifold. Moreover, the embeddings show intra-class diversity and inter-class separability, indicating that the distilled datasets capture meaningful variability without sacrificing discriminability.\n\nIt conducts ablation studies to investigate the influence of feature layer selection in representativeness guidance. The cases consistently reveal that the final output layers are suboptimal for representativeness guidance, as they prioritize distribution alignment over representativeness.", "weaknesses": "Diffusion models are widely adopted in dataset distillation to extract features and obtain information. It is similar to use diffusion as priors in this paper. It is better to discuss related works and highlight the differences.\n\nIt is better to compare with more baslines such as [R1,R2,R3], which also adopts diffusion for dataset distillation. \n\n\n[R1] CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillation\n\n[R2] Taming Diffusion for Dataset Distillation with High Representativeness\n\n[R3] Dataset Distillation via Vision-Language Category Prototype", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHdMVc4Lpy", "reviewer": "Reviewer_wTWR", "summary": "This paper proposes Diffusion As Priors (DAP), a novel framework for dataset distillation that leverages inherent priors in pre-trained diffusion models. The authors establish a theoretical connection between diffusion model objectives and dataset distillation requirements, identifying three key priors: diversity, generalization, and representativeness. The main contribution is formalizing representativeness using Mercer kernel-induced distances and incorporating this as guidance during the reverse diffusion process without requiring model retraining. Extensive experiments on ImageNet-1K and its subsets demonstrate that DAP achieves state-of-the-art performance while maintaining cross-architecture generalization.\n\nThe key novelty is decomposing the conditional score function as: $\\nabla_x \\log p(x|R) = \\nabla_x \\log p(x) + \\nabla_x \\log p(R|x)$, where the first term captures diversity/generalization priors from the pre-trained diffusion model, and the second term introduces representativeness through energy-based guidance using kernel-induced distance measurements.", "questions": "- Could you provide more justification for why kernel-induced distance is the right metric for representativeness? Have you considered alternative metrics like Maximum Mean Discrepancy (MMD) or Optimal Transport distances? How would these compare theoretically and empirically?\n\n- Are there scenarios or dataset characteristics where DAP underperforms? For instance, does it struggle with fine-grained classification, imbalanced data, or out-of-distribution classes?\n\n- The linear kernel is chosen \"due to its tractability\" but Table 8 shows RBF performs comparably. Could you discuss the theoretical implications of different kernel choices more deeply? Does kernel selection depend on dataset characteristics?\n\n- When the method does not have access to full training samples during sampling but instead a handful of subset of them, how will the proposed method perform?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a principled framework connecting diffusion model objectives to dataset distillation requirements through clear mathematical formulations and proofs.\n\n- Extensive experiments across multiple datasets (ImageNet-1K, ImageNette, ImageWoof, ImageIDC), architectures (ConvNet, ResNet, MobileNet, EfficientNet, Swin), and protocols (hard-label, soft-label) demonstrate broad applicability.\n\n - Unlike methods requiring fine-tuning or external training, DAP leverages pre-trained diffusion models directly, making it practical and computationally efficient (no additional training cost).", "weaknesses": "- While the Mercer kernel framework is mathematically sound, the paper does not convincingly argue why minimizing kernel-induced distance in feature space is the optimal objective for representativeness in DD. Alternative formulations (e.g., maximum mean discrepancy, optimal transport) could be equally valid.\n\n- Table 10 reveals significant computational costs during sampling, with speed increasing from 15-36 seconds per iteration depending on data size. This overhead could be prohibitive for large-scale applications. The paper acknowledges this but doesn't propose solutions.\n\n- The method requires access to the full training dataset during sampling for representativeness guidance, which somewhat limits the practical benefits of distillation\n\n- The paper primarily uses linear kernels with brief exploration of RBF (Table 8). Other kernel choices  and their theoretical implications are not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Diy97a1OvV", "reviewer": "Reviewer_1VVS", "summary": "The paper introduces a novel framework that employs diffusion models as powerful generative priors for dataset distillation, the task of compressing a large dataset into a compact synthetic subset while preserving downstream task performance. Unlike traditional approaches that directly optimize synthetic samples in pixel or feature space, this method leverages the diffusion process to model the underlying data manifold, enabling the generation of representative and diverse samples without costly bi-level optimization. Furthermore, the framework enhances representativeness by encouraging the synthetic data to align with real samples in the feature space via a Mercer kernel-based similarity measure. Experimental results on the ImageNet benchmark demonstrate that the proposed approach consistently outperforms state-of-the-art baselines in classification accuracy.", "questions": "1. **Clarification of $x^{\\text{train|c}}_t$ selection.**\nIn Algorithm 1, how are the samples $x^{\\text{train|c}}_t$ obtained or selected before the DAP sampling process?\n\n\n2. **Diversity Comparison with Baselines.**\nThe proposed method (Diffusion as Prior, DAP) demonstrates substantially greater diversity compared to baseline methods such as MGD³ and IGD, as illustrated in Figure 1. Interestingly, while MGD³ explicitly incorporates mechanisms to enhance diversity, the proposed method does not directly encourage the generation of diverse samples. Could the authors elaborate on the underlying reasons for this observed improvement in diversity?\n\n3. **Quantitative and Qualitative Comparison.**\nCould the authors provide a more detailed comparison between DAP and the baseline methods in terms of diversity, representativeness, and classification performance under different IPC (images per class) settings?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper presents a comprehensive literature review and demonstrates promising experimental results, highlighting the effectiveness and practical potential of the proposed approach.", "weaknesses": "Although the proposed method is straightforward and effective, several key implementation details are missing, making the reported performance difficult to reproduce. For instance, the paper does not clearly explain how the training samples are selected to pair with synthetic samples when computing the kernel distance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BcTxgkL4LM", "reviewer": "Reviewer_URQY", "summary": "This paper interprets diffusion models as possessing three key characteristics: diversity, generalization, and representativeness. Building on this interpretation, the authors propose a training-free framework aimed at enhancing the representativeness of diffusion models. The improvement is achieved by incorporating a distance metric based on the Mercer kernel into the backward process of diffusion models.", "questions": "Why didn’t the authors provide an analysis of the computational cost? Since the proposed framework is claimed to be training-free, it should inherently offer computational advantages over training-based methods. However, without quantitative comparisons (e.g., runtime, memory usage, or efficiency metrics).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed DAP method substantially improves the quality of distilled datasets and leads to higher test accuracy.\n2. The paper provides a thorough theoretical analysis that clearly establishes the connections between diffusion priors and the dataset distillation (DD) task..", "weaknesses": "1. Sections 3.1 and 3.2 mainly restate that diffusion models exhibit diversity and generalization, which are well-known properties. This content would be more appropriate as background material in the introduction rather than the method section, as it does not present any novel contributions.\n2. The introduction of the kernel function in Section 3.3.1 lacks clarity. Its purpose and motivation are not well explained, and the connection between representativeness and the kernel function is missing. Although its application becomes evident in Section 3.3.2, the earlier introduction in 3.3.1 is confusing when read sequentially.\n3. In Algorithm 1, the procedure describes how to generate a single sample using representative guidance. However, it is unclear how multiple synthetic images per class (i.e., IPC images) are generated. Additionally, the explanation of how these generated images collectively maintain the three claimed characteristics (diversity, generalization, and representativeness) remains insufficient, given the inherent randomness in individual generations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)