Dataset Distillation as Pushforward Optimal Quantization
Authors: Hong Ye Tan
Emma Slade
Venue: ICLR 2026 Poster (2026)
Keywords: dataset distillation
optimal quantization
clustering
latent diffusion
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We theoretically interpret dataset distillation as a measure approximation problem, showing consistency when using diffusion models to generate data, and achieving SOTA or competitive results on ImageNet-1K.

Abstract: Dataset distillation aims to find a small synthetic training set, such that training on the synthetic data achieves similar performance to training on a larger training dataset. Early methods solve this by interpreting the distillation problem as a bi-level optimization problem. On the other hand, disentangled methods bypass pixel-space optimization by matching data distributions and using generative techniques, leading to better computational complexity in terms of size of both training and distilled datasets. We demonstrate that by using latent spaces, the empirically successful disentangled methods can be reformulated as an optimal quantization problem, where a finite set of points is found to approximate the underlying probability measure. In particular, we link disentangled dataset distillation methods to the classical problem of optimal quantization, and are the first to demonstrate consistency of distilled datasets for diffusion-based generative priors. We propose Dataset Distillation by Optimal Quantization (DDOQ), based on clustering in the latent space of latent diffusion models. Compared to a similar clustering method D4M, we achieve better performance and inter-model generalization on the ImageNet-1K dataset using the same model and with trivial additional computation, achieving SOTA performance in higher image-per-class settings. Using the distilled noise initializations in a stronger diffusion transformer model, we obtain competitive or SOTA distillation performance on ImageNet-1K and its subsets, outperforming recent diffusion guidance methods.

Published as a conference paper at ICLR 2026
DATASET DISTILLATION AS PUSHFORWARD OPTIMAL QUANTIZATION
Hong Ye Tan UCLA University of Cambridge
Emma Slade
Tangram Therapeutics
GSK.ai
hyt ucla. edu
emma slade @aladeetangrant..... com
ABSTRACT
Dataset distillation aims to find a small synthetic training set, such that training the synthetic data achieves similar performance to training on a larger training dataset. Early methods solve this by interpreting the distillation problem as a bi-level optimization problem. On the other hand, disentangled methods bypass pixel-space optimization by matching data distributions and using generative tech- niques, leading to better computational complexity in terms of size of both training and distilled datasets. We demonstrate that by using latent spaces, the empirically successful disentangled methods can be reformulated problem, where finite set of points is found to approximate the underlying proba- bility measure. In particular, we link disentangled dataset distillation methods to the classical problem of optimal quantization, and are the first to demonstrate consis- tency of distilled datasets for diffusion--bsse generative priors. We propose Dataset Distillation by Optimal Quantization (DDOQ), based on clustering in the latent space of latent diffusion models. Compared a similar clustering method D achieve better performance and inter-model generalization dataset using the same model and with trivial additional computation, achieving SOTA performance in higher image-per-class settings. Using the distilled noise initializations a stronger diffusion transformer model, we obtain competitive or SOTA distillation performance on ImageNet- IK and its subsets, outperforming recent diffusion guidance methods.
optimal quantization
the ImageNet-IK K
INTRODUCTION
Training powerful neural networks requires a large amount of data, and thus induces high computa- tional requirements. Dataset distillation (DD) targets this computational difficulty by changing the data, as opposed to other parts of training such as optimization or architecture (Wang et al., 2018). The DD objective consists of finding synthetic training set, such that training neural network the synthetic data yields similar performance.
There several closely related notions of reducing computational load when training models on datasets. Core-set methods find a subset of training data (as opposed to synthetic data) that achieve good training performance (Mirzasoleiman
2020 Feldman 2020). Model distillation, sometimes known as knowledge distillation, aims to train smaller model that predicts the output of larger model (Gou al.) [202]] Polino al. 2018). Importance sampling methods accelerate training by weighting training data, finding examples that are more influential for training (Paul et al. 2021). For more detailed surveys on dataset distillation methods and techniques, we refer Yuu
et al. 2023 Sachdeva & McAuley 2023).
1.1 BI-LEVEL FORMULATION OF DATASET DISTILLATION
Denote training set (more generally, distribution of training data) by T, and the expected and empirical risks (test and training loss) by and L respectively, evaluated for some parameter The goal of DD is to find synthetic dataset S (of given size) minimizing the test loss diserepancy

---

Published as a conference paper at ICLR 2026
Training data distribution
Distilled data distribution {(o,
(I)
Encoding
(3)
Decoding
Clustering
Low-dimensional latent space
Distilled latent points and weights
Figure 1: Sketch of the proposed method pipeline. Using an encoder/dccoder model, we map our high dimensional data to low-dimensional space, which then clustered using k-means. The clustered latent points and weights are then decoded to obtain the distilled data. This work argues that the weights are important when decoding; furthermore, "disentangled" distillation using an encooder.llusterddecooorr framework is asymptotically consistent.
(Sachdeva & McAuley 2023):
S arg min Raarg min L(S)) Raarg min ((T))
(1)
This formulation computationally intractable. Approximations include replacing the minimum discrepancy objective with maximum test performance, replacing the learning algorithm & with an inner neural network optimization problem, and solving the outer minimization problem using gradient methods. Common heuristic relaxations to the bi-level formulation (I) include meta-leame beta-lcarning (Wang et al. 2018 Deng & Russakovsky 2022), distribution matching (Zhao Bilen 2023), and trajectory matching (Cazenavette et al.) 2022). Other methods include neural feature matching (Zhou al. 2022 Loo et al. 20.2)) 2022) and the corresponding neural tangent kernel methods (Nguyen et al. 2021 2020). representative matching (Liu al. (203b),, and group robustness (Vahidian et al.) 2024). For better scaling, [Cazenavette et al. (2022); Moser et al. (2024) consider using generative priors such GANs to generate more visually coherent images, increasing performance and replacing the need for optimization with a neural network inversion task.
While the bi-level formulation follows naturally from the qualitative problem statement of dataset distillation, there are two main drawbacks, namely computational complexity and model architec- ture dependence. The dimensionality of the underlying optimization problems limit the applicability on large scale datasets, which are particularly useful for computationally limited applications. For example, the ImageNet-IKK dataset consists of 1.2M training images, totalling over 120GB of memory (Deng et al. 2009). The full dataset ImageNet-21 consists of over 14M images and takes up around 1.2TB of memory, which generally infeasible to train expert models on, and makes backpropagation through network training steps impossible.
1.2 DISENTANGLED AND DIFFUSION METHODS
Yin et al. (0023) is the first work to "disentangle" the bi-level optimization framework into three separate problems, named Squeeze, Recover and Relabel (SRe²L). In particular, the inner neural network optimization problem is replaced with matching statistics Curriculum Data Augmentation (CDA) uses adaptive training to get more performance (Yin & Shen 2024). Liu et al. (2023a) considers optimizing images such that neural network features are close to Wasserstein barycenters of the training image features. Sun al. 2004 proposes Realistic Diverse
monchhhommlizationn layers.
2

---

Published as a conference paper at ICLR 2026
and Efficient Dataset Distillation (RDED), which replaces the latent clustering objective with patch-based adversarial objective. Su et al. (2024) considers clustering directly in the latent space of latent diffusion model (LDM) (Rombach et al. 2022), named Dataset Distillation via Disentangled Diffussio Model M). This avoids backpropagation when distilling and has constant memory usage with respect images per class (IPC), direct advantage over the linear memory sealing of optimization-basee methods. Recent state-of-the-ar methods consider fine-tuning diffusion models have better statistics (Gu al. 2024), and guiding the diffusion to maximize the influence of the distilled points (Chen et al. 2025). y similarly to active learning.
While dataset distillation has had extensive experimental effort, few theoretical justifications or computable formalizations exist in the literature. [Sachdeva & McAuley (2023) proposes a high-level formulation based on minimizing the difference in test loss between learning on the full dataset versus the synthetic dataset. KKngurtsev et al., (2024) considers dataset distillation as dependent on the desired inference task (typically classification with cross-entropy loss for image data), interpreting trajectory matching as mean-field control problem. No prior literature on dataset distillation addresses theoretically whether or not the distilled datasets are reasonable approximations of the input training data distribution. In this work, address convergence in measure space for disentangled methods, demonstrating consistency and convergence of the distilled datasets.
We summarize the contributions of this work follows.
1. We theoretically justify the disentangled dataset distillation framework, exploiting its struc- ture to show that these methods converge to the true data distribution as the number of distilled points increases, using classical notions of optimal quantization and Wasserstein dis- tance. Motivated by the empirical usage of clustering in latent spaces, we show in Theorem [] that optimal quantizations induce convergent approximations of gradients of population risk. Furthermore, Corollary 1 showws the approximation rate is given by O(K where d is the dimension of the latent space and K is the number of quantization points. This motivates the usage of wwwdimmnsional latent space to model the data distribution.
2. We propose Dataset Distillation by Optimal Quantization (DDOQ) in Algorithm I DD algorithm based on clustering in a -iwddennniioal latent space. Compared recent disentangled method D'M, our proposed method has smaller Wasserstein distance between the distilled latent points and latent data distribution, indicating better approximation.
3. We algorithmically compare our proposed method with D'M and various common disen- tangled baselines on the ImageNet-1K dataset using the same generative diffusion model backbone, demonstrating significantly better classification accuracy at varied IPC bud- gets and better cross-architectuus generalization. To demonstrate the potential of DDOQ, we additionally use the stronger diffusion transformer (DiT) backbone, used in recent ceftusioo-bbsee methods. We provide a central comparison with SOTA disentangled and diffusion-baut based distillation methods, yielding competitive or better results than existing SOTA methods on ImageNet-IK IK and its subsets.
This work is structured as follows. Section 2 covers related background and convergence results, in- cluding optimal quantization and diffusion models. Thcorem [ in Section demonstrates consistency of the optimal quantizers when passed through diffusion- based generative priors to the image space, and motivates adding automatically-leamed weights in the prototyping phase. Section 3.1 details the data distillation pipeline in sequential manner, from distillation to training new models using the computed weights. Section Section contttin experiments of our proposed method against the previously SOTA D'M method as well as other recent SOTA baselines on the large-scale ImageNet-1K dataset.
BACKGROUND
Define (R be the set of probability measures on Rd with finite second moment, not necessarily admitting a density with respect to the Lebesgue measure. We use W, to denote the Wasserstein-2 distance between two probability distributions in Pa (2(R (Santambrogio 2015).
3

---

Published as a conference paper at ICLR 2026
2.1 OPTIMAL QUANTIZATION
For a probability measure E Pa Pa (Re), an optimal quantization (or vector quantization) at level K is of points (I1, JK C Idd such that the p-averaged Euclidean distance to the quantized points is minimal. This can be formulated as the minimizer of the (quadratic) distortion, defined as follows.
Definition (Quadratic distortion). For a quantization grid (II, UK) E (IR") the corresponding Voronoi cells are
C. = by E Rd lly zell min ly 2,1b; i K. K.
(2)
Given a measure y E P2(TR) the (quadratic) distortion function Up takes tuple of points (I1, RK) and outputs the average squared distance to the set:
(11,,.,,
min - u(dx)= Ex- [min |X xx.1?].
We will write HK mean the distortion function at level K, i.e. with domain (IR and drop the subscripts where it is clear. An optimal quantization is a minimizer of G.
Note that the assumption that μ E P2(RR)) has finite second moments implies that the quadratic distortion is finite for any set of points, and that optimal quantization exists (Pages) 2015). We note that the optimal quantization weights are uniquely determined by the quantization points. This gives equivalence of the distortion minimization problem to the Wasserstein minimization problem, when restricted to measures of finite support (Pages 2013).
(r1,..., JK). Assume that the (probability mea- sure is null on the boundaries of the Voronoi cells p(0C,) 0. Then the measure v that minimizes
Proposition 1. Suppose have a quantization
the Wasserstein-2 distance (14) and satisfies supp Ix is vk E ((x)))(a))) where ((x;) denotes the Dirac delta distribution at I;. Moreover, the optimal coupling given by the projection onto the centroids.
Proof. Deferred to Section
In other words, finding points that minimize the quadratic distortion is equivalent to finding a K- finitely supported (probability) measure, minimizing the Wasserstein-2 distance to the underlying measure.
The case where the approximating measure is a uniform Dirac mixture is called the Wasserstein barycenter problem (Cuturi & Doucet 2014). The Wasserstein barycenter has higher error than the optimal quantization, but admits an easily computable dual representation.
Remark 1.
The quantizer can be shown to have nice approximation properties when taking expectations of functions. A prototypical example for DD would have f be the gradient neural network with respect to some loss function. This implies that data distribution μ and its quantization v induce similar training dynamics.
Proposition Let f Rdd be an s-Lipschitz For a probability measure ps E P2(Rd) that assigns no mass to hyperplanes, and quantization x (I1,,.., Ix), ) let v Σ (CC)) (((()(())))) be the corresponding Wasserstein-optimal measure with support in difference between the population risk E, and the weighted empirical risk [.y is bounded as
Proposition 7 The
- S 12
Proof. Deferred to Section
2.1.1 SOLVING THE OPTIMAL QUANTIZATION PROBLEM
To find an optimal quantizer, we use the competitive learning vector quantization (CLVQ) algorithm (Ahalt et al. 1990). This arises directly from gradient descent on the quadratic distortion. The gradient of the distortion has a representation in terms of pecentroids of the corresponding Voronoi cells, given explicitly in Section C.5
4

---

pT, wish to Published as a conference paper at ICLR 2026
The CLVQ algorithm is presented in the appendix as Algorithm It consists of iterating: (i) sampling from μu, (ii) computing the nearest cluster centroid, and (in)) updating the cluster centroid with weighted average. CLVQ is equivalent to the mini-batch k-means method when the step-sizes Ti are chosen to be the reciprocals of the number of points per-cluster (Sculley) 2010) Pedregosa et al. 2011). We can thus interpret mini-batch k-means finding local minima of the optimal quantization problem.
The CLVQ algorithm produces points x (.,, FK ). but it remains compute the associated weights approximating the measures of the Voronoi cells p [.)) as in Proposition This can be done
an online manner within the same iterations (Pagès 2015).
Proposition 3 (Bally & Pagès 2003 Prop. 7). Assume that the measure
(B")) for some
> O, and that it assigns no mass to hyperplanes. Assume further that the grids x(1) produced by
CLVQ converge to a stationary grid x", i.e. VG(x*) = O, and that the step-sizes satisfy Σ 7/k = to0
and Σ 1/x 1+ < oo for some O. Then,
1. The companion weights we converge almost surely the limiting weights μ(Cz)):
2. The moving average of the empirical quadratic distortion converges the limiting distortion:
Σ
| - ==-11=+-((x)) ISISK
Using these weights, we target the problem of approximating optimal quantizers, rather than the Wasserstein barycenter problem. The addition the weights reduces the distortion in the latent space. In the following section, we show that this reduced distortion carries over to the image space, which leads to better training fidelity as given using Proposition
2.2 SCORE-BASED DIFFUSION
To connect the quantization we need to consider properties the the latent space with the quantization error in the image space, process. In particular, we focus on score-based
diffusion models, seen as discretizations of particular noising SDEs (Song al. 2020). Consider the following SDE, where W is a standard Wiener process:
dx fxx E) di + g(() dw.
The reverse of the diffusion process is given by the reverse-time SDE (Haussmann Pardoux 1986) where the density of x at time > 0 is given by pt.
dx (z, t) " (z)) di + s(t)dW,
(6)
and W is reverse time Wiener process. For increasing noising schedule (t)) noise-scale (().. the variance-exploding SDE (VESDE, or Brownian motion) and variance- variance-preserving SDE (VPSDE, or Ornttein-Uhlenbeck process) are given respectively by
(t)) dw and dr dt
dr =
2 s()) - 3B(() aw.
These SDEs are related to early diffusion models. Specifically, VPSDE corresponds to denoising score matching with Langevin dynamics (Song Ermon 2019). and VESDE to denoising diffusion probabilistic models (Sohl-Dickstein et 2015, Ho al., (2020). Using this particular structure, we may obtain convergence results as seen in the next section. We note that by time-rescaling, we may assume without loss of generality that the noising schedule is linear o² (t) t or the noise-scale is constant 3(t) = 1.
The goal in question: given Wasserstein-2 convergence of the marginals 1/7 (k)
derive bound expectations B, B,,g 2, p. [S], for some small fixed 8 E (0, 7) and Rd - I satisfying some regularity conditions. In other words, we wish to show that generative diffusion preserves closeness of data distributions. Such bound would directly link to training neural networks with surrogate data, e.g. by taking ff to be the gradient loss function with respect to some network parameters.
5

---

Published as a conference paper at ICLR 2026
Remark 2. Having 8 = 0 may not be well defined because of son-smottssssss and blowup of the score at time for singular measures (Pidstrigach) 2022 Yang et al. 2023).
Remark 3. Working with weak convergence is necessary due to the singular empirical measures.
Pidstrigach| (anggachhh2222 2022) demonstrates that the backward SDE process satisfies a data- clata-processigg inequality, showing that the f-divergence after buckwards diffasson is at most the f-divergence at marginal time T. However, [-divergences require absolute continuity of the compared marginal with respect the underlying diffused distribution, which is equivalent absolute continuity with respect to the Lebesgue measure by the Hörmander condition. This rules out singular initializations such as empirical measures, which arise in dataset distillation.
3 DATASET DISTILLATION AS OPTIMAL QUANTIZATION
Our main result Theorem gives consistency of dataset distillation for a score-based diffusion prior in the image space. Later,
this in Section 3.1 present DDOQ as modification of the D'M method. By simply changing the clustering objective from Wasserstein barycenter to an optimal quantization by adding weights, we can effectively reduce the Wasserstein distance the data distribution. Moreover, from Proposition 3 the weights are automatically determined during the k-means clustering process.
Theorem 1. Consider the VESDE/Brownian motion or the VPSDDE/Omsstiin--UHlenbeee process
dr = dW or dz dz= 2 + ddW
(8)
2
For any initial data distribution μ P2(RR) with compact support bounded by R >> O. the backwards diffusion process is well posed. Suppose that there are two distributions per, Vr at time that undergo
the reverse diffusion process (with fixed initial reference measure μ) up time L = 5 E (0, T) to produce distributions 148, Vg- There exists a (universal explicit) constant C = C(5, T, d) E I2 is an T--lipschitz function, then the difference in expectation
(0, +00) such that f T2d satisfies
||E Er [fIll S. CLWW2HT, (T).
In dataset distillation terms, f will typically be replaced by the gradient of a loss function. The above result suggests that a distilled image dataset can be given by passing distilled latent dataset through the generative reverse SDE process. Moreover, when training neural network on a distilled dataset given by optimal quantization, the gradients on the distilled dataset and full training dataset at each step will automatically be similar. This bypasses the hcuristics needed in bi-level DD formulations, and avoids fine-tuning generation-time guidance of the diffusion models.
Theorem [ combined with the asymptotic rates of Graf & Luschgy (2000) gives convergence rates as the number of quantization points increases. We note that this can be further be combined with the convergence of optimal quantization rates of empirical measures such as Theorem 3 in Section C.4 In particular, the next result shows that the number of points increases, have convergence the underlying data distribution in image space, giving consistency.
Corollary 1. Suppose μ (2 has compuct support and is diffused through either the Brownian motion or MusstinnUUhlenbeck process up to time T to produce marginal pr. Let v (K) be optimal quantizers of μT at level K for K E N. For fixed 5 E (0, let v/ (K) denote the corresponding backwards diffusion at time T - 5. Then, for any L-Lipschiiz f and as K + oc,
||E (g [flll = LO(K))/d)
(10)
3.1 PROPOSED METHOD: DATASET DISTILLATION BY OPTIMAL QUANTIZATION
We have seen that clustering gives consistent approximation to the latent distribution. Using the generative diffusion model and Theorem [ we obtain a consistent approximation to the original data distribution in the image space. We propose Dataset Distillation by Optimal Quantization (DDOQ). We detail the steps in text below and summarize in Algorithm
Suppose given latent diffusion model (LDM). i.c. (conditional) encoder-decodee pair and diffusion model on the latent space (Rombach et al. 2022). Let K be the target number of images per class. Constructing and using the distilled dataset consists of the following steps in sequence.

---

Published as a conference paper at ICLR 2026
Algorithm 1: Dataset Distillation by Optimal Quantization (DDOQ)
Data: Training data and labels (T, C), pre-trained encoder-decoder pair (E, D), text encoder T, latent diffusion model U4 target IPC count K
/* Step 1: encode
Initialize latent points == E
/* Step 2: cluster
x/
Compute and save k-means cluster centers (L) and cluster counts 1/ 1,.., K, L C:
Compute weights w(() x (1) /E,
/+ Step 3: decode
Compute class embeddings emb - (L), LE L;
Compute and save class distilled images Sr, - tet = D ( emb) k = 1,... "..,KkK K}; Result: Distilled images S = ULECS, Sr.
network fo: Data: Labels for training data E S, loss function E: (r, 0) + IR
To train
/x Step Train new model (Validation
Train network using loss function ming ((.,,,)) {(,, V,
Table I:
Wasserstein-2 distances of the distilled latents compared to the encoded training data on classes of ImageNet-IK, We observe the weighting drops the Wasserstein-2 distance significantly.
Test class
Avg reduction
D'M DDOQ
49.67
51.14
47.16
IPC 10
-15.7%
42.97
42.71
39.14
D*M
47.34
49.10
45.28
IPC 50
-16.1%
DDOQ
40.41
41.01
37.53
1. Encoding, We use the encoder of the LDM to map training samples from the image space to the latent space, giving empirical samples from the SDE marginal (4r- This done per-class in the context of classification.
2. Clustering. We then the CLVQ (mini-batch k-means) algorithm on these samples to compute the K centroids and corresponding weights, as in Proposition 3 This gives an empirical distribution u(k) that approximates jar. Equivalently, it consists of tuples of points and weights (I;; w such that VT
(K) = Σ, w,ō(x;) pr.
3. Decoding/Image synthesis. Given the clustered centroids in the latent space, the generative part of the L.DM is employed to reconstruct images. This comprises the image component of the distilled dataset. The weights of the latent points are assigned directly to the weights the corresponding generated image.
4. Training new models. Training a network fo from scratch requires: distilled images æ, some corresponding labels y, and the weights each image. For loss function f(x, y, 0) such as cross-entropy or KL divergence, the loss for each sample is weighted by w. The complete loss function is given by
min Σ w A(x, W, 0).
(2.9-10)
Within Section 1 the encoder and decoder are given by a pre-trained encoder-decoder model, combined with diffusion model in the latent space. Compared with D we include (automatically determined) weights when training new network in the final step, indicated by the variable length arrows. These are justified by considering the expectation of network gradients with respect to the training and distilled distributions. As seen in Table [ the inclusion of the weights significantly decreases the Wasserstein distance of the distilled points to the data distribution, when tested on classes of I
7

---

Published as a conference paper at ICLR 2026
Figure 2: Example distilled images of the "jeep" class in ImageNet- 1K along with their k- a-means weights below. There are little to no qualitative features that can be used to differentiate the low and high weighted images, mainly due the high fidelity of the diffusion model. However, the weights are indicative of the distribution of the training data in the latent space of the diffusion model.
Table Comparison of top-1 classification performance on the ImageNet- IK dataset at various IPCs. method DÍM, due to the
We observe that the proposed DDOQ outperforms the clustering- addition of weights to the synthetic data. The maximum performance for all methods should 69.8 as the soft labels are computed using pre-trained ResNet-18 18 model. In particular, we achieve 30% reduction in error gap using ResNet- 101 at IPC 200.
IPC
Method
ResNet-18
ResNet-50
ResNet-101
IPC
Method
ResNet-18
ResNet-50
ResNet-101
Full
TESLA
SRe'L
46.820.2
55.620.3
60.820.5
SRc"L
21.3
28.420.1
30.9
CDA
53.5
61.3
61.6
10
RDED
42.01o.1
48.311.
50
RDED
56.5101
61.2
D'M
27.9
33,5
34.2
D'M
55.2
62.4
63.4
DDOQ
33. 23.000
34.420.99
36.7
DDOQ
56.2
62.5
63.6
69.8
SRe'I.
52.8
61.0mo.4
62.8
SRe'L.
57.040.4
64.640.3
65.9.0.3
CDA
58.0
65,1
65.9
CDA
63.3
67.6
68.4
100
D'M
59.3
65.4
66.5
D'M 62.6
67.8
68.1
DDOQ
60.1
65.9_0.15
66.7
DDOQ
63.4
68.0
68.610.08
We note that there is flexibility in the choice of label y when training model. For example, the soft-label synthesis of D'M or RDED employs another pre-trained classifier V,, such as ResNet. Using this, the soft-labels are given by y V (x), opposed to one-hot encodings of the corresponding classes.
4 EXPERIMENTS
We compare with two different latent diffusion architectures, namely the original latent diffusion model utilizing UNets (Rombach et al. 2022). then the stronger diffusion transformer (DiT) architec- ture (Peebles & Xie 2023). We first use the LDM compare the pure algorithmic differences of DDOQ with D and related baselines. Then, we use DiT to demonstrate the potential of DDOQ compared newer state-of-the-art baselines.
4.1 UNET BACKBONE
To validate the proposed DDOQ algorithm, we directly compare with the previous otate-of-the-art disentangled methods D'M (Su et al. 2024) and RDED (Sun et al. 2024 on the ImageNet- 1K dataset. Baseline figures are reported as given in their respective works. For low IPC, we also report the TESLA method, which is SOTA bi-level method based on MTT (Cui et al., 2023), but unscalable past IPC=10. RDED achieves strong results for low IPC using its aggregated images and special training schedule, while CDA improves upon SRe'L using time-varying augmentation. For consistency, soft labels are computed using the pre-trained PyTorch ResNet-18 model, and new ResNet- 18,50,10 5011 models are trained using the distilled data. We provide direct comparison of the distilled data performance Table 2 for the IPCs E {10, 50, 100, 200). Variances for DDOQ averaged over five models trained on the same distilled data.
We observe that while RDED is very powerful the low IPC setting due to the patch-based distilled images, which effectively gives the information of 4 (down-sampled) images in training sample.
8

---

Published as a conference paper at ICLR 2026
Table 3: Generalization performance of D'M and the proposed DDOQ method for different soft-label teachers and student architectures at IPC 50. We observe that DDOQ has uniformly better cross- architecture generalization for convolutional teacher and student architectures, and slightly worse performance for student models using the transformer architecture Swin-T.
Student Network
Teacher Network
ResNet-18
MobileNet-V2
EfficientNet-B0
Swin-T
D
55.2
47.9
55.4
58.1
ResNet-18 18
DDOQ
56.2
52.1
58.0
57.4
D
47.6
42.9
49.8
58.9
MobileNet-V V2
DDOQ
47.7
45.6
52.5
56.3
D
27.5
21.9
26.4
38.1
Swin-T
DDOQ
28.5
24.1
29.3
36.0
However, the gap quickly reduces for IPC 50, getting outperformed by the clustering- baased D*M and proposed DDOQ methods with more powerful models like ResNet-101. Results for IPC 100/200 not available online for RDED and we omit them due to computational restriction.
The proposed DDOQ algorithm is uniformly better than D*M, with the most significant increase in the low IPC setting. Moreover, DDOQ surpasses all the compared SOTA disentangled DD methods for IPC 100 and 200. For low number of quantization points, the gap in Wasserstein distance of the Wasserstein barycenter and the optimal quantizer to the data distribution may be large. As indicated in Theorem [ lower Wasserstein distance means more faithful gradient computations on the synthetic data, which may explain the higher performance with minimal algorithmic change, as well as implicitly allowing for gradient matching in existing bi-level methods.
Table 3 shows the generalization performance of the distilled latent points. The PyTorch pre- trained MobileNet- V2 and Swin-T networks are used create the soft labels. We then evaluate the distilled images and soft labels with three convolutional architectures and the Swin-T transformer architecture. We observe that DDOQ not only able to generalize different model architectures, but also uniformly outperforms D'M on all convolutional student architectures. The slightly worse performance of DDOQ when using a transformer architecture for the student may be due to more precise hyperparameter tuning requirements.
To illustrate the weights, Figure plots ten example images from the "jeep" class when distilled
using K 10 IPC. We observe that there is very large variance in the weights (1.) / / 13 indicating the presence of strong clustering in the latent space. Nonetheless, there is no qualitative evidence that the weights indicate "better or worse" training examples, rather indicating the structure in the latent space.
4.2 DIT BACKBONE
To compare the potential of DDOQ, we use a stronger generative model, namely the diffusion transformer (DiT) (Peebles & Xie 2023). This architecture achieves uniformly better sample quality compared to the LDM used in the previous section, in terms of Inception Score and Fréchet inception distance. We denote the method with DiT backbone as DDOQ-DiT.
We compare with the SOTA DD methods based on diffusion guidance, namely Minimax (Gu et al. 2024) and Influence Guided Diffusion (IGD) (Chen al. 2025), which both use DÎT. These methods guide the decoding process using some fine-tuning or batch statistics. In contrast, DDOQ-DiT directly modifies the initialization the latent space. In addition, we compare with samples directly generated with DiT from random initializations, labelled "DiT'. We compare ImageNet-II IK,, well the 10-class subsets ImageNette and Image Woof. The baseline figures are taken from Chen et al. (2025) which report higher numbers than Gu et al. (2024).
Table demonstrates that we significantly outperform the diffusion -usiianne based methods on the full ImageNet-11 K, well DDOQ-DIT with the UNet backbone. Moreover, competitive-

---

Published as a conference paper at ICLR 2026
Table 4: Comparisons on ImageWoof and ImageNette using the ResNetAP-10 architecture, and ImageNet- IK using ResNet-18. DDOQ-DiT outperforms the guided diffusion methods at low IPC and is competitive at higher IPC, namely outperforming the compared methods on ImageNet- IK.
Dataset
IPC
DiT
DiT-IGD
Minimax
Minimax-IGD
DDOQ-DiT
Full
29.0
41.0
39.641.2
43.3
48.8
Image Woof
87.2
50
55.8
62.7
59.8
65.0
65.4
10
62.8 20.8
66.5
63.2
65.3
68.2
ImageNette
94.6
50
76.9
81.0+1.2
78.2
82.341.1
79.8
10
39.640.4
45.5
44.3
46.2
53.0
ImageNet-IK
69.8
50
10.9
59.8
58.610.3
60.3
62.7.0.1
with or better than the baselines on the 10-class subsets, namely outperforming in the low IPC setting and in the more difficult ImageWoof subset. Moreover, the stronger diffusion model significantly benefits the low IPC ImageNet- 1K setting, increasing ResNet- test accuracy from 33.1 to 53.0. An ablation on the effect of different weights during training is given in Section
5 CONCLUSION
This work proposes DDOQ, a dataset distillation method based on optimal quantization. Inspired by optimal quantization and Wasserstein theory, we theoretically demonstrate consistency of the distilled datasets in Theorem [ when using standard diffusion-bar baaeed generative models generate the synthetic data. Experiments show the proposed method is competitive or better than SOTA on large-scale ImageNet experiments.
We have presented theoretical justification for disentangled dataset distillation methods, which rely on clustering the mechanism for approximating the underlying data distributions. More specifically, we justify the combination of low-dimensional latent space and diffusion model. Consistency or convergence of other dataset distillation frameworks such bi-level methods and diffusion guidance are still open questions.
Future work could include sharper bounds in Theorem that exploit the sub- Gabssaasiinttt of the diffused distributions or manifold hypothesis. Other interesting directions could be relating the weightings of the synthetic data the hardness of learning the data, such as in (Joshi & Mirzasoleiman 2023), further increasing the performance using different training regimes such as curriculum learning. or extending the theoretical analysis inexact score matching. Possible empirical work could investigate correlations between the weights and influence (Pruthi et al. 2020
ACKNOWLEDGMENNTS
This work was done while HYT was on placement in GSK.ai. HYT acknowledges support from GSK and the Masason Foundation.
REFERENCES
Stanley C Ahalt, Ashok Krishnamurthy, Prakoon Chen, and Douglas E Melton. Competitive learning algorithms for vector quantization. Neural networks, 3(3):277-290, 1990.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326 1982.
Vlad Bally and Gilles Pagès. A quantization algorithm for solving multidimensional discrete-time optimal stopping problems. Bernoulli, 96)):1003-1049 2003.
Jean-Baptiste Bardet, Nathaël Gozlan, Florent Malrieu, and Pierre-André Zitt. Functional inequalities for Gaussian convolutions of compactly supported measures: Explicit bounds and dimension dependence. Bernoulli, 24(1)3333-353, 2018.
10

---

Published as a conference paper at ICLR 2026
George Cazenavette, Tongzhou Wang. Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence- guided diffusion for dataset distillation. In The Thirteenth International Conference on Learning Representations, 2025.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- 1 with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
Marco Cuturi and Arnaud Doucet. Fast computation of Wasserstein barycenters. In International conference on machine learning, pp. 685-693. PMLR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEFE, 2009.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. Advances in Neural Information Processing Systems, 35:34391- 34404, 2022.
Dan Feldman. Core-sets: Updated survey. Sampling techniques for supervised or unsupervised tasks, pp. 23-44, 2020.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 1296)):189-11881 2021.
Siegfried Graf and Harald Luschgy. Foundations of quantizaaioo for probability distributions. Springer
Science & Business Media, 2000.
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang. Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15793-15803, 2024.
Martin Hairer. On malliavin's proof of hörmander's theorem. Bulletin des sciences mathematiques, 135(6-7):650-666, 2011.
Ulrich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability, pp. 1188-1205, 1986.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 68511 2020.
Lars Hörmander. Hypoelliptic second order differential equations. Acta Mathematica, 119(1): 147-171, 1967.
Anselm Hudde, Martin Hutzenthaler, and Sara Mazzonetto. A stochastic gronwall inequality and applications to moments, strong completeness, strong local lipschitz continuity, and perturbations. In Annales de l'Institut Henri Poincaré- Probabiiiss Statistiques, volume 57, pp. 603-626, 2021.
Siddharth Joshi and Baharan Mirzasoleiman. Data-efficient contrastive self self-supervised learning: Most beneficial examples for supervised learning contribute the least. In International conference on machine learning, pp. 15356-15370. PMLR, 2023.
J Kieffer. Exponential rate of convergence for lloyd's method i. IEEE Transactions Information Theory, 28(2):205-210, 1982.
Vyacheslav Kungurtsev, Yuanfang Peng, Jianyang Gu, Saeed Vahidian, Anthony Quinn, Fadwa Idlahcen, and Yiran Chen. Dataset distillation from first principles: Integrating information extraction and purposeful learning. arXiv preprint arXiv:2409.01410, 2024.
Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions Information Theory, 52(10):4394 4412, 2006.

---

Published as a conference paper at ICLR 2026
Haoyang Liu, Yijiang Li, Tiancheng Xing, Vibhu Dalal, Luwei Li, Jingrui He, and Haohan Wang. Dataset distillation via the wasserstein metric. arXiv preprint arXiv:2311.1853 2023a.
Yanqing Liu, Jianyang Gu, Kai Wang. Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17314-17324, 2023b.
Yating Liu and Gilles Pagès. Convergence rate of optimal quantization and application to the clustering performance of the empirical measure. Journal of Machine Learning Research, 21(86): 1-36, 2020.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 3871 13891, 2022.
Paul Malliavin. Stochastic calculus of variation and hypoelliptic operators. In Proc. Intern. Symp. SDE Kyoto 1976, pp. 195-263. Kinokuniya, 1978.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6960-
PMLR, 2020.
Brian B Moser, Federico Raue, Sebastian Palacio, Stanislav Frolov, and Andreas Dengel. Latent dataset distillation with diffusion models. arXiv preprint arXiv:2403.0388) 2024.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge- regression. arXiv preprint arXiv:2011.000500 2020.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5199 2021.
Introduction to vector quantization and its applications for numerics. ESAIM: proceed- Gilles Pagès.
ings and surveys, 48:29-79, 2015.
Gilles Pages and Jun Yu. Pointwise convergence of the lloyd algorithm in higher dimension. SIAM Journal on Control and Optimization, 54(5):2354-2382, 2016.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances in neural information processing systems, 34: 20596-20607, 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830) 2011.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Jakiw Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-35865, 2022.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint
2018.
David Pollard. Quantization and the method of k-means. IEEE Transactions Information theory, 28(2):199-20) 1982.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
12

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Noveen Sachdeva and Julian McAuley. Data distillation: Learning Research, 2023.
survey. Transactions on Machine
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäüserr, NY, 55(58-63):94, 2015.
David Sculley. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, pp. 1177-1178, 2010.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXhv:2011.13456 2020.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D'4M: disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5809-5818, 2024.
Dataset distillation via
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9390-9399, 2024.
Saeed Vahidian, Mingyu Wang, Jianyang Gu, Vyacheslav Kungurtsev, Wei Jiang, and Yiran Chen. Group distributionally robust dataset distillation with risk minimization. arXiv preprint arXiv:2402.04676, 2024.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018.
Zhantao Yang. Ruili Feng, Han Zhang, Yujun Shen, Kai Zhu, Lianghua Huang, Yifei Zhang, Yu Liu, Deli Zhao, Jingren Zhou, al. Lipschitz singularities in diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation via curriculum data synthesis in large data era. Transactions on Machine Learning Research, 2024.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. Advances in Neural Information Processing Systems, 35:9813-995 3-227, 2022.
13

---

Published as a conference paper at ICLR 2026
A DETAILS ON HEURISTIC BI-LEVEL DISTILLATION
1. (Meta-learning.) This uses the assumption that more data produces better results, replacing the risk matching objective with a risk minimization objective and removing the dependence T (Wang et al. 2018 Deng & Russakovsky 2022). The learning algorithm 4 (
with initial parameter Oo is given by unrolling gradient steps 0z+1 = 0r 1y Ls (0x). The distilled dataset minimizes the training loss trained on the distilled set up T iterations:
arg min Eg,-pa (Or)).
(11)
2. (Distribution matching.) Inspired by reproducing kernel Hilbert spaces and the empirical upproximations using random neural network features, /Zao & Bilen (2023) proposes an alternate distillation objective that is independent of target loss. The minimization objective
is
arg min Eg 8
Σ (O(
||| Σ vo(x))l?
(12)
FET
where Vh are randomly initialized neural networks. This can be intuitively interpreted as matching the (first moment of) neural network features over the synthetic data.
3. (Trajectory matching.) For fixed network architecture, this method aims to match the gradient information the synthetic and true training datasets from different initializations (Cazenavette et al., (022). The heuristic is that similar network parameters will give similar performance.
addition, the gradients are allowed to be accelerated by matching small number of synthetic training steps with a large number of real data training steps: for N < MM steps, the Matching Training Trajectory (MTT) objective is (with abuse of notation):
T-M
100 TT Ml?
Σ
arg min Egg pp
(13)
110 07 112
S CT
1=
where 04 - of
0f+: 1/ ( 02 07.
Practical computational approximations include pre-training large number of teacher networks from different initializations.
B ADDITIONAL DEFINITIONS
The Wasserstein-2 distance is defined as
Wz(u, D) ( inf (TEI(H..)
1/2
J || v112 dr(2,, v)
(14)
where T(u, v) denotes the set of all couplings, i.e. joint probability measures on R.d Rd with marginals /a and v.
C OPTIMAL QUANTIZATION
C.I ADDITIONAL BACKGROUND
Optimal quantizers converge in Wasserstein distance rate O (K-//d) to their respective distribu- tions as the number of particles increases (Graf & Luschgy 2000). Moreover, any limit point of the optimal quantizers is an optimal quantizer of the limiting distribution (Pollard [982). Moreover, it can be shown that if a sequence of distributions converges in Wasserstein distance A4n - p. then so do the errors in quantization for fixed quantization level (Liu & Pages 2020 Thm.
C.2 PROOF OF PROPOSITION 1
...,, &g} Assume that the (probability) measure P is null on the boundaries of the Voronoi clusters p(²(C.)) 0. Then the measure that minimizes the Wasserstein-2 distance (14) and satisfies supp C {I1, ..., FKk } is VK DK (x.)) Moreover, the optimal coupling is given by the projection onto the centroids.
Proposition 4. Suppose we have a quantization X = {II,...,
14

---

Published as a conference paper at ICLR 2026
Proof. For any coupling p(, between and μl, certainly have that
|fl:- vll d7(x, y) 2 (ll--)))77(,,)))
/f dist(x, v) dz(x,)) - dist(x, v) du(().
where the first inequality comes from definition of distance a (7-a.s.) and the support condition on v, and the equality from the marginal property of couplings. The final term is attained when y has the prescribed form: the coupling is c(z,, y) Iyec, and the corresponding transport map is projection onto the set x (defined p.a.s, from the null condition). This shows a lower bound of (14) that is attained.
C.3 PROOF OF PROPOSITION 2
Theorem 2. Let f IR - R be an L-Lipschitzfuunciinn.
function. For probability measure μ E P2( that assigns mass hyperplanes, and quantization (21,..., Ik), let v Σ ((C)))(z..) be the corresponding basserstein-optimae measure with support in x, as in Proposition
L
The difference
between the population risk E,, [ʃ] and the weighted empirical risk Ey [J] is bounded
[x [ S Tg(())//2
(15)
Proof. Since μ assigns no mass to hyperplanes, we may decompose into Voronoi cells.
Upc f dp - [ fdv
K Σ
f(x)
f(1;) du(a))
C.
i=1
t=1 Cc, T.[ d)(())
L
f min || -3 du(x)
S L
min || x, 12d
Rd
The first equality comes from definition of v, and the inequalities from the Lipschitz condition and Hölder's inequality respectively.
For given quantization quantization level K, the quadratic distortion thus satisfies respectively:
((x) 1/2 inf (W2(M, (4) probability measures supp C Cx};
(16)
arg min (xx) arg min (W)(k, probability measures v, /supp v| S K}. x, =
(17)
C.4 CONVERGENCE OF OPTIMAL QUANTIZATION
The following result gives convergence rate, assuming slightly higher regularity of the underlying probability measure.
Proposition (Liu & Pagès 2020 Let O, and suppose μ P2+t There exists universal constant Ca,, E (0, Fo0) such that for every quantization level,
kk I Can 02+n(1)K
(18)
where o,(μ)) = min E, a 1'
The following non-asymptotic result considers the convergence of optimal quantizers for a sequence of probability measures, converging in the Wasserstein sense.
15

---

Published as a conference paper at ICLR 2026
Algorithm 2: CLVQ
Data: initial cluster centers Ti .... 2K step-sizes (7:)i20, +- 0
(0)
(0)
Initialize weights w (WI, ..., wx) (1/K, 1/K);
while not converged do
Sample X₃ ll;
Select "winner" kwin E arg
|| 1 ||;
Update (6+1) +- (1 (1-7))ak k:)ak
(6) + 7 if k: = kwin. otherwise F4 Update weights Wk f (1 W: + 7 l==kwn
(1+1) t- 2k
i < 1
end
Result: quantization uK Ek wx((xz))
Theorem 3
(Liu & Pagès 2020 Thm. 4). Fix quantization level K 1. Let pay f E (2 with support having least K points, such that W2(Ja, aas cc. For each E N, les x be an optimal quantizer of Pa. Then
(x)(x(") inf ( S 4e We (m p)) + 4W2 (Pn p0)
(19)
where K,,, = [infx (x) is the optimal error.
Remark 4. Convergence to optimal quantizer is only guaranteed in the case of log-concave distribution in one dimension (Kieffer) 1982) Liu & Pagès 2020). In higher dimensions, convergence
stationary (but not optimal) grid is possible (Pages & Yu 2016 Pagès 2015).
C.5 GRADIENT INTERPRETATION OF CLVQ
Proposition (Differentiability of distortion Pagès 2015 Prop. 3.1). Let x (f1,...,.* xK) ) E (RRa)K be such that the x; are pairwise distinct, and assume that μ((C.)) O. Then, the quadratic distortion is differentiable with derivative
Vg(() (2 Cula)
(xi (ve- u(de)
(20)
i=l....K
i.e., the gradient for quantization point T; points away from the p-centroid its Voronoi cell.
The gradient step for some step-sizes 7 (0, 1) reads
(k++) 1)
x (k) 7 ()((()))
x()) Hull(supp /1)*
(21)
where Hull denotes the convex hull. Recalling that the gradient (20 is peexpectation over (z)z). the corresponding stochastic Monte Carlo version of the above gradient descent yields the CLVQ
algorithm
x (k+1) = (k)) Yhe (Ix (R)
X
(22)
ISiSk"
Note that the computation of this requires the ability to sample from μ4, well being able to compute the nearest neighbor X₂ to the quantization set x (equivalent to the inclusion X₃ E ((K)).
C.6 LLOYD I ALGORITHM
Lloyd I. This consists of iteratively updating the centroids with the fu-centroids, given by the μI- average of the Voronoi cells. Clearly, if this algorithm converges, then the centroids are equal to the μ-centroids and the grid is stationary. This is more commonly known the k-means clustering algorithm, employed in common numerical software packages such as scikit-leane (Pedregosal
al. 201T). Convergence of the Lloyd-T algorithm be found in e.g. (Pages & Yul 2016).
16

---

Published as a conference paper at ICLR 2026
Algorithm 3: Lloyd I (k-means)
Data: Probability distribution ju with finite first moment, initial cluster centers z
2 K
k 0:
while not converged do
Compute Voronoi cells C,²) {y RR || yll min, ||x, v1lk:
Replace cluster centers with ju-centroids x; (k+1) + (S p((
k: -- k +
end
D BACKGROUND ON WELL-POSED DIFFUSIONS
Suppose that the true data distribution on the image space is given by μ E P(R.). assumed to have
bounded support. Then, by the Hörmander condition, the law of a random variable (X)) evolving under either the VPSDE or the VESDE will admit a density pa(x)) for all / > 0 with respect to
the Lebesgue measure, that is smooth with respect to both and & (Hörmander) 1967). Using the
following proposition, we have well-definedness of the backward SDE (Anderson 1982 Haussmann Pardoux 1986).
Proposition 7. For the forward SDE assume that there exists some > 0 such that
1. f(x, g(t) are measurable, and f is uniformly Lipschitz: [l, t) - fly, t)ll - K| vll for all x, y Rd;
2. [1, O)I + lo(1) S K(1 ||=]|):
3. The solution X, of 55) has a C. density Pt (c))) all > 0, and
S(n T
J J lpd|l + (x)) < too Via (0,T), R 0;
4. The score V le is locally Lipschitz on (0,7) x Rd
Then the reverse process X is solution of the 16 and moreover, the solutions of 6 are unique law.
Now given that the backwards SDE is indeed a diffusion, the data processing inequality uses the Markov property and states that the divergence after diffusion is less than the divergence before diffusion (Liese & Vajda 2006). This is summarized in (Pidstrigach, [2022 Thm. 1).
Theorem 4. Denote the initial data distribution by μo μ4, and let pr be the distribution of random variable X, satisfying the forward SDE (5) on [0, Tİ. Assume the above assumptions, and let Y, satisfy the backwards SDE 6 on [0, T with terminal condition YT DT. Denote by pt and v the marginal distributions at time I. E [0, T] of X, and Yú, and assume that UT 4. pT. Then:
1. The limit Yo = limm- Y, exists a.s., with distribution vo < 40.
2. For any f-divergence Dy,
Dy(po,r) S Dy vT) and Dy(Vos Mo) S Dy(yr,PT).
(23)
This theorem shows that for f-divergence, such as total variation distance or Kullback-Leibler divergence, convergence of the marginals at time implies convergence of the backwards-diffine marginals time 0 (also any t < T). However, this requires absolute continuity of the initial marginal distribution 2/r with respect to fr, equivalently, w.r.t. Lebesgue measure. This precludes useful bounds for singular initializations of vr, such as empirical measures.
17

---

Published as a conference paper at ICLR 2026
E D'M ALGORITHM
Algorithm 4: D'M (Su et al. 2024)
Data:
Training data and labels (T, C), pre-trained encoder-decoder pair (E, D), text encoder T, latent diffusion model Ur target IPC count K
Initialize latent variables Z = E());
for label I. L do
Initialize latent centroids = I,.., K; Initialize update counts of = 1, ho 1, 1,... K; * Compute prototypes with k-means for minibatch Z do for 2 E do
0 + arg ming || - =
/* Update learning rate
_ vi +1;
"
/* Update centroid
27 (1 +/
end
end
12
Compute class embedding y = T[L);
Compute class distilled images Sr. = (D o u1 k-= :==1...,, I,..,,.K K}:
13
14
end
Result: distilled images S ULEC St.
PROOF OF MAIN THEOREM
We first require lemma that controls diffusions for compact measures. In particular, the score is (weakly) monotonic.
Lemma 1 (Bardet al. 2018)). Let μ. E P(Rd) be probability measure with compact support, say bounded by R. Let (z) (2-1)-477 exp(- | |/ be the density of the standard normal distribution in Rdd Then Pt is the density * gt. it satisfies
R²
-
( - y. logpr (x) - logp:(u)) S
11
12
Proof. From (Bardet al. 2018 Sec. 2.1), the density Pt can be written as
(-(1) [|x|1|
+ wo(cs))
(x(x) = (2rt)dd22 exp
2t
where
(x,2) u(dz) log Cv,
Wi(x) = - log
exp
Rdd
with Cz(x) = Sg. oxp(-1lel3//2))mddd and v(dx) = C, exp(-1le (00ll=l1//2))uuddx) Moreover,
R
0 0 - <
la-
(24)
f2
Therefore,
d log (2Tt)
||,||/
log (x.) =
W((x)
2t
has Hessian satisfying
R22 p
1
2 (z)
la.
Therefore, V log P. satisfies the desired monotonicity condition. Note that R can be strengthened to diam(supp /1).
18

---

Published as a conference paper at ICLR 2026
We next require the following proposition, which can be thought of stochastic version of Gronwall's inequality. We present a simple version of the even stronger result available in HHdde
let al. (202I).
Proposition 8 (Hudde et al. 2021 Lem. 3.6). Consider the diffusion
dr = f(z, t) dt + g(x, dW.
(25)
Suppose that there exists measurable [0, T] + oo] satissfyng So c(t) dt < to0, and that for all t [0, and x, y E IRd,
(x y, f(x, () f(v.1)) + =1l,(,,1)))) - g(v, 1) S +[1){ - yl?
(26)
Then for processes XT, X starting at æ, respectively under (2) it holds for all E (0, T`) that
|x - (( S || vll exp (%" o(s) ds
(27)
Our goal is to "pushforward" the convergence of the Wasserstein distance from the latent space (of op- timal quantizations) through the diffusion model (backwards SDE (6)) into the image space/manifold. We proceed with the proof.
Theorem 5. Consider the VESDE/Broonnin motion
dæ dW
(28)
or the VPSDE/Omsteiin-UUlnnbeee process
dx 2 dt + dw.
(29)
Then, for any initial data distribution μ E P2( with compact support bounded by R 0, the assumptions for Proposition 7 hold and the backwards diffusion process is well posed.
Suppose farther that there are two distributions pT Vr time T that undergo the reverse diffusion process (with fixed initial reference measure μ) up to time t = E (0.T) to produce distributions M&, vg. Then there exists a constant C C(8, R. d) E (0.++0)) such that if f Bd - R" is an L-Lipschitz function, then the difference in expectation satisfies
|| - Em [fIll S (LW2(UT,UT).
(30)
Proof. The main idea is push Wasserstein-optimall coupling through the backwards SDE, then using Proposition 8 and Proposition First fix a á E (0, 2'. and let supp be bounded by R. Let Gr (2/1)-44/2 cxp [z/l2//r) denote the distribution of the Gaussian (0, tIa) in dimensions. By the Hörmander condition, the densities of a random variable X, with initial distribution Xo undergoing the Brownian motion or Ornstein-Uhlenbee
2
process exist, and furthermore the forward and backward SDE processes are diffusions (Malliavin) 1978 Hairer 2011 Pidstrigach] 2022).
Step (Monotonicity of the drifts.) For the Brownian motion,
(x(x) (p(2) (z).
For the Ouhstein-Ullenbeck process, the solution from initial condition Xo = Ti is
2t = zoe -12 + W1-e-1
The law of X, is thus /2(e *91 -1--ex(-()) where ((e5/22))) has support bounded by Re2
Apply Lemma I with μ for the Brownian motion, and p(ef/2r) for the Ornstein-Uhlenbeck pro-
cess. The corresponding backward SDEs (in forward time) for the Brownian motion and Ornstein- Uhlenbeck process as given by the forward time versions of 6 are
dx = v -lo + dW for Brownian motion;
I + V (1ogpr-0(1)) de dw for the OU process.
dx
2
19

---

Published as a conference paper at ICLR 2026
where (r) is the law of Xy-1for [O,
Step 2. (Lipschitz w.r.t initial condition after diffussion) For the backwards SDEs, since the score is (weakly) monotone and the diffusion term is constant, the backwards SDEs satisfy the monotonicity condition in Proposition 8 Hence, there exists a constant C' such that for any Y Y2" evolving according to the backwards SDE,
B Y7- S Cl|a: |a vll-
(31)
From the monotonicity condition and Proposition g the constants can be chosen to be as follows, noting the diffusion term is constant:
/ R2
1
log Cou = (''
S' R (1- (1- 1 1- dt.
1
log Crrowiian =
dt,
(32)
82
Step (Lift to function expectation.) Now let Y,, Y, be two diffusions, initialized with distributions Yo pr, Ye vr. Let y E T(pr,vr) be any coupling. Define the "lifted coupling" ły on the
measurable space ((Rd R.d) Ω2 (TR." RR)) s F), where FP) is the underlying (filtered)
probability space of the diffusion, as the pushforward of the diffusion:
(Yo + Yr-S, Yo + YT) @
(33)
Marginalizing over I., this is (probability) measure on Rd x Rdd since the backward SDE paths continuous: the backward SDEs admit the following integral formulation, where (rr, Y,) is the drift term of the backward SDE:
Y = Yo + 1-/ Mr. Y.) dr + W.-
Moreover, & is a coupling between - ~ 148 and V ug. We compute:
|| // Blf(z) f(v) df
[B [f] Eus ExIlf
S T. // (|Y7_s 11--s-y-lll (7-,1d
S CT // 11 ((,))
CL /ʃ |a vll d> S CL
(ff 1/2
The desired inequality follows by taking infimums over admissible couplings y l'(HT,
G APPROXIMATE TIMINGS
All times are done on Nvidia A6000 GPUs with 48GB of VRAM. We note that synthesis time as reported in SS et al. (2024); Sun et al. (2024) do not include the time required to generate the latent variables, and thus are not sufficiently representative of the end-to-end time required to distill the dataset.
Table 5: Time required for each step of dataset distillation ImageNet-IK. Synthesis requires application of the Stable Diffusion V1.5 model each distilled latent variable, and soft label requires application of the pre-trained ResNet- 18 model to each distilled image. Memory usage is constant between IPCs due to equal batch size.
Step
Time (IPC 10)
Time (IPC 100)
(Latent clustering) (Synthesis)
8 hours
8 hours
2 hours
day
3 (Soft label)
hour
16 hours
(Training ResNet18)
2.5 hours
9 hours
20

---

Published as a conference paper at ICLR 2026
H EXPERIMENT HYPERPARAMETER
We detail the parameters when training the student networks from the distilled data. They are mostly similar Su et al. (024)..
For consistency and a more direct comparison with previous methods, we use the pre-trained Py Torch ResNet- 18 model to compute the soft labels, using the same protocol [uu al. 2024) After computing the soft labels using the pre-trained ResNet-18 model, we train new ResNet-18, ResNet- 50 and ResNet-101 models to match the soft labels. The data augmentation is also identical, so that the only differences are the addition of the weights to the training objective and some minor hyperparameter tuning for the new objective. The latent diffusion model chosen for latent generation and image synthesis is the publicly available pre-trained Stable Diffusion V1.5 model, the same as D
Table 6: Hyperparameter setting for ImageNet- IK experiments.
Setting
Value
Network
ResNet
Input size
224
Batch size
1024
Training epochs
300
Augmentation
Raidommes
Min scale
0.08
Max scale
1
Temperature
20
Optimizer
AdamW
Learning rate
2e-3 for Resnet18, 1e-3 otherwise
Weight decay
0.01
Learning rate schedule
M6++1 m//4 at epoch 250
Variance reduction (heuristic). We note that the variance the number of cluster assignments can vary significantly, sometimes up to two orders of magnitude, such as in Figure After normalizing the cluster counts Step of Algorithm small, and do not contribute much to the neural network training. To reduce this effect, we use the per-centroid weights as follows,
give weights w E (0, 1), most weights are very
K
w
(L)
VK,
(L) U3
(L)
(34)
j j=1
I ABLATION ON WEIGHTING
To test the effect of the heuristic weights used in 34 we consider the setting of the DiT backbone Section 4.2 We consider three different weightings within the training in Step 4:
1. Constant weights 1. This is equivalent to D*M.
2. The heuristic weighting strategy 34
3. The direct cluster weights, given by wy ε 2/ where un are the cluster counts from the k-means, normalized such that the sum over each class
We the same cluster images and only differ the weights. Table 7 details the test accuracy for IPC 10 on ImageNet-IK with various weights, applied with a ResNet- teacher and ResNet-18 student architectures. We observe that the heuristic weighting strategy outperforms D'M across different learning rates. Moreover, the direct cluster weights are able to obtain higher test accuracy, albeit being more sensitive to the choice of learning rate.
21

---

Published as a conference paper at ICLR 2026
Table 7: Ablation across different weighting strategies when training new student network with soft labels. We observe that the heuristic weight used in the main experiments always outperforms D,M. The direct cluster weighting more sensitive to learning rate, but eventually outperforms the heuristic weighting for higher learning rates.
Learning rate
le-3
2c-3
5e-3
le-2
No weighting (D'M)
52.1
52.5
51.5
52.2
Heuristic weight (3)
52.3
52.6
53,2
53.6
Direct cluster weighting
44.1
50.3
54.6
55.6
22


Reviews:
[{"review_id": "XYx6oTvfmk", "reviewer": "Reviewer_gQzu", "summary": "This paper provides a theoretical interpretation of dataset distillation (DD) by framing it as an optimal quantization problem in latent space. The authors show that modern disentangled DD methods can be understood as approximating the true data distribution via finite quantization of its latent representation.\nBuilding on these observations, they propose Dataset Distillation by Optimal Quantization (DDOQ), which performs weighted clustering (via CLVQ) in the latent space of a pretrained diffusion model and uses the resulting centroids and weights to reconstruct distilled images. Empirically, DDOQ achieves improved performance over prior works such as D4M and RDED on ImageNet-1K, ImageNette, and ImageWoof, with notable advantages in medium IPC regimes and better inter-model generalization.", "questions": "- **More ablations**: Could the authors provide an ablation study on ImageNet-1K (e.g., at IPC 10) that directly compares: (a) no weights (D4M baseline), (b) the theoretically motivated weights, and (c) your proposed heuristic weights? Without these ablations, it is difficult to assess whether the theory is useful in practice or if an arbitrary heuristic is the true source of improvement.\n- **Source of performance gains**: Can you quantify the relative error reduction from (a) moving from D4M to DDOQ on a fixed U-Net backbone, versus (b) moving from a U-Net to a DiT backbone using a fixed D4M algorithm? This would help disentangle the contribution of your method from the contribution of the stronger (or different) generative model.\n- **OQ utility**: Beyond justifying the use of weights, does the OQ perspective offer any other suggestions for concrete algorithmic improvements? For instance, the $\\mathcal{O}(K^{-1/d})$ error rate suggests a trade-off between the number of prototypes K and latent dimension d. Does your theory provide any guidance on how to choose d for a given distillation budget K? If not, what is the practical utility of the theory beyond re-describing weighted k-means?\n- **Necessity of weighting in high-dimensional latent spaces**: The paper suggests that the weighting scheme mitigates the mismatch between the limited number of quantization points (IPC) and the high dimensionality of the latent space. Could the authors elaborate on this relationship quantitatively? For instance, is there an empirical or theoretical threshold linking the number of latent samples K and latent dimension d beyond which weighting becomes less critical (e.g., K>>d)?\n- **Higher IPC regimes**: If the number of distilled images increases substantially (e.g., IPC@1000), do the authors expect the weighting adjustment to remain influential, or would the raw cluster frequencies already approximate the latent density sufficiently? Clarifying this could help understand whether the proposed weighting primarily benefits low-data regimes or remains beneficial asymptotically.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Theoretical contribution**: The key theoretical contribution is the formal link between quantization theory, Wasserstein distance, and dataset distillation consistency. Theorem 1 demonstrates that score-based diffusion preserves the distributional closeness between the raw data and its quantized latent approximation, resulting in consistent gradient expectations during training on the distilled dataset. Corollary 1 further establishes asymptotic convergence rates $\\mathcal{O}(K^{-1/d})$ for the approximated data distribution in image space, offering the consistency guarantees for diffusion-based DD methods.\n- **Clean problem framing**: Reformulating dataset distillation as a pushforward optimal quantization problem is an insightful contribution that bridges classical quantization theory and modern generative modeling.\n- **Comprehensive experiments**: Evaluation covers both UNet and DiT-based diffusion models, multiple IPC regimes, and cross-architecture generalization. The results are consistent and competitive with SOTA baselines.", "weaknesses": "- **Heuristic implementation > theoretical analysis**: The core theoretical argument is that optimal quantization (non-uniform weights) is superior to finding the Wasserstein barycenter (uniform weights). However, the final implementation does not utilize these theoretically derived weights. Instead, it employs an ad-hoc heuristic (Eq. 34) output for \"variance reduction.\" This choice is not justified by the theory and is not ablated. This undermines the central claim that the theory guides the method's design. Rather, the theory appears to be more of a post-hoc justification for the general idea of weighting, while the actual implementation relies on an unexplained heuristic.\n- **Lack of practical guidance**: \n    - The paper introduces weighting as a core contribution but provides little analysis of its behavior, interpretability, or sensitivity during training.\n    - The bound in Theorem 1 contains a constant C with dependencies that are not analyzed in a practical context (e.g., how to choose diffusion time T to optimize the bound). The convergence rate $\\mathcal{O}(K^{-1/d})$ is standard for quantization but does not lead to new insights on how to choose an optimal latent dimension d or number of prototypes K for a given budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUWW3SIsZG", "reviewer": "Reviewer_Zzhq", "summary": "The paper aims to improve representative selection for classes in dataset distillation. It reframes the selection process as an optimal quantization problem and supports the approach with theoretical proofs and experimental validation. The objective is to provide a more principled and theoretically grounded way of selecting better representatives for dataset distillation for sample efficient training. This combination of theory and empirical study gives the work potential value for the dataset distillation research.", "questions": "1. The bad performance of the Swin-T transformer as a student is attributed to hyperparameters. Providing results with optimal hyperparameters will be helpful to identify if transformer architecture poses some limitation to this work or disentangled distillation in general.\n\n2. Proposition 1 and 2 use the notation δ_{x_i}. Please clarify whether this denotes a Dirac delta function (i.e., a unit point mass at x_i) or some other function indicating the Voronoi cell’s probability mass.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Frames the dataset distillation problem as an optimal quantization problem and supports the suggested improvements with appropriate theoretical proofs. The theoretical formalization connects optimal quantization error with downstream expectation differences in the image domain, which helps bridge diffusion-based priors and representative selection in a mathematically grounded way.\n\n2. Theorem 1 gives an explicit upper bound for the Wasserstein distance in latent space, which they show is related to the expectation difference in image-space functions after reverse diffusion. This link between latent space distance and image-space fidelity is useful and not previously formalized for these disentangled distillation methods.\n\n3. The complexity of the proposed method is low, but the performance gains are consistent and meaningful compared to baselines.\n\n4. Experiments covering other approaches for the same latent diffusion backbone and comparison of different prior selection methods for DiT are good to show that the gains are from the proposed approach and translate to other backbones.", "weaknesses": "1. The theoretical foundation simply builds upon existing proofs from quantization theory, so the novelty in theoretical contributions is limited. The actual improvements suggested are limited and incremental, as operationally, it simply adds weights to cluster centers.\n\n2. A weighted loss is used while training the student models, which makes it unclear if the improvements are from better diffusion prior selection via clustering or better optimization of student model via weighted loss. An ablation study clarifying this will improve the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0sRwCPZD5", "reviewer": "Reviewer_98qT", "summary": "This paper proposes Dataset Distillation by Optimal Quantization (DDOQ), which reframes disentangled dataset distillation as an optimal quantization problem. The method leverages latent diffusion models (LDMs) to represent data distributions in a low-dimensional latent space, performs weighted clustering (CLVQ) to approximate the latent data measure, and uses the generative decoder to reconstruct synthetic training samples.", "questions": "Could the authors provide a table comparing the computational cost (e.g., memory usage, and GPU hours) of DDOQ against other baseline methods such as D4M and RDED?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper introduces a new conceptual connection between dataset distillation and optimal quantization under the Wasserstein distance framework.\n2. The authors propose to use the Wasserstein distance between the distilled latent representations and the original latent data distribution as a quantitative indicator of how well the synthetic dataset approximates the real data distribution.\n3. The proposed method, DDOQ, improves performance in higher images-per-class (IPC) settings.", "weaknesses": "1. The introduction looks like a related work and fails to clearly introduce the research gap and specific contributions of the paper. As a result, it is difficult for readers to understand what is novel about this work.\n2. The proposed DDOQ method is a modification of D4M, replacing uniform clustering with a weighted k-means step. The overall pipeline (latent clustering, diffusion decoding, and weighted training) remains conceptually similar to previous methods. The framing of the approach as “optimal quantization” seems more like a theoretical reinterpretation than a fundamentally new algorithmic contribution.\n3. In Section 3.1, it is not explicitly shown how optimal quantization principles are concretely applied to dataset distillation. The practical novelty seems the introduction of weights, but these weights are simply normalized cluster weights (as shown in Algorithm 1 in line 278) and not directly derived from optimal quantization theory.\n4. The paper is mathematically dense and lacks intuitive explanation or visual figures. A framework diagram or flowchart illustrating the DDOQ pipeline would greatly help readers grasp how the theoretical formulation connects to the practical implementation.\n5. In Table 2, the caption claims that “the maximum performance for all methods should be 69.8,” but this number does not appear in the table, leading to confusion. Additionally, the DDOQ performance at IPC=10 is substantially worse than RDED, indicating that the proposed method may not generalize well across different IPC settings.\n6. The paper lacks a detailed ablation study analyzing the effects of key hyperparameters, such as the latent dimension, the number of quantization points K, and the weighting scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dC7D4nsSar", "reviewer": "Reviewer_Pn8q", "summary": "The paper reframes disentangled dataset distillation as an optimal quantization problem in latent space, adds per-centroid weights (a small but material tweak), and proves a diffusion-pushforward consistency bound; empirically the paper proposes DDOQ , and it beats D4M and is competitive with DiT-guided methods at higher IPC on ImageNet-1K.", "questions": "1. What exactly drives the gains, weights or quantizer? Please add a D4M + weights variant and a DDOQ w/o weights ablation to separate effects.\n\n2. What is the effective latent dimension d used by LDM/DiT in your pipeline, and how do W₂ and accuracy scale with K (log–log plots) to test the K⁻¹/ᵈ prediction?\n\n3. Please replicate Table 2 with a ViT-B and Swin-T soft-label teacher to test robustness of conclusions beyond ResNet-18’s 69.8 cap.\n\n4. Report end-to-end cost (distill + train) for D4M vs DDOQ vs guidance-based baselines, at IPC10/50/200, on fixed hardware.\n\n5. Any insights (e.g., label entropy, weight dispersion, LR/BS sensitivity) on why Swin-T lags with DDOQ in Table 3?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.Casting disentangled DD as optimal quantization ties practice to classical theory; the pushforward bound (Theorem 1) is simple and actionable.\n\n2.Adding weights to latent centroids is near-free and consistently helps (lower W₂; small but real accuracy gains over D4M).\n\n3.Multi-IPC, multiple students (R-18/50/101; MobileNet-V2; EfficientNet-B0; Swin-T) and two generators (LDM, DiT), with cross-teacher/student table.\n\n4.Strong results with DiT. Beats Minimax/IGD on full ImageNet-1K at low IPC; competitive on ImageNette/Woof.\n\n5.Clean pipeline and well-written paper.", "weaknesses": "1. The empirical lift mainly comes from adding weights to D4M-style clustering; it’s an elegant repackaging more than a new core mechanism. A D4M+weights control would isolate weighting vs. barycenter vs. quantizer effects.\n\n2. Theorem 1 requires δ>0 (no bound at exact image time), compact support, and Lipschitz test functionals; constants depend on (δ,T,R,d). Useful, but the gap to practical (non-Lipschitz) training signals remains.\n\n3. Teacher dependence and fairness. Main UNet/LDM tables fix ResNet-18 soft labels (cap at 69.8), but teacher swaps materially affect generalization (Table 3) and DiT tables use different baselines. A teacher-swap on ImageNet-1K main table would de-bias conclusions.\n\n4. Results focus on accuracy; there’s little about wall-clock/GPU hours for distillation + training across methods/backbones. (RDED omitted at high IPC due to compute, another signal that cost matters.)\n\n5. DDOQ underperforms D4M for Swin-T students at IPC50; this suggests hyperparameter or representation-mismatch issues that deserve diagnosis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)