Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation
Authors: Xiao Cui
Yulei Qin
Wengang Zhou
Hongsheng Li
Houqiang Li
Venue: NeurIPS 2025 poster ()
Keywords: Dataset Distillation
Optimal Transport
Primary area: other

Abstract: Dataset distillation seeks to synthesize a compact distilled dataset, enabling models trained on it to achieve performance comparable to models trained on the full dataset. Recent methods for large-scale datasets focus on matching global distributional statistics (e.g., mean and variance), but overlook critical instance-level characteristics and intraclass variations, leading to suboptimal generalization. We address this limitation by reformulating dataset distillation as an Optimal Transport (OT) distance minimization problem, enabling fine-grained alignment at both global and instance levels throughout the pipeline. OT offers a geometrically faithful framework for distribution matching. It effectively preserves local modes, intra-class patterns, and fine-grained variations that characterize the geometry of complex, high-dimensional distributions. Our method comprises three components tailored for preserving distributional geometry: (1) OT-guided diffusion sampling, which aligns latent distributions of real and distilled images; (2) label-image-aligned soft relabeling, which adapts label distributions based on the complexity of distilled image distributions; and (3) OT-based logit matching, which aligns the output of student models with soft-label distributions. Extensive experiments across diverse architectures and large-scale datasets demonstrate that our method consistently outperforms state-of-the-art approaches in an efficient manner, achieving at least 4\% accuracy improvement under IPC=10 settings for each architecture on ImageNet-1K.

Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation
Xiao Cuil.2
Yulei Qin
Hongsheng Li University of Science and Technology of China
Wengang Zhou'
Houqiang Li!
CUHK MMLab
Independent Researcher
cuixiao2001 1ma.:000000mma11.mmmmmmmmm1m111m ustc. edu. en, yuleichin@126. com {zhug, listt edu. en, helißee. cuhk, edu, hk
Abstract
Dataset distillation seeks to synthesize a compact distilled dataset, enabling models trained on to achieve performance comparable models trained the full dataset. Recent methods for large-scale datasets focus on matching global distri- butional statistics (e.g., mean and variance), but overlook critical instance-level characteristics and intraclass variations, leading to suboptimal generalization, We address this limitation by reformulating dataset distillation as Optimal Trans- port (OT) distance minimization problem, enabling fine-grained alignment at both global and instance levels throughout the pipeline. OT offers a geometrically faithful framework for distribution matching. It effectively preserves local modes, intra-class patterns, and fine-grain grained variations that characterize the geometry of complex, sigh-dimensional distributions. Our method comprises three components tailored for preserving distributional geometry: (1) OT-guided diffusion sampling, which aligns latent distributions of real and distilled images; (2) label-image- aligned soft relabeling, which adapts label distributions based on the complexity of distilled image distributions; and (3) OT-based logit matching, which aligns the output of student models with soft-label distributions. Extensive experiments across diverse architectures and large-scale datasets demonstrate that our method consistently outperforms state-of-the-ard approaches in an efficient manner, achiev- ing at least 4% accuracy improvement under IPC=10 settings for each architecture on ImageNet-IK.
Introduction
The expansion of data has fueled advances in deep learning, but also introduced prohibitive costs in storage, computation, and energy [1] 2 3 To address these challenges, dataset distillation aims to synthesize a small set of training samples to expedite model training while maintaining comparable performance [4] Such distillation not only improves accessibility and cost-efficiency. but also facilitates practical applications such as knowledge transfer [5]. federated learning 7.. and continual learning [8] [9. Moreover, it provides valuable lens to investigate the theoretical principles underlying training efficiency and representation capacity in deep learning [T]
Traditional dataset distillation methods can be broadly categorized into optimization-base 12 13 14 15) and distribution-mate matching-based approaches [16] 17 18 19. Despite their effectiveness, these methods remain largely restricted to small-scale, low-resolution datasets such MNIST [20). CIFAR (21) or downsampled ImageNet subsets [22]. This limitation stems from the prohibitive
Corresponding authors: Wengang Zhou and Houqiang Li.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

OT Distance Minimization:
Step I: O O'ggidded Diffusion Sampling
Performance Comparison on ImageNet
OT-Guided Aligunent
Mobileltet-V2
Wipuoe Plra
Latent Space
Outeel Wi Distilled Images
Real Imagges
IDC
DIT-C0
W [pime: Marn)
Step Label-Image--eee Allgnedd Relabeling
Outs
Dislance
ResMot:11
Evarractio Factor
W(Jtues Pasall
Real Data (with label) [FUPC]]
Not
Distiled Dala
Step 3: OT-based New Model Logit Matching
Batch-wise OT-based,
Same variance Different geometry
Same mean
W(Paali Plae
(wtt))
Distnbuttons
Label
Cap W(F.a) = (P',
Logit Alignment
New Model
Outpuls
EfficientNet-Bg
Tränsforner
(a) Motivation
(b) Our OT Distance Minimization Framework
(c) Performance
Figure 1: (a) Distributions with identical mean variance may differ geometrically, causing biases in global-statistics--un optimization. (b) We decompose the OT objective into three stages: OT-guided diffusion sampling, -eeeeeaaa-- soft relabeling, and OT-based logit matching. (c) Our method consistently outperforms prior approaches across architectures on ImageNet- (IPC = 10).
computational cost of alternating optimization between the distilled data and the condensation model [23] and the reliance on integrated image representations that demand costly pixel-level
refinement 244 Recent efforts have explored generative and model-inversion- based techniques to
overcome these scalability bottlenecks. Model-inversion-be based methods (2 26 27 28 originally
proposed under data-free distillation framework [29] 30), rely entirely on global satch-normalization
statistics extracted from pretrained models. While simple, this design imposes an inherent limitation:
it fundamentally lacks the ability to recover or preserve instance-level, local distributional structures. In contrast, generative-model- sampling process, showing potential to approximate the true data distribution more faithfully.
methods [24
33 leverage real image samples during the
However, existing generative approaches have yet to fully realize this promise, as they still solely focus on matching global gradient statistics. Besides, the fine-grained distributional structures are diversity guidance, resulting in local mode collapse
not properly captured by sosine-immllaitt--eee and distributional mismatch in the distilled set. To address this limitation, we propose principled reformulation grounded in Optimal Transport (OT), which enables Ane-grained distributional ge- ometry alignment between real distribution and model output distribution. Specifically, we define distributional geometry alignment as preserving distribution-level global and local structures (e.g.. coarse-grail -eeneee patterns and subclass densities), rather than image-level features. Our key insight is that each real data point encapsulates rich intra-class semantic variation, such as the distinctive traits of different subclasses or local modes within the same class. OT inherently provides geometrically faithful and perceptually aligned measure of distributional differences [34], making especially promising for preserving and transferring these fine-grained semantic structures during distillation.
Building upon this, we formulate dataset distillation as an OT distance minimization problem. As shown in Figure [ to make the alignment process tractable and iilimizzationfriennlli we decompose the total OT distance into three complementary objectives that altogether contribute to its minimiza- tion: (1) instance-level transport in image latent space, (2) label-image alignment in label space, and (3) batch-wise logit alignment between new model predictions and soft targets. Such decomposition ensures alignment through the sequential stages of dataset distillation, ranging from image generation and label assignment to student model training. In the first stage, latent space transport is achieved by continuously computing the OT distance between the accumulated synthetic images (including the newly generated samples) and the real image batches at each sampling step. The gradients from this computation are used to guide the diffusion sampling process. In the second stage, we align the complexity of the synthetic image distribution with that of the soft label distribution, thereby narrowing the OT distance between the distilled data and the real data. In the final stage, we transfer the rich distributional geometric information embedded in the distilled set to new student models by minimizing the batch-wise OT distance between the student outputs and the soft-label distributions.
We evaluate our method across diverse range of architectures, including ResNet, MobileNet, EfficientNet, Swin Transformer, ConvNet, and ConvNeXt. Our approach consistently outperforms

---

state- oftte-oftthe-at methods across all datasets, architectures, and IPCs, achieving particularly strong results on ImageNet- IK [22 under the challenging IPC=10 scenario. Our contributions are threefold:
We propose novel perspective of dataset distillation by formulating the task as OT dis- tance minimization problem. We decompose the objective into three tractable components.
We systematically enhance distributional geometry alignment through key stages of the pipeline, including image sampling, soft label relabeling, and student model logit matching.
We demonstrate the effectiveness and generalizability of our method across broad range of datasets and model architectures, which significantly surpass existing techniques.
2 Related Works
Numerous studies have investigated dataset distillation: initial works target low-resolution, small- scale datasets, while more recent methods address large-scale, higher-resolution scenarios.
2.1 Small-scale distillation methods
Traditional dataset distillation methods can be broadly classified into optimization based and Iitributtionmatcchie (DM)-basee approaches. Optimization- based methods [12] [3] 35
14 [5]
adopt a bi-level optimization framework, where model parameters are updated in the outer loop while synthetic data are refined in the inner loop to match gradients or trajectories. In contrast, DM-based directly align the feature distributions of real and synthetic data, thereby avoiding costly nested optimization. However, all these methods exhibit high model depen- dence on the condensation model, which limits the versatility of the distilled datasets in generalizing across different architectures [38] 39]. Also, they incur significant time and memory costs due to three factors: (1) treating synthetic data as fixed entities, (2) requiring exhaustive pixel-level refine- ments, and (3) relying on real data for image refinement. As a result, traditional dataset distillation approaches are predominantly applied small-scale datasets.
methods [16] [ 18] 33 37 [9]
2.2 Large-scale distillation methods
Recent methods for large-scale, high-resolution datasets fall into two main categories: model- inversion-basee and generative-mode
methods. radellinverrion-baseee methods [27] 28 23 41 42 43) compress the real dataset into a compact model representation, eliminating the need for real data during image refinement. This reduces memory overhead and allows scaling to large datasets such as ImageNet- 1 (22. However, the lack of real data the reconstruction process results in the loss of fine-grad fine-grained instance-level information, which hinders the distilled dataset from accurately capturing the structural properties and instance-specific characteristics of the real distribution. senerative-model-basee models to avoid pixel-level refinements. While generating one sample at a time reduces memory overhead and avoids treating data fixed entities, independently synthesizing each sample prevents the distilled dataset from maintaining a coherent overall distribution, thereby limiting its ability to capture the full diversity and structural relationships of the real data distribution. Due to their inherent exclusion of real data during reconstruction, model-inversion based methods fail to preserve fine-gra fine-grained structures of the real distribution; accordingly. we adopt the snnerative-modele paradigm as our starting point. We address the shortcomings of both families by proposing an OT framework that ensures distributional geometry alignment throughout the distillation process.
28]
methods [32
45,
46 24
leverage pretrained generative
3 Preliminaries
Given images x ~ qxx). define the induced latent distribution gz by Zo= = E(x), 2n q2 (2o), where
is the encoder mapping images into latent space. A latent diffusion model learns Po 92 (zo). from which we can efficiently sample. Let D be a decoder that reconstructs images via x = ((zo). The forward noising process corrupts the clean latent Zo via Gaussian perturbations:
Varzo + VI are, e~ (0,,
(I)
3

---

Table 1: Distributions involved in dataset distillation, All are defined over the joint space X' J.
where X' denotes the image space and y denotes the label space.
Distribution
Image Source
Label Source
Description
Mune (x, y)
Real (full) Images
Ground -ruth y(x)
True data-labo distribution
Vaaiill
(soft) (x, y)
Distilled (generated) Images
Teacher soft label t(x)
Distilled data with soft label
V distill
(hard) (x))
Distilled (generated) Images
One-hot label Yonebot (x)
Distilled data with hard label
View y)
Distilled (generated) Images
Student logit output s(x)
Output of the model trained
where dr controls the noise schedule. The reverse process reconstructs clean samples via a parame- terized denoising function Es (zc t), iteratively refining noisy inputs with update function s:
2 szz, t, (Z,, f))-
Guided diffusion |4 modifies this process by introducing an auxiliary guidance function G(z t) that adjusts the sampling trajectory. This allows generation to be conditioned on labels, structural priors, or more abstract objectives by modifying s(z,, t, EΦ) to optimize for task-specific constraint.
Influence- Guided Diffusion (IGD) (4 leverages guided diffusion for dataset distillation by modifying the reverse sampling process to generate vraniingopptimal data. Instead of passively sampling from (x)), IGD introduces trajectory influence function and diversity function into the diffusion process to prioritize samples. Given a guided diffusion framework, IGD modifies the sampling update as:
2 s(z,, L, cΦ) p. V. Gr 1) 7: v,, (Zt),
where (z(,,,) t) represents the influence function for global distributional trajectory matching, and (Z() enforces diversity to prevent redundancy in the distilled dataset.
4
Methods
4.1 Problem Statement
Dataset distillation aims to construct a compact distilled dataset S Vaisuu y) (this means that Vaisüll denotes the empirical distribution over dataset S) from real, full dataset T = Murue (X. (x,y). such that |S| < IT|. A student model trained on S should mimic the performance of training on T, i.e. the output distribution Vnew (x,y) of the student model should remain close the ground- truth distribution Mune y). We formulate the dataset distillation objective as an OT minimization problem, where minimize the Wasserstein distance W (Atme, Vnew) between the ground-truth and student-induced distributions. The key distributions involved in the formulations are summarized in Table We provide a detailed description of symbols in Appendix [
4.2 Reconstructing the Optimal Transport Distance
We now provide a theoretical decomposition of our objective, W(Anee View), by introducing two key principles: (1) the triangle inequality partitioning the discrepancy introduced before and after distilled set construction, and (2) a multiplicative contraction term reflecting the benefit of soft labels over hard labels. Unlike the commonly used measures such as KL divergence and cosine similarity, which do not satisfy the triangle inequality, the Wasserstein distance W(.,)) is a proper metric on the space of distributions. This property allows to decompose the total discrepancy
W ((Pree Pnew) S W (Htrue disill
(hard) + W (y distill PPew).
(hard)
As soft labels better approximate the true label distribution than one-hot assignments, recent works adopt soft labels instead of hard one-hot assignments, leading to the following relaxed upper bound:
W W(Pan Puree; Vnew) S W(
V (soft)) distill
+ (I (soft) disttll (neww)
(5)
Dataset Discrepancy
Login Matching Error
The first term captures the mismatch between the distilled and real data distribution. The second term reflects the losit-wise alignment between the student model's output distribution and the soft labels

---

of the distilled data. Directly minimizing the first term is challenging due to too many variables to optimize. To analyze it further, we model the soft label advantage via a multiplicative relation:
(saft) W (Muoe distill
(paner (hard))
(soft)) distill
distill
(6)
The contraction factor a (v disull) measures how much soft labels reduce the discrepancy between label and image distributions compared to hard labels alone. The contraction is achieved by matching the complexity of the teacher-provided soft label distributions to that the distilled image distribution. (hard) and Mrue use one-hot (hard) labels, their Wasserstein distance can be computed class-wise, by independently solving optimal transport between images of the same category:
Since both disgill
W( V (hard) = Ey [( (x y), V/
(hard)
(x v)
where Ey denotes the expectation over label classes, which measures the average conditional Wasser- stein distance across classes. Putting everything together, we arrive at structured upper bound:
W (Horue: 1//ww)
S By W[Pno v),
ddistil (x ())
ar (saft)
(soft) + ((V /neww
(8)
where each term is controlled by a distinct design choice: OTT-guided diffusion sampling, label-image- aligned soft label relabeling, and OT-based logit matching between the student model and the distilled
dataset. This decomposition allows principled basis of our method, which explicitly targets at
minimization of each component. Our pseudocodes are provided in Appendix B
4.3 OT-guided Diffusion Sampling (OTG)
In the remainder of this section, we optimize the three terms in Equation 8 sequentially. We now concentrate on the first term. For each class 6, we minimize the class-conditional OT distance W(Pne | c), disill [hard] (x c)) through diffusion guidance: we compute the OT distance between the distilled images and the real images the latent space as the guiding function. At each diffusion step during the generation of the n-th latent zy. we draw a random batch of class-c samples from dataset T and encode them into latent representations Zg. We then employ the following guidance function:
Gw(z,) W(M, 20] Z
where M denotes previously sampled n-1 latents for class c, and represents the pythonic concatenation. We denote [M zi] as M The OT matrix pd can be efficiently approximated:
1 Vi. (10)
A argmin P. (.D n. zZ)) >p)(P), where ΣPd 7
Vj. Σ
P
where /(P) is the entropy of P, (:--) denotes the Frobenius inner product, A > 0 is the entropy regularization weight, DMM; Z4) represents the cost matrix measuring the pairwise distance between the real latent representations Ziy and the sampled Mę Without loss of generality, the nn-norm cost matrix and initialize the candidate transport matrix K" as:
D
Dij (M, ZTy) = M 2yD ll,, K = exp(-
(11)
A1
Next, Sinkhorn normalization is applied through iterative updates to K:
K <- diag (Ki-II, @ (n11z+1) K K <- K'diag (
T 1 1z 2 ( 1,))
-1
(12)
where @ denotes element-wise division, (.)T indicates matrix transpose. After T iterations, the optimal transport matrix 'dd obtained, and we can compute the Sinkhorn distance (an approximation of the OT distance) W as:
pd = KT. W( D3) (P. 22), - -E))
(13)
5

---

For the entire guided diffusion sampling, we follow the previous approach to combine the terms of trajectory and diversity functions with our OT function. The iteration of = TD + 1 yields zú: (14)
zi-1 = s(zi, (.)) - pr Vx; 1) Vx;z - Bi V Gw;G(zz))
where Pt. "t and 91 are weights, (z 1) and (D(z;) are trajectory and diversity functions, re- spectively. By minimizing the OT distance in the image distribution space, we account for the contribution of individual real images and incorporate both global and local structural information, thereby promoting fine-grained geometric alignment between distributions. Finally, we use decoder D to convert all latent representations into images, forming the distilled image set Sx.
4.4 Label-Image-Aligan Soft Label Relabeling (LIA)
We now focus on the contraction factor (eoft)) ), which characterizes the alignment between the complexity of the soft label distribution and that of the distilled image distribution (Appendix G.2 for details). Since the representational capacity of the distilled dataset is primarily governed by the number of images per class (IPC), we adopt an IPC-aware strategy that minimizes the overall OT distance. In low-IPC regimes, the distilled image distribution is less expressive and more prone to overfitting. Assigning overly complex soft labels in such cases can introduce distributional mismatch and degrade alignment. To mitigate this, we employ a smaller number of representative teacher models to produce simplified, low-entropy soft labels that offer well-calibrated supervision. In contrast, high-IPC regimes enable the distilled dataset to support greater semantic diversity. Accordingly, we leverage larger and more diverse set of teacher models to generate fine- fine-grained soft label distributions, which better capture the intrinsic structure of the true label space. Formally, for each synthetic image, we assign the soft label the averaged output from a set of IPC-dependent teachers:
1
t(x:)= [T(IS NET((S2I)
Σ
Fx(x,)
Σ Fi(x:); for each x; Sx: KET(IPC)
(15)
[T(IPC)
where F2 denotes the logit output function of the (-th teacher, t(x:) denotes soft label for image Xs, and T(IPC) is subset of teacher models selected minimize the contraction factor a. This strategic relabeling ensures that the soft label distribution faithfully matches the capacity of the distilled images, reducing the discrepancy term W(D disull Mume) and thereby improving alignment.
(soft)
For a fair comparison with prior methods [27 28 we adopt the same region-level soft label storage strategy in FKD 48
4.5 OT-based Student Model Logit Matching (OTM)
After obtaining a distilled set that preserves rich geometric structures of the real data, we transfer this information to student models (i.c., new models) by aligning the distribution of their logits with that of the real dataset. We achieve this alignment by minimizing the last term in the upper bound of Equation g We consider a batch of b samples and denote the soft labels of this batch and the logit output of student model as t and s, respectively. Most traditional divergence measures operate on a per-sample basis and match logits independently. thereby failing to capture inter-sample relationships. To address this limitation, we employ batch-wise OT distance that aligns logits while capturing global distributional structure. Specifically, similar to Section 4.3] we use the Sinkhorn method to efficiently solve for the OT matrix Pdu, with entropy regularization i((P) weighted by A2:
A argmin (P, C(t,s)) - X2h(P), where Σ Vj. Σ b Vi. P
(16)
Here, C E h is the batch-wise cost matrix, where each entry Ct measures the distance between the soft label t(x,) and the synthetic output s(x,). Specifically, we employ the f--norm:
Ciy(t, = t(x,) s(x,) llp LSD = W(t, s) = (P*2, C).
(17)
Adapting Equations 11 12] and 13 to current dimensions, we compute pda and the corresponding
batch-wise Sinkhorn distance loss Lsp- For a fair comparison with previous methods 40 23 we
use the cross-entropy loss LEE, the MSE loss LMSE. and the Sinkhorn loss LSD for distillation:
Σ Ky
(x;), + Rig CMsE (t(x,), s(x:)) + B2LsD,
(18)
where K1, K2, and Bg are scalar weights, Blonchor (x;) denotes the hard label for the distilled image x;-

---

Table 2: Performance comparison on ImageNet- IK 22 with ResNet-18. The numbers in parentheses for "Ours"
"Ourr"" represent the number of training epochs on the distilled set for new models.
Comparsion with generative-mone model-hased methods.
IPC
DPM
D'M
TDSDM
DDPS 46
DIT-IGD
(1000)
10
23.6+0.11 32.2=0.1
27.9:0.7
44.5:-0.4 39.4=0.3
39.610.4 52.9066
42.1--033 59.4+0.2
42.1+0.3 59.40.2
45.5440.5 59,8+0,3
52,9:10.1 61.9:0.55
08.6-0.1
50
55.2+0.3
64.2:0.4
Comparsion with model inversion based methods
IPC
SRe/I.
40
2X
2d
41
EDC
25
(300)
(1000)
10
21.3+0.6
31,4+0.5
42.0+0.1
33,5+0,3
32. 12.1-
48.6+-0.3 58.0.10.2
46.0+-0.6
52,9++0,1
58.6-0.1
50
61.9.0.5
Table 3: Other architecture performance comparison on ImageNet- IK [2]
MobileNet-Ve
Efficienteet-00
Swin Transformer
ConvNeXt
Method
IPC10
IPC50
IPC10
IPC50
IPC10
IPC50
IPC10
IPC50
SR 27
10.24-2.6
31.8+0.3
11.4=2.5
34.8-0.4
4.8+00.6
42.1+0.3
4.14-0.4
48.8++0,2
RDED 28
40.4+00.1
53.3+0.2
31.0-0.1
58.5-004
42.3+0.6
53.2+0.8
48.3+00.
65.440.4
EDC 23
45.0=10.2
57.8+0.1
51.1=0.3
60.9=0.2
46.0+0.55
57.9+0.3
54.410.2
66.640.2
DIT-IGD
39.2.10.2
57.8.0.2
47.710.1
62.0.0.11
44.1110.6
58.610.55
51.910.2
66.8.00.5
Ours (300)
51,0++.66
61.0+0.4
56.7-0.2
64.4=0.1
50.2++.2
68.2+0.1
61.2+00.1
70.2++0.8
Ours (500)
54.6110.
63.0:10.4
59.6-0.2
66.0=0.6
56.2+11.
69.4:0.1
64.5.10.3
71.11:1111
Ours (1000)
57.610011
63.9.1.22
62.4 0.1
66.8
63.7:10.2
7..5:0.1
67.0110.1
71.8:10.9
5 Experiments
5.1 Experimental Settings
Dataset. Given our primary focus on large-scale dataset distillation, we evaluate method the full ImageNet-1k 1K dataset [22]. To ensure comparability across varying category scales, we further conduct experiments
widely used subsets, ImageNet-1100 100 [33 and ImageNette (49). To construct comprehensive benchmark covering both low-resolution and high-resolution settings, we additionally include CIFAR-100 [21]. The dataset descriptions are presented Appendix
Network architectures. To evaluate the generalization capability of our method, we experi- ment with diverse of network architectures, including convolutional neural networks (CNNs), transformer-bast models, and hybrid models. Specifically, we consider CNN-based architectures, including ResNet [50], MobileNet EfficiienN (52). and ConvNet [53]: a transformer-basee model, the Swin Transformer [54]; and the hybrid architecture ConvNeXt [55]. This selection provides a comprehensive evaluation across diverse architectural paradigms and inductive biases.
Baselines. We compare our approach with a broad range of dataset distillation methods. Specifically, we include traditional methods such as DM [16), IDC
[13 and DATM (56]; moodel-nnverionn based
methods including S I. (27] G-VBSM 40 RDED 28 CDA (26] SC-DD 41 EDC [23] CV-
DD 25), and DELT [ as well as generative-moder modeiie--mdill-aeeeee methods such D
31, D (2),
TDSDM 33, DIT [4), Minimax [45), DDPS [6). and IGD (24]. We report the top-1 test accuracy
of models trained on distilled datasets with different IPC (Images Per Class) settings to ensure fair and consistent comparison. Each network is trained five times from scratch to report error bars.
Implementation details. To ensure fair evaluation, we follow the configurations of IGD 24 and EDC 23), maintaining consistency in training procedure and hyperparameter settings. For the OT components, we set a1 1, 71 e {1000, 3000), 02 = 0.1, and 7/2 = 0.1. For simplicity, we set p (fr-norm). More details are in Appendix
5.2 Results and Discussions
extensively evaluated our generative OT framework on ImageNet- 1K (22.. comparing it against maae-offthe--at dataset distillation methods, including both generative model- and model-inversion-baal approaches, across various architectures and IPC settings. Table 2 presents results on ResNet-18 [50]. Our method significantly outperforms prior methods
Results on ImageNet-IK.
7

---

Table 4: Performance comparison on ImageNette [49].
Model
ConvNet-6
ResNetAP-10
ResNet-18
IPC
10
50
100
100
10
50
Hard Label
Random
46.0=0.5
7.8
79.940.8
$4.2#
17.3=10
81.140.6
55.8+1.0
75.8=11
82.010.4
DM [16
4918=1.1
70.3+0.8
78.5+088
60.2++0.7
76.7+11.1
80,940077
60.9+0..7
75.0=1.0
81,5+0.4
IDC-1 13
48.2=1.2
72.4+:0.7
80.6+1.1
60.4+00.6
77.4=0.7
81.5+122
61.0++088
77.5=1.0
81.7+008
DIT 44
56.2=13
74.110.6
78.240.3
62.8.0.8
76.9:0.5
80.1
62.5109
75.2=007
77.4+0.7
Minimax 45
58.2=0.9
76.940.8
81.1+0.3
63.2:+1.0
78.2+0.77
81.5+10
64.940.6
08.1=06
81,340,7
DiT-IGD
6..9-199
80,9+099
84.5+0.7
66.5++1.1
81.0+1.2 1.2
85,2++0.8
67.7+003
80.40088
84,4+0888
Ours
6.90=0.9
83.11100
86.5+00.5
0..0=
0.68
86.4:10.6
69.11:199
84.6=0.4
85.9:0.2
Soft Label
SRe'L 27
29.4:13.00
40.9=0.3
50,2+0,4
RDED 88
63.5=0.6
84.3+0.3
89.2+00.7
60.8:00.5
80.5
89.31:0.6
61.410.4
80.4=04
89.6:1.0
D'M 32
03.
84.410.4
89.6#0.2
06.2:1..3
84.7=0.5
90.24::.3
57.410..
84.8=0022
90.410.7
DDPS
62.5:002
8..
90,240,2
DDPS
46
60.4+00.
85.8-044
91.6+0.4
DIT-IGD*
24
69.6=110
1.0
86.7:10.9
89,9:10.6
1.3
86.8+
90.6:1:066
74.8:10.7
86.4=099
90.7.10.5
Ours
0.3
89.11:0.9
91.3-10.2
77.8:10.8
89.7
91.6110.3
79.0:10.3
0.33-0.3
92.0:1066
94.3+-0.5
94.61:0.5
95.3-0.6
Table 5: Performance comparison on ImageNet- 100
13
Table 6: Performance comparison on
Model
IPC
SRc (27
RDED [8]
DELT 57]
Ours
CIFAR-100 (21] using ConvNet-3 [53
10
9.5.1..4
36.010.3
28.2+:15
47.7
IPC
10
50
100
ResNet-18
50
27.0:10.4
61.610.1
67.91006
72.610.1
DM 16
29.7-0.3
43.6+0.4
47.140..4
100
30.41.0.3
74.540.
75.1+002
79.2+0.1
M3D [8]
42,410.2
50,910.7
52.1.10.6
10
6.4.1..1
33.910.11
22.41333
36.3::.55
DATM 56
47.2-04 55.04002
57.540.2
ResNet-101 101
50
25.7++0.3
66.0+-066
70.8+22.3
74.3+-0,2
SRc'L 27
24.5=0.4
45.2+:0.
46.6+:0.5
100
27.61:0.2
73.51:0.8
77.6+1.8
81.610.1
RDED 28
48.1:2.33
57.010.1
58.110.4
4.540.4
23.64-0.7
1588+0.2
43.2+-02
D'M
45.0-0.11
48.8+-0.3
50.340.2
MobileNet
50
18.4+0.2
51.5+00.8
55.0+1188
69,5+00,3
DIT-IDG
24
45.810.6
53.9100.6
55.910.4
100
22.1+00.3
70.8+1.11
76.74+0.3
7..0
Ours
50.7
57.5
58.7
at 300 epochs. When training is extended to 1000 epochs, performance further improves. This shows that our distilled images and soft labels contain sufficient information for continued optimiza- tion. Beyond ResNet, we evaluated generalization on MobileNet-V V2 [51], EfficientNet-Bi [52]. Swin Transformer [54), and ConvNeXt (55) (Table 3). Our framework consistently surpasses prior approaches across all architectures. The larger performance gain at lower IPC settings highlights its ability to better preserve fine-graines distributional details. When IPC is low, existing dataset distillation methods struggle to cover the full data distribution, leading to significant discrepancies between the learned distribution and the real distribution. In contrast, our approach explicitly aligns the latent space distribution, logit-level semantic consistency, and label-image relationships, ensuring that even with limited synthetic samples, our distilled set comprehensively represents the real data.
Results on ImageNet subsets. To further compare with prior works and to evaluate our method under reduced-category settings, we conduct experiments class selection and class count. As shown in Tables and 5 our method consistently outperforms all baselines. Notably, we observe significant performance improvements under both hard label and soft label settings. This demonstrates that OT-guided sampling effectively captures fine-grained sample information, contributing to the learning of the new model. During the subsequent OT distance minimization phases, this extracted information is systematically transferred to the new model. resulting in enhanced performance. Robustness tests are conducted Appendix G.8
ImageNet subsets, varying both
We evaluate our method on CIFAR- 100 21 to assess its generalizability on low-resolution datasets, as summarized in Table 6 To ensure a broad comparison, we include traditional low-resolution oriennted methods, along with model-inversion based and interative-moeel based methods, both specially designed for large-scale datasets. Unlike most existing methods that specialize in either low-resolution or high-resolution datasets, our approach achieves otate-of-the-art performance on ImageNet (22 while maintaining superior results on CIFAR. This further highlights the robustness of our Or-driven strategy in preserving distributional characteristics across scales.
Results on CIFAR-100.
8

---

Table Ablation Study on ImageNette [49 under IPC=10. Note:
Table 8: Mean runtime per class (sampling) or per epoch (match- ing) on ImageNet- K [2
Here, "wo LIA" denotes soft relabeling with the teacher ensemble from IPgh-IPC settings, without adapting to the current IPC.
Hard Label
Soft Label
Stage
IPC-50
Model
wlo OTG
OTG
wio OTG
w/o LIA
w/o OTM
Full
wlo OTG
97.1s
537.4s
Samp.
ConvNet-6
61.9
67.0
72.5
74.3
73.2
74.5
w OTG
97.74
540.3s
ResNetAP-10
66.5
68.0
74.2
76.4
75.9
77.8
ResNet-18
67.7
69.1
77.2
77.8
77.5
79.0
Match.
OTM
23.36
126.6s
Table 9: Distilled set generation time (IPC=10, ImageNet- Table 10: Effect of a ImageNette 1K, 8x4090). PreS: Presample, PostS: Postsample.
49
Teachers
ResNet-18 0.906
w/o LIA
LIA
EDC 23
3h PreS +3h PostS 5h Recover 0.4h Relahel
0.903
0.643
Ours
3.4h Diffusion Sample 0.3h Relabel
Avg. Acc.
76.0
76.2
77.1
Impact of different components. Our OT-guided diffusion sampling effectively transfers the geometric structure the image space distribution the distilled images. This alignment is further enhanced by the Label-Imane Mabel-Image-AMee Image-Aligned Soft Relabeling, which narrows the distributional gap between the distilled and real data. During student model training, the OT-based student logit matching module faithfully propagates this information to the new model. This further reinforces alignment between the original distribution Pine and the learned distribution Vnew- As shown Table each component involved in minimizing the OT distance plays a critical role, underscoring the necessity of aligning distributions throughout the entire pipeline. More validations are provided in Appendix G.3
Runtime analysis. As shown in Table 8 the additional time overhead introduced by our OT constraint is consistently less than 1%. Table Saaleeprovidee breakdown of the time required for each step in generating the distilled set for both our method and the state-of-the-at model-inversion- based method, EDC 23 Our approach is notably faster than EDC, which further demonstrates its efficiency.
Discussion of contraction factor a. Table 10 reports the values of a measured under different soft label generation strategies. Our LIA strategy significantly reduces the OT distance between the distilled data and the real data, allowing the distilled data to capture more information of the real distribution. This leads substantial performance improvement. More discussion Appendix G.2
Sensitivity analysis. As shown Table 11 our method delivers consistently high accuracy over broad range of OT-rrllted hyperparameter settings, demonstrating low sensitivity. This robustness eliminates exhaustive tuning and enables straightforward deployment across diverse scenarios. also makes our approach readily scalable. Additional results and analyses are available in Appendix G.
Real
DTT
Ours
Figure 2: Comparison of generated images from different methods on ImageNet- 100 (IPC = 10).

---

Table 11: Impact of different OT hyperparameters.
81
A1
82
2
ConvNet
ResNetAP
ResNet
Hard Label
1000
67.0++0.9
68.0+0.3
1..1
10000
66.3+-0.7
68.5+-0.3
68.9+008
10
1000
05.8:10.5
67.510.5
68.7
Soft Label
1000
0.1
0.1
74.5++0.3
77.8+008
79.0+0.3
1000
0.1
1
74.6110.8
76.310.8
78.2.1..0
1000
0.1
7..310.5
78.110.3
77.410.2
50
50
40
Accuracy
45
Accuracy
20
IGD
IGD
EDC
10
EDC
Ours
Ours
25
100
150 Epoch
200
250
300
%
50
100
250 300 Epoch
150
200
Figure 3: Training accuracy curves of ResNet-18 (left) and MobileNet (right). IGD: DIT-IGD.
Visualizations. Figure iell@illlusuurr illustrates qualitative comparison among DIT 44 DiT-IGD 24 and our method. DiT often produces visually similar outputs that lack semantic diversity. DiT-IGD introduces diversity without aligning with the underlying real data distribution, leading non-representative incorrect generations. Furthermore, its influence estimation based solely on intraclass averaged statistics, which results in perceptible blurring. In contrast, instance-specific characteristics and ame-grained distributional structures, thereby enabling faithful approximation of the real data manifold. We also present the test accuracy at each logit-matching step in Figure 3 Our method achieves faster convergence and consistently higher accuracy, especially in early epochs, demonstrating superior sample informativeness and stronger distribution alignment when compared to EDC 23 and DiT-IGD 24] Please refer to Appendix H for more visualizations.
approach explicitly models both
Conclusion
We propose principled framework for generative large-scale dataset distillation by formulating it as an OT distance minimization problem. Our approach explicitly decomposes the total OT distance into three interpretable components and systematically minimizes each ensure comprehensive distributional alignment. This allows new models trained on distilled data to behave similarly to models trained on the full dataset, regardless of architecture. Extensive experiments across diverse datasets and model architectures validate the effectiveness and generalizability of our method.
Broader Impact. Our distilled datasets lower carbon footprints associated with new model training, fostering sustainable development. They also enable efficient learning in federated and continual learning scenarios, enhancing data privacy and model adaptation across distributed systems.
This work was supported by the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
Acknowledgement
Competing Interests
The authors declare competing interests.
10

---

References
[1] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: comprehensive review. arXiv preprint 0rXiv:2301.07014, 2023.
[2] Ping Liu and Jiawei Du. The evolution of dataset distillation: Toward scalable and generalizable solutions. arXiv preprint arXiv:2502.05677. 2025.
[3] Xiao Cui, Qi Sun, Min Wang, Li Li, Wengang Zhou, and Houqiang Li. Layoutene: Leverag- ing enhanced layout representations for transformer-basee complex scene synthesis. ACM Transactions on Multimedia Computing, Communications and Applications, 2025.
[4] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexci A Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018.
[5] Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepin- version. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8715-8724, 2020.
[6] Yuqi Jia, Saeed Vahidian. Jingwei Sun, Jianyi Zhang, Vyacheslav Kungurtsev, Neil Zhenqiang Gong, and Yiran Chen. Unlocking the potential of federated learning: The symphony of dataset distillation via deep generative latents. In European Conference on Computer Vision (ECCV), pages 18-33. Springer, 2024.
[7] Ze Chai, Zhipeng Gao, Yijing Lin, Chen Zhao, Xinlei Yu, and Zhiqiang Xie. Adaptive backdoor attacks against dataset distillation for federated learning. IEEE International Conference on Communications (ICC), pages 461 1611-440 4619, 2024.
[8] Enneng Yang, Li Shen, Zhenyi Wang, Tongliang Liu, and Guibing Guo. An efficient dataset condensation plugin and its application continual learning. Conference on Neural Informa- tion Processing Systems (NeurIPS), 36, 2023.
[9] Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memory- constrained online continual learning. In AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 12217-12225, 2024.
[10] Lechao Cheng, Kaifeng Chen, Jiyang Li, Shengeng Tang, Shufei Zhang, and Meng Wang. Dataset distillers are good label denoisers in the wild. arXiv preprint arXiv:2411.1 1924, 2024.
[11] Dongyao Zhu, Bowen Lei, Jie Zhang, Yanbo Fang. Yiqun Xie, Ruqi Zhang, and Dongkuan Xu. Rethinking data distillation: Do not overlook calibration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4935-4945, 2023.
[12] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations (ICLR), 2020.
[13] Jang-Hyun Kim, Jinuk Kim, Scong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Wo Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parame- terization. arXiv preprint arXiv:2205. /4959, 2022.
[14] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 4750-4759, 2022.
[15] Wenliang Zhong, Haoyu Tang, Qinghai Zheng, Mingzhu Xu, Yupeng Hu, and Liqiang Nie. Towards stable and storage-efficient dataset distillation: Matching convexified trajectory. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[16] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6514-6523, 2023,
[17] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 7856-7865, 2023.
11

---

[18] Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. M3d: Dataset condensation by minimizing maximum mean discrepancy. In AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 9314-9322, 2024.
[19] Shaobo Wang, Yicun Yang. Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: A minmax perspective. arXiv preprini arXiv:2502. 20653, 2025.
[20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE (Proc. IEEE), 86 (6(11))22 (122111):2782222 2324, 2002.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
[22] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
[23] Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[24] Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, and Wei Wang. Influence- guided diffusion for dataset distillation. In International Conference on Learning Representa- tions (ICLR), 2025.
[25] Jiacheng Cui, Zhaoyi Li, Xiaochen Ma, Xinyue Bi, Yaxin Luo, and Zhiqiang Shen. Dataset distillation via committee voting. arXiv preprint arXiv:2501.07575 2025.
[26] Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. Transactions on Machine Learning Research (TMLR), 2024.
[27] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[28] Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[29] Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepin- version. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8715-8724, 2020.
[30] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-ineremental learning. In IEEE International Conference on Computer Vision (ICCV), pages 9374-9384, 2021.
[31] Ali Abbasi, Ashkan Shahbazi. Hamed Pirsiavash, and Soheil Kolouri. One category one prompt: Dataset distillation using diffusion models. arXiv preprint arXiv:2403.07142 2024.
[32] Duo Su, Junjie Hou, Weizhi Gao, Yingjic Tian, and Bowen Tang. D'4: Dataset distillation via disentangled diffusion model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[33] Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. arXiv preprint arXiv:2310. 10402, 2023.
[34] Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. AAAI Conference on Artificial Intelligence (AAAI), 2025.
12

---

[35] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. IEEE International Conference on Computer Vision (ICCV), pages 17314-17324, 2023.
[36] Hansong Zhang, Shikun Li, Fanzhao Lin, Weiping Wang. Zhenxing Qian, and Shiming
Ge. Dance: Dual-view distribution alignment for dataset condensation. arXiv preprint arXiv:2406.0106 2024.
[37] Xiao Cui, Yulei Qin, Wengang Zhou, Hongsheng Li, and Houqiang Li. Optical: Leveraging optimal transport for contribution allocation in dataset distillation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2025.
[38] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Yan Zhu. Generalizing dataset distillation via deep generative prior. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[39] Jun- YeonY Moon, Jung Uk Kim, and Gyeong- Moon Park. Towards model- dataset condensation by heterogeneous models. arXiv preprint arXiv:2409. 14538, 2024.
Gyeong-Moon
[40] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. General- ized large-scale data condensation via various backbone and statistical matching. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[41] Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distilla- tion: A good compression is all you need. arXiv preprint vrXiv:2404.07976, 2024.
[42] Haoyang Liu, Yijiang Li, Tiancheng Xing, Peiran Wang, Vibhu Dalal, Luwei Li, Jingrui He, and Haohan Wang. Dataset distillation via the wasserstein metric. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1205-1215 2025.
[43] Xiao Cui, Yulei Qin, Xinyue Li, Wengang Zhou, Hongsheng Li, and Houqiang Li. Rethinking long-tailed dataset distillation: A uni-level framework with unbiased recovery and relabeling. arXiv preprint arXiv:2511. 1.1818..888.... 2025.
[44] William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4195-4205, 2023.
[45] Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang. Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[46] Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Min Zhang, and Bin Chen. Efficient dataset distillation via diffusion-driven patch selection for improved generalization. arXiv preprint arXiv:
2024.
[47] Jiwen Yu, Wei Wang. Xiaobo Zhang, Bo Huang, and Yi Wang. Freedom: Training-free energy-guided conditional diffusion model. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 23174-23184, 2023.
[48] Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. arXiv preprint arXiv:21 2021.
[49] Jeremy Howard. A smaller subset of 10 easily classified classes from imagenet, and a little more french. Project URL: https://github. http::/gghuub.co/ffsttaaaa/ com/fustai/mmagenttt, 4, 2019.
[50] Kaiming He,, Xiangyu Zhang, and Shaoqing Ren. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, Jun. 2016. IEFE.
[51] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4510-4520, Salt Lake City, UT, USA, Jun.
2018. IEEE
13

---

[52] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), pages 6105-6114 14. PMLR, 2019.
[53] Keiron O'Shea and Ryan Nash. An introduction convolutional neural networks. arXiv preprint arXiv: /51 1.08588, 2015.
[54] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference on Computer Vision (ICCV), pages 10012-10022. 2021.
[55] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1976-11986, 2022.
[56] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via fffficulty-aligneee trajectory matching. In International Conference on Learning Representations (ICLR), 2024.
[57] Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, and Shitong Shao. Delt: A simple diversity -riivn earlylate training for dataset distillation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2025.
[58] Cédric Villani and Cédric Villani. The wasserstein distances. Monograph: Optimal Transport: Old and New, pages 93 111111,7 2009.
[59] Jingwei Zhang, Tongliang Liu, and Dacheng Tao. An optimal transport analysis generaliza- tion in deep learning. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 34(6):2842 225-33333.,,, 2021.
[60] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning (ICML), pages 214-223, 2017.
[61] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. Conference on Neural Information Processing Systems (NeurIPS), 30, 2017.
[62] Artan Sheshmani, Yi-Zhuang You, Baturalp Buyukates, Amir Ziashahabi, and Salman Aves- timehr. Renormalization group flow, optimal transport, and diffusion-based generative model. Physical Review E (Phys. Rev. E), 111(():01534, 2025.
[63] Dylan Wheeler and Balasubramaniam Natarajan. Conceptual Icarning and causal reasoning for semantic communication. IEEE Transactions on Cognitive Communications and Networking (TCCN), 2025.
[64] Weilin Chen, Jie Qiao, Ruichu Cai, and Zhifeng Hao. On the role of entropy- loss for learning causal structure with continuous optimization. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2023.
[65] Tung Le, Khai Nguyen, Shanlin Sun, Nhat Ho, and Xiaohui Xic. Integrating efficient optimal transport and functional maps for unsupervised shape correspondence learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 23188-23198, 2024.
[66] Yonghao Liu, Fausto Giunchiglia. Ximing Li, Lan Huang. Xiaoyue Feng, and Renchu Guan. Enhancing unsupervised graph few-shot learning via set functions and optimal transport. arXiv preprint arXiv:2501.0560 2025.
[67] Eduardo Fernandes Montesuma, Adel El Habazi, and Fred Ngole Mboula. Unsupervised anomaly detection through mass repulsing optimal transport. arXiv preprint arXiv:2502. 12793, 2025.
[68] Jifei Luo, Xiao Cui, Hantao Yao, and Changsheng Xu. Neighbor-aware geodesic transportation for neighborhood refinery. OpenReview Preprint, 2025.
14

---

[69] Ali Baheri, Zahra Sharooei, and Chirayu Salgarkar. Wasserstein adaptive value estimation for actor-critic reinforcement learning. arXiv preprint arXiv:2501.10605 2025.
[70] Pascal Klink, Carlo D' Eraao, Jan Peters, and Joni Pajarinen. On the benefit of optimal transport for curriculum reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
[71] Yixing Lan, Xin Xu, Qiang Fang, and Jianye Hao. Sample efficient deep reinforcement learning with online state abstraction and causal transformer model prediction. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2023.
[72] Qiyuan Zhang, Shu Leng, Xiaoteng Ma, Qihan Liu, Xueqian Wang, Bin Liang, Yu Liu, and Jun Yang. Cvar-constrained policy optimization for safe reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2024.
[73] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Conference Neural Information Processing Systems (NeurIPS), 26, 2013.
[74] Jiafei Lyu, Mengbei Yan, Zhongjian Qiao, Runze Liu, Xiaoteng Ma, Deheng Ye, Jing- Wen Yang, Zongging Lu, and Xiu Li. Cross-domain offline policy adaptation with optimal transport and dataset constraint. International Conference on Learning Representations (ICLR), 2025.
[75] Zhichen Zeng. Boxin Du Si Zhang, Yinglong Xia, Zhining Liu, and Hanghang Tong. Hi- erarchical multi-marginal optimal transport for network alignment. In AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 16660-16668 2024.
[76] Okan Koc, Alexander Soen, Chao-Kai Chiang, and Masashi Sugiyama. Domain adaptation and entanglement: an optimal transport perspective. arXiv preprint arXiv:2503.08/55, 2025.
[77] Khai Nguyen, Hai Nguyen, Tuan Pham, and Nhat Ho. Lightspeed geometric dataset distance via sliced optimal transport. arXiv preprint 2tt::5501.890/ 2025.
[78] Lian-Bao Jin, Na Lei, Zhong- -uan Luo, Jin Wu, Chao Ai, and Xianfeng Gu. Semi-discrete optimal transport for long-tailed classification. Journal of Computer Science and Technology (0):))),0())2552266,, 2025.
[79] Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. Sinkd: Sinkhorn distance minimization for knowledge distillation. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2025.
[80] Xiao Cui, Yulei Qin, Yuting Gao. Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. Sinkhorn distance minimization for knowledge distillation. In International Joint Conference on Language Resources and Evaluation and International Conference on Computational Linguistics (LREC-COLING pages 14846-14858, 2024.
[81] Hyungjin Chung. Jeongsol Kim, Michael T. McCann, Marc L. Klasky, and Jong Chul Ye, Dif- fusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.1466 2022.
[82] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 8780- 8794, 2021.
[83] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.122 12598, 2022.
arXiv preprint
[84] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv: 22..0420 2022.
[85] Bahjat Kawar, Jiaming Song, Michael Elad, and Stefano Ermon. Denoising diffusion restora- tion models. In Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 23593-23606, 2022.
15

---

[86] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Conference on Neural Information P Processing Systems (NeurIPS), volume 35, pages 25683-25696, 2022.
[87] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusson models as auug-and-play priors. In Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 14715-14728, 2022.
[88] Nithin Gopalakrishnan Nair and Vishal M. Patel. Steered diffusion: A generalized framework for plug-and-play conditional image generation. In IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pages 20850-20860, 2023.
[89] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 843 032,,52,115 2023.
[90] Xiao Cui, Qi Sun, Wengang Zhou, and Houqiang Li. Exploring GPT-4 vision for text-to- image synthesis evaluation. In Tiny Papers Track at the International Conference Learning Representattonss (ICLR), 2024.
[91] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arxiv:2010.02500, 2020.
[92] Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[93] Yifan Wu, Jiawei Du, Ping Liu, Yuewei Lin, Wei Xu, and Wenqing Cheng. Dd-robustbench: An adversarial robustness benchmark for dataset distillation. IEEE Transactions Image Processing (TIP), 2025.
[94] Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950, 2020.
[95] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Learning Representations (ICLR), pages 7472-7482, 2019.
[96] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. 2018.
[97] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Conference on Neural Information Processing Systems (NeurIPS), 31, 2018.
[98] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), pages 7472-7482. PMLR, 2019.
[99] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research (JMLR), 9(1 (1:179999-26555 2008.
[100] Xiao Cui, Wengang Zhou, Yang Hu, Weilun Wang, and Houqiang Li. Heredity- aware child face image generation with latent space disentanglement. arXiv preprint arXiv:2108./1080
2021.
16

---

Appendix
A Overview
This appendix provides comprehensive supplementary materials to further elaborate on our method's theoretical foundations, experimental setup, and empirical findings. It includes the following sections:
Section B: More Related Work. Detailed discussions on prior studies, with an emphasis optimal transport and guided diffusion sampling
Section C Dataset Descriptions. Comprehensive descriptions of all datasets used in our experiments, including ImageNet-IK [22], ImageNette [49), ImageNet-100 [13], and CIFAR-I00 [2]]
Section D: Symbol Table. A complete summary of key mathematical notations, hyperpa- rameters, and definitions referenced throughout the paper.
Section E: Pseudocode. Step-by-step pseudocode for the proposed pipeline, detailed proce- dures for OT-alignment in different stages, and the calculation process for the contraction factor
Section F Implementation Details. Full specifications of hyperparameter settings, training schedules, and augmentation strategies across all datasets used in our experiments.
Section G: Further Experimental Analyses. Additional experiments, including sensitivity studies G.I), in-depth analysis of the contraction factor (..2), comparisons with alternative
distance metrics (G.3), runtime analysis (G.4), data coverage evaluation (G.5), expanded comparisons with other baselines (G.6 and G.7 robustness evaluation under adversarial attacks (.8)) and evaluation under extremely low-IPC settings|G. G.9
This section constitutes the of the appendix, offering deep empirical analyses of the contraction factor a and the robustness properties of the distilled models. Other experiments farther strengthen and extend the key findings presented in the main text.
Section H More Visualization Results. Additional qualitative visualizations, including t-SNE plots and synthesized images, to assess semantic coverage and distributional diversity.
Section Limitations. Critical discussion of the imitations of framework.
Section J Broader Impact. Reflections on the broader societal, ethical, and practical implications of our dataset distillation method.
Together, these supplementary materials provide a complete and transparent view of our method, support full reproducibility, and offer additional insights that complement and strengthen the main paper.
B More Related Work
Optimal transport OT theory provides a principled mathematical framework for comparing probability distributions by computing the minimal required to transform one distribution into another. Compared to KL divergence and Jensen-Shannon (JS) divergence, OT provides a more geometrically faithful measure of distributional differences, particularly when dealing with distributions with non-overlapping supports [58] [9]. The Wasserstein distance, also known as OT
distance, effectively quantifies distributional discrepancies and has been widely applied in image
generation [60)] 661, 62), causal discovery [63] 64), unsupervised learning [655] 66 67 68), and reinforcement learning (69 770
71 72]. However, its exact computation is intractable for high- dimensional data due to prohibitive complexity. To overcome this, the Sinkhorn distance introduces entropy regularization, making OT computation more efficient and numerically stable (73). This regularized variant extends OT applications to domain adaptation 74] 75, 76), classification 77. 78). and knowledge distillation 34
80 In this work, we propose a generative model-based OT framework designed to achieve precise distributional alignment throughout the dataset distillation process. Our approach optimizes the distilled dataset to minimize the OT distance between any student model's 's output distribution and the real data distribution, ensuring improved generalization.
17

---

Guided diffusion sampling Guided diffusion sampling enhances the generative capabilities of pre-trained diffusion models by incorporating external guidance during the reverse process to steer generation toward desired semantics [8I]. Early methods, such as classifier guidance [82] inject gradients from pre-trained classifiers into the sampling process to condition generation. However, this approach necessitates domain-speent classifiers trained on noisy intermediate latents, which are often impractical. Classifier- free guidance [83] addresses this limitation by training the model with both conditional and unconditional objectives, enabling control without external models. Building upon this, Wang et al. [84] introduced linear operator-based guidance, constraining the diffusion process to the null space of known measurement operators: nonetheless, this strategy faces challenges extend guidance to inverse problems through iterative optimization and blug-and-play conditioning. Concurrently, methods like
in generalizing to nonlinear mappings. Subsequent works [85] 86 87
those proposed by Gopalakrishhan Nair et al. [88], Yu et al. [47. and Bansal et al. [89] introduce generic guidance functions by injecting gradients from task-specific losses computed on denoised intermediate states, thereby broadening applicability without necessitating model retraining. Inspired by these methods, Influence- fuddde Diffusion (IGD) 24 leverages guided diffusion for dataset distillation by modifying the reverse sampling process to generate training-optimal data. However, its reliance on matching global distributional trajectories and introducing diversity through random perturbations often leads to suboptimal alignment, neglecting discriminative yet informative local characteristics in favor of global averaging, To overcome this limitation, we propose an optimal transport- based guidance strategy that explicitly aligns the geometric structure of real and synthetic distributions, achieving fine-gre mne-grained consistency in guided diffusion sampling.
C Dataset Description
ImageNet-IK ImageNet-IK K 22 also known as the ILSVRC 2012 dataset, is large-scale image classification benchmark comprising 1,000 object categories. It contains approximately 1.28 million training images, 50,000 validation images, and 100,000 test images. The dataset is organized according to the WordNet hierarchy, with each synset corresponding to a distinct semantic concept. ImageNet-1K has been instrumental in advancing deep learning research and remains standard benchmark for evaluating image classification models in large-scale settings.
ImageNette ImageNette [49] is curated subset of ImageNet, consisting of 10 relatively easy categories, including "tench", "English springer", "cassette player", "chain saw", "church", "French horn", "garbage truck", "gas pump". "golf ball", and "parachute". It was introduced to facilitate rapid experimentation and prototyping of image classification models, particularly under limited computational budgets. All images are resized resolution of 224 x 224 pixels, providing lightweight yet meaningful benchmark for distillation and robustness studies.
Net-eNee-100 ImageNet- 100 13 is another subset derived from ImageNet-IK. KK, comprising 100 randomly selected classes. Hach class typically contains around 1,000 training images and 300 test images, maintaining relatively balanced distribution. ImageNet-100 provides manageable yet challenging benchmark for evaluating classification performance, especially in scenarios where computational efficiency and rapid iteration are prioritized.
CIFAR-100 CIFAR-100 21 is a widely used benchmark dataset for image classification, extending the number of classes from 10 (CIFAR-10) to 100. Each class contains 600 images, with all images having a resolution of 32 x 32 pixels. Despite its compact size, CIFAR- 100 presents a significant classification challenge due its high intra-class variability and fine fine-grained label structure, making it a valuable resource for developing and assessing lightweight classification models.
D Symbol Description
To enhance elarity, a detailed description of mathematic symbols in the present study is provided in
Table [2]
18

---

Table 12: Descriptions of all symbols, functions, and hyperparameters introduced in the main paper.
Symbol
Definition
E
Encoder to transform image into the latent space
D S T
Decoder to reconstruct latent code back to the image space
Distilled (synthetic) dataset
Real (full) dataset
X Zo
Image
Latent code of clean sample
Z zi
Latent code of noisy sample at time step f
Latent code of class-c noisy sample at time step t Noise schedule controlling the perturbation at time step Gaussian noise
ar
E
Denoising function parameterized by Ø Reverse diffusion update function
G Gr
Guidance function in guided diffusion Influence function for general alignment Diversity function enforcing diversity distilled data Guidance function based optimal transport Previously sampled n. latents for class
GD
Gw
M
M
Concatenation of M and the latent zi under sampling Latent representations of class e from the real dataset Norm order
Zi
Ph
Optimal transport matrix for guided diffusion sampling with regularization
D
Latent space cost matrix for guided diffusion sampling
K'
Transport matrix at the t-th step of Sinkhorn normalization Sinkhorn iterations
T To Ad C Ft t s
Diffusion denoising iterations
Optimal transport matrix for logit matching with regularization Batch-wise cost matrix for logit matching
The logit output function of the t-th teacher
Soft label for batch
Student model output for a batch Soft label for the i-th image in batch One-hot hard label for the i-th image in batch Batch size
t;
Yonehot (x:)
h(P)
The entropy of P Distilled data distribution Distilled data distribution with soft label Distilled data distribution with hard label
Vdistill
soft
distill
1/ distill
hard
Muue
Real dataset distribution
Vnew
Output of the student model after training on the distilled set
W (Jrres Vnew)
Wasserstein distance between real dataset and student model output
Pt.
Weight for influence function in the reverse sampling process
Tit
Weight for diversity function the reverse sampling process
B1
Weight for the optimal transport guidance in the reverse sampling process
A
Entropy regularization weight for optimal transport matrix Entropy regularization weight for logit matching
X2 (soft)
a Maaill
Contraction factor quantifying the benefit of soft labels Images per class in the dataset Set of teacher models Distilled image set Weight for cross-entropy loss in logit matching Weight for mean squared error loss in logit matching Weight for Sinkhorn distance loss in logit matching
IPC
T
Sx
Ki
R2
B2
LeE
Cross-entropy loss
LASE
Mean squared error loss Sinkhorn distance loss
LSD
19

---

E
PseudoCode
We present the pseudocode for our pipeline Algorithm [ The detailed calculation the optimal transport (OT) distance for OT-guided Diffusion Sampling is provided in Algorithm 2 while the OT-bbased Student Model Logit matching is outlined in Algorithm 3 For efficient computation, approximate the contraction factors using features in the latent space, enabling dimensionality reduction while preserving critical information.
Algorithm OT-based Generative Dataset Distillation Framework
Require: Real dataset T = {(x,, 3:3}. teacher models T, target IPC, diffusion model G, encoder E, decoder D, student model S
Ensure: Distilled dataset Sx and trained student model S
for each class 1 to C' do
2: Encode real samples: Z <- [((x; V4- = c})
3: for sample index n = IPC do
4: Sample latent zi,, using diffusion model G
5: for == TD to do
6: Compute OT-guidance Gw(zf) w.r.t. Zg and previously sampled latents M
2-1
7: Update latent using guidance: zi-1 f- s(z;, - Pr V9, - 7/2 V9n 31 V9w
8: end for
9: Append - to M
10: end for
11: end for
12: Decode all latents: Sx <- D(M for all c
13: Select teacher set T(ICC) according to IPC level
14: for each image X; E Sx do
15: Generate soft label: t(x;) + [T Eier Fr(x:)
16: end for
for each training batch Sx do
18: Get soft labels t and student outputs S(B)
19:
Compute batch-wise OT loss LSD f- W(t,
20:
Compute per-sample CE and MSE loss: LCE = Σ LCE C.CE:(aooohan (Jonchot, s), LMSE LMse(t,
21:
Total loss: = Kj Ece + K2LMME + CyD
22:
Update student model S using gradient descent
23:
end for
24:
return Sx. S
Algorithm 2 Computation of OT-based Guidance for Image Latent Sampling
Require: Previously sampled latents - current latent zf, random batch of real class latents
Bg regularization weight '1 iteration number T
Ensure: Optimal transport distance as guidance value Gw
1: Concatenate latent: M +- [M 1 z
2: Compute cost matrix: D M Zg 1))l
3: Initialize kernel matrix: K expl D/A1)
4: Set iteration counter
5: while & < T do
6: Row normalization: K + diag K1, @ (n 112;1)
K
. diag (k 11271 @ (lZzl 1m) -1
7: Column normalization:
8: Increment iteration counter +- ++1
9:
end while
10: Final transport matrix: pd <- K
Compute guidance: Gw(2g) (Pd1 (.) Ens P.y
return Gw (2g)
20

---

Algorithm 3 Computation of Batch-wise OT for Student Logit Matching
Require: Teacher output t, Student output s, Hyper-parameter A2. Maximum number of iterations T
Ensure: Sinkhorn loss LsD
1: Apply softmax: t < Softmax(t), 8 Softmax(s)
2: Compute distance matrix Ci,(t,)) = || t(x;) s(x;) llr
3: Compute kernel matrix K <- exp
C
X2
4: Set iteration counter 6-0 0
5: while T do
6: Row normalization: K + K 2 (11,1))
7: Column normalization: K t K Ø (1.1g K)
8: Increment iteration counter + +t]
9: end while
10: Sinkhorn loss LSD + (K, C) Σis Cij
return LSD
Algorithm 4 Class-wise OT Distance in Label-Image Space for Contraction Factor a Calculation
Require: Real latent sets ZT E R-dixd distilled latent sets Zs Rxxxd
One-hot labels Hy E {0, 1 soft labels Ss E [0, 1] Regularization parameter E > 0, iterations T
Ensure: Average classwise OT distance Lavg
Compute pairwise cost matrix C₁₁ + ( Zs(i)lpp
2: Initialize list of valid class distances W
3: for each class c 1 to C do
4: â - H c. < Ss(:.c]
5: if E, à = 0 E, b, = 0 then
6: continue
7:
end if
8:
a a/ a, a
9:
b b b/ E, by
10: K+ exp( C/s)
11:
Initialize - 1/N
12:
Initialize + 1/N₂
13:
fort 10=11 to T do
14:
u u a/(K
5) v v 1--T/ .1 5)
16:
end for
17:
7 f diag(u) K diag(v)
18:
L. C.+DX Ei, Vis Cij
19: Append L. to list L
20: end for
21: W
[2] Σ. fa.
22: return W
21

---

F
Implementation Details
To ensure a fair and rigorous evaluation, we adopt the training protocols and experimental configu- rations established by IGD [4 and EDC [23] maintaining full consistency in model architecture, optimization settings, and evaluation pipelines. Following Minimax [45] and IGD, we utilize a latent DiT model from Pytorch's official repository and open-source VAE model from Stable Diffusion. DDIM [9]] with 50 denoised steps is used as the vanilla sampling method for generation. Also, all hyperparameters related to trajectory and diversity guidance are directly inherited from IGD, while the settings for student model logit matching follow those of EDC, with the exception of parameters introduced by our optimal transport (OT) framework. Notably, most of the OT-specific hyperparame- ters are set to fixed values across all datasets, and we observe that they require minimal tuning to achieve strong performance. This demonstrates the robustness of our method and its low sensitivity to OT parameter variations, Comprehensive hyperparameter configurations for all benchmark datasets including ImageNet-IK, ImageNette, ImageNet-100) and CIFAR-100 detailed in Tables 13 respectively.
15 and 16
Table 13: Hyperparameter setting on ImageNet-IK 22].
Config
Value
Explanation
Guided Diffusion Sampling
k
5
Pt VI ar
V (#alt)
7t B1
120 1
Weight for Diversity Guidance
Weight for OT Sampling Guidance
X1
1000
Entropy Regularization Weight
T
20
Sinkhorn Iterations, Same for Logit Matching
Soft Label Relabeling
Epochs Batch Size
300, 500, 1000
300 for comparison with most baselines Use 100 when IPC = 50
50
T(IPC 10)
ResNet-18, ShuffleNet ResNet-18, MobileNet, EfficientNet, ShuffleNet
NA
T(IPC 50)
NA
Student Model Logit Matching
Optimizer Learning Rate
Adam W
NA
0.001
Only use le-4 for Seiin-Tranffrmer
EMA Rate
0.99
Control EMA-based Evaluation Inherit from EDC Weight for LSD Entropy Regularization Weight
K1, A2
1, 0.025
B2
0.1
X
0.1
Scheduler
Smoothing LR Schedule RandomResizzcdrrpp
2
Augmentation
NA
RandommHoiizomtaaFiig
G Further Experimental Analyses
G.1 More Sensitivity Analysis
In the main text, we presented a preliminary sensitivity analysis of key hyperparameters. To further evaluate their impact on performance, we conduct extensive ablation studies, with results summarized in Figures 4 [5] 6 7 and Table Overall, we observe that our method exhibits strong robustness to most hyperparameter settings: performance remains stable across broad range of values. We select the hyperparameters by considering trade-offs among different architectures. Specifically, increasing the value of 31 slightly improves the performance of ConvNet, but degrades that of ResNet- 18. We therefore set f3, to balance this trade-off. Similarly, increasing 82 enhances performance on AP, but negatively affects both ResNet- 18 and ConvNet. Thus, we choose 32 = 0.1 to
22

---

Table 14: Hyperparameter setting on ImageNette [49].
Config
Value
Explanation
Guided Diffusion Sampling
k
5
Pt k. V1 of
(e)))))))))
V (r(Ro19
50 when IPC=10
74
120 when IPC=50 or 100
Weight for Diversity Guidance
B1
Weight for OT Sampling Guidance
1000 when IPC=10
A1
Entropy Regularization Weight
3000 when IPC=50 or 100
T
20
Sinkhorn Iterations, Same for Logit Matching
Soft Label Relabeling
Epochs
1000
Same for reproducing IGD
50 when IPC=10
Batch Size
NA
100 when IPC=50 or 100 ResNet-18, MobileNet ResNet-18, MobileNet, EfficientNet, ShuffleNet
T(PC 10)
NA
T(PPC 100)
Same for IPC=50
Student Model Logit Matching
Optimizer
AdamW
NA NA
Learning Rate
0.001
EMA Rate
0.99
Control EMA-baser Evaluation
K1, K2
1, 0.025
Inherit from EDC
B2
0.1
Weight for LSD Entropy Regularization Weight
A2
0.1
Scheduler
Smoothing LR Schedule RandAugment RandomResizedCrop RandommHorizonaalFiip
S <=
Augmentation
NA
80.0
80.0
77.5
77.5
75.0
75.0
Accuracy (%)
Accuracy (%)
72.5
70.0
70.0
67.5
67.5
65.0
ConvNet-6
65.0
ConvNet-6
ResNet-AP
AP
62.5
62.5
ResNet-18
ResNet-18
60.0
60.0
102
10a
10a
8
10
À
81
Figure 5: Effect of Xι (entropy regulariza- tion weight for sampling) on ImageNette 49 (IPC=10).
Figure 4: Effect of 3, (OT sampling weight) on ImageNette 49 (IPC=10).
achieve the best average performance. In contrast, reducing either 31 or 82 consistently harms all model variants, which highlights the importance and effectiveness of our OT-based regularization terms. Since the latent and feature spaces differ in scale, we apply separate scaling factors λ and X2 to normalize their contributions. As shown in the figures, setting X 1000 and X2 0.1 yields a favorable trade-off across architectures. Taken together, our approach demonstrates two complementary aspects of robustness: (1) most OT-related hyperparameters exhibit consistent
23

---

Table 15: Hyperparameter setting on ImageNet- 100
13
Config
Value
Explanation
Guided Diffusion Sampling
(x).))))
k
5
Pt VT axe V2 (2.10)
120
Weight for Diversity Guidance
B1
1
Weight for OT Sampling Guidance
A1
1000
Entropy Regularization Weight
T
20
Sinkhomn Iterations, Same for Logit Matching
Soft Label Relabeling
Epochs Batch Size
300
NA
100
NA
T(IPC 10)
ResNet-18, ShuffleNet ResNet-18, MobileNet, EfficientNet, ShuffleNet
NA
T(IPC 100)
Same for IPC=50
Student Model Logit Matching
Optimizer Learning Rate EMA Rate
AdamW
NA NA
0.001
0.99
Control EMA-based Evaluation
K1; K2
1, 0.025
Inherit from EDC
B2
0.1
Weight for LSD
Aa
0.1
Entropy Regularization Weight
Scheduler
Smoothing LR Schedule RandomResizedCrop
Augmentation
NA
RandomHHorizonaalFiip
Table 16: Hyperparameter setting on CIFAR-100 100
21
Config
Value
Explanation
Guided Diffusion Sampling
k
5
Pr.
Oue
V (8.
120
Weight for Diversity Guidance
81
Weight for OT Sampling Guidance Entropy Regularization Weight
A1
1000
20
Sinkhorn Iterations, Same for Logit Matching
Soft Label Relabeling
Epochs Batch Size T(IPC = 10)
1000
NA
50
NA
ResNet-18, ShuffleNet ResNet18, ConvNet, MobileNet, WRN, ShuffleNet
NA
T(IPC- 100)
Same for IPC=50
Student Model Logit Matching
Optimizer Learning Rate EMA Rate
AdamW
NA NA
0.001
0.99
Control EMMA-asse Evaluation Inherit from EDC
K2
1, 0.025
B2
0.1
Weight for LsD Entropy Regularization Weight
0.1
Scheduler
Smoothing LR Schedule RandAugment RandomResizzedCrop RandomHoorizonalFFiip
(=
Augmentation
NA
24

---

80.0
80.0
77.5
75.0
75.0
Accuracy (%)
Accuracy (%)
72.5
72.5
70.0
70.0
67.5
67.5
65.0
ConvNet-6
65.0
ConvNet-6
ResNet-AP
ResNet-AP
62.5
62.5
ResNet-18
ResNet- 18
60.0
0.2
0.4
0.6
0.8
60.0
0.2
0.4
Ag
0.6
0.8
1.0
82
Figure 7: Effect of X2 (entropy regularization weight for logit matching) on ImageNette [49 (IPC=10).
Figure 6: Effect of 32 (OT matching weight) on ImageNette [49] (IPC=10).
Table 17: Effect of 7 on ImageNette
[9
(IPC=10).
T
5
10
20
50
100
ResNet-18
77.9
78.6
79.0
78.8
78.9
ConvNet-6
73.8
74.3
74.5
74.7
74.6
behavior across different scenarios, requiring little to no manual adjustment, and in practice we only select X from the set f1000, 3000) while keeping all other OT-related hyperparameters fixed (see Section[ for default values); and (2) performance remains stable even when these parameters vary within reasonable ranges, eliminating the need of careful tuning.
G.2 Further Analysis of the Contraction Factor
We provide formal characterization and empirical analysis of the contraction factor a, which quantifies the degree to which soft labels reduce the discrepancy between the label and image distributions compared hard labels. This factor plays a critical role in interpreting the effectiveness of soft supervision in dataset distillation. For efficient computation, approximate the contraction factors using features in the latent space, enabling dimensionality reduction while preserving critical information.
Definition and Computation To compute a, we compare the conss-coniiional optimal transport (OT) distances from the real dataset to two variants of the distilled dataset: one annotated with soft (soft) labels distil and one with hard labels (hard) The contraction factor is then defined as the relative
improvement in transport distance under soft supervision: a W( (soft) //W (Ptree Maiiill (hard) )
Let ZT E N xd and Zs E Xd denote latent embeddings extracted from real and distilled images, respectively. We construct the pairwise cost matrix using an Lp norm:
C ZT Zs(i)lp,
C N
(19)
For each class c E (1...., C), extract the marginal label distributions over samples: a(e)
Hy
c) for real hard labels, and (ce) = Ss(:44 c) for distilled soft labels, where H, E (0, 1 x and
Ss [0, N2xC These are normalized into valid probability vectors:
(e)
(())
(0)
((e)
(20)
Σ o
Σ,
We then perform entropic regularized OT using the Sinkhorn algorithm. The Gibbs kernel is defined as:
= exp
(21)
25

---

Table 18: Effect of on ImageNet-1K 22 (IPC=10). Config A: ResNet- 18. Config B: ResNet-18. MobileNet, EfficientNet, ShuffleNet. Config C: ResNet-18, MobileNet, AlexNet, ShuffleNet. Config
D: ResNet-18, ShuffleNet.
Teachers
Config A
Config B
Config C
Config D
1.00
0.99
0.95
0.93
Ace (ResNet-18)
50.3
52.3
52.7
52.9
Acc (Swin)
47.2
47.8
49.2
50.2
Table 19: Effect of a on ImageNet-IK IK 22 (IPC=50). Config A: ResNet- 18. Config B: ResNet-18. MobileNet, EfficientNet, ShuffleNet. Config C: ResNet- 18, MobileNet, AlexNet, ShuffleNet. Config D: ResNet-18, ShuffleNet.
Teachers
Config A
Config
Config C
Config
0.97
0.16
0.97
1.00
Ace (ResNet-18)
62.3
61.9
60.8
60.5
Acc (Swin)
65.5
68.2
65.5
65.3
where controls regularization strength. The scaling vectors u and are initialized uniformly as:
u - 1/N1, v + 1/N2,
(22)
and iteratively updated as:
a(r)
u [+1
(23)
Kv + 8"
b(e)
(24)
K fuu+++2 0
where 5 ensures numerical stability. After T iterations, the transport plan is:
(0) diag(u) K diag(v),
(25)
and the classwise OT cost becomes:
L. = (7 (0) C), Σ
Vig
(c)
Ci..
(26)
1
Averaging over the valid class set C (i.e., classes with non-zero support in both distributions) yields:
1
W (pinee D. distill
fe.
(27)
C
GEC
To compute the counterpart W W(Prue (hard) dssill ), we replace Ss with its hard label projection S E
{0, 1) N2xC and repeat the same computation.
Empirical Insights We conduct several additional experiments, with results shown in Tables [8
and 19 Our empirical analysis provides several important observations regarding the role of the contraction factor in guiding effective distillation. We first find that is highly sensitive to the diversity and calibration quality of the teacher ensemble, and this sensitivity is modulated by the IPC (images per class) setting. In low-IPC regimes (e.g., IPC=10), using overly complex inconsistent
(sufft)) leading to smaller a
teacher predictions increases the optimal transport distance W /isiil values and ultimately harming the generalization ability of the distilled dataset. Conversely, when IPC sufficiently high (e.g.. IPC=50), stronger and more expressive teacher distributions better capture the semantic structure of the real data, resulting in larger values and improved alignment between real and synthetic distributions.
Second, we observe that deliberately reducing thereby explicitly minimizing the overall optimal transport distance, leads to significant improvements downstream model performance. This effect is particularly evident under both settings, where configurations with smaller (c.g.. Config D)
26

---

Table 20: Comparison of our optimal transport-bant based distance with other measures for guided
diffusion sampling on ImageNet- [22]
IPC
MMD
MMD (RKHS)
Ours
49.4
50.3
52.9
50
60.4
60.6
61.9
Table 21: Comparison between the sample-wise and batch-wise optimal transport distance for student model logit matching on ImageNet- IK [22]. OOM: CUDA out of memory.
ResNet-18
MobileNet-V2
Level
IPC=10
IPC=50
IPC=10
IPC=50
Sample-wise
50.6
60.8
OOM 51.0
OOM
Batch-wise
52.9
61.9
61.0
achieve better Top-1 accuracy across both the ResNet-18 18 and Swin Transformer. These results empirically confirm that shrinking the distributional gap through controlling a facilitates more efficient and effective knowledge transfer. Third, in high-IPC regimes, when single teacher is used (e.g., Config A), student models that share the same architecture as the teacher can fully exploit the teacher's architectural biases, achieving strong performance. However, such tight alignment may limit generalization to unseen architectures. By appropriately contracting a, we encourage the distilled dataset to encode more transferable, architecture architectur--iddeendeen features, thereby improving the student's ss adaptability to diverse downstream architectures. Overall, these findings validate a as principled and tunable indicator of distillation quality, and highlight the importance of strategic contraction strategies tailored to both teacher complexity and downstream generalization targets.
G.3
Comparison with Other Distance Measure
For guided diffusion sampling Unlike conventional metrics such as cosine similarity, KL diver- gence, or mean squared error, which rely on explicit instance-level alignment, optimal transport (OT) enables distribution-leve aligmment without enforeing one-to-one correspondences. This property makes OT particularly well-suited for dataset distillation scenarios, where the number of synthetic samples significantly smaller than that of the original dataset, and direct pairing is often infeasible or suboptimal. While Maximum Mean Discrepancy (MMD)-based measures have also been adopted for distribution alignment without requiring exact correspondences, they primarily focus on matching global distributional statistics and fail to capture fine-grainee pairwise relations between individual contrast, our OT-based formulation explicitly
instances the real and synthetic distributions. models such pairwise interactions and thus facilitates more accurate and semantically consistent guidance during the diffusion sampling process. As shown in Table 20 our method consistently outperforms MMD and MMD with reproducing kernel Hilbert spaces (RKHS) baselines ImageNet- IK under both low-IPC and high-IPC settings. These results underscore the importance of modeling instance-level correspondences for effective guidance and highlight the superiority of OT in capturing the geometry complex data distributions.
For student model logit matching Table 21 illustrates the consistent superiority of batch-wise OT distance over sample-wise OT distance. This result highlights that batch-wise Sinkhorn distance more effective in transferring the distributional geometry captured by the distilled set from label- image space to the newly trained student models. The sample-wisg logit matching approach treats each instance independently, failing account for the global structure and correlations within a batch.
Table 22: Performance comparison of logit matching methods on ImageNette (IPC=10).
Network
ConvNet-6
ResNetAP-10
ResNet-18
MMD
72.6
75.3
76.4
KI.
73.0
75.4
77.6
OTM
74.5
77.8
79.0
27

---

Table 23: Distribution coverage comparison among different methods.
Threshold
DiT
44
DiT-IGD
24
Ours
10
40.2
40.8
41.6
12
54.6
56.3
57.5
In contrast, our batch-wise formulation preserves inter-sample relationships, enabling more faithful distributional alignment and resulting in more robust knowledge transfer. Moreover, when dealing with datasets containing large number of classes (e.g.. 1,000 classes in ImageNet-IA IK), the batch-wise approach substantially reduces memory consumption and avoids the CUDA out-of-memory issues frequently encountered by sample-wise matching, further enhancing its scalability.Also although KL. and MMD serve as simpler divergences, they are inherently limited. KI. divergence is applied per sample and ignores inter-sample relationships, while MMD matches only global statistics. In contrast, our OTM applies batch-wis wise OT alignment between student logits and soft labels, capturing the joint distributional structure of samples. This enables OT faithfully preserve inter-sample geometry and match structural uncertainty in the soft labels, which KL and MMD overlook. As shown in Table 22 this leads clear performance gain:
G.4 More Discussions on Runtime
While EDC 23 reduces the runtime during the recovery phase compared to previous work, it does not fully optimize for multi-GPU parallelism across its various processes. Specifically, the pre- sampling and post-sampling phases during initialization do not benefit from multi-GPU parallelism, parallelizing these steps does not result in substantial time reduction. Moreover, the recovery phase is inherently constrained by data loading and model-inversion methods, and beyond four GPUs, further increases in parallelism yield minimal improvements runtime. In contrast, our approach is designed to optimize each class separately, with the sampling process dependent only on images sampled from the same class and the corresponding real images. As a result, our method scales more efficiently with the number of GPUs, with runtime decreasing nearly inversely proportional to the number of GPUs. Furthermore, when the need for high IPC arises, our method can be adapted to split high-IPC tasks into several Iower-IPC ones for parallel processing, maintaining strong parallel efficiency and further enhancing applicability in real-world scenarios.
G.5 Data Coverage Analysis
To assess the representational fidelity of the distilled dataset, we adopt a coverage-based evaluation metric. Specifically, for each data point the original dataset, we determine whether has least one nearest neighbor in the distance dataset within a predefined distance threshold. This metric reflects how well the surrogate data captures the underlying structure of the original distribution. As shown in Table 23 our method consistently achieves higher coverage compared to baseline methods across multiple thresholds. The improvements are observed over both the original DiT [44 model and DiT-IGD (24), indicating that our approach provides better distributional alignment. Notably, the performance gap widens as threshold increases, further validating the robustness of our distilled data in covering diverse modes of the original dataset.
G.6 Comparison with DWA
DWA 92 enhances diversity by adjusting the statistics of the squeezed network based on each generated sample. However, it still relies solely on global statistics, specifically the mean and variance associated with batch normalization (BN), and thus fails capture the rich instance-level information and geometric distributional structures inherent in the real dataset. Visualizations from the DWA paper further illustrate that, while the directed weight adjustment improves the diversity of the distilled dataset, the distribution remains concentrated, failing to adequately cover the majority of real data distribution. models, and the results clearly demonstrate significant performance advantage of our approach. This further emphasizes the importance of leveraging fine-grained instance-level information for achieving improved model performance and more faithful distributional alignment.
Table 24| we compare our method with DWA across multiple student
28

---

Table 24: Comparison with DWA
92 on ImageNet- IK [22].
ResNet-18
MobileNet-V2
EfficientNet-BO
Method
IPC10
IPC50
IPC10
IPC50
IPC10
IPC50
DWA 92
37.9:t0.2
55.2=0.2
29.1=:0.3 01.0+
51,610.5 61.0+0.4
37.4+0.5
56.3+0.4
Ours
52.9+0.11
61.9+-0.5
56.7++0.2
6..4+-0.1
Table 25: Comparison between WMDD 42 IK [22] under different IPC settings.
and our method on ImageNette 49
and ImageNet-
Dataset
ImageNette
ImageNet-1K
IPC
10
50
10
50
WMDD
64.8
83.5+0..3
57.6+0..5 52.9-1.11
38.2:10.2
Ours 79.0
89.3+t0.3
61.9-10.5
G.7 Comparison with WMDD
Although both our method and WMDD 42 utilize optimal transport (OT), they differ significantly in both methodology and motivation, leading to distinct formulations and implementations.
WMDD 42
is distributionn-attee based distillation method that applies OT in single, offline step to compute a Wasserstein barycenter over the real data's feature distribution. This barycenter then used a fixed target throughout training, where synthetic images are optimized to match it using standard 1.2 loss in the feature space. contrast, we introduce fundamentally different generative paradigm where OT is not a static, one-off computation, but a dynamic guidance mechanism integrated throughout the entire data synthesis and training pipeline. Specifically, OT guides the sampling of synthetic images by aligning latent representations, regulates the soft label relabeling process by matching label complexity to the image distribution, and structures the training loss of the student model by aligning its logits to the relabeled targets.
The motivation behind WMDD to replace the use simple data summaries, such as the feature means often targeted by MMD-based methods, with more geometrically meaningful summary, namely the Wasserstein barycenter, derived from the Wasserstein metric. contrast, our method is driven by the need to address inherent limitations in generative distillation pipelines, which often fail to preserve the fine-gres graingd geometry of the real data distribution iestribution-paaticuulla intra-class variations and local modes. These aspects explicitly addressed in our framework through multi-stage, OT-guided design.
We compare the top-1 accuracy of our method with WMDD under different images-per-class (IPC) settings on both the ImageNette and ImageNet- 1K datasets in Table
G.8 Robustness Evaluation
To assess the robustness of student models trained with distilled datasets, we follow the evaluation protocol established OD-RobustBencc (93], utilizing adversarial attacks implemented in the Tor- chAttacks library [94] As shown in Tables 2nd 27 we evaluate models trained on ImageNette (49 under IPC=10 and IPC=50 settings, measuring both standard test accuracy and adversarial robustness
against a variety of attack methods.
Our method consistently achieves higher clean accuracy and substantially improves robustness compared to MTT [14] across different perturbation budgets (jæl 4/255 and [s| 8/255). These improvements can be attributed the distributional properties enforeed by our optimal transport bbT)d distillation framework. By minimizing the OT distance between the synthetic and real data distributions, our method preserves not only class-level statistics but also fine- mme-grained, instance- level geometric structures. This leads to the learning of semantically faithful and smoother decision boundaries, which are inherently more resilient to adversarial perturbations. Moreover, our OT-guided diffusion sampling produces visually more coherent and perceptually realistic images compared to other types of approaches. The generated synthetic samples better preserve the semantic integrity and natural variability of the original data, providing stronger perceptual signals during model training.
29

---

As result, the student model benefits from more robust feature space that aligns well with human perception, further enhancing adversarial robustness beyond purely decision-boundary- boundary-leve effects.
In contrast, methods that primarily match global statistics or rely on heuristic trajectory guidance, such MTT, often produce synthetic datasets lacking such structural fidelity, resulting in brittle decision boundaries that are more vulnerable to attacks.
From theoretical perspective, prior works 195 96 have established strong connection between adversarial robustness and the sharpness decision boundaries: sharper, more irregular boundaries tend to amplify adversarial vulnerability, whereas flatter, smoother boundaries promote robustness. By aligning not only global distributions but also the local transportation cost between real and synthetic samples. OT encourages the distilled student model to form flatter and more coherent decision surfaces aligned with the real data geometry.
Moreover, from loss landscape perspective, minimizing the OT distance guides optimization towards flatter minima, where small input perturbations induce minimal output changes. This connection well supported by prior studies [97 198), which show that flatter loss surfaces correlate strongly with improved adversarial robustness. Together, these empirical and theoretical insights demonstrate that preserving distributional geometry via optimal transport provides principled and effective pathway for enhancing the adversarial robustness of models trained on distilled datasets.
Table 26: Performance comparison on DD-R Re-uobbutteech 933 evaluated on ImageNette 49 under perturbation budget of |s] 4/255. Results for MTT 14 are directly copied from the DD- RobustBench benchmark.
IPC=10
IPC=50
Attack Methods
MTT
[4
Ours
MTT [4]
Ours
Clean Accuracy
66.4
69.1
67.7
84.6
FGSM
10.8
20.8
24.0
PGD
4.6
9.2
2.6
9.8
CW
4.6
12.0
1.4
14.8
VMI
5.4
9.0
2.0
11.2
Jitter
12.2
20.4
13.0
23.8
Table 27: Performance comparison on DD-RobustBench [93 evaluated on ImageNette (49 under
perturbation budget of |s| 8/255. Results for MTT 14 RobustBench benchmark.
directly copied from the DD-
IPC=10
IPC=50
Attack Methods
MTT
14
Ours
MTT
[4
Ours
Clean Accuracy
69.1
67.6
84.6
FGSM
0.8
11.0
1.8
14.8
PGD
0.2
2.8
1.2
15.0
CW
0.2
9.6
0.2
6.8
VMI
0.2
0.8
2.0
Jitter
11.4
12.4
9.8
14.6
G.9 Evaluation on Low-IPC Settings
We have conducted additional experiments on ImageNet- IK 22 for the challenging settings of IPC=1, IPC=2, and IPC=5. The results of these experiments presented in the Table 28
Importantly, the IPC=1 setting, since only one synthetic image is generated per class, the OTG process cannot leverage previously distilled samples for alignment. Instead, for each class, we compute the OT distance between its single synthetic candidate and the corresponding real images in the latent space to guide generation.
30

---

Table 28: Performance comparison different methods on ImageNet-1K under small IPCs (1, 5). Best results are in bold.
Method
IPC
DM
FrePo
TESLA
SRe2L
RDED
EDC
DiT-IGD
Ours
1.5
7.5
7.7
0.4
6.6
12.8
10.7
15.9
1.7
9.7
10.5
16.5
5
23.8
39.5
38.6
45.7
H
More Visualization Results
H.1 T-SNE Results
To assess the effectiveness of our OT-guided diffusion sampling, we present the I-SNE 99 results in Figure The diversity in IGD is driven solely by simila-similarity based diversity guidance, without leveraging the distributional structure of the real dataset. This limitation leads to insufficient coverage of critical regions in the true data distribution, such as the central region of the green (Cassette player), the lower part of the blue (Tench), and the middle-upper section of the purple (Church) areas. Consequently, several important subclasses are absent from the distilled dataset, resulting in the new model failing to learn relevant intra-class variations and important subclass-specific information. In contrast, approach iteratively computes the optimal transport distance between the real dataset and the distilled set, explicitly incorporating both intra-class structures and finer substructures the real data. This enables our distilled dataset to capture a broader range of essential submodalities and regions, facilitating a more comprehensive transfer of information to the new model, and minimizing information loss. By employing the optimal transport distance during the new model's 's training, we ensure the effective transfer this enriched information, leading to significant improvements in model performance.
additional supervision signal
(a) IGD
(b) Ours
Figure 8: Visualization study for sample distributions of distilled datasets (IPC=10) generated by IGD [24] and Ours versus the original ImageNette [49] dataset. The dark points represent the distilled set, while the light points represent the real (original) set. The diversity in IGD 24 is driven solely by random diversity guidance, lacking awareness of the real data distribution. As result, it fails to cover critical regions such as such as the central region of the green class (Cassette player), the lower part of the blue class (Tench), and the middle-upper section of the purple class (Church). In contrast, our method incorporates both intra-class structures and fine-grained substructures of the real data, which allows it to effectively cover most subclass regions.
H.2 Distilled Images
Figures and 10
provide additional visual comparisons between IGD 24 and our method, as well as standalone visualizations of our distilled dataset. Our method effectively captures the structural information of the real data distribution, resulting high-fidelity samples with semantic diversity that faithfully reflects the underlying real-world distribution.
31

---

Lampshade
Blue Jean
Smoothing Tron
Doberman
Komondor
Christmas Stocking
Wardrobe
Volvet
Vault
Toilet Seat
Sunscreen
Spotlight
IGD
Ours
Figure 9: Additional visualization comparing the distilled datasets generated by IGD approach on ImageNet-II IK 22 (IPC=10).
24 and our
32

---

Yurt
Guacamole
Ice Cream
French Loaf
Paddlewheel
Megalith
Palace
Alp (high mountain)
Promontory
Goldfish
Jeep
Figure 10: Additional visualizations for our distilled set on ImageNet- 1K
22
(IPC=10).
33

---

I Limitations
Our current framework inherits the trajectory influence guidance mechanism from IGD 24 which, while effective for improving the general and global alignment of sampled data, introduces substantial computational overhead. Specifically, the additional sampling steps required to maintain trajectory consistency significantly slow down the generation process compared to the vanilla DiT [44], which operates without such constraints. In future work, we aim to reformulate guidance process to retain benefits while reducing the reliance on explicit trajectory tracking, thereby enabling faster and more scalable sampling.
Broader Impact
Our work aims to reduce dataset size while maintaining performance, enabling model training with significantly lower computational and storage costs. This can lower the entry barrier for institutions with limited resources and promote environmentally sustainable AT development [100]. Moreover, our distilled datasets have the potential to facilitate efficient learning in federated and continual learning scenarios, thereby enhancing data privacy and supporting model adaptation across distributed systems. However, as with most data-driven approaches, there exists risk that the distilled data may retain amplify biases present the original datasets. This could lead unintended consequences, particularly in sensitive applications. Additionally, by accelerating the deployment of compact models, our method may inadvertently contribute insufficiently audited systems being widely adopted. We emphasize the importance of responsible deployment, including bias auditing, fairness-awas aware design, and transparency, and encourage future work explore these aspects more thoroughly.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction accurately reflect the paper's contributions to generative large-scale dataset distillation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss this in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
35

---

Justification:
This paper does not include theorems or lemmas.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:| Yes]
Justification: Yes, we provide sufficient implementation details in Section 5.1 and Ap- pendix F
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived
well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification: We provide code in the supplementary materials, and provide anonymous Github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nipscc//pie wwcppbblccgguiicuceccccccoe.oopp tetggiddddeeeoeebbbiiinnnn
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setting is described in Section 5.] and Appendix F Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars Tables 2, 3, 5, 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
37

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resouces in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in the paper, including datasets, publicly available. Proper credits are given to the creators or original owners of these datasets where applicable. The licenses and terms of use for these datasets explicitly mentioned and respected in accordance with their respective guidelines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
39

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We have attached our code and user instructions in the supplementary materials
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
40

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euups..... wwwparippsscccoooeeeeeee cc/Conferencess///////e for what should or should be described.
41


Reviews:
[{"review_id": "omgGTLSQpm", "reviewer": "Reviewer_y7XW", "summary": "This paper proposes a novel dataset distillation method based on Optimal Transport (OT) aimed at achieving geometrical alignment between real and distilled data by minimizing OT distance. The core idea of the approach is to preserve local modes, intra-class patterns, and fine-grained variations in high-dimensional distributions, thereby enhancing the quality and generalization ability of the distilled data.", "questions": "Q1.  In Experiment Table 11, I noticed that the values of \\( \\lambda_1 \\) are set to 1000 and 10000, which seems unreasonable. As $\\lambda_1$ increases, the solution to the Sinkhorn algorithm becomes more uniform, leading to a weaker correlation with the image information. I suggest that the authors provide experiments with smaller values of \\( \\lambda_1 \\) to alleviate our concerns about the first part of the guidance not having a significant effect.\n\nQ2. I found that Section 4.5, \"OT-based Student Model Logit Matching,\" is somewhat similar to a recent paper [2]. I would appreciate it if the authors could provide a comparison and highlight the differences between the two.\n\n\n[2] SelKD: Selective Knowledge Distillation via Optimal Transport Perspective. ICLR2025", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation in Figure 1(a) is interesting, and the use of Wasserstein Distance in this context is reasonable.\n\n2. I appreciate the explanation in Section 4.2, \"Reconstructing the Optimal Transport Distance,\" where the author summarizes the three methods of the paper through an approximated inequality.\n\nWeaknesses:\n\n1. In Eq. 13, the author refers to $W=⟨D,K⟩$ as the optimal transport distance, which is not entirely accurate. Strictly speaking, it is an approximation of the Wasserstein Distance, or more specifically, the Sinkhorn distance, since $K$ is just an approximation of the exact solution. For more details, see [1].\n\n2. The author proposes three improvements, but I am unclear about which one plays a decisive role.\n\n3. I do not fully understand the significance of Label-Image-Aligned Soft Label Relabeling. I would appreciate it if the author could provide some diagrams to help readers understand, instead of a long explanation in text.\n\n\n\n[1] Sinkhorn distances: Lightspeed computation of optimal transport", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. Our responses are as follows.\n\n>\tQ1: In Eq. 13, the author refers to $W = \\langle D, K \\rangle$ as the optimal transport distance, which is not entirely accurate. Strictly speaking, it is an approximation of the Wasserstein Distance, or more specifically, the Sinkhorn distance, since $K$ is just an approximation of the exact solution. \n\nA1: We thank the reviewer for the accurate observation. We agree that $\\mathrm{W} $ is the Sinkhorn distance — an entropy-regularized approximation of the true Wasserstein distance. In our initial presentation, we used the term \"optimal transport distance\" to maintain notational simplicity, as the Sinkhorn method is a standard and computationally necessary approximation for high-dimensional data where the exact Wasserstein distance is intractable. In the final version, we will revise our phrasing to avoid any ambiguity. Specifically, we will **replace “optimal transport distance” with “Sinkhorn distance (an approximation of the OT distance)”** when referring to Eq. 13 and related components.\n\n>\tQ2: The author proposes three improvements, but I am unclear about which one plays a decisive role.\n\nA2: We sincerely thank you for this insightful question. It touches upon the core philosophy of our method's design, and we appreciate the opportunity to clarify the interplay between the three main components of our framework.\n\nWe wish to first clarify that the three components—(1) OT-guided Diffusion Sampling, (2) Label-Image-Aligned Soft Relabeling, and (3) OT-based Logit Matching—are not designed as independent \"improvements.\" Instead, they are three deeply interconnected and indispensable stages of a single, unified framework. Our core contribution is to reformulate the entire dataset distillation task as an end-to-end OT distance minimization problem, aiming to minimize $\\mathrm{W}(\\mu_{\\text{true}},\\nu_{\\text{new}})$. The three components the reviewer mentioned are precisely the optimization sub-goals derived from a principled, theoretical decomposition of this main objective (as shown in Equation 8 of our paper). They work synergistically to serve the final goal of distributional geometry alignment across the entire pipeline.\n\nTo empirically validate that each component plays a critical, non-redundant role, we conducted a detailed Ablation Study, with the results fully demonstrated in Table 7 (page 8). The results clearly show that:\nThe full framework achieves the highest accuracy.\nRemoving any single component leads to a significant performance drop. \nThis experiment strongly proves that all three components are integral to our framework's success. There is no single \"decisive\" component; rather, the strength of our method lies in their synergistic integration, which systematically aligns distributional geometry from image generation to student model training. We believe it is this holistic design, where each part addresses a specific term of the overall OT objective, that ultimately leads to our state-of-the-art performance.\n\nWe will make sure to further emphasize this synergistic relationship in the final version of our paper.\n\n>\tQ3: I do not fully understand the significance of Label-Image-Aligned Soft Label Relabeling. I would appreciate it if the author could provide some diagrams to help readers understand, instead of a long explanation in text.\n\nA3: We thank the reviewer for suggesting to include diagrams. Below, we present pseudo-diagrams and empirical evidence to clarify our method.\n\nMotivation: match soft-label distribution complexity to distilled image distribution capacity\n\nHigher IPC →  More complex distilled image distribution (higher diversity and expressiveness)  \n\nLarger teacher ensemble (when IPC is high)  → More complex soft label distributions with richer semantic modes\n\nCausal chain:\n\nGiven an IPC setting (e.g., 10 or 50)\n\n → LIA selects an IPC-adaptive teacher ensemble\n\n → Soft-label distribution complexity matches synthetic image capacity\n\n → Contraction factor $\\alpha$ in Eq. (6) decreases\n\n → OT upper bound in Eq. (8) tightens\n\n → Top-1 accuracy improves\n\nAs illustrated above, our Label-Image-Aligned Soft Label Relabeling (LIA) module adaptively selects a teacher ensemble based on the IPC setting. This ensures that the **complexity of the soft-label distribution matches the expressive capacity of the distilled images, thereby directly reducing the contraction factor  $\\alpha$ in Eq. (6). A smaller $\\alpha$ leads to a tighter OT upper bound in Eq. (8), which in turn yields better student model performance**.\n\nWe validate both the effectiveness of LIA in reducing $\\alpha$ and the downstream benefit of lower $\\alpha$ on student accuracy, across small- and large-scale datasets. \n\n- **LIA ⟶ Smaller $\\alpha$:**\n  - In **Table 10** (*ImageNette*), applying LIA reduces  $\\alpha$ from 0.906 to 0.643 under IPC=10.\n  - In **Tables 17 and 18** (*ImageNet-1K*), teacher configurations selected via LIA also consistently yield the lowest $\\alpha$ among all variants.\n\n- **Smaller $\\alpha$ ⟶ Higher Accuracy:**\n  - In all three tables, the setting with the **smallest $\\alpha$** also achieves the **best Top-1 accuracy**, confirming that $\\alpha$ is not only reduced but consequential to performance.\n\nThese results confirm that LIA reduces the OT alignment gap by controlling the contraction factor $\\alpha$, thus contributing directly to the overall generalization quality of the student model. We will insert the above diagrams (with proper styling) into the appendix and cross‑reference it in Section 4.4 to enhance readability.\n\n>\tQ4: In Experiment Table 11, I noticed that the values of ( \\lambda_1 ) are set to 1000 and 10000, which seems unreasonable. As λ1 increases, the solution to the Sinkhorn algorithm becomes more uniform, leading to a weaker correlation with the image information. I suggest that the authors provide experiments with smaller values of ( \\lambda_1 ) to alleviate our concerns about the first part of the guidance not having a significant effect.\n\nA4: We sincerely thank the reviewer for this thoughtful observation. Our choice of $\\lambda_1$ is driven by two key considerations: (1) the numerical stability of the Sinkhorn kernel and (2) alignment with the scale of the latent space in our framework. Specifically, the cost matrix $\\mathbf{D}$, computed in the latent space, have a **mean value typically falling in the range of 2000 to 3000**. Given that the transport kernel is computed as $\\exp(-\\mathbf{D} / \\lambda_1)$, using a small $\\lambda_1$ (e.g., 10 or 100) would result in extreme exponentials (e.g., $\\exp(-300)$), causing the kernel to underflow to near-zero entries. This in turn leads to a degenerate or uninformative transport plan, effectively nullifying gradient flow and rendering the guidance ineffective.\n\nTo ensure that the kernel matrix remains well-conditioned and that the resulting transport plan captures meaningful structure, $\\lambda_1$ must be chosen on a scale commensurate with the cost values in $\\mathbf{D}$. In our experiments, we found $\\lambda_1 = 1000$ to be a stable and effective setting, balancing expressiveness with numerical tractability. This choice is empirically supported by the sensitivity analysis presented in Appendix G.1 (Figure 5, page 22), where we evaluate a wide range of $\\lambda_1$ values from 100 to 10,000. As shown, performance degrades significantly for small $\\lambda_1$ values, while peaking around $\\lambda_1 = 1000$. Hence, our selected values are not arbitrary; they are grounded in both theoretical insight and empirical validation.\n\nWe will revise the appendix to explicitly include a note on the scale of $\\mathbf{D}$ and its influence on $\\lambda_1$, to better clarify this design choice. \n\n>\tQ5:  I found that Section 4.5, \"OT-based Student Model Logit Matching,\" is somewhat similar to a recent paper [2]. I would appreciate it if the authors could provide a comparison and highlight the differences between the two.\n\nA5:  We thank the reviewer for pointing out the relation to SelKD. While both methods utilize optimal transport (OT), they are designed for fundamentally different tasks, leading to distinct goals and formulations.\nSelKD introduces a flexible framework for **selective knowledge distillation**, where a student model is trained to learn only a subset of the teacher’s knowledge. It employs OT to **reconstruct a class-subset distribution by projecting the full teacher distribution onto the selected classes in a transport-cost-minimizing way**. The student is then aligned to this projected distribution using **KL divergence**, rather than directly minimizing the transport cost itself. This strategy is **well-suited for selective knowledge distillation or open-set problems**. However, in **standard closed-set full-class tasks** where all classes are retained, such teacher-label reallocation is **unnecessary**, rendering the **OT component of SelKD less relevant and offering limited benefits in this setting**. \n\nIn contrast, our work focuses on dataset distillation, where the goal is to synthesize a small dataset that enables new models to generalize across the **full label space**. The OT-based logit matching in Section 4.5 is one component within our broader framework. We apply the batch-wise Sinkhorn distance between student logits and generated soft labels to enhance supervision within each distilled mini-batch, effectively **capturing the distributional structure across all classes while preserving inter-sample geometry**.\n\nTo provide a direct comparison, we created a variant of our framework by replacing our OT-based Logit Matching (OTM) module with the loss function proposed by SelKD for the standard closed-set task. We denote this configuration as 'Ours (w/ SelKD loss)' :\n\nTable. Performance comparison on ImageNette (IPC=10)\n| Network|ConvNet-6|ResNetAP-10|ResNet-18|\n|-|-|-|-|\n|Ours (w/ SelKD loss)|72.6|74.2|75.8|\n|Ours|**74.5**|**77.8**|**79.0**|"}, {"review_id": "Mi9AYaEy6L", "reviewer": "Reviewer_kJL2", "summary": "This paper observes that recent methods for large-scale dataset distillation primarily focus on matching global distributional statistics (e.g., mean and variance) while overlooking critical instance-level characteristics and intra-class variation, which limits generalization. To address this, the authors reformulate dataset distillation as an Optimal Transport distance minimization problem, enabling fine-grained alignment at both global and instance levels. OT provides a geometric framework for distribution alignment, preserving local modes, intra-class structure, and high-dimensional distributional geometry. Their method comprises three components designed to preserve distributional structure: 1) OT-guided diffusion sampling to align the latent distributions of real and distilled images. 2) Label-image-aligned soft relabeling, which adjusts label distributions according to the complexity of distilled image distributions. 3) OT-based logit matching to align student model outputs with the soft-label distribution. Experiments across multiple architectures and datasets show that the proposed approach outperforms existing methods, achieving ~4% accuracy gain under the IPC=10 setting for each architecture on ImageNet-1K.", "questions": "Please see the *Strengths And Weaknesses* part above. My primary concern is whether OT provides a meaningful advantage for this task, and why it is truly needed. I would prefer not to see it used merely for novelty (e.g., just because no prior DD papers have used it) without concrete justification or practical benefit. In addition, some implementation details are missing, and I hope the authors will address these concerns thoroughly in the rebuttal.", "limitations": "The authors have discussed some of their method's limitations in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Applying Optimal Transport to generative dataset distillation appears novel as I am not aware of prior work using it in this context.\n2. The paper is generally well-written.\n3. The idea of aligning both global and instance-level distributions in the distillation process is interesting.\n4. The authors have included their code in the submission, which enhances the reliability of their results and shows good improvements over previous methods.\n\n\nWeaknesses\n1. The paper misses some technical details. For example, in Section 4.4 (label-image-aligned soft relabeling), the authors do not clarify whether soft labels are generated at the global image level or like SRe2L at the region level. These different operations could lead to significant differences in performance.\n2. In Tables 2 and 3, results from different training epochs (300, 500, 100) are presented in the same table. It would be more intuitive to split these into separate tables to allow for direct comparison with other baselines trained for the same number of epochs.\n3. The motivation for using Optimal Transport seems reasonable, I am concerned this may be another instance of a \"forced A+B\" combination, where a well-known technique is applied simply for novelty rather than clear necessity or demonstrated advantage.\n4. The overall pipeline appears relatively complex, especially the OT-guided diffusion sampling part, which may reduce its utilization in practice.", "was_revised": true, "final_justification": "The authors' response has generally addressed my concerns. My initial major doubt was whether combining Optimal Transport (OT) with dataset distillation was truly necessary. The authors explained that the benefit lies in capturing critical instance-level features and fine-grained geometric structures, which I find to be a reasonable justification.", "rebuttal": "We thank the reviewer for the constructive comments. Our responses are as follows.\n\n>\tQ1: The paper misses some technical details. For example, in Section 4.4 (label-image-aligned soft relabeling), the authors do not clarify whether soft labels are generated at the global image level or like SRe2L at the region level. These different operations could lead to significant differences in performance.\n\nA1：For a **fair comparison** with prior methods (e.g., SRe2L, RDED, G-VBSM, CDA, SC-DD, CV-DD, EDC, D$^4$M, TDSDM, Minimax, DDPS, DiT-IGD), we adopt the same region-level soft label storage strategy as in FKD [1]. Notably, **except for D$^3$M, all baselines in Tables 2 and 3 follow this convention**. We will explicitly clarify this detail in Section 4.4 of the final manuscript, to avoid confusion and ensure technical transparency.\n\n[1] A fast knowledge distillation framework for visual recognition. ECCV, 2022\n\n>\tQ2：In Tables 2 and 3, results from different training epochs (300, 500, 1000) are presented in the same table. It would be more intuitive to split these into separate tables to allow for direct comparison with other baselines trained for the same number of epochs.\n\nA2: Thank you for highlighting the clarity issue in Tables 2 and 3.\n\nWe agree that facilitating direct comparison is important. Our primary motivation for presenting the 300-, 500-, and 1000-epoch results together is to demonstrate that the distilled data is sufficiently rich to support continued performance improvement with longer training. The 300-epoch results are included for a direct and fair comparison, as 300 epochs represent the most commonly adopted (and typically the shortest) training schedule in prior work on ImageNet-1K. However, as you note, this standard is not universal; for instance, Minimax and DDPS use 2000 epochs for IPC=10, 1500 for IPC=50, and 1000 for IPC=100, while SC-DD uses 600 epochs for each IPC.\n\nTo address your concern while transparently presenting the full scope of our findings, we will revise the table layout in our final manuscript. Specifically, we will adopt the following structure:\n\n1. We will add a dedicated \"Epochs\" column to clearly and unambiguously state the training epochs for every method.\n\n2. We will apply a light‐gray background to all 300‑epoch entries so readers can identify the primary, apples‑to‑apples comparison at a glance.\n\n3. We will revise the table captions to explicitly state that the 300-epoch results serve as the primary point of comparison, while the 500- and 1000-epoch results demonstrate the continued improvement of our method.\n\n>\tQ3: The motivation for using Optimal Transport seems reasonable, I am concerned this may be another instance of a \"forced A+B\" combination, where a well-known technique is applied simply for novelty rather than clear necessity or demonstrated advantage.\n\nA3: We thank the reviewer for this critical question, as it touches upon the core of genuine research innovation. We wish to clarify that our framework is not a \"forced combination\" of Optimal Transport (OT) and dataset distillation for novelty's sake. Instead, our use of OT is a principled response to a fundamental limitation we identified in prior state-of-the-art methods. Existing advanced approaches in large-scale dataset distillation primarily focus on matching global statistics, such as mean and variance. However, this approach has an inherent weakness: **it overlooks the critical instance-level features and fine-grained geometric structures** within the data distribution, such as the various local modes or intra-class variations. By overlooking geometric details, the resulting distilled datasets fail to capture the real data's underlying structure and diversity, which in turn limits the generalization capability of models trained on such data. To overcome this bottleneck, a new theoretical framework capable of fundamentally understanding and preserving distributional geometry is needed, and Optimal Transport is the ideal choice for this task.\n\nTherefore, Optimal Transport is not an ancillary module in our method but its foundational principle throughout. We **reformulate the entire dataset distillation task as an end-to-end OT distance minimization problem**. This objective is then ingeniously decomposed into three synergistic stages of the distillation pipeline, ensuring geometric distribution alignment from start to finish. First, in the image generation stage, we employ \"OT-guided diffusion sampling\" to align the latent space distributions of real and generated images, thereby preserving complex local patterns. Second, in the label assignment stage, our \"label-image-aligned soft relabeling\" strategy adapts the complexity of the label distribution to match that of the distilled images, which is directly tied to minimizing the overall OT distance via a contraction factor. Finally, during student model training, we use \"OT-based logit matching,\" which aligns model outputs with soft labels at the batch level, effectively preserving the relational structure between samples that per-sample metrics would ignore.\n\nThe necessity and superiority of this OT-centric approach are not merely theoretical but are robustly and clearly demonstrated in our experiments. Most critically, our ablation studies in Table 7 provide direct evidence against the \"forced combination\" concern. The results show that when **removing any of the proposed OT-guided components (OTG, LIA, or OTM), the performance of all tested architectures drops significantly**, proving that these modules are indispensable for achieving our reported performance. This superior alignment translates into substantial performance gains, where our method consistently outperforms prior art, **achieving at least a 4% accuracy improvement on ImageNet-1K under the IPC=10 setting across all tested architectures**. Furthermore, the visualizations in Figure 2 confirm that our method produces **more diverse and representative samples that closely approximate the real data manifold**, whereas prior approaches often yield perceptibly blurred or semantically inaccurate generations. \n\nWe also demonstrate that **our individual OT-based modules are superior to common, simpler alternatives for their specific tasks**. For the sampling stage, Table 19 shows our **OT-Guided Sampling (OTG) significantly outperforms guidance based on MMD (Euclidean) and MMD (RKHS)**. This performance gap arises because metrics like MMD matches only aggregate moments and fails to explicitly model instance-level correspondences. In contrast, OT establishes pairwise couplings between real and synthetic samples, **enabling finer-grained alignment that better preserves the geometric structure of the data manifold**. For the relabeling stage, Tables 10, 17, and 18 demonstrate that our **Label-Image-Aligned (LIA) relabeling module consistently yields lower values of the contraction factor $\\alpha$** compared to randomly or statically selected teacher ensembles. This **reduction in $\\alpha$ directly translates into substantial performance improvements**. For the logit matching stage, the following results on ImageNette (IPC=10) show that our **OT-based Matching (OTM) consistently outperforms both KL Divergence and MMD**:\n\nTable. Performance comparison of logit matching methods on ImageNette (IPC=10).\n\n|Network|ConvNet-6| ResNetAP-10|ResNet-18|\n|-|-|-|-|\n|MMD|72.6|75.3|76.4|\n|KL|73.0|75.4|77.6|\n|Ours (OTM)|**74.5**|**77.8**|**79.0**|\n\nPer-sample methods like KL divergence are applied individually to each sample and ignore inter-sample relationships, while global metrics like MMD match only global statistics. In contrast, our **batch-wise OT (Sinkhorn) alignment** (OTM) captures the **joint distributional structure** of the logits and faithfully preserves **inter-sample geometry**, which other methods overlook.\n\nIn conclusion, we demonstrate that OT is a principled and effective method for resolving the geometric deficiencies of existing distillation methods, with its necessity and advantages fully validated by our experimental results.\n\n>\tQ4: The overall pipeline appears relatively complex, especially the OT-guided diffusion sampling part, which may reduce its utilization in practice.\n\nA4: We thank the reviewer for their valuable feedback regarding the perceived complexity of our pipeline. While the pipeline includes multiple stages, we demonstrate that it remains lightweight and straightforward to use. In fact, the OT-guided diffusion sampling part is implemented with minimal overhead. **Runtime profiling in Table 8 shows < 1 % extra cost.** Furthermore, the complete generation pipeline is notably faster than competing state-of-the-art methods such as EDC, with increasing advantage as the number of GPUs increases, highlighting its practicality for large-scale applications.\n\nBeyond its computational efficiency, we designed the framework for straightforward adoption. A notable strength of our method is that **a fixed set of OT-related hyperparameters generalizes well across datasets and model architectures**. Moreover, performance remains robust even under moderate parameter variations, as shown in Figures 4–7, demonstrating **low sensitivity and eliminating the need for exhaustive tuning**. This robustness significantly reduces the burden of deployment in new scenarios.\n\nThe necessity of each component, including the OT-guided sampling, is empirically validated by our ablation study, where removing any stage results in a significant degradation of performance. This confirms that the pipeline's structure is not unnecessarily complex but rather essential for preserving the fine-grained distributional geometry that enables our method to significantly outperform prior work. We will ensure these points on efficiency, ease of use, and justified design are more prominently highlighted in the final manuscript to better convey the practical value of our approach."}, {"review_id": "CQ9BdtQaIw", "reviewer": "Reviewer_i9LC", "summary": "This paper proposes a dataset distillation method based on optimal transport  to align the distribution between real and synthetic data. It introduces a three-stage sampling framework: an OT-guided sampling, a soft label relabeling, and an OT-based logit matching, to preserve both global and fine-grained structure. Experiments on ImageNet and other benchmarks show strong performance, especially in low-data settings.", "questions": "1. The notation is confusing. For example, writing “S ≡ ν_distill(x, y)” and “T ≡ μ_true(x, y)” mixes up datasets (which are collections of points) with distributions (which are probabilistic objects). Later, in Eq. (5), both terms use ν_distill, but one is about image data and the other is about label logits. These are very different things, so reusing the same symbol makes it hard to follow the logic.\n\n2. It’s unclear how Eq. (8) is derived from Eq. (7). Why the inequality hold?", "limitations": "1. The paper feels a bit over-sold. The idea of distribution matching is in a heavy optimal transport fashion. The optimal transport are in three parts of the pipeline, but only the sampling step clearly needs it. The other two (label relabeling and logit matching) could probably be done with simpler methods like MMD or KL divergence. Also, the idea of using OT for dataset distillation isn’t new — see:\nLiu et al., \"Dataset Distillation via the Wasserstein Metric\", arXiv:2311.18531 (2023).\n\n2. The results look good, but they need long training to get there. For example, the best results on ImageNet-1K require 1000 training epochs, which weakens the claim that this is an efficient method. The paper should discuss performance under shorter training more directly.\n\n3. Small IPC cases (<10) are not well covered. IPC=10 is already a relatively generous setting. Many real-world use cases care about much lower IPC, like 1 to 5. It would be helpful to see how the method performs in these tougher scenarios.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nThe performances is good. The model outperforms baselines across datasets and architectures, especially at IPC=10.\n\nWeaknesses\n1. The paper is a bit over-selling on simple idea. OT is used in all stages, though simpler methods may work for some. and OT idea in dataset distillation is not new.\n2. The notation is ambiguous, the same symbol is  used for both data and label distributions, causing ambiguity.", "was_revised": true, "final_justification": "As they have addressed most of my concerns, I will increase my rating to accept.", "rebuttal": "We thank the reviewer for the constructive comments. Our responses are as follows.\n\n>\tQ1: The notation is confusing. For example, writing “S ≡ ν_distill(x, y)” and “T ≡ μ_true(x, y)” mixes up datasets (which are collections of points) with distributions (which are probabilistic objects). Later, in Eq. (5), both terms use ν_distill, but one is about image data and the other is about label logits. These are very different things, so reusing the same symbol makes it hard to follow the logic.\n\nA1: First, regarding writing the dataset $\\mathcal{S}$ as the distribution $\\nu_{\\text{distill}}(\\mathbf{x},\\mathbf{y})$, this is indeed an academic shorthand where we treat a finite dataset as its empirical distribution [1].  To avoid ambiguity, we will clarify this convention explicitly in a footnote in the final version (e.g., “$\\nu_{\\text{distill}}$ denotes the empirical distribution over dataset $\\mathcal{S}$”).\n\nSecond, regarding the seeming reuse of symbols for \"image data\" and \"label logits\" in Equation (5), we want to clarify that, as explicitly defined in Table 1, both Wasserstein distance terms operate over the full joint distribution $(\\mathbf{x},\\mathbf{y})$. That is, the notation $\\nu_{\\text{distill}}$ consistently denotes the same object: the distilled joint data-label distribution. Our framework's two-stage \"relay race\" design helps to understand this:\n\n-\tStage 1: $\\mathrm{W}(\\mu_{\\text{true}}, \\nu_{\\text{distill}}^{(\\mathrm{soft})})$ measures the gap between the real data and our distilled data. It asks, \"How well do our synthetic (image, soft label) pairs represent the true distribution?\" This is optimized by our image generation (OTG) and label assignment (LIA) modules.  \n\n-\tStage 2: $\\mathrm{W}(\\nu_{\\text{distill}}^{(\\mathrm{soft})}, \\nu_{\\text{new}})$ measures the gap between our distilled data and the final student model's output. The logits are merely the mechanism for optimizing this stage, while the objective remains matching the joint distributions.\n\nTo formally eliminate this ambiguity at the symbolic level throughout the paper, we will explicitly write (x,y) in all relevant distribution notations in our revision, for example, by writing $\\mathrm{W}(\\mu_{\\text{true}}, \\nu_{\\text{new}})$ as $\\mathrm{W}(\\mu_{\\text{true}}(\\mathbf{x},\\mathbf{y}), \\nu_{\\text{new}}(\\mathbf{x},\\mathbf{y}))$.\n\n[1] G. Peyré and M. Cuturi. Computational Optimal Transport: With Applications to Data Science. Foundations and Trends in Machine Learning, 2019.\n\n>\tQ2: It’s unclear how Eq. (8) is derived from Eq. (7). Why the inequality hold?\n\nA2: Equation (8) is formed by substituting Equations (6) and (7) back into Equation (5):\n\n- First, replace $ \\mathrm{W}(\\mu_{\\text{true}}, \\nu^{(\\mathrm{soft})}_{\\text{distill}}) $ in Equation (5) with its equivalent from Equation (6).\n- Then, replace $ \\mathrm{W}(\\mu_{\\text{true}}, \\nu^{(\\mathrm{hard})}_{\\text{distill}}) $ with its class-wise expectation from Equation (7).\n\nThis leads to the final structured upper bound in Equation (8).\n\n>\tQ3: The paper feels a bit over-sold. The idea of distribution matching is in a heavy optimal transport fashion. The optimal transport are in three parts of the pipeline, but only the sampling step clearly needs it. The other two (label relabeling and logit matching) could probably be done with simpler methods like MMD or KL divergence. Also, the idea of using OT for dataset distillation isn’t new — see: Liu et al., \"Dataset Distillation via the Wasserstein Metric\", arXiv:2311.18531 (2023).\n\nA3: Thank you for your insightful feedback. Our contribution is not in merely using OT, but in structurally integrating OT across sampling, relabeling, and logit‑matching as a single end‑to‑end OT distance minimization objective, which is essential for preserving distributional geometry. Results show the necessity of each component and the superiority of our whole pipeline.\n\nOn the Logit Matching component, KL and MMD indeed serve as simpler divergences, but they are inherently limited. KL divergence is applied per sample and ignores inter-sample relationships, while MMD matches only global statistics. In contrast, our OTM applies **batch-wise OT alignment** between student logits and soft labels, capturing the **joint distributional structure** of samples. This enables OT to faithfully preserve **inter-sample geometry** and match structural uncertainty in the soft labels, which KL and MMD overlook. As shown below, this leads to a clear performance gain:\n\nTable. Performance comparison for logit matching on ImageNette (IPC=10).\n\n|Network|ConvNet-6| ResNetAP-10|ResNet-18|\n|-|-|-|-|\n|MMD|72.6|75.3|76.4|\n|KL|73.0|75.4|77.6|\n|Ours (OTM)|**74.5**|**77.8**|**79.0**|\n\nThe LIA module serves as a **contraction-control strategy** grounded in our OT framework, and its effectiveness is validated by extensive experiments in Tables 10, 17, and 18. It adaptively selects teacher ensembles based on the IPC setting and controls the **soft label distribution complexity** to match image expressiveness. This design directly **minimizes the OT-derived contraction factor $\\alpha$**, which tightens the OT upper bound (Eq. 8), thereby enhancing the effectiveness of supervision. In contrast, KL and MMD are not only less expressive but are also incompatible with our OT-based framework. They **cannot guide teacher ensemble selection nor participate in contraction-aware optimization**, which are essential to our LIA module.\nTherefore, **neither LIA nor OTM can be substituted by KL or MMD.**\n\nRegarding Liu et al. (\"Dataset Distillation via the Wasserstein Metric,\" ICCV 2025, hereafter WMDD), while both WMDD and our approach leverage OT, they differ fundamentally in both scope and formulation. WMDD is designed to for **distribution-matching-based dataset distillation** and applies OT only once at the beginning of training to compute a static Wasserstein barycenter. This barycenter is then matched using a conventional L2 loss.\nIn contrast, our method introduces a holistic framework for **generative dataset distillation**. Our method employs dynamic OT guidance in three stages—sampling, relabeling, and logit matching—forming a unified, multi‑stage pipeline that optimizes the same OT objective end to end. Also, WMDD uses OT for **offline initial computation**, whereas we employ OT **actively throughout training** to preserve evolving distributional geometry.\nThis distinction results in a consistent performance gap, as shown in the table below:\n\n|Dataset|ImageNette|ImageNette| ImageNet-1k|ImageNet-1k|\n|-|-|-|-|-|\n|IPC|10|50|10|50|\n|WMDD|64.8$\\pm$0.4|83.5$\\pm$0.3|38.2$\\pm$0.2|57.6$\\pm$0.5|\n|Ours|**79.0$\\pm$0.3**|**89.3$\\pm$0.3**|**52.9$\\pm$0.1**|**61.9$\\pm$0.5**|\n\nWe will include a discussion of WMDD in the related work section and incorporate it as a new baseline in the experimental evaluation in the revised manuscript.\n\n>\tQ4: The results look good, but they need long training to get there. For example, the best results on ImageNet-1K require 1000 training epochs, which weakens the claim that this is an efficient method. The paper should discuss performance under shorter training more directly.\n\nA4: Thank you for raising this important point regarding efficiency. We would like to clarify that the 1000-epoch results were presented not as a requirement of our method, but as part of the **post-distillation evaluation** to demonstrate the information richness and long-term utility of the distilled dataset. However, such **training of the downstream model is not necessary** to achieve strong performance.\nAs detailed in Table 2, when training a new ResNet-18 for only 300 epochs, our method achieves 52.9% accuracy on ImageNet-1K (IPC=10). This result significantly outperforms leading baselines under the same settings, such as the top model-inversion method EDC (48.6%) and the top generative method DiT-IGD (45.5%). \n\nFurthermore, the overall efficiency of our method is highlighted by its **rapid convergence during downstream training** and **low-cost distilled data generation**, which is a **central goal in dataset distillation**. The **new model training curves in Figure 3** show that our method \"achieves faster convergence and consistently higher accuracy, especially in early epochs\" when compared to EDC and IGD. This demonstrates the high quality and informativeness of our distilled samples. In addition to training dynamics, the **generation of the distilled set itself is also highly efficient**. Table 8 shows that the adopted OT computation only brings **negligible extra computation** to the distillation process. Our runtime analysis in Table 9 reveals that our approach is notably **faster than EDC**. \n\n>\tQ5: Small IPC cases (<10) are not well covered. IPC=10 is already a relatively generous setting. Many real-world use cases care about much lower IPC, like 1 to 5. It would be helpful to see how the method performs in these tougher scenarios.\n\nA5: We thank the reviewer for this insightful suggestion. We have conducted additional experiments on ImageNet-1K for the challenging settings of IPC=1, IPC=2, and IPC=5. The results of these new experiments are presented in the three tables provided below.\n\nImportantly, in the IPC = 1 setting, since only one synthetic image is generated per class, the OTG process cannot leverage previously distilled samples for alignment. Instead, for each class, we compute the OT distance between its single synthetic candidate and the corresponding real images in the latent space to guide generation.\n\nTable: Performance on ImageNet-1K (IPC=1)\n|Method|DM|FrePo|TESLA|SRe2L|RDED|EDC|DiT-IGD|Ours|\n|-|-|-|-|-|-|-|-|-|\n|Acc|1.5|7.5|7.7|0.4|6.6|12.8|10.7|**15.9**|\n\nTable: Performance on ImageNet-1K (IPC=2)\n|Method|DM|FrePo|TESLA|RDED|EDC|DiT-IGD|Ours|\n|-|-|-|-|-|-|-|-|\n|Acc|1.7|9.7|10.5|16.5|22.8|20.6|**25.9**|\n\nTable: Performance on ImageNet-1K (IPC=5)\n|Method|RDED|EDC|DiT-IGD|Ours|\n|-|-|-|-|-|\n|Acc|23.8|39.5|38.6|**45.7**|\n\nOur results consistently surpass prior state-of-the-art methods."}, {"review_id": "CV2WUZ9H6w", "reviewer": "Reviewer_Xuuh", "summary": "This paper proposes to incorporate optimal transport as the distance measurement between the synthetic dataset and the original dataset for dataset distillation. Specifically, the authors decompose the OT distance between these two sets into three terms and address them at different distillation stages. First, they adopt an OT-guided diffusion sampling to better maintain the structure of the original data distribution. Second, soft labels are generated based on the complexity of distilled image distributions. Last, an OT-based logit matching is employed for training student models. A detailed ablation study is conducted to illustrate the effectiveness of these three terms. Overall, the proposed method achieves state-of-the-art performance across multiple benchmarks with a marginal computational burden attached.", "questions": "1. What is the difference between this paper and [1] from the motivation and method design perspective?\n\n   [1] Dataset Distillation via the Wasserstein Metric (Haoyang Liu et al., 2023)\n2. How is the Sinkhorn iteration $T$ decided? \n3. Can the authors provide empirical evidence that introducing more models would lead to more diverse soft labels?\n4. Some extra questions for experiments:\n   - The authors adopt several different training settings (300 epochs, 500 epochs, and 1,000 epochs). Are the same group of soft labels incorporated for all these settings?\n   - In Table 7, does the column \"w/o OTG\" corresponds to DiT-IGD?", "limitations": "The authors have included a limitation discussion and proposed possible solutions to tackle the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The method design is based on reasonable motivations: the instance-level characteristics and intraclass variations are not taken into consideration at the same time by the previous dataset distillation methods. \n2. The authors apply a detailed analysis of the decomposition of OT distance between the synthetic set and the original set. Subsequently, the authors propose three components to tackle them, respectively. \n3. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. I'm surprised to see that the adopted optimal transport computation only brings negligible extra computation to the distillation process. \n4. The writing is overall clear and easy to understand. \n\n## Weaknesses\n1. There was another paper utilizing optimal transport in the context of dataset distillation [1]. I can see the difference between this paper and [1]. It would further enhance the paper quality to provide a comparison and discussion. \n\n   [1] Dataset Distillation via the Wasserstein Metric (Haoyang Liu et al., 2023)\n2. It takes $T$ Sinkhorn iterations to obtain the optimal transport matrix. However, there lack of a parameter analysis of $T$. Is it designed based on empirical results or previous literature?\n3. The soft labeling is a bit confusing. With a small number of teacher models (e.g., two models in this paper for IPC=10), when the models have conflicts against each other, won't that introduce more uncertainty? When more models are introduced, the voting mechanism will weaken the different opinions of each model. The authors claim that more views will introduce greater semantic diversity. It would be better to include certain empirical results to support this.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I would like to maintain the original score of accept.", "rebuttal": "We thank the reviewer for the constructive comments. Our responses are as follows.\n\n>\tQ1: There was another paper utilizing optimal transport in the context of dataset distillation [1]. I can see the difference between this paper and [1]. It would further enhance the paper quality to provide a comparison and discussion.\n\nA1: We thank the reviewer for pointing out the related work and agree that including a comparison will further improve the paper. Although both our method and WMDD utilize optimal transport (OT), they differ significantly in both methodology and motivation, leading to distinct formulations and implementations.\n\nLiu et al. (\"Dataset Distillation via the Wasserstein Metric,\" ICCV 2025, hereafter WMDD) is a **distribution-matching-based** distillation method that applies OT in a **single, offline step** to compute a Wasserstein barycenter over the real data’s feature distribution. This barycenter is then used as a fixed target throughout training, where synthetic images are optimized to match it using a **standard L2 loss** in the feature space. In contrast, we introduce a fundamentally different generative paradigm where OT is not a static, one-off computation, but a **dynamic guidance mechanism integrated throughout the entire data synthesis and training pipeline**. Specifically, OT guides the sampling of synthetic images by aligning latent representations, regulates the soft label relabeling process by matching label complexity to the image distribution, and structures the training loss of the student model by aligning its logits to the relabeled targets.\n\nThe motivation behind WMDD is to replace the use of simple data summaries, such as the feature means often targeted by MMD-based methods, with a more geometrically meaningful summary, namely the Wasserstein barycenter, derived from the Wasserstein metric. In contrast, our method is driven by the need to address inherent limitations in generative distillation pipelines, which often fail to preserve the fine-grained geometry of the real data distribution—particularly intra-class variations and local modes. These aspects are explicitly addressed in our framework through a multi-stage, OT-guided design.\n\nWe compare the top-1 accuracy of our method with WMDD under different images-per-class (IPC) settings on both the ImageNette and ImageNet-1K datasets. As shown in the table below, our method consistently outperforms WMDD across all scenarios.\n\n| Dataset | ImageNette | ImageNette | ImageNet-1K | ImageNet-1K |\n|-|-|-|-|-|\n|IPC| 10| 50| 10 | 50 |\n|WMDD| 64.8$\\pm$0.4| 83.5$\\pm$0.3| 38.2$\\pm$0.2| 57.6$\\pm$0.5|\n|Ours|**79.0$\\pm$0.3**|**89.3$\\pm$0.3**|**52.9$\\pm$0.1**|**61.9$\\pm$0.5**|\n\n>\tQ2: It takes $T$ Sinkhorn iterations to obtain the optimal transport matrix. However, there lack of a parameter analysis of $T$. Is it designed based on empirical results or previous literature?\n\nA2: Thank you for the insightful question. We have conducted a targeted analysis of the number of Sinkhorn iterations $T$, as it directly influences the quality of the approximated transport matrix and the overall effectiveness of our OT-based alignment components.\n\nWe summarize the results of this analysis in the table below:\n\nTable. Effect of $T$ on ImageNette (IPC=10)\n|$T$|5|10|20|50|100|\n|-|-|-|-|-|-|\n|ResNet-18|77.9|78.6|79.0|78.8|78.9|\n|ConvNet-6|73.8|74.3|74.5|74.7|74.6|\n\nAs shown, increasing $T$ from 5 to 20 results in notable performance gains, particularly due to improved convergence of the OT plan. However, beyond $T$=20, the improvement plateaus, with accuracy gains becoming marginal while unnecessarily increase computational costs.  We therefore use $T$=20 by default for all reported experiments, balancing convergence quality and computational efficiency.\n\n>\tQ3: The soft labeling is a bit confusing. With a small number of teacher models (e.g., two models in this paper for IPC=10), when the models have conflicts against each other, won't that introduce more uncertainty? When more models are introduced, the voting mechanism will weaken the different opinions of each model. The authors claim that more views will introduce greater semantic diversity. It would be better to include certain empirical results to support this.\n\nA3: We thank the reviewer for raising this thoughtful question. In fact, our method does not aim to reduce uncertainty by suppressing disagreement across teachers, nor do we rely on majority voting. Instead, we adopt a principled approach based on aligning the **distributional complexity** of soft labels with the **representational capacity** of the synthetic images.\n\nConcretely, using fewer teacher models results in soft label distributions that are less complex and contain fewer semantic modes. Under **low IPC** settings, where synthetic images are limited in diversity and expressiveness, a simpler soft label distribution is more suitable. It avoids over-regularizing weak representations with overly complex supervision and leads to better alignment in the joint image-label space.\n\nConversely, under **high IPC** settings, where the synthetic image distribution is richer and contains more semantic structure, we intentionally introduce more teacher models to increase the complexity of the soft label distribution. This promotes stronger semantic supervision and better fits the higher-capacity image representations.\n\nOur Label-Image-Aligned Soft Label Relabeling (LIA) module adaptively selects a teacher ensemble based on the IPC setting. This ensures that the **complexity of the soft-label distribution matches the expressive capacity of the distilled images, thereby directly reducing the contraction factor  $\\alpha$ in Eq. (6). A smaller $\\alpha$ leads to a tighter OT upper bound in Eq. (8), which in turn yields better student model performance**.\n\nThis adaptive strategy is not just a heuristic; it is directly and empirically validated by our analysis of $\\alpha$ in Appendix G.2 (page 25):\n\n-\tFor Low IPC (Table 17, IPC=10): A smaller ensemble of 2 teachers (Config D) achieves a lower contraction factor $\\alpha$ (0.93) and the highest accuracy (52.9% on ResNet-18). Increasing the ensemble to 4 diverse teachers (Config B) is detrimental, worsening both $\\alpha$ to 0.99 and accuracy to 52.3%. This confirms that for low-capacity image sets, excessive label complexity is harmful.\n\n-\tFor High IPC (Table 18, IPC=50): The trend reverses as the image set's capacity increases. The larger, more diverse ensemble of 4 teachers (Config B) now achieves a dramatically smaller $\\alpha$ (0.16) and the best generalization performance (68.2% on Swin Transformer). The simpler 2-teacher ensemble is now suboptimal.\n\nTogether, these results validate our design: **matching soft-label distribution complexity to image capacity minimizes $\\alpha$, reduces OT distance, and improves performance**. We will clarify this mechanism and include illustrative diagrams (including a motivation diagram and a causal flow) in the final version.\n\n**Additional diversity check.** To qualitatively assess whether a larger teacher ensemble introduces greater semantic diversity, we report the average Shannon entropy of the soft labels (ImageNette, IPC = 10):\n|Ensemble|Entropy|$\\alpha$|\n|-|-|-|\n|4 teachers|1.91 nats|0.903|\n|2 teachers|1.82 nats|0.643|\n\nThe results suggest the **presence of increased semantic diversity and richness when using more teacher models**, observed at the individual sample level. However, any single‑sample statistic cannot enumerate mode count, reveal inter‑sample structure, or fully describe the geometry of the label distribution. **It is therefore unsuitable as an optimization proxy**. In practice, we recommend: (1)  selecting a small set of ensemble configurations based on IPC, following our LIA guideline; and (2) choosing the one with the smallest $\\alpha$, which directly tightens the OT upper bound (Eq. 8).\n\n>\tQ4: The authors adopt several different training settings (300 epochs, 500 epochs, and 1,000 epochs). Are the same group of soft labels incorporated for all these settings?\n\nA4: No, the effective soft labels are different for each training epoch.\nTo ensure a **fair comparison with state-of-the-art baselines**\t(e.g., SRe2L, RDED, G-VBSM, CDA, SC-DD, CV-DD, EDC, D$^4$M, TDSDM, Minimax, DDPS, DiT-IGD), we follow the methodology similar to FKD [1]. This approach utilizes a unique set of soft labels for each training epoch, which are generated based on different stochastic data augmentations. This provides the non-repetitive supervision necessary for effective long-term training.\n\n[1] A fast knowledge distillation framework for visual recognition. ECCV, 2022\n\n\n>\tQ5: In Table 7, does the column \"w/o OTG\" corresponds to DiT-IGD?\n\nA5: For hard label settings: Yes, the “w/o OTG” column corresponds to DiT-IGD, which uses guided diffusion without OT guidance.\n\nFor soft label settings: No, the correspondence does not hold. DiT-IGD lacks all the OT components proposed by our method (OTG, LIA, OTM)."}]

Author rebuttal:

Meta-review: This paper proposes an optimal transport-based framework for generative dataset distillation, aligning distributional geometry between synthetic and real data. The method is well-motivated, technically sound, and empirically validated across multiple benchmarks, showing competitive or superior performance compared to prior dataset distillation approaches. Reviewers agreed that the work is original and based on resonable motivations. Main concerns included generalizability to additional experiment settings (low-IPC), sensitivity to OT hyperparameters, and the need to better distinguish the contribution from the related work “Dataset Distillation via the Wasserstein Metric” (arXiv:2311.18531, 2023). The rebuttal provided satisfactory clarifications on these points. Overall, the paper represents a novel, technically solid, and practically promising contribution to generative dataset distillation, and is recommended for acceptance.
Decision: Accept (poster)