Title: Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dataset Distillation

Optimal Transport
Abstract: Dataset distillation seeks to synthesize a compact distilled dataset, enabling models trained on it to achieve performance comparable to models trained on the full dataset. Recent methods for large-scale datasets focus on matching global distributional statistics (e.g., mean and variance), but overlook critical instance-level characteristics and intraclass variations, leading to suboptimal generalization. We address this limitation by reformulating dataset distillation as an Optimal Transport (OT) distance minimization problem, enabling fine-grained alignment at both global and instance levels throughout the pipeline. OT offers a geometrically faithful framework for distribution matching. It effectively preserves local modes, intra-class patterns, and fine-grained variations that characterize the geometry of complex, high-dimensional distributions. Our method comprises three components tailored for preserving distributional geometry: (1) OT-guided diffusion sampling, which aligns latent distributions of real and distilled images; (2) label-image-aligned soft relabeling, which adapts label distributions based on the complexity of distilled image distributions; and (3) OT-based logit matching, which aligns the output of student models with soft-label distributions. Extensive experiments across diverse architectures and large-scale datasets demonstrate that our method consistently outperforms state-of-the-art approaches in an efficient manner, achieving at least 4\% accuracy improvement under IPC=10 settings for each architecture on ImageNet-1K.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to incorporate optimal transport as the distance measurement between the synthetic dataset and the original dataset for dataset distillation. Specifically, the authors decompose the OT distance between these two sets into three terms and address them at different distillation stages. First, they adopt an OT-guided diffusion sampling to better maintain the structure of the original data distribution. Second, soft labels are generated based on the complexity of distilled image distributions. Last, an OT-based logit matching is employed for training student models. A detailed ablation study is conducted to illustrate the effectiveness of these three terms. Overall, the proposed method achieves state-of-the-art performance across multiple benchmarks with a marginal computational burden attached.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The method design is based on reasonable motivations: the instance-level characteristics and intraclass variations are not taken into consideration at the same time by the previous dataset distillation methods. \n2. The authors apply a detailed analysis of the decomposition of OT distance between the synthetic set and the original set. Subsequently, the authors propose three components to tackle them, respectively. \n3. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. I'm surprised to see that the adopted optimal transport computation only brings negligible extra computation to the distillation process. \n4. The writing is overall clear and easy to understand. \n\n## Weaknesses\n1. There was another paper utilizing optimal transport in the context of dataset distillation [1]. I can see the difference between this paper and [1]. It would further enhance the paper quality to provide a comparison and discussion. \n\n   [1] Dataset Distillation via the Wasserstein Metric (Haoyang Liu et al., 2023)\n2. It takes $T$ Sinkhorn iterations to obtain the optimal transport matrix. However, there lack of a parameter analysis of $T$. Is it designed based on empirical results or previous literature?\n3. The soft labeling is a bit confusing. With a small number of teacher models (e.g., two models in this paper for IPC=10), when the models have conflicts against each other, won't that introduce more uncertainty? When more models are introduced, the voting mechanism will weaken the different opinions of each model. The authors claim that more views will introduce greater semantic diversity. It would be better to include certain empirical results to support this.\n\nQUESTIONS:\n1. What is the difference between this paper and [1] from the motivation and method design perspective?\n\n   [1] Dataset Distillation via the Wasserstein Metric (Haoyang Liu et al., 2023)\n2. How is the Sinkhorn iteration $T$ decided? \n3. Can the authors provide empirical evidence that introducing more models would lead to more diverse soft labels?\n4. Some extra questions for experiments:\n   - The authors adopt several different training settings (300 epochs, 500 epochs, and 1,000 epochs). Are the same group of soft labels incorporated for all these settings?\n   - In Table 7, does the column \"w/o OTG\" corresponds to DiT-IGD?\n\nLIMITATIONS:\nThe authors have included a limitation discussion and proposed possible solutions to tackle the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dataset distillation method based on optimal transport  to align the distribution between real and synthetic data. It introduces a three-stage sampling framework: an OT-guided sampling, a soft label relabeling, and an OT-based logit matching, to preserve both global and fine-grained structure. Experiments on ImageNet and other benchmarks show strong performance, especially in low-data settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe performances is good. The model outperforms baselines across datasets and architectures, especially at IPC=10.\n\nWeaknesses\n1. The paper is a bit over-selling on simple idea. OT is used in all stages, though simpler methods may work for some. and OT idea in dataset distillation is not new.\n2. The notation is ambiguous, the same symbol is  used for both data and label distributions, causing ambiguity.\n\nQUESTIONS:\n1. The notation is confusing. For example, writing “S ≡ ν_distill(x, y)” and “T ≡ μ_true(x, y)” mixes up datasets (which are collections of points) with distributions (which are probabilistic objects). Later, in Eq. (5), both terms use ν_distill, but one is about image data and the other is about label logits. These are very different things, so reusing the same symbol makes it hard to follow the logic.\n\n2. It’s unclear how Eq. (8) is derived from Eq. (7). Why the inequality hold?\n\nLIMITATIONS:\n1. The paper feels a bit over-sold. The idea of distribution matching is in a heavy optimal transport fashion. The optimal transport are in three parts of the pipeline, but only the sampling step clearly needs it. The other two (label relabeling and logit matching) could probably be done with simpler methods like MMD or KL divergence. Also, the idea of using OT for dataset distillation isn’t new — see:\nLiu et al., \"Dataset Distillation via the Wasserstein Metric\", arXiv:2311.18531 (2023).\n\n2. The results look good, but they need long training to get there. For example, the best results on ImageNet-1K require 1000 training epochs, which weakens the claim that this is an efficient method. The paper should discuss performance under shorter training more directly.\n\n3. Small IPC cases (<10) are not well covered. IPC=10 is already a relatively generous setting. Many real-world use cases care about much lower IPC, like 1 to 5. It would be helpful to see how the method performs in these tougher scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper observes that recent methods for large-scale dataset distillation primarily focus on matching global distributional statistics (e.g., mean and variance) while overlooking critical instance-level characteristics and intra-class variation, which limits generalization. To address this, the authors reformulate dataset distillation as an Optimal Transport distance minimization problem, enabling fine-grained alignment at both global and instance levels. OT provides a geometric framework for distribution alignment, preserving local modes, intra-class structure, and high-dimensional distributional geometry. Their method comprises three components designed to preserve distributional structure: 1) OT-guided diffusion sampling to align the latent distributions of real and distilled images. 2) Label-image-aligned soft relabeling, which adjusts label distributions according to the complexity of distilled image distributions. 3) OT-based logit matching to align student model outputs with the soft-label distribution. Experiments across multiple architectures and datasets show that the proposed approach outperforms existing methods, achieving ~4% accuracy gain under the IPC=10 setting for each architecture on ImageNet-1K.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Applying Optimal Transport to generative dataset distillation appears novel as I am not aware of prior work using it in this context.\n2. The paper is generally well-written.\n3. The idea of aligning both global and instance-level distributions in the distillation process is interesting.\n4. The authors have included their code in the submission, which enhances the reliability of their results and shows good improvements over previous methods.\n\n\nWeaknesses\n1. The paper misses some technical details. For example, in Section 4.4 (label-image-aligned soft relabeling), the authors do not clarify whether soft labels are generated at the global image level or like SRe2L at the region level. These different operations could lead to significant differences in performance.\n2. In Tables 2 and 3, results from different training epochs (300, 500, 100) are presented in the same table. It would be more intuitive to split these into separate tables to allow for direct comparison with other baselines trained for the same number of epochs.\n3. The motivation for using Optimal Transport seems reasonable, I am concerned this may be another instance of a \"forced A+B\" combination, where a well-known technique is applied simply for novelty rather than clear necessity or demonstrated advantage.\n4. The overall pipeline appears relatively complex, especially the OT-guided diffusion sampling part, which may reduce its utilization in practice.\n\nQUESTIONS:\nPlease see the *Strengths And Weaknesses* part above. My primary concern is whether OT provides a meaningful advantage for this task, and why it is truly needed. I would prefer not to see it used merely for novelty (e.g., just because no prior DD papers have used it) without concrete justification or practical benefit. In addition, some implementation details are missing, and I hope the authors will address these concerns thoroughly in the rebuttal.\n\nLIMITATIONS:\nThe authors have discussed some of their method's limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel dataset distillation method based on Optimal Transport (OT) aimed at achieving geometrical alignment between real and distilled data by minimizing OT distance. The core idea of the approach is to preserve local modes, intra-class patterns, and fine-grained variations in high-dimensional distributions, thereby enhancing the quality and generalization ability of the distilled data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The motivation in Figure 1(a) is interesting, and the use of Wasserstein Distance in this context is reasonable.\n\n2. I appreciate the explanation in Section 4.2, \"Reconstructing the Optimal Transport Distance,\" where the author summarizes the three methods of the paper through an approximated inequality.\n\nWeaknesses:\n\n1. In Eq. 13, the author refers to $W=⟨D,K⟩$ as the optimal transport distance, which is not entirely accurate. Strictly speaking, it is an approximation of the Wasserstein Distance, or more specifically, the Sinkhorn distance, since $K$ is just an approximation of the exact solution. For more details, see [1].\n\n2. The author proposes three improvements, but I am unclear about which one plays a decisive role.\n\n3. I do not fully understand the significance of Label-Image-Aligned Soft Label Relabeling. I would appreciate it if the author could provide some diagrams to help readers understand, instead of a long explanation in text.\n\n\n\n[1] Sinkhorn distances: Lightspeed computation of optimal transport\n\nQUESTIONS:\nQ1.  In Experiment Table 11, I noticed that the values of \\( \\lambda_1 \\) are set to 1000 and 10000, which seems unreasonable. As $\\lambda_1$ increases, the solution to the Sinkhorn algorithm becomes more uniform, leading to a weaker correlation with the image information. I suggest that the authors provide experiments with smaller values of \\( \\lambda_1 \\) to alleviate our concerns about the first part of the guidance not having a significant effect.\n\nQ2. I found that Section 4.5, \"OT-based Student Model Logit Matching,\" is somewhat similar to a recent paper [2]. I would appreciate it if the authors could provide a comparison and highlight the differences between the two.\n\n\n[2] SelKD: Selective Knowledge Distillation via Optimal Transport Perspective. ICLR2025\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}