Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation
Authors: Xinhao Zhong
Shuoyang Sun
Xulin Gu
Chenyang Zhu
Bin Chen
Yaowei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: decoupled dataset distillation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper point out the unfair performance comparison overlooked within the field of decoupled dataset distillation

Abstract: Dataset distillation aims to generate compact synthetic datasets that enable models trained on them to achieve performance comparable to those trained on full real datasets, while substantially reducing storage and computational costs. Early bi-level optimization methods (e.g., MTT) have shown promising results on small-scale datasets, but their scalability is limited by high computational overhead.
To address this limitation, recent decoupled dataset distillation methods (e.g., SRe$^2$L) separate the teacher model pre-training from the synthetic data generation process. These methods also introduce random data augmentation and epoch-wise soft labels during the post-evaluation phase to improve performance and generalization. However, existing decoupled distillation methods suffer from inconsistent post-evaluation protocols, which hinders progress in the field. In this work, we propose **R**ectified **D**ecoupled **D**ataset **D**istillation (RD$^3$), and systematically investigate how different post-evaluation settings affect test accuracy. We further examine whether the reported performance differences across existing methods reflect true methodological advances or stem from discrepancies in evaluation procedures. Our analysis reveals that much of the performance variation can be attributed to inconsistent evaluation rather than differences in the intrinsic quality of the synthetic data. In addition, we identify general strategies that improve the effectiveness of distilled datasets across settings. By establishing a standardized benchmark and rigorous evaluation protocol, RD$^3$ provides a foundation for fair and reproducible comparisons in future dataset distillation research.

Published as a conference paper at ICLR 2026
RECTIFIED DECOUPLED DATASET DISTILLATION: A CLOSER LOOK FOR FAIR AND COMPREHENSIVE EVAL- UATION
Xinhao Zhong' Shuoyang Sun Bin Chen Yaowei Wang
Xulin Gu
Chenyang Zhu
Harbin Institute of Technology, Shenzhen Peng Cheng Laboratory
"Tsinghua Shenzhen International Graduate School, Tsinghua University
xhh21223egmail com, 24s1511520stu, hit. edu. en, 2101107206stu wew. edu. en, chenyangzhu.co catpsaiuuss@mmmiii.. csmm chenbin2021@hie www.hhhttt..c.....mmmeeeeeeeeeee edu. on, wangywepcl.ac.o
ABSTRACT
Dataset distillation aims to generate compact synthetic datasets that enable mod- els trained on them to achieve performance comparable to those trained on full real datasets, while substantially reducing storage and computational costs. Early bi-level optimization methods (e.g., MTT) have shown promising results on small- scale datasets, but their scalability is limited by high computational overhead. To address this limitation, recent decoupled dataset distillation methods (e.g., SR separate the teacher model pre-training from the synthetic data generation process. These methods also introduce random data augmentation and epoch-wiss soft labels during the post-evaluation phase to improve performance and generaliza- tion. However, existing decoupled distillation methods suffer from inconsistent post-evaluation protocols, which hinders progress in the field. this work, we propose Rectified Decoupled Dataset Distillation (RD*), and systematically in- vestigate how different post-evaluation settings affect test accuracy. We further examine whether the reported performance differences across existing methods reflect true methodological advances or stem from discrepancies in evaluation procedures. Our analysis reveals that much of the performance variation be attributed to inconsistent evaluation rather than differences in the intrinsic quality the synthetic data. addition, we identify general strategies that improve the effectiveness of distilled datasets across settings. By establishing standardized benchmark and rigorous evaluation protocol, RD provides a foundation for fair and reproducible comparisons in future dataset distillation research. Our code is available /nttps://qitiee com/ hww.gg11g1113
INTRODUCTION
Deep learning has rapidly advanced in recent years, with large-scale models trained on extensive datasets achieving impressive performance across diverse domains- notably computer vision He et al. (2016); Dosovitskiy et al. (2020) and natural language processing DDeliin et al. (2018); Brown et al. (2020). However, training models on large-scale datasets typically incurs prohibitive computational and memory costs, posing significant challenges for deployment, especially in resource- constrained environments. Dataset distillation (DD) Wang al. (2018) has emerged as a promising direction to address this issue by enabling the creation of compact synthetic datasets that retain the utility of the original data. Early coammation-machiinn methods Zhao & Bilen (20216 2023);
Cazenavette al. (2022) have achieved reliable performance small-scale datasets Krizhevsky (2009), but their nested optimization structures imposed substantial time consumption, thereby limiting applicability larger datasets Deng et al. (2009)
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Recently, decoupled dataset distillation methods Yin et al. (2024); Su et al. 2024): Sun et al. (2024a)
have been proposed to address this issue by separating model pre-traiming from data synthesis, significantly reducing computational costs. They further enhance performance by incorporating epoch-wise soft labels from teacher models during post-evaluation, achieving state-of-the-an re- sults large-scale benchmarks such as ImageNet-1K [eng al.] (2009). Existing decoupled approaches Yin et al.] (2024) can be categorized into three paradigms based on their synthetic data generation mechanisms: optimization-basse
selection-based, and generation-basee meth- ods. All these approaches share the common requirement of pre- pre-training teacher models (either classifiers or generative models like diffusion models Song et al. (202I) to achieve decoupling. Specifically, optimization-bant
methods Yin al. (2024); Shao et al. (2024a); Yin Shen (2024); Du et al. (2024); Shao et al. (2024b)
-27.3%
HO.775
Valisiatio Accuracy (%6
perform pixel-level optimization of synthetic datasets using pre-trained classifiers, guided by cross-entropy loss and Batch Normalization (BN) layer statistics. In contrast, selection-based methods Sun et al.] (2024a); Zhong et al. (2024b) utilize classifiers
40
-5
30-
generative models to ex- tract class-relevant visual regions directly from original images. On the other hand, generation- based methods Su et al. (2024); Gu et al. (2024) fine-tune generative models or optimize visual- textual embeddings to synthesize new images. Unfortunately, current research faces several sig- nificant challenges: First, inconsistent evalua- tion settings across various compression ratios, target datasets, and cross- arosttacch models pose substantial comparability barriers for re- searchers. Second, existing studies often over- look methodological commonalities, leading to incomplete comparisons that consider only subsets of the three aforementioned paradigms. More im- portantly, the inherent evaluation setting sensitivity in the post-evaluation phase results in performance comparisons being conducted under inconsistent settings, giving rise to confounded performance gains and significantly hindering the structured progress of this field. As shown in Figure [ the performance gap reported by previous methods exceeds 27%. However, under unified and simplified settings, the actual improvements drop to less than 7%. This observation underscores key challenge in dataset distillation: Claimed performance gains must be carefully disentangled to assess whether they arise from the core distillation mechanism or from auxiliary enhancements unrelated to the distillation process itself.
25-
Reported
Reciifil
SKea1
CDA G-VASM
DWA
EDC
Mininas
D/M
RDED
Distillation
Methed
Figure 1: Performance comparison of various distillation methods evaluated by ResNet-18 on ImageNet-IK under IPC=10. Previous methods achieve significant 27.3% performance improve- ment being influenced by multiple factors. After fairly reevaluating all methods under a unified set- ting, we obtained a rectified 6.7% performance enhancement.
To tackle these challenges, we introduce Rectified Decoupled Dataset Distillation (RD*), unified and comprehensive baseline framework under consistent post-evaluation settings that ensures fairesss. Specifically, conduct an in-depth investigation of the varied post-evaluation settings employed by prior methods, focusing on key parameters such as batch size and learning rate decay. Moreover, establish standardized evaluation protocol for decoupled dataset distillation, covering three critical dimensions: target datasets, compression ratios, and cross-aree moss-architecturg generalization. We then systematically replicate and re-evaluate the true performance and generalization capabilities of synthetic datasets generated by various methods. Our findings reveal that simply aligning evaluation settings suffices to eliminate substantial performance differences among synthetic datasets. The rectified results demonstrate that some reported performance gains primarily stem from improved post-evaluation settings rather than genuine enhancements the quality of synthetic datasets.
Building upon RD3, we highlight additional evaluation dimensions (e.g., time consumption) beyond test accuracy that are of greater importance. In addition, we identify several simple yet impactful techniques, such as using alternative initialization for optimization-bar based methods, that substan- tially influence test accuracy and may inadvertently introduce unfair advantages in future studies. quantify and mitigate performance variations induced
This systematic exploration enables by implementation-speeiie modifications. To the best of our knowledge, this work represents the first exhaustive evaluation of representative decoupled dataset distillation methods under fully
2

---

Published as a conference paper at ICLR 2026
standardized experimental conditions. We anticipate that RD will provide robust foundation for meaningful comparisons and accelerate methodological advancements in this emerging field.
2
RELATED WORKS
2.1 BI-LEVEL DATASET DISTILLATION
Given a large-scale dataset T {(x;, (i)
= dataset distillation aims to generate compact yet
informative synthetic dataset S = [(ss vi) Jir- which preserves as much class-relevant information
as possible while ensuring |S] < (7). With S, one can train a model from scratch with parameters
0s arg min (xyee (((foxx))
(1)
0
where I(:--) represents the loss function and So represents classifier parameterized by 0. Similarly, we define 0 for the original dataset T. The primary objective can be formulated as:
sup [((for (x). v) 1(fos (x), S. (x.
To achieve this, DD Wang et al. (2018) introduced a meta-learning approach based on a nested computation graph. However, the unrolled computation process incurs significant time costs. As an alternative, recent studies adopt a bi-level optimization framework that matches various proxy statistics between S and TT formulated
S = arg min D(fe (S), (T)).
where :) represents different distance metrics used for matching, and for denotes the correspond- ing feature extractor. DC (Zhao al., 202T) and DCC (Lee et al. 2022) minimize the distance between gradients in progressively trained network, while DM (Zhao & Bilen 2021a). CAFE (Wang)
ctt al. 2022), and DataDAM (Sajedi et al. 2023) focus on matching feature embeddings. Similarly, MTT (Cazenavette et al. 2022), TESLA (Cui et al. 2023), and DATM (Guo et al. 2023) align training trajectories to enhance learning consistency. Despite the significant achievement on small datasets (e.g., CIFAR10), bi-level distillation methods could not scale to the large-scale datasets (e.g..
ImageNet-1h KK) due to prohibitive computational cost (Cui et al. 2022).
2.2 DECOUPLED DATASET DISTILLATION
Recent decoupled methods have significantly reduced computational complexity by decoupling the training processes of proxy models from synthetic dataset generation, while still achieving robust performance on large-scale datasets. Based on the different generation mechanisms, we categorize decoupled dataset distillation methods into three primary paradigms as follows.
Optimization- based. L first introduced the decoupled optimization method by minimizing cross- entropy loss on synthetic datasets through pre-trained classifiers and aligning batch normalization (BN) statistics between synthetic and original datasets, which can be formulated as:
S, arg min[ XCBN(fef. s:)],
where À denotes the weighting factor for the BN loss CaN. Building upon this, CDA Yin & Shen (2024) integrates curriculum learning into the optimization process and dynamically adjusts hyper- parameters during the post-evaluation phase. DWA Du al. (2024) adopts real data initialization while further decomposing LaN. significantly enbancing the diversity of the synthetic dataset through pre-trained model perturbation. G-VBSM Shao al. (2024a) utilizes multiple pre-trained models as teacher networks, simultaneously matching class-wise BN and convolutional statistics, and in- corporates ensemble soft-labels and MSE loss during evaluation. EDC Shao et al. (2024b) further smooths the loss landscape in synthetic datasets and employs specialized evaluation ppaass settings, positioning itself as the state-of-thes of-the--rttttttt (SOTA).
Generation-basee With the advancement of generative diffusion models, several methods have been developed to produce synthetic datasets by optimizing different components of the diffusion process. Minimax Gu et al. (2024) employs DiT model Peebles & Xie (2023) pre-trained on ImageNet- K,
3

---

Published as a conference paper at ICLR 2026
fine-tuning it with "minimax" criterion. Then the diffusion model is used to directly generate images. However, DiT has the limitation of only generating class label-conditione images. As a result, Minimax cannot be applied to datasets that are Out-of-Distribution (OOD), such CIFAR- 10.
In contrast, D Su et al. (2024) uses Stable Diffussin (SD) pre-trained on LAION Schuhmann et al. (2022) the backbone. It first performs k-means clustering visual embeddings to obtain class centroids. These centroids are then combined with text prompts to generate synthetic datasets. D substantially enhances synthetic dataset diversity through SD and overcomes the OOD issue. Selection-based. Subsequent studies have proposed generating synthetic datasets by identifying and cropping class-relevant visual regions, thereby reducing redundancy in large-scale datasets.
RDED Sun et al. (2024a) performs random cropping on randomly sampled images and then ranks all patches in ascending order based on classification loss from pre-trained classifier. Additionally, RDED concatenates multiple patches to form single image to improve representativeness.
Subsequent methods have extended this paradigm by focusing on increasing the diversity of selected patches to improve generalization. FocusDD Hu et al. (2025) utilizes pre-trained ViT Dosovitskiy et al. (2020) as the selector and incorporates class-irrelevant background patches. DPS Zhong et al. (2024b) employs SD as the selector, identifying class-relevant regions via differential text prompts (with and without class labels). Please refer to Appendix B for more detailed literature reviews.
2.3 EPOCH- WISE LABEL MATCHING
Early information-matchiii methods like MTT and subsequent improvements achieved superior performance using hard labels under extreme data compression scenarios, primarily applied to small- scale datasets. Recent studies Qin et al. (2024) suggest epoch-wise soft labels can better facilitate student model learning from synthetic datasets large-scale settings, which can be formulated as:
g'$ arg min LKL (fe, (A(S)) (fo, fog (A(S))),
(5)
8Ee
where f represents the classifier at training epoch A(.) denotes the random data augmentation, and T.LL represents the Kullhack-Leibler (KI.) divergence. However, current distillation methods' epoch-wise soft-label implementations involve substantial misalignment: CDA employs smaller batch sizes, RDED utilizes smoothed learning rate with stronger data augmentation, while G-VBSM and EDC generate hybrid soft labels through multiple teacher models. These implementation variances create significant obstacles for fair performance comparisons that urgently require resolution.
3 UNIFIED EVALUATION FRAMEWORK
We select well-known and satte-of-the-aite (SOTA) methods as baselines and categorize them into three groups: (1) Optimization-ban based methods Yin et al. (2024); Yin & Shen (2024): Shao et al. (2024a)
: Du al. ): [Su al. (024);; (2) Generation-based methods (Shao et al. (2024b) Gu
ct al. (2024) (3) Selection-basee methods Sun et al. 20044a). Notably, although Minimax Gu et al.
(2024) originally employs hard labels for evaluation, we include it in our consideration due to its
applicability to large-scale datasets.
All subsequent methods adjust several evaluation-phase settings on the basis of SRePL., including (1) reduced training batchsize and increased training epochs, (2) carefully selected optimizer, (3) incorporation of extra loss function regularization, (4) hybrid soft labels and (5) various data augmentations. Based on current knowledge, We are the first to make a thorough investigation across different methods. The difference with related works are shown in Appendix A
3.1 DATASETS AND NETWORKS
Datasets. We adopt six standard image datasets: (1) CIFAR-IO/100 Ki
(20zz)))skyy(0009), (2009), both of which
have 50K 32 x 32 training images and 10K testing images from 10 and 100 classes. (2) ImageNet- 1K Deng et al. (2009), consisting over 1,200,000 training images with various resolution from 1,000 classes. (3) IinylmageNet Le & Yang (2015). subset of the ImageNet- IK with 200 classes. The training split contains 100K images, while the validation and test set include 10K images, All the images possess resolution of 64 64. (4) ImageNette and CoaeeeooooCaaaaaaeeeee et al. (2022). two
4

---

Published as a conference paper at ICLR 2026
widely used Seaars-grrined and fine-gra -eee-gga subsets of ImageNet- 1K, including 10 classes derived from ImageNet- IK. For all datasets, we conduct a comprehensive evaluation with IPC (image per class) from to 100, which previous methods have not fully evaluated.
Network Architectures. We follow the settings used in previous works Yin et al. (024 Sun et al. (024a),, employing ResNet- 18 He et al. (20T)) as the backbone network and applying soft labels across all settings. For the ResNet series, we additionally utilize ResNet-50/101 as more complicated evaluation models. For generalization evaluation, while employing CNN architectures like EfficientNet TTn & Le (2019) and MobileNet Howard (2017), we introduce Swin-T(Liu et al. (202T) and ViT-B Dosovitskiy al. (2020) from the ViT series evaluation models, providing timely and comprehensive assessment.
3.2 POST-EVALUATION SETTINGS
We adopt a standard setting with a single pre- trained ResNet-18 18 and KL divergence gener- ate soft labels and optimize the student model for simplification and fairness Yin et (2024);
Yin & Shen (2024); Du et al. (2024): Sun et al.
(2024a); Su et al. (2024). We summarize the unified settings comparison to previous works and explain their motivations. Incremental post- evaluation impacts are presented in Figure 2 please refer to Appendix C for more detailed im-
10
plementations and Appendix D for more results.
Simplreed
Botch Size
Specs
Ausmmettaiion
Schedulr
Training Epoch. Early information informattin-matccinng methods /Zhao et al. (202T); (Zhao & Bilen (2023): Cazenavette et al. (2022) typically train on synthetic datasets for around 1,000 epochs and evaluate the performance under overfitting, yet this setting is impractical for large-scale dataset applications. While most decoupled dataset distillation methods adopt 300 training epochs as widely used evaluation protocol, our preliminary experiments reveal that certain methods [Sun et al. (2024a); (hao et al. (2024b) accelerate model convergence, which introduces bias in absolute performance comparisons. Therefore, we implement 400 training epochs during evaluation. As the impact on performance remains minimal yet align the converge iterations.
Figure 2: Performance comparison between SRe2L. L and RDED on ImageNet- KK under IPC=10 evalu- ated by ResNet- 18 with the same post-evaluation settings. The incremental techniques added from left to right lead to different performance impact.
shown in Figure 2
Batch size. For few-shot learning tasks like dataset distillation, batch size (BS) exerts intriguing and significant impacts on experimental outcomes. SRe*L employs BS=1024, while CDA uses BS=128 and demonstrates that smaller batch size yields notable improvements. Building upon this, RDED adopts varying BS sizes under different conditions. CV-DD Cui al. (2025) further explore an extreme setting with BS=16. We further simplify and propose an optimized setting to balance the performance and efficiency: when evaluating synthetic datasets with ResNet-18, we uniformly set BS=50 across all settings unless |S| < 50, leading to nearly a 10% performance improvement across various methods as shown Figure 2 For generalization tasks, we increase BS 100 mitigate gradient fluctuations induced by small batch sizes.
Smoothing Learning Rate (LR) Scheduler. For large-scale datasets like ImageNet-IK,, existing methods employ Adam optimizer with an initial learning rate of 0.001 and a cosine annealing scheduler for optimization. RDED and EDC further implement smoothing LR scheduler to enhance the performance. Recent work CV-DD manually selects the scheduler smoothing factor across different settings. Through preliminary experiments, we establish universal yet competitive setting: using Ç = 1 with R8 evaluation model for finer-graines tuning as shown in Figure 2 while adopting Ç = 2 for different architectures. Please refer to Appendix K for a intuitional comparison.
Data Augmentation. Under the premise of ensuring teacher-student model alignment through soft labels, previous methods universally enhanced synthetic dataset diversity via CutMix, Random Resized Cropped and Random Horizontal Flipped, achieving substantial performance gains. Building upon this, RDED and its optimized variant EDC introduce additional augmentation by exchanging patches with patch-concatenated images and expanding the crop ratio from 0.08 to 0.5. These
5

---

Published as a conference paper at ICLR 2026
modifications collectively yield positive performance impacts as shown in Figure Consequently, adopt RDED's 's data augmentations as the universal standard and apply it across all methods. Detailed analysis are shown in Appendix M
4 EXPERIMENTAL RESULTS AND ANALYSIS
4.1 Do METHODOLOGICAL DIFFERENCES TRANSLATE TO PERFORMANCE DISCREPANCIES?
Under the unified and fair settings provided by the RD3 framework, we systematically reevaluated all decoupled dataset distillation methods, with the results presented in Table and Table 2 Among optimization-hase hased methods, EDC consistently demonstrates superior performance across all datasets and compression ratios, establishing itself as the representative method for this category. We subsequently compare EDC with generation-basee and selection geleetion-based methods
ResNet-
Optimization
Generation
Selection
Dataset
IPC
SRe²t.
CDA
G-VBSM
DWA
EDC
Minimax
D
RDED
16.2 +0.7
16.4 #0.6
17.5 10.7
18.3 1-0.3
26.6 00.5
13.4 #0.8
22.5 +:0.7
10
29.7 +088
30.6 +0.6
31.5 +0.4
33.1 +0.4
40.5
+0.6
34.7 +0.4
37.3 +044
CIFARIO
53.9 10.5
54.5 =0.7
50.4
59,9 10.4
64.8 10.3
61.9 10.4
63.3 +0,2
100
69.2 +0.3
68.8 +0.6
71.2 +0.44
72,3 +0.1
74.4 +0.2
77.7 +0.2
75.7 +1144
6.9 1:0.6
6,7 1:0.5
7.6 #:0.8
7.5 ::0.6
15.4 10033
6.6 +:0.8
11.8 1.7
32.6 +0.5
33.5 +0.5
38.9
41.3 +0.55
46.6 +0.4
CIFARI00
50
54.4 10.7
56.2 =0.4
58.2 1:0.4
62.1 1:0.6
65.2 1:66
64.3 :0.4
64.1 10.4
100
59.6 +0.4
60.7 +0.3
63.3 4-0.4
64.2 4-0.4
69.1 +0.55
68.9 +0.2
67.5 +0.2
6.1 +:0.8
7.1 1:0.5
6.2 +:0.
6.8 :00.8
10.2 10.6
9.8 10..7
3,9 1:0.8
11.1 0.9
34.2 +0.9
37.5 +0.6
37.3 +-0.3
38.3 +0..5
42.1 +0.6
39.4 +0.4
36.7 +0.6
44.2 -0.4
TinyImageNes
50
52.5 0077
$3.0 =066
53,7 1:0.6
54.2 1:033
$7.1 10.4
54.4 10.4
53.8 10.4
58.7 0.44
+0.5
55.7 +0.3
56.5 +.0.4
56.8 +-0.5
61.5 40.3
56.1
$7.6
26.6 1:07
25.4
28.9 10.66
29.7 1:099
33.6 1.55
28.8 1:0.5
27.7 10.6
31.4 0.66
ImageNette
10
56.7 +0.6
$4.6 +0.4
61.6 +0.4
64.3 +0.44
70.6 +0.4
66.6 +0.5
66.3 +0.5
63.8 +0.55
79.0 10.3
77.8 -0.3
81.4 100.7
83.2 +0.5
86.7 10.3
85.2 1:033
86.5 :0.22
86.8 0.66
85.2 =022
84.7 =0.5
87.7 10.3
87.1 10.1
90.3 20.
89.3 =0.2
90.7 =0.1
89.6
12.2 1:9
14.6 0.6
14.4 10.4
16.5 1:0.5
24.4 0.03
23.8 1:0.5
19.7 10666
20.3 -0.5
10
26.8 +0.5
25.7 +0.5
34.5 40.5
36.1 4-0.5
42.3 40.6
45.5 +.06
35.4 +0.5
46.5 +0.6
Image Woof
10.5
59.7
65.5 10.5
10.7
72.6 104
72.2 10.4
69.8
72.0
100
69.5 =0.4
68.7 =0.4
10.5
70.8
20.2
79.2 10.1
80.3 00.3
78.6 =0.4
4.1 10.1
0.
4.2
0.9
0.5
6.8 :1: 0.3
5.4 1: 0.4
7.6 : 0.5
ImageNet- IK
40,2
4.52+0.5
41.5
42.5 0.7
45,9
45.4
46.3
50
55.2 +: 0.2
0.6
56.6
57.7 + 0.5
60.1 1 0.3
60.4 0,2
60.2 0.4
58.9 1:9:::0 0.7
100
59.7
1 0.4
0.2
6.5
0.51
0.1
62.2 0.5
63.5 = 0.2
0.4
Table 1: Performance comparison across various datasets with well-known decoupled distillation methods. The highlight results denote the best performance achieved under different settings within our fair framework. "" denotes the second performance, and "_" denotes the results could not obtained with certain settings.
On CIFAR-10/100 EDC achieves dominant performance advantages in most scenarios, underper- forming D'M in only two specific compression ratio settings. However, the performance superiority diminishes when EDC is applied to higher-resolution datasets and more complex data domains. In contrast, Generation-based methods exhibit competitive performance on both ImageNette and Woof D'M particularly benefits from its image diversity advantages in large IPC settings. While Minimax maintains stable performance across all settings, it is limited by its label space and cannot be applied datasets other than ImageNet- 1K and subsets. Notably, in a low IPC setting(e.g. IPC=1), D'M shows severe limitations, especially on the representative fine-grained dataset Image-Woof, while other methods maintain stable performance, showing that D'M cannot effectively condense class-relevan features under extreme settings.
The most challenging dataset ImageNet- I reveals a distinct phenomenon: each of the four methods achieves the best performance under different IPC settings. RDED exhibits performance degradation with increasing IPC due to its limited diversity, mirroring trends observed in Image-Nette and Image- Woof. D'M and Minimax still demonstrate better scalability in IIgh-IPC settings. Surprisingly. EDC maintains a competitive performance across all settings. We provide an additional qualitative analysis in Appendix O and visualizations in Appendix P

---

Published as a conference paper at ICLR 2026
ImageNet-IK
Optimization
Generation
Selection
IPC
Rectified
SRe²L
CDA
G-VBSM
DWA
EDC
Minimax
D'M
RDED
12.8
6.6 0.2
4.1
4.2 0.8
4.2
4.5 = 0.9
7.0 0.5
5.4 = 0.4
0.5
(1.01)
21.3 0.6
33.5 0.3
31.4 0.5
37.9 0.2
48.6 =
44.3 0.5
27.9
4.
10
40.2 +: 0.3
41.2 0.5
41.5 0.6
42.5 +: 0.7
46,9 + 0.6
45,9 +: 0.7
45.4
46.3 4: 0.2
A
(18.9
(7.71)
(10.1 t)
(4.6
(1.50)
(1.6 ()
(17.5
(4.3
46.8 0.2
$1.8 0.3
55.2 35.2..2.. 0.2
0.3
55.2 ::
50
/
55.2 = 0.2
56.7
55.7 0.4
59.2
60.1 = 0.3
60.4 = 0.2
60.2
58.9 +.7
(8.4 ()
(3.21)
(3.9
(4.0 f)
(2.1
(1.8 ()
(5.0
(2.41)
52.8 0.3
0.2
56.6 0.2
0.5
59.3 0.0
100
59.7 = 0.4
60.6
61.5 0.4 0.4
62.1 0.5
63.2 = 0.1
62.2 = 0.5
63.5 + 0.2
61.5 0.4
(6.9 11 (2.61)
(4.9
(4.4
(4.2
Table 2: Comparison of reported accuracy obtained from original papers and re-evaluated by RD on ImageNet-IKK IK. "_" denotes the missing values in previous works.
Summary: The observed performance differences among distillation methods primarily at- tributable to inconsistencies in post-evaluation settings rather than inherent differences in data quality, and no individual method consistently oulperforms the others.
4.2 WHAT METRICS BEYOND TEST ACCURACY MATTER FOR EVALUATING QUALITY?
Effectiveness Efficiency. Another critical evaluation metric that has been systematically overlooked in previous studies is the time con- sumption for dataset generation. Existing per- formance comparisons remain incomplete due to their limitations within specific method cate- gories (e.g., optimization-basee based). Given the min- imal performance variations observed under our RD3 framework, efficiency emerges as a crucial evaluation criterion that needs comprehensive comparison.
035
LAA
(63.2%, 52h)
43.5
1/62.1%
(61.5% 1h)
(61.5%, 694h)
1/611.%% 256h)
(60.6%% 112h)
(59.7% 106h)
265
Total Training
For optimization- based and sraecton-bbsedd methods, total time consumption equals per- image processing cost multiplied by total im- methods require additional computation cost for diffusion model fine-tuning Gu et al. (2024) or category centroid calculation Su et al. (2024) beyond basic generation intuitive and equitable efficiency comparison, measured generation time under the most challenging IPC=100 setting. Notably, our evaluation excludes classifier training time required by optimization and selection methods, meaning their actual deployment costs would be substantially higher than the results we reported. All the experiments are conducted on a single Nvidia RTX-3090. The visualization shown Figure reveals that while performance differences remain marginal, time consumption varies by orders of magnitude (i.c., up to 100x).
Figure 3: Comparison of the effectiveness and efficiency of all the decoupled distillation meth- ods. Upper-left quadrant representing optimal effectiveness--effiienny balance.
age count. Generation-barn
time. To establish
Generalization Ability. To systematically investigate the intrinsic differences among different methods, we evaluate the corresponding synthetic datasets using diverse evaluation architectures. Unlike G-VBSM and EDC utilizing ensemble models to generate hybrid soft labels, we exclusively employ single ResNet-18 for soft label generation to ensure maximum faimess in evaluation. Table presents the experimental results of IPC=50 on ImageNet- The performance variance across the ResNet family remains within 5%, with discrepancies decreasing as network depth increases.
7

---

Published as a conference paper at ICLR 2026
Method
ResNet-18
ResNet-50
ResNet-101
MobileNet-V2
EfficientNet-B0
Swin-V2-T
ViT-B-16
SR
55.2
62.8 +: 0.1
63.6 +
0.5
55.3 +
55.3 0.5
53.4
CDA
56.7 0.6
63.1 0.2
64.2 0.3
50.2 50.2 0.2
56.0 0.3
56.6 0.3
0.3
G-VBSM
56.6
63.3 = 0.4
63.8 0.3
48.7 0.1
56.1 0.1
58.2
0.
DWA
0.5
63.3 1 0.2
64.1
52.1 0.2
57.3
$7.9.02
55.5
EDC
0.3
66.4 + 0.3
66.0 0.2
54.9
59.6 0.4
62.4 0.3
6..6
Minimax
60.4 + 0.2
65.0 + 0.3
64.6 + 0.5
53.8 +.18+0.1
59.9 + 0.3
61.2+
62.3
62.3+0.2
D*M
60.2
66.0 + 0.3
66.5 1 0.5
55,8
0.2
61.4 0.3
62.2 0.3
63.7 0.3
RDED
58.9
65.2 = 0.3
65.9 0.2
53.5
= 0.3
58.7 = 0.4
61.3 0.6
61.4 0.3
Table 3: Generalization ability of synthetic dataset ImageNet- IK under IPC=50. All the soft labels
by
to ensure fainwesSRe".
G-VBSM
EDC
D'M
RDED
Noise
40.2
41.2
41.5
38.6
36.5
4.2
7.6
A
(3.9
(9.1
IPC=1
12.2
12.8
15,5
13.9
15.6
(8.1 1)
(8.6
(8.5
(8.5 f)
(8.0
Random
41.8
42.6
46.4
42.5
45.6
A
(1.6
(1.4
(1.9
(L.3
40.2
41.5
46,9
45.4
46.3
IPC=10
40.9
42.3
47.9
46.1
47.5
RDED
41.5
42.0
46.1
43.1
46,9
(0.7 1)
(0.8 f)
(1.0 1)
(0.7
(1.2.))
(1.3
(0.8 1)
(4.5 1)
(0.6 1)
55.2
56.6
60.1
60.2
58.9
D'M
40.9
418
44.8
42,2
45,4
IPC=50
51,2
52.4
57.1
56.3
56.4
4
(0.7
(0.6
(3.3 ()
(0.3 ()
(1.5.)
(4.0 ()
(4.2 L)
(3.0 J)
(3.9
(2.5
Table 4: Performance comparison of optimization- Table 5: Performance impact of using multiple based methods with different initialization on teacher models to generate hybrid soft label on ImageNet-IK under IPC=10. + and t indicate ImageNet- K. The highlight results denote the best the change direction of A compared to the default performance achieved across methods under differ- settings "_" of various distillation methods.
settings.
Performance divergences become more pronounced when testing other CNN-based models, with all methods achieving the poorest performance on MobileNet-BO..
Our extensive experiments conducted transformer- architectures reveal that most methods outperform the ResNet-18 baselines, demonstrating effective knowledge transfer. However, we observe that substantial performance degradation for SRee1., CDA, and DWA ViT-B-16, potentially attributable to their limited image diversity. which hinders ViTs' learning capacity. In contrast, the superior diversity of D enables it achieve SOTA performance across most settings. More experimental results about generalization are shown in Appendix and Appendix F
Summary: comparison marginal differences test accuracy, computational efficiency should be considered a more critical criterion for evaluating different methods. Furthermore, generalization capability which often exhibits more pronounced variation offers more informative metric.
4.3 WHAT SUBTLE FACTORS INFLCENCE THE FIDELITY OF DISTILLED DATASETS?
Alternative Initialization. As thoroughly demonstrated in early information matching studies Zhao & Bilen (202 1): Liu et al. (2023), the initialization of synthetic datasets often plays a critical role the field of dataset distillation. However, for existing aptimization-base decoupled distillation methods, the use of superior initialization has become an underacknowledgee practice. We systemati- cally investigate the impact of initialization on optimization-bant methods and explore potential combinations of different distillation methods by different initializations.
As shown in Table different initializations exert substantial influence on specific methods. Notably, both DWA and EDC. which primarily aim to enhance dataset diversity, exhibit significant performance G- G-VBSM demonstrates significant
degradation when using Gaussian noise imitialization.oonwee performance improvements when initialized with random sampling RDED-ggnneatee images, occasionally outperforming EDC in certain settings. This is attributed to the inherent uncertainty introduced by its multi-teacher model matching mechanism during optimization. We provide qualitative analysis in Appendix H
Hybrid Soft Label. Despite G-VBSM and EDC emphasized that the involvement of multiple teacher models in image optimization necessitates the use of hybrid soft labels generated by these models during the relabel phase. the performance benefits of using hybrid soft label with other methods has not been explored. We implement hybrid soft labeling across five representative methods and evaluated them ResNet-18 18. Experimental results shown in Table 5 reveal remarkable performance
8

---

Published as a conference paper at ICLR 2026
IPC
Loss
SRe*l.
G-VBSM
EDC
D
RDED
KL MSE-GT
4.1
4.2
7.0
5.4
7.6
(
3.6]
4.81
6.8
5.71
7.8
ResNer-001 ((-1)
ReNNe-101 (5-1)
KL MSEGT
40.2
41.5
46.9
45.4
46.3
10
40.97
42.31
47.91
47.51
46.81
KL.
55.2
56.6
60.1
60.2
58.9
MSE-GT
56,41
57.81
60.81
61.3
60.21
KI.
59.7
61.5
63.2
63.5
61.5
100
MSE-GT
59.5)
61.91
64.1
62.71
Table 6: Performance impact of using different loss functions on ImageNet-
SRteL
G-VESM
DWA
Mininas
D'M
RDED
Diettatis Matkel
1 K. 4. and f indicate the change direction compared to KI divergence.
Figure 4: Performance on ImageNet- IK under IPC=10 with different smoothing factor Ç.
Label
IPC
CIFARIO
CIFARIO0
TinylmageNet
ImageNette
ImageWoof
ImageNet- 1K
16.4 + 0.4
3.7 + 0.6
1.9 + 0.7
18.2 + 0.5
10.2 :: 0.2
0.7 :: 0.4
10
23.5 + 0.3
10.6 0.4
4.1 0.3
43.7+ 0.4
23.2- 0.5
6.1- 0.6
Hard
31.6 0.3
23.1 0.3
10.1 0.5
65.0+ 0.4
35.5+ 0.2
25.8+ 0.3
100
40.6+ 0.3
36.9 0.5
17.2+ 0.2
72.8+ 0.2
41,8+ 0.1
40.3+ 0.1
25.51 0.4
1.6
6.64:0.7
23.2 0.7
16.3 0.4
5,2 0.5
10
42.3# 0.3
50.5 0.2
09.3
64.5 0.3
36.5 0.4
45,81 0.1
Soft
50
66.41 0.1
68.3 0.2
57.31 0.4
88.51 0.2
69.31 0.5
61.81 0.2
80.1 0.5
70.9 0.2
59.9 0.2
90.81 0.1
74.81 0.3
64.11 0.2
Table 7: Performance of random sampling with hard label and soft label. The random images show a strong performance especially with soft label.
gains, SRe²1. achieve a staggering twofold improvement with hybrid labels under IPC=1, while RDED and D'M also demonstrate significant enhancements even though their generation processes do not involve any other teacher models. Although the improvement magnitude decrease at IPC=10, consistent performance gains were still observed. The full results are shown in Appendix This evidence confirms that hybrid soft labeling is a universal and impactful technique.
Summary: There exist general techniques aimed at refining the quality of synthetic datasets along two dimensions (i.e., the images and soft labels), which can lead to substantial performance variations without changing existing methods.
4.4 WHICH OVERLOOKED VARIABLES UNDERMINE FAIR EVALUATION IN EVALUATION??
Optimization Objective. We investigate the impact of another previously misaligned loss function selection across all methods. G-VBSM replaces the commonly used KL divergence (g.(-11-) (-II-) in
other decoupled distillation methods Yin al. (2024p: h: Yin Shen (2024); [Su al.] (2024) with MSE+ y x GT (i.e., combining mean squared error and ground truth alignment). This modification draws from two insights: (1) The theoretical perspective proposed Kim et al. (202I) that KL
divergence becomes equivalent to MSE as r xo. (2) The standard knowled goowedgee distillation practice of incorporating ground truth alignment as regularization.
To eliminate confounding variables, we systematically apply different loss function to all distillation methods. We set =0.025, which is value empirically validated as acceptable G- G-VBS''s original paper. Experimental results shown in new loss function does not guarantee consistent performance gains, it produces positive effects in most settings. Moreover, these improvements could potentially be amplified through optimal y selection. A comprehensive results are provided in Appendix [ and we provide the experimental results under hard label setting in Appendix G and diverse knoledge distillation techniques in Appendix NN This finding suggests that the impact of loss function requires proper ablation studies when comparing against baselines in future research.
that under our RD3 framework, while the
Optimization Scheduler. SRe2L first demonstrated that employing Adam optimizer with cosine annealing on large datasets enhances stability and performance. Building upon this foundation, RDED and EDC adopted smoothing learning rate. The mathematical formulation of this schedule is given by n L+cos(ri/(N)) where 7/ represents the learning rate epoch and N represents the

---

Published as a conference paper at ICLR 2026
total epoch number. However, recent work CV-DD Cui al. (2025) achieved superior performance through manual adjustment of Ç across different datasets, compression ratios, and evaluation model architectures.
To isolate the influence of Ç, we first evaluate all methods using both ResNet-18 and ResNet-101 as evaluation models. Our analysis visualized in Figure reveals that with identical teacher and student models, faster learning rate decay facilitates earlier entry into fine-tuning phases. While evaluating with larger models (e.g.. ViT-B-16), excessively small learning rates often hinder effective learning, resulting in an unconverged solution at the end of optimization.
reveals
Summary: Consistent performance improvements can be achieved solely through adjustments the training protocol, even when the synthetic dataset is held fixed. To ensure fairness, all subsequent methods should adopt unified training setting, regardless of their specific motivations.
WHAT CONSTITUTES THE OVERLOOKED BASELINE IN PREVIOUS COMPARISON?
Compared to traditional information-matchiee optimization methods, a significant performance gain in decoupled dataset distillation methods arises from their use of multi-round soft labels. To investigate the true efficacy of this method, we explored constructing the generated dataset using directly sampled random images.
Experimental results shown in Table reveal surprising observation: under the soft label paradigm, simple random sampling outperforms all existing decoupled dataset distillation methods CIFAR- 10/1000 ImageNette, and ImageNet-IK. This outcome aligns with findings in existing literature Xiao et al. (2025). Further analysis demonstrates that for coarse-grained datasets like ImageNette and IeageNet-1K KK, randomly sampled images maximize diversity while maintaining alignment with the teacher model's learned knowledge, thereby achieving strong soft label consistency, Conversely, on fine-grained datasets such as TinyImageNet and ImageWoof, existing decoupled methods excel by generating highly representative images that facilitate student model learning, whereas random sampling often introduces ambiguous patterns that degrade performance.
Under the hard label setting, Minimax and RDED surpass random sampling across most datasets and compression ratios. This advantage stems from their ability to produce realistic and representative images that align with category distributions, thereby aiding student model training. In contrast, optimization- methods and D M, lacking explicit knowledge alignment via soft labels from teacher models, generate images with weak correlations to hard labels. This limitation severely hinders student models from learning accurate representations, resulting in performance far inferior to random sampling.
The field of decoupled dataset distillation has historically overlooked the fact that randomly sampled images with soft labels constitute a powerful baseline, which in some cases all existing distillation methods. Future work should pay greater attention to this phenomenon and adopt random sampling strong comparative benchmark.
outperform
CONCLUSION
In this work, we revisit common inconsistencies in experimental settings used compare decoupled dataset distillation methods and highlight the importance of establishing fair and comprehensive evaluation protocols. To this end, we introduce RD3, systematic re-evaluation framework that distinguishes true methodological improvements from performance gains driven by favorable hyper- parameter tuning. Our empirical analysis reveals that many reported advances are largely attributable to hyperparameter optimization rather than substantive algorithmic innovations. Building these insights, we further investigate the prevalence of evaluation inconsistencies and provide refined performance assessments. Our findings offer actionable guidance for future work aimed at genuinely improving the quality of synthetic datasets.
Ack Acknowledgement. This work is supported in part by the National Natural Science Foundation of China under grant 62576122, 62301189 189, and Shenzhen Science and Technology Program under Grant KIIID202400031000 1037000000
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4750-4759 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. General- izing dataset distillation via deep generative prior. arXiv preprint arXiv:2305.0/6 2023.
Jeffrey A Chan-Santiago, Praveen Tirupattur, Gaurav Kumar Nayak, Gaowen Liu, and Mubarak Shah. Mgd Mode-guided dataset distillation using diffusion models. arXiv preprint arXiv:2505.18963 2025.
Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang. Xiaobo Zhang, and Wei Wang. Influence- guided diffusion for dataset distillation. In The Thirteenth International Conference on Learning Representations, 2025.
Jiacheng Cui, Zhaoyi Li, Xiaochen Ma, Xinyue Bi, Yaxin Luo, and Zhiqiang Shen. Dataset distillation via committee voting. arXiv preprint arXiv:2501.0757,, 2025.
Justin Cui, Ruochen Wang. Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation benchmark. arXiv preprint arXiv:2207.09633 2022.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- 1k with constant memory. In International Conference on Machine Learning, pp. 65960 PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.111 11929, 2020.
Jiawei Du Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. arXiv preprint 1rX1v:2409.17612. 2024.
Jinnann Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang. Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15793-1011 15803, 2024.
Xulin Gu, Xinhao Zhong, Zhixing Wei, Yimin Zhou, Shuoyang Sun, Bin Chen, Hongpeng Wang, and Yuan Luo. Temporal saliency-guided distillation: A scalable framework for distilling video datasets. arXiv preprint arXiv:2505. 20694, 2025.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via sefficultt--alinne trajectory matching. arXiv preprint arXiv:2310.0577 2023.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2026
Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applica- tions. arXiv preprint arXiv:1004.0486,, 2017.
Youbing Hu, Yun Cheng, Olga Saukh, Firat Ozdemir, Anqi Lu, Zhiqiang Cao. and Zhijun Li. Focusdd: Real-world scene infusion for robust dataset distillation. arXiv preprint arXiv:2501 006405 2025.
Taehyeon Kim, Jaehoon Oh, Nak Yil Kim, Sangwook Cho, and Se-Young Yun. Comparing kullback-leibler divergence and mean squared error loss knowledge distillation. arXiv preprint wrXiv:2105.08919 2021.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung. Sangdoo Yun, and Sungroh Yoon. Dataset condensa- tion with contrastive signals. In International Conference on Machine Learning. pp. 12352-12366 PMLR, 2022.
Zekai Li, Xinhao Zhong, Zhiyuan Liang, Yuhao Zhou, Mingjia Shi, Ziqiao Wang, Wangbo Zhao, Xuanlei Zhao, Haonan Wang. Ziheng Qin, Dai Liu, Kaipeng Zhang, Tianyi Zhou, Zheng Zhu, Kun Wang, Guang Li, Junhao Zhang, Jiawei Liu, Yiran Huang, Lingjuan Lyu, Jiancheng Lv, Yaochu Jin, Zeynep Akata, Jindong Gu, Rama Vedantam, Mike Shou, Zhiwei Deng, Yan Yan, Yuzhang Shang, George Cazenavette, Xindi Wu, Justin Cui, Tianlong Chen, Angela Yao, Manolis Kellis, Konstantinos N. Plataniotis, Bo Zhao, Zhangyang Wang. Yang You, and Kai Wang. Dd- ranking: Rethinking the evaluation of dataset distillation. GitHub repository, 2024. URL https Ranking
com./b.com//i //github. com /NUS NUS -PPC-AI-Late
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 77314-17324 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Ding Qi, Jian Li, Jinlong Peng, Bo Zhao, Shuguang Dou, Jialin Li, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Cairong Zhao Fetch and forge: Efficient dataset condensation for object detection. Advances in Neural Information Processing Systems, 37:1192833 19111192831111 119300, 2024.
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth a thousand images in dataset distillation. arXiv preprint arXiv:2406. 10485, 2024.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 17097-17107. 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 22255778-2224 2022.
Yuzhang Shang, Zhihang Yuan, and Yan Yan. Mim4dd: Mutual information maximization for dataset distillation. Advances in Neural Information Processing Systems, 36, 2024.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16709-16718, 2024a.
Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. arXiv preprint arXiv:2404. 13733, 2024b.
12

---

Published as a conference paper at ICLR 2026
Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In European conference on computer vision, pp. 673-690. Springer, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D 4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5809-5818, 2024.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9390-9399, 2024a.
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15731-15740, 2024b.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 61 14. PMLR, 2019.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12196-12205, 2022.
Tongzhou Wang, Lun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 11X1:://11.10955 2018
Lingao Xiao, Songhua Liu, Yang He, and Xinchao Wang. Rethinking large-scale dataset compression: Shifting focus from labels to images. arXiv preprint 1xXiv:2502.06434.. 2025.
Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowledge distillation self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
17185 17194, 2023.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8715- 8724, 2020.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation via curriculum data synthesis large data era. Transactions Machine Learning Research, 2024.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from new perspective. Advances in Neural Information Processing Systems, 36, 2024.
Weijia Zhang, Dongnan Liu, Weidong Cai, and Chao Ma. Cross-view consistency regularisation for knowledge distillation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 2011-2020 2024.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. arXiv:2/ 10.0418.. 2021a.
arXiv preprint
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685. PMLR, 2021b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 1(2)3, 2021.
13

---

Published as a conference paper at ICLR 2026
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp- 11953-11962, 2022.
Xinhao Zhong, Bin Chen, Hao Fang, Xulin Gu, Shu-Tao Xia, and En-Hui Yang. Going beyond feature similarity: Effective dataset distillation based on class-aware conditional mutual information. arXiv preprint arXiv:2412.09945. 2024a.
Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Jianlong Wu, and Bin Chen. Efficient dataset distillation via diffusion-driven patch selection for improved generalization. arXiv preprint ar 2.99599, 2024b.
Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Meikang Qiu, Shuhan Qi, and Shu-Taa Tao Xia. Hier- archical features matter: A deep exploration of progressive parameterization method for dataset distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30462-30471, 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A COMPARISON WITH EXISTING EVALUATION WORKS
A.1 DC-BENCH
At the time of its release, DC-BENCH Cui et al. (2022) did not include decoupled data distillation methods, such methods had not yet been proposed. Consequently, the benchmark only covered basic bi-level distillation methods. Moreover, due to their substantial computational and memory requirements, these methods not scalable to large datasets or higher IPC settings, limiting the coverage and applicability DC-BENCH compared to our work.
A.2 PCA
PCA Xiao et al. 2025) re-evaluates existing optimization-han based decoupled distillation methods under the CDA setting ImageNet- IK, revealing that their performance often falls below that of random sampling and proposing data adjustment strategies to address this gap. In contrast, our work establishes more comprehensive definition of decoupled distillation methods, explicitly incorporating generation based methods. We further demonstrate that existing decoupled distillation methods do consistently underperform random sampling across all datasets, especially on fine- grained datasets (e.g., ImageNet-Wond Additionally, our study provides a deeper analysis of how various evaluation settings influence the performance of all types of decoupled distillation methods, offering insights that inform future improvements.
A.3 DD-RANKING
DD-Ra ID-Ranking Li et al. (0224 introduces evaluation metric by computing accuracy gaps between distilled and randomly sampled datasets under different configurations to unify the evaluation of both bi-level and decoupled distillation methods. However, it does not offer standardized benchmark framework nor investigate the performance discrepancies among different synthetic datasets. While our method make an in-depth analysis on how the various settings influence the test accuracy of different decoupled distillation methods.
B LITERATURE REVIEW
B.1
S
SR (Yin et al. 2024) first proposed the decoupled concept, drawing method from data-free knowl- edge distillation Yin et al. (2020) to completely disentangle proxy model training from data optimiza- tion processes, thereby reducing the substantial time overhead required by traditional information-
matching based optimization methods Wang et al. (2018); Zhao Bilen (202aa b); Cazenavette
et al. (2022); Wang et al. (2022): Shang et al. (2024); Zhong et al. (2024a). Since inheritss
the data-free distillation framework, it employs Gaussian noise initialization which poses significant challenges for optimizing towards real data distribution. To address this, SRe simultaneously aligns dynamic BN statistics from synthetic datasets with teacher network's 's frozen BN information during optimization, thereby further constraining the generated dataset's distribution. Additionally, to resolve parameter dependency issues caused by single proxy model usage, SRe"L pioneered the introduction of epoch-wicd soft labels during student model training to maximize knowledge transfer and alignment. The method further incorporates data augmentation operations like CutMix and RandomResizedCrop during alignment phases to enhance dataset diversity and boost performance. Subsequent methods have expanded the pipeline to object detection (Qi et al. 2024) and video classification (Gu et al. 2025 tasks.
B.2 CDA
Building upon LLs foundation, CDA Yin Shen (2024) introduces curriculum learning into the image optimization process by implementing adaptive progressive cropping from small to large
15

---

Published as a conference paper at ICLR 2026
scales in generated datasets, thereby achieving sifficulty-graduaate optimization schemes. This framework further modifies hyperparameters SRe²L's training evaluation procedures. Specifically, reducing BS yields substantial performance improvements. However, CDA regrettably omits thorough experimental analysis of these setting modifications while failing to isolate and validate the actual performance contributions from the curriculum learning component through ablation studies.
B.3 G-VBSM
Although SRe"L significantly enhances student model performance by utilizing labels during the relabel phase for knowledge transfer, it inadvertently causes generated datasets to overfit to single teacher parameters, thereby compromising generalization capability. To address this limitation, G- VBSM (Sao et al. (2024a) introduces model pools comprising diverse architectures to optimize generated datasets, effectively reducing dependency on specific parameters and architectural settings. The method extends alignment objectives beyond Batch Normalization statistics to include convolutional features during optimization, while shifting the optimization scale from IPC to category- level for enhanced intra-class data diversity. G-VBSM further proposes more effective loss functions during evaluation phases to constrain information-rich datasets, explicitly requiring soft labels to be generated through collaborative predictions from architecturally heterogeneous teacher models. Notably, while maintaining SRe2L's original hyperparameter settings (e.g.. batch size) in post- evaluation phases, G-VBSM's novel settings remain untested on datasets generated by SRe"L, leaving unresolved whether these modifications specifically cater to its own optimization characteristics. And the additional matching strategy introduced during recover phase leads to ten times time consumption.
soft
B.4 DWA
Since SReeL. employs Gaussian noise initialization for generated datasets, the optimization process must rely on the mean values in BN statistics to approximate the original data distribution, which severely compromises the diversity of generated datasets. DWA Du et al.) (2024) initially samples from the original dataset as initialization, then decouples the mean and variance components in the BN-based loss function while allocating greater optimization weights to the variance component. Subsequently, it introduces weight perturbations to teacher models to further enhance dataset diver- sity. Notably. DWA not only achieves substantial performance improvements by adopting CDA's parameter settings, but also significantly boosts computational efficiency through true initialization that accelerates optimization convergence. Using better initialization has consequently emerged simple yet effective performance enbancement technique in subsequent research.
B.5 EDC
Building upon G-VBSM, EDC Shao et al. (2024b) introduces systematic improvements across three critical phases. During dataset generation, EDC advances beyond DWA's random sampling initializa- tion by employing
images starting points. This strategic initialization effectively constrains redundant degrees of freedom arising from multi-teacher collaborative optimization while dramatically accelerating convergence. The framework innovatively incorporates flatness regulariza- tion through rigorous analysis of loss landscapes during optimization, achieving sharpness-aware minimization. For relabeling phases, EDC implements refined settings including reduced batch sizes as the same as RDED did and enhanced teacher model selection for improved soft label blending. The changes during evaluation stage include Further batch size reduction, precision-tuned learning rate schedulers, and EMA-based assessment mechanisms for performance refinement. Despite achieving multi-fold performance gains specific settings, EDC critically overlooks two crucial aspects: (1)
Systematic verification of proposed techniques" generalizability beyond distillation contexts. (2) Failure to disentangle performance improvements between dataset quality and evaluation protocol enhancements. This methodological gap exacerbates existing inconsistencies in decoupled dataset distillation frameworks, where performance metrics become confounded by optimized evaluation hyperparameters.
16

---

Published as a conference paper at ICLR 2026
B.6 MINIMAX
With rapid advancements in diffusion models, these architectures have been successfully integrated into dataset distillation frameworks. Unlike conventional parametric distillation approaches that employ GANs Cazenavette et al. (2023); Zhong al. (2025), diffusion-based methods directly generate images through learned stochastic processes rather than pixel-level optimization, simulta- neously enhancing generalization capabiliitiss and reducing computational overhead. Minimax Gu al. (2024) utilizes DiT pretrained on ImageNet-11 K foundational models, implementing novel regularization strategy that expands feature distances to the most similar samples while contracting distances to dissimilar counterparts during diffusion model fine-tuning. This optimization ensures generated samples effectively approximate the original dataset distribution. Although Minimax incorporates CutMix for performance enhancement, it intentionally omits epoch-wise soft labels. Given its demonstrated scalability to large-scale datasets, we formally categorize Minimax within the decoupled dataset distillation and conduct comprehensive performance re-evaluation with soft label.
B.7 DOM
While Minimax demonstrates notable performance on ImageNet and its subsets, the diffusion model fine-tuning process incurs substantial temporal costs. This method employs DiT models and relies on one-hot labels as categorical prompts, fundamentally restricting its applicability to other datasets like CIFAR-100100. D'M Su al. (2024) addresses these constraints by building upon Stable Diffusion (SD), text-to-image generation diffusion model. The D'M pipeline processes datasets through VAE encoders to obtain visual embeddings, conducts latent space clustering for class centroid derivation, and finally synthesizes images by combining these centroids with corresponding textual prompts. Although D surpasses SRc*L in generating semantically coherent images through diffusion mechanisms, its performance remains inherently dependent the generation-based model's 's capabilities. The unconstrained visual embeddings frequently deviate from SD's latent data distribution, resulting in category-irreleant image generation. While such anomalies may enhance dataset diversity when employing soft labels, they significantly impair performance on fine-grained datasets where precise feature representation is crucial, ultimately leading accuracy degradation.
B.8 RDED
For high-resolution datasets such as ImageNet-1K IK and its subsets, RDED Sun et al. (2024a) initially performs random cropping on original images and subsequently employs a pre-trained classifier to score and rank patches based loss magnitude, ultimately stitching multiple high-scoring patches into composite images. In contrast, for low-resolution datasets like CIFAR- 10/100, the framework directly scores and sorts original images through classifier evaluation while omitting cropping and concatenation operations. Distinet from alternative methods, RDED achieves remarkable computa- tional efficiency by eliminating the training process entirely, with its synthesized datasets maintaining central positioning within the original data distribution. The framework further enhances synthetic dataset performance through implementation of reduced batch sizes and optimized learning rate decay schedules, demonstrating superior adaptability across varying resolution domains.
B.9 EMERGING METHODS
Recent advancements in decoupled dataset distillation continue to emerge with notable methodologi- cal innovations. Here, we briefly summarize the subsequent methods.
DELT Xing (2022) addresses the trade-off between intra-class diversity and representational fidelity inherent in optimization-bant based approaches by initializing with RDED -eED-eeneatted datasets and selectively optimizing partial samples during training, thereby achieving enhanced performance with improved efficiency.
CV-DD Cui et al. (202)) employs multi-teacher model classification losses for joint optimization of synthetic datasets while establishing strengthened baseline through our proposed universal techniques integrated with S framework. Regrettably, these enhancements remain absent in other baseline implementations, leading to suboptimal solutions. In generation-based approaches,
17

---

Published as a conference paper at ICLR 2026
IGD Chen et al. (2025) leverages DiT with influence dunction-guidee optimization to amplify
dataset representativeness,, complemented by gradient-infone informed strategies for diversity augmentation. Regarding selection based methods,
DDPS Zhong et al. (2024b) identifies RDED's classification model-driven evaluation as severely compromising diversity, instead adopting diffusion model-guided loss differentials calculated with text prompt under labeled and unlabeled conditions to localize class-relevant regions.
DD Hu et al. (2025) utilizes pre-trained ViT patch extractors with attention-drive visual saliency mapping, while incorporating irrelevant background images to further diversify synthetic datasets.
MGD3 Chan-Santiago et al. (2025) introduces a plugin for diffusion model inference that guides the denoising direction with mode signals, encouraging the generation process to focus on more class-ifformative and prominent regions. During evaluation, MGD adopts hard-label settings from Minimax and soft-label settings from D*M.
Although these methods collectively advance dataset distillation effectiveness,, their comparative anal- yses frequently neglect baseline setting aligmment and essential ablation studies, thereby accentuating the critical necessity of our proposed systematic evaluation framework.
C IMPLEMENTATION DETAILS
Since the generation process of different methods is extremely different, we do not report the corresponding hyper-parameters for simplified version. Overall, the re-generation of synthetic dataset follows the consistent settings of previous works. The only variations occur during post- evaluation phase, and we list the implementation details follow.
Implementation Details for Post Evaluation ResNet-18 18
Implementation Details for Post- Dost-Evaluation Other Architectures
Optimizer
Adamw
Optimizer
Adammw
Learning Rate
0.001
Learning Rate
0.001
Loss Function
KL-Divergenee
Function
KL-Divergence
Batch Size
or ((S) 50)
Batch Size
100 or ([S) 100)
Epochs
400
Epochs
Scheduler
Cosine Annealing
Scheduler
Cosine Annealing
Smoothing Factor
Smoothing Factor
<=
Augmentation
PatchShuffle, RundomResizedCrop.
Augmentation
PatchShuffle.
RandomResizzeCropp
Horizontal Flip, CutMix
Horizontal Flip, CutMix
Table 8: Hyperparameters for post-evaluation Table 9: Hyperparameters for post-evaluation task
on 18 across various datasets.
other architectures across various datasets..
C.I RESNET-18
For ResNet- 18, since the teacher and student models share identical architectures, specific hyperpa- rameters must be employed during training to achieve optimal performance. As shown in Table 8 using smaller BS enables the student model to acquire more precise knowledge through soft labels. Simultancously, employing smaller Ç values accelerates learning rate decay, allowing the student model to enter fine-tuning phases faster for acquiring refined knowledge.
C.2 CROSS ARCHITECTURE
For other architectural settings where significant disparities exist between teacher and student models, particularly for ViT-based architectures that are substantially larger than ResNet-18, 18, knowledge alignment through soft labels often proves challenging. Therefore, two complementary strategies required shown in Table 9 Larger batch sizes mitigate gradient fluctuation effects to better approximate the original dataset distribution. Larger Ç values maintain learning rates at higher ranges during initial phases facilitate effective convergence learning.
18

---

Published as a conference paper at ICLR 2026
C.3 DIFFERENT EVALUATION SETTINGS INTRODUCED PREVIOUS METHODS
For clear comparison, summarize the post-evaluation settings used for each method in Table I0 This provides strong support for the necessity of our work. We acknowledge that adjusting evaluation settings may lead to improved performance, and encourage future work design task-specific enhancement techniques, it need to be emphasized that any changes made during the post-evaluation phase must be tested across all baselines assess their impact on performance.
Config
SRe"I.
CDA
G-VBSM
DWA
EDC
Minimax
D'M
RDED
Label Loss
Soft
Soft
Hybrid Soft
Soft
Hybrid Soft
Soft
Soft
Soft
KL
KL
MSE-GT
KL
MSE-GT
CE
KL
KL
Batchsize
1024
128
1024
128
100
256
1024
100
(O)
CutMix Patch Shuffle
CutMix
Data Augmentation
CutMix
CutMix
CutMix
CutMix
CutMix
CutMix
Patch Shuffie
Table 10: Different evaluation settings introduced by previous methods. The genuine quality improve- ment is conflicted by unaligned settings.
ImageNet
Optimization
Generation
Selection
Model
IPC
SRe"1.
CDA
G-VBSM
DWA
EDC
Minimax
RDED
4.7 +002
4.3 +
4.6 0.2
48 1 0.4
7.3 0.3
7.7 0.3
8.2 : 0.3
ResNet-50
48.5 0.4
4.2 0.3
50.1 0.4
53.9
54.1 0.2
53.3
53.2
50
62.8 = 0.2
63.1 0.5
0.3
63.3 = 0.2
65.2.20.2
65.0 = 0.1
66.0 + 0.2
65.2 = 0.1
64.9 0.4
65.2
+.2::00 0.2
65.5 0.1
65.6 = 0.5
66.9 - 0.1
67.1 = 0.2
67.4 + 0.2
66.9
3.4 3,4
3.1
3.6
4.0
+.3
6.0 - 0.3
4.7
6,8 : 0.3
ResNet-101
45.1 0.2
4.2
0.44
48.7 0.4
52.8
54.8 0.2
53.5 4 0.4
53.6
50
63.6 - 0.2
64.2
63.8 + 0.4
64.1 = 0.4
66.0 0.5
65.6 0.2
66.5
+ 0.4
65.9 0.3
65.6 0.1
66.1 0.3
66.4 0.3
66.5
0.3
67.4 0.5
67.6 0.1
67.9
0.2
67.5
Table Performance comparison on ImageNet-1K with decoupled distillation methods evaluated by ResNet-50 and ResNet-101
D UNIFIED AND FAIR FRAMEWORK ACROSS VARIOUS METHODS
To investigate whether our proposed unified RD3 framework introduce potential bias, provide the performance variations of all methods under this gradual setting in the table below. It can be observed that all methods exhibit similar patterns of performance improvement, which further supports the fairness and consistency of our proposed evaluation protocol. The consistent patterns observed across all methods under massive changeable settings, provide strong evidence for the fairness of our proposed RD framework.
Config
1R
CDA
G-VBSM
DWA
EDC
Minimax
D'M
RDED
Simplified
26.6
27.2
27.1
28.7
31.3
30.7
30.5
30.9
Aligned Batchsize
35.2
36.4
36.9
37.7
41.2
40.4
40.7
41.2
Aligned Data Augmentation
38.3
39.4
39.7
40.6
43.1
42.7
42.5
43.9
Aligned LRS (C)
40.2
41.2
41.5
42.5
46.9
45.9
45.4
Table 12:
The performance across various distillation methods with incremental settings. All the methods exhibit the same pattern on performance improvement.
E MORE PERFORMANCE ON RESNET SERIES
To systematically investigate the performance characteristics of synthetic datasets generated by various methods, we conduct comprehensive evaluations using deeper architectures (i.e., ResNet-50 and ResNet-100 101) that share structural homology with teacher models, assessing performance across multiple compression ratios ImageNet- IK. Our empirical analysis shown in Table [] reveals that performance disparities across methods diminish proportionally with model depth escalation, with maximum accuracy variance reduced merely 2.3% under ResNet-10" 101 evaluation, revealing
19

---

Published as a conference paper at ICLR 2026
that the substantial performance variations reported in existing literature predominantly stem from inconsistent evaluation protocols, and making the efficiency more essential than the effectiveness.
Furthermore, when doubling image quantity to IPC=50 and IPC=100 settings, synthetic datasets demonstrate negligible performance enhancements on ResNet-50/101 compared to ResNet-18 base- lines, suggesting current synthesis techniques fail to adequately preserve the original data distribu- tion's 's topological characteristics. The conspicuous absence of challenging boundary samples and failure in faithful reconstruction of daas-diisriminative features indicate that current generation- based mechanisms cannot effectively capture distribution extremities. This fundamental limitation in synthesizing distributionally faithful samples, particularly edge-case exemplars, highlights a critical research direction for subsequent investigations in distillation methods.
Method
ResNet-18
ResNet-50
ResNet-101
MobileNet-V2 V2
EfficientNet-B0
Swin-V2-T
ViT-B-16
SRePL
55.2
57.5 1 0.2
59.5 1
31.5 0.4
44.8 1 0.4
55.0 0.2
$1.3 31.3..0. 0.1
CDA
58.8 + 0.3
60.1 + 0.3
33.6
46.7 0.3
57.1
0.2
G-VBSM
0.2
58.2 0.4
60.2
0.2
47.2
5..
0.4
DWA
0.5
59.4 0.1
61.2
30.2+00.4
47.7 0.2
58.9 0.5
54.7
EDC
60.1
0.2
62.3
3.1
50.5
62.0 0.4
0.3
Minimax
0.2
62.2 0.3
61.6
37.
51.6 0.1
61.9 0.2
61.8
D'M
0.4
63.1 + 0.2
62.5 0.3
39.9 0.3
52.0 0.1
63.0
62.6 0.4
RDED
0.7
62.3 = 0.2
6.8
0.3
50.0 = 0.3
61.8 + 0.2
60.7 = 0.3
Table 13: Generalization ability of synthetic dataset ImageNet- IK under IPC=50 with 50 batch size in post-evaluation phase. The performance degradation is obvious on certain model architectures.
F VARYING GENERALIZATION ABILITY
We conducted two supplementary experiments to further investigate the generalization capabilities of synthetic datasets. First, under IPC=50 setting, we adjusted BS from 100 to 50. Experimental results shown in Table reveal significant architectural disparities in BS sensitivity: CNN-based models exhibited 3% 3%-4% performance degradation on ResNet-50/101, while MobileNet-V2 and EfficientNet-BB BO architectures suffered over 15% performance drop, indicating substantial variance in gradient fluctuation tolerance across architectures, particularly in data-efficient learning scenarios like dataset distillation. Conversely, ViT-based models demonstrated remarkable stability with merely 1% degradation on VIT-B-16 and even performance improvement on Swin-V2-T variants, confirming ViT's training stability given fixed dataset settings.
Subsequently, we evaluated cross-architecture generalization under IPC=10 as shown in Table [4 For CNN-based models, performance degradation remained acceptable compared to IPC=50 basclines. showing comparable decline patterns to ResNet- observations. However, ViT-based architectures suffered catastrophic performance collapse, with both Swin- V2-T and ViT-B-16 16 variants experiencing over 40% accuracy reduction. This phenomenon aligns with established observations regarding Vision Transformers' limited efficacy in low-sample regimes, simultaneously presenting critical challenges for achieving successful knowledge transfer from CNN-optimized distilled datasets to ViT architectures under high compression ratios.
Method
ResNet-18 18
5e
ResNet- 101
MobileNet-V?
EfficientNet-BO
Swin-V2-T
ViT-B-16
S
40.2 + 0.3
48.5
: 0.3
45.1 + 0.2
33.0 +: 0.3
43.3 : 0.5
15.5 0.2
11.2 0.2
CDA
41.2 + 0.6
49.2
+ 0.3
46.6 0.3
33.4 0.4
42,7 + 0.4
16.3
10.2
G-VBSM
41.5 0.6
49.5 i 0.3
46.4
34,5 0.4
43.8 1 0.4
1.34
11.8
DWA
4.7
50.1 0.3
48.7+
3.5
45.4 0.1
18.8 +.2
13.6+0.2
EDC
0.6
53.9 0.3
5288
3.2
48.4 0.3
0.5
22.1
Minimax
45.9
54.7 1
52.4 0.4
1.5
49.6
28.4 0.1
D'M
0.6
53.3 0.2
53.5
0.4
47.9
22.6
22.1
RDED
0.2
53.2 + 0.3
53.7 +
40.2
0.4
48.2 0.4
28.1
22.8+0.1
Table 14: Generalization ability of synthetic dataset on ImageNet-IK under IPC=10. The ViT-based models show extremely low performance with high compression ratio.
20

---

Published as a conference paper at ICLR 2026
ResNet118
Optimization
Generation
Selection
Dutuset
IPC
SRe"1.
CDA
G-VBSM
DWA
EDC
Minimax
D
RDED
10.5 20.5
10.3 #0.4
9.7 #0.6
9.2 10.5
18.8 107.
17.1 1:055
12.2 1:06
CIFARIO
14.6
16.6
18.1 40.2
23.1
22.8
50
15.6 #0.4
13.7 #0.4
17.2 =0.5
22.2 #0.5
29.2 -0.4
30.8 =0.55
35,3 +.03
100
18.2 +0.4
18.5 +0.4
20.3 +0.3
27.5 +0.2
39.4 +0.3
38.5 +-0.3
41.6 +0.5
+0.4
1.6 10.4
1.330.3
2.1 1:.5
3.7 40.4
2.04
4.4 0.77
3.2 +0.3
3.0 +0.6
4.1 +006
3.8 +0.7
12.4 +0.55
8.7 4-066
11.6 +0.4
CIFARI00
50
4,9 10.4
5.4 1.33
5.0 10.5
5,9 10.4
21.4 1.0.5
+:0.3
23.6 1:055
100
7.5 +0.3
7.440.4
7.8 +0.3
8,9 =0.5
30.2 1006
28.7 :0.4
32.5 1003
+0.5
100
1.3
1,9
3.3
2.2
10.6
3.2 =0.5
TinyImageNet
10
1,9 +0.5
2..4
2.9 #0.4
4.3 40.6
9.7 +0.5
6,3 10.5
4.7 4..4
10.6 11055
+0.4
7.3 -033
7.250.5
10.5
20.3 +.0.5
18.4 +0.3
=0.5
22.8 1:0.7
100
#0.3
12.7
13.3
=0.2
27.8 1044
25.3 10.4
10.6
30.7 1:0.3
18.2 =0.4
+0.5
18.3 1:0.5
+0.7
26.7 10066
18.9 +0,4
22.4 +0.3
22.5
10
20.2 10.4
20.5
20,5
29.3 20.3
38.6 10.7
39.1 10.4
40.4 10.3
34.6 1006
IetgeNette
10.1
27.8
28.3
32.5 +0.5
46.4 +0.4
+0.64
100
30.0 +0.4
31.5 +00.
31.2 +0.4
37.3 #0.5
52.7 +0.5
68.5 1:0.6
66.4 +..3
59.4 +0.5
11.7 #0.6
12.2
11.3 +0.6
12.5 #0.77
12.6 +0,5
16.8 +004
13,6 +0.5
14.4 +0.4
12.4 +0.3
13.2 +0.5
16.8 +0.6
23.7 +0.4
24.3 +0.22
22.4 +0.4
21.1 +0.5
ImageWont
13.6
14.3 10.6
23.7
25.8 +0.5
31.4 10.3
31.2 10.4
100
18.5 +0.5
19.1 #0.6
17.8 10.3
28.3 #0.2
30.4 10.4
49.7 1:0.3
42.2 +0.4
42.9 +0.3
0,3 +0,2
0.2
0.2
0.4 0.5
0.6 0.3
1.3 : 0.7
0.7 0.5
1.1 0.4
+.2
0.2
12.4
ImageNet- IK
50
3,5 + 0.3
0.9
0.2
6.1
17.4 0.5
33.4 + 0.4
18.9 2 0.5
31.7 0.4
100
4.6
+ 0.2
7.1
15.2
3.20022
16.9
21.1
0.3
42.1 + 0.1
26.1 = 0.2
40.1
+ 0.4
Table 15: Performance comparison across various datasets with well-known decoupled distillation methods using hard label. All the methods exhibit significant performance degradation.
G HARD LABEL PERFORMANCE
To systematically investigate the role of soft labels decoupled dataset distillation, we conducted experiments replacing soft labels with one-hot labels during evaluation while keeping other settings unchanged.
Experimental results shown in Table 1.5 reveal that optimization-base methods exhibit intolerable performance degradation across all datasets. Due to their exclusive reliance on single teacher models during optimization, the generated images tend to overfit to specific parameters. Simultaneously, using only cross-entie cross-entropy loss and matching BN statistics fails to effectively help randomly initialized student models learn meaningful categorical information. This forces optimization- based methods to completely depend on teacher-generated soft labels for knowledge transfer, creating significant deployment challenges for lightweight and simplified distillation implementations.
For generation-basee methods, while Minimax remains inapplicable to ImageNet and external subsets, its integration with pretrained DiT models enables generation of near-photorealistic images preserving substantial category-related features on ImageWoof, ImageNette, and ImageNet-IK datasets. This facilitates effective learning of mapping relationships between generated images and their labels in student models. However, D'M's complete dependence on latent distributions in Stable Diffusion results in significant divergence from target dataset distributions. especially on TinylmageNet. Without soft label guidance, D'M's excessive diversity hinders accurate student learning. Both generation-basee methods perform poorly on TinylmageNet datasets, indicating resolution differences exacerbate distributional inconsistencies.
Selection- based method demonstrate superior performance across most datasets by preserving au- thentic category-related visual features through real image selection. On fine-grained datasets like ImageWoof RDED enhances dataset representativeness through strategic simple image selection. However, coarse- coarse-grained datasets like ImageNette, oversimplified images impair student learning, resulting in substantial performance gaps compared to generation peaerattio-bbsedd methods. These findings suggest future improvements should focus on developing automated mechanisms identify dataset
21

---

Published as a conference paper at ICLR 2026
distribution characteristics and impose corresponding constraints, potentially enabling universal algorithms adaptable to various dataset types.
Noise
Random
RDED
Minimax
D
(a) SRe LL
(b) DWA
(c) EDC
Figure 5: Visual comparison of class "ostrich" with different distillation methods using various initialization.
H VISUAL COMPARISON OF INITIALIZATION
To visually demonstrate the impact of different initializations on optimization -based methods, we present the corresponding visualizations in Figure It is evident that varying initializations substan- tially influence the final synthesized images.
For SR limited visual divergence across initializations arises because its optimization process aligns closely with cross-entropy loss and BN statistics, prioritizing distributional alignment over diversity. The modest performance improvement primarily stems from generated images better matching the data distribution learned by the teacher model.
For DWA, initializing with real images imposes distributional constraints during optimization. While this introduces perturhations to the teacher model and disentangles BN statistics to enhance diversity, it also causes performance degradation when using noise distribution-shifted D'M images, the teacher model struggles to transfer knowledge accurately under such conditions.
For EDC, the collaboration of multiple teacher models and fewer optimization iterations leads to severe performance deterioration when noise-based initializations are employed. As shown in the figure, images generated by EDC in this scenario resemble noise. However. when initialized with more representative samples, EDC's diversity becomes constrained. With fewer optimization steps, the generated images closely resemble the initialization, preserving category- relevann details and consequently improving performance.
With the qualitative and quantitative analysis above, We identify the utilization of more powerful initialization should not be considered as strong contribution.
I HYBRID LABEL EXTENSION
As shown Table 16
we further investigate the impact of hybrid soft labels across additional datasets and compression ratios. To align with the setting of EDC, we employ ResNet-18, ConvNet-W: 128, WideResNet-16-2 MobileNet-V V2, and ShuffleNet- TinylmageNet, while ResNet- 18, MobileNet-v2, ShuffleNet V2-X0-5, and AlexNet is utilized to produce hybrid soft labels for ImageNet- IK. is observable that, regardless of the target dataset's scale, hybrid soft labels yield substantial performance improvements under IPC=1 and IPC=10.
to generate hybrid soft labels for
22

---

Published as a conference paper at ICLR 2026
IPC Hybrid
SRe"L.
CDA
G G-VBSM
DWA
EDC
Minimax
D 'M
RDED
TmyyInggeNet
1
/
6.1
7.1
6,2
6.8
10.2
9.8
3,9
11.1
13.8 ")
14.6 (7.5 1)
14.7 (8.5 t)
15.3 (8.5 7)
19.5 (9.3 f>
18.2 (8.4 t>
(8.6 f)
19.1 (8.0 f>
10
/
34.2
37.5
37.3
38.3
42.1
39.4
36.7
44.2
38.7 (4.5 -)
42.2 44.2 t)
(2.4 f)
43.6 (5.3 ()
48.0 (5.9 1)
44.6 (5.2 1)
41.1 (4.4 f)
49.2 (5.0 1)
52.5
53.0
53.7
54.2
57.1
54.4
53.8
58.7
50
/
48.9 (3.6 ))
48.7 (4.3 ()
49.3 (4.4
48.8 (5.4 U)
52.3 (4.8 ()
50.1 (4.3 ()
49.2 (4.6 J)
53.1 (5.6 ()
ImageNet-IKK IK
4.1
4.2
4,2
4,5
7.0
6.8
5.4
7.6
1
/
(2.2 7)
12.5 (8.3 f)
12.8 (8.6 1)
13.6 (9.1 ()
15.5 (8.5 f)
15.7 1)
13.9 (8.5 f)
15.6 (8.0 f)
/
40.2
41,2
41.5
42.5
46,9
45.9
45,4
46.3
40.9 (0.7 () 42.1 (0.9 1)
42.3 (0.8 t)
43.7 (1.2 ()
47.9 f)
46.8 (0.9 f)
46.1 (0.7 T)
47.5 (1.2 t)
/
55.2
56.7
56.6
57.7
60.1
60.4
60.2
58.9
51.2
52.4 (2.4 ()
54,9 (2.8
57.1 (3.0
56.8 (3.6 L)
56.3 (3.9 ()
56.4 (2.5 L)
Table 16: performance of using hybrid label on TinylmageNet and ImageNet- lk. The performance gain decrease with the growing IPC.
Notably, even for methods devoid of proxy model involvement, such Minimax and D'M, hybrid soft labels consistently enhance performance. We hypothesize that at lower IPC levels, diverse teacher models effectively augment dataset diversity through soft labels, thereby improving performance despite architectural discrepancies with the student model.
However, under IPC=50, hybrid soft labels induce significant performance degradation. We attribute this phenomenon to the fact that the generated images already ensure sufficient diversity, whereas overly heterogeneous soft labels hinder the student model's ability to learn precise categorical infor- mation from the distilled dataset, leading to performance decline. Synthesizing these observations, we emphasize that for dataset distillation tasks, the optimal selection of soft label formulations must be adaptively tailored to specific settings. Furthermore, exploring superior strategies for teacher model ensemble design across different methods remains critical direction for future research.
IPC
Loss
SRe²L
CDA
G-VBSM
DWA
EDC
Minimax
D
RDED
KI.
4.1
42
422
4.5 04.1
7.0
6.8 1.3
5.4 0.7 ()
7.6
GT
03.8 J)
0.20 J)
0.2 (4.0
0.6 (6.4
1.1 (6.5 J)
MSE-GT
3.6 l)
4..40.41) 1)
(0.6 t1
5.2 (0.7 f)
688 (0.2 D>
7.57 1)
5.7 (0.3
7.8 (0.2
KL
40.2
412
41.5
42,5
46.9
45,9
45.4
46.3
GT
(38.8 D
1.5 (39.7 L)
(10.3 D)
1.9 (10.6 L)
6.2 (10.7 ()
9.2 (36.7 1)
(39.8 1)
12.4 (33.9 D
MSE-GT
(0.9 1)
42.0 (0.8 1)
(0.8 1)
43.1 (0.6 t>
47.9 (1:0 1)
47.2 (1.3 f)
47.5 (2.1 f)
46.8 (0.5 1)
55.2
56.7
56.6
57.7
60.1
60.4
60.2
58.9
50
GT
3.5 (51.7 D)
5.9 (50.8 !)
7.2 (49.4 ()
6.1 (51,6 L)
17.4 (42.7 D)
33.4 (27.0 L)
18.9 (41.3 J)
31.7 (27.2.))
MSE-GT
(1.4
(1.5
57.8 1)
59.1 f)
60.8 (0:7 1))
61.5
61.3
60.3
KL
39.7
60.6
61.5 15.2 (16.3
62.1 16.9 (15.2
63.2
62.2
63.5
61.5
100
(56.1
(53.3
21.1 (42.1
42.1 (20.1
26.1 (37.4 J)
40.1 (21.4
MSE-GT
59.5 (0.2 ()
60.1 (0.5 L)
61.9 (0.4 f)
62.0 (0.1 L)
64.1 10:9 1)
63.0 (0.8 1)
62.9 (0.6 L)
62.7 (1.2 1)
Table 17: performance of using different loss functions on ImageNet- 1K. The performance could be further enhanced with the appropriate loss fuction.
J Loss FUNCTION CONSIDERATION
As shown in Table 17 we evaluate the performance of student models under different loss functions across all methods, with experimental results presented in the table. When using only the cross- entropy loss with ground-truth labels (equivalent to the hard label paradigm), all methods exhibit significant performance degradation, as detailed in our analysis of hard label performance. In contrast, when combining the gross-entropy losses between student outputs and both teacher outputs and hard labels as joint loss function, effective performance improvements are achieved across most settings. For experimental simplicity, we did not extensively tune the weights of the two cross-entropy losses, suggesting that optimized parameter settings could yield further enhancements. Future work should
23

---

Published as a conference paper at ICLR 2026
focus designing effective loss functions tailored the characteristics of the distilled dataset, thereby facilitating improved knowledge acquisition by student models.
5Rt
RDb
5000
Valde Notdetieeaarra Ncarra
V-40
V-400
17am Rotte11
18
30000 2-201
Ne500 (8)21)
Noxa
Calds
RandenCCrp
Raskesllp
Pastdkactte
Ege
Argincetation
Figure 6: Comparison of the learning rate decay with different post-evaluation epoch and smoothing factor. Our PD framework provide more refined parameter selection.
Figure 7: Comparison of the impact of using dif- ferent data augmentation ImageNet-IK under IPC=10. CutMix and RandomCrop play an essential role in enhancing performance.
K LR SCHEDULER ANALYSIS
The visual comparative analysis of learning rate decay strategies employed by RD3 across different settings and those used in prior works is illustrated in the figure. Methods like SRe2L and CDA adopt a cosine decay strategy with (=1, while RDED and EDC propose that using (=2 further enhances performance. Recent work CVDD suggests adapting Ç based on dataset compression ratios and evaluation models. Under extended training epochs, we implement refined ( selection according to evaluation models and visualized it shown in Figure for ResNet, Ç=1 is chosen, positioning the learning rate curve between historical settings. This ensures faster decay without premature optimization termination, thereby achieving additional performance gains. For other evaluation models, we employ larger learning rates than all previous methods under equivalent training epochs. Consequently, student models learn with larger step sizes despite imperfect knowledge alignment, enabling escape from local optima while accelerating convergence. To maintain framework simplicity, we did not exhaustively optimize selection, suggesting that adjusting Ç in specific scenarios could potentially achieve superior performance.
L SOFT LABEL TEMPERATURE
Under the unified setting provided by RD3 we investigate whether the soft label tempera- ture differentially impacts performance based on variations in dataset generation processes and data distributions. Experimental results, il-
SRe'L RDED
Validation Accuracy (%)
40
D
35
lustrated in Figure & reveal consistent trends across three representative methods. At exces- sively low temperatures, all methods exhibit pro-
nounced performance degradation, attributed to the concenttett output distribution of the teacher model, which resembles hard labels and fails to provide nuanced prior knowledge for the student model. When temperatures exceed 20, performance plateaus or even declines certain methods, as overly smoothed soft labels from the teacher model obscure categorical discriminability. These observations align with phenomena identified in our experiments with different loss functions, further substantiating that under the soft label paradigm, variations in generated images do not fundamentally alter behavioral characteristics, with soft labels predominantly governing the efficacy of knowledge transfer.
13
25
Temperature
Figure 8: The performance of ResNet-18 trained on different temperature settings ImageNet-1K under IPC=10.
24

---

Published as a conference paper at ICLR 2026
M IMPACT OF DATA AUGMENTATIONS
To investigate the generalizability of data augmentation, we designed ablation studies on various methods, with results illustrated in Figure Without any data augmentation, all decoupled dataset dis- tillation methods exhibit extremely poor performance. With CutMix and RandomResizedCrng added, both augmentation strategies substantially enhance the performance of all methods. Cropping images and compositing patches from different images significantly improve dataset diversity, enabling teacher models to convey richer knowledge via soft labels. Further deploying RandomHorizontalFliy yields additional around 1% improvement, demonstrating that simple flipping operations still contribute meaningfully. Finally, we tested the generalization of the PatchShuffle strategy proposed in RDED. While PatchShuffle randomly replaces patches across images, specifically designed for RDED's patch-composed images, surprisingly. applying PatchShuffle to SRe*L whose optimization process is patch-agnostic still achieves 1% performance gains. This confirms that diverse augmenta- tion operations universally enhance dataset diversity and boost performance. Consequently, future work should explicitly disclose whether additional data augmentations are employed and include corresponding ablation analyses.
N IMPACT OF DIVERSE SOFT LABEL ENHANCEMENT TECHNIQUES
In the context of decoupled dataset distillation, the of soft labels serves the foundation for applying data augmentation (c.g., CutMix) during the post-evaluation phase. Therefore, our intention highlight that any absolute performance gain claimed by newly proposed methods over existing baselines must be evaluated under identical post-evaluation settings, including the use of soft labels and data augmentation strategies. To investigate the impact of different knowledge distillation tech- niques to the dataset distillation, we have incorporated additional soft label augmentation techniques during the post-evaluation phase. The experimental results on ImageNet-18 under IPC=10 are shown in Table 18 As observed, soft label augmentation can act as general performance booster. However, when applied without proper constraints, it may lead unfair comparisons. This observation further exposes the problem we have identified in the current decoupled dataset distillation literature and underscores the necessity of our proposed work.
Config
S
CDA
G-VBSM
DWA
EDC
Minimax
D*M
RDED
None
40.2
41.2
41.5
42.5
46.9
45.9
45,4
46.3
DKD
Zhao et al.
2022
41.3
42.4
42.2
43.3
47.8
47.1
46.5
47.4
NKD
Yang el al.
2023
41.1
42.7
42.6
43.0
48.5
47.9
46.9
48.0
LSKD
Sun et al. 2024b
41.5
43.0
42.2
43.4
48.7
48.4
46.8
48.4
CRLD
Zhang el al.
2024
41.8
43.2
42.5
43.7
45.1
48.8
46.2
48.2
Table 18: Impact of soft label enhancement techniques. is clear to see that using the more powerful knowledge distillation methods could lead to consistent and significant performance improvement across various distilled datasets.
O QUALITATIVE INTERPRETATION
0.1 TRAINING DYNAMIC ANALYSIS
Under our proposed framework, we observe significant performance differences in generated datasets from various methods during student model training. The training accuracy and test accuracy curves of student models during evaluation are shown in Figure 9 Using randomly sampled images as the baseline, we note that the real training process achieves high generalization due to extensive data augmentation, reflected in the large gap between the two curves.
For optimization-basee methods, since the generated datasets are optimized through cross-entropy loss and global BN statistics, the images tend to be overly simplistic for student models. This results in high training accuracy but low test accuracy. The gap between training and test accuracy gradually narrows method performance improves. The only exception is G-V G-VBSM, whose generated datasets exhibit increased complexity but lack explicit guidance due to the introduction auxiliary models during optimization.
25

---

Published as a conference paper at ICLR 2026
Fra Puttader
(n) Random
(b) SRe*L
(c) CDA
(d) G-VBSM
(c) DWA
Tele
(g) Minimax
(h) D
(i) RDED
Figure 9: Training dynamics comparison between various training datasets on ImageNet under IPC=10. Different methods exhibit significant difference of gap between training and test accuracy.
Generation- bbaed methods demonstrate training dynamics nearly identical to real datasets, confirming that diffusion models can effectively approximate real data distributions while preserving categorical information and generating diverse images. Future work should explore how to identify and produce more beneficial data distributions based on this foundation.
In contrast, selection-based method prioritize images based on classifier accuracy, achieving the high- est training accuracy. Despite this, RDED shows competitive performance under high compression ratios, surpassing all methods except EDC. However, under lower compression settings, RDED's performance declines sharply due to insufficient dataset diversity, highlighting critical direction for future optimization.
O.2 T-SNE VISUAL ANALYSIS
We explore the differences in synthetic datasets produced by various distillation methods from another perspective. By visualizing the feature distributions of generated datasets and the original dataset using t-SNE, as shown in Figure 10 we can intuitively observe variations in data distributions across methods.
For optimization-basee methods, except for EDC, whose data distribution aligns closely with the original dataset, other methods exhibit significant distribution shifts. In fine-grat fine-grained categories, images generated by SRe²L, CDA, and DWA become overly simplistic, preventing teacher models from providing effective guidance. while for G-VBSM, although its generated dataset demonstrates dispersed intra-class distributions, inter-class distances are inappropriately minimized in incorrect directions.
Generation- based approaches (i.c., D and Minimax) show data distributions largely consistent with the original dataset, confirming that diffusion models effectively approximate real data distributions, thereby achieving competitive performance under low compression ratios.
The selection-base method RDED, which selects images based on classifier accuracy, generates data distributions similar to optimization-bast methods. However, since its dataset still comprises original images, it maintains favorable intra-class diversity. Nevertheless, under low compression ratios, RDED also faces challenges of distribution shifts.
26

---

Published as a conference paper at ICLR 2026
(4) Random
(b) SRe*1.
Cd)
G-VRSM
EIX:
(8) Minimax
(h) D'M
B RDED
Figure 10: T-SNE visualizations of top 10 classes in ImageNet- 1K from different synthetic datasets under IPC=100. The dark dots and light dots denote the synthetic datasets and real dataset respectively. For clear comparison, we additionally provide the distribution of only real dataset shown in (a).
27

---

Published as a conference paper at ICLR 2026
P VISUALIZATION
We provide visualizations the images sampled from real dataset and synthetic datasets from different methods, as illustrated Figures I 19
Figure 11: Visualization of top 10 classes in ImageNet-1K IK from real dataset.
28

---

Published as a conference paper at ICLR 2026
Figure 12: Visualization of top 10 classes in ImageNet- IK from SR under IPC=100.
29

---

Published as a conference paper at ICLR 2026
Figure 13: Visualization of top 10 classes in ImageNet- IK from CDA under IPC=100.
30

---

Published as a conference paper at ICLR 2026
Figure 14: Visualization of top 10 classes in ImageNet- 1K from G-VBSM under IPC=100.
31

---

Published as a conference paper at ICLR 2026
Figure 15: Visualization of top 10 classes in ImageNet- IK from DWA under IPC=100.
32

---

Published as a conference paper at ICLR 2026
Figure 16: Visualization of top 10 classes in ImageNet- IK from EDC under IPC=100.
33

---

Published as a conference paper at ICLR 2026
Figure 17: Visualization of top 10 classes in ImageNet- IK from Minimax under IPC=100.
34

---

Published as a conference paper at ICLR 2026
Figure 18: Visualization of top 10 classes in ImageNet- IK from D'M under I0C=100.
35

---

Published as a conference paper at ICLR 2026
Figure 19; Visualization of top 10 classes in ImageNet- IK from RDED under IPC=100.
36


Reviews:
[{"review_id": "3EPFXbc6V5", "reviewer": "Reviewer_7EFG", "summary": "This paper introduces a unified and systematic evaluation framework for dataset distillation to compare decoupled distillation methods.", "questions": "1. Should convergence speed be considered a characteristic of a method? If some methods converge faster than others, using longer training epochs may negatively affect their performance.\n2. The cross-architecture results are interesting. Would replacing the soft label network with a different network change the results?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The evaluation is systematic and practical, providing valuable guidance for empirical method comparison and design.", "weaknesses": "1. Lacks fundamental analysis—most observations are based on empirical results and hypotheses.\n2. Uses a fixed hyper-parameter set for all methods. As an evaluation framework, this approach is unconvincing since different methods may have different optimal hyperparameters.\n3. Some design choices require additional justification (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9yqEAh5Iw8", "reviewer": "Reviewer_McZF", "summary": "The paper argues that evaluation across existing dataset distillation methods are inconsistent. In consequence, it is unclear whether performance gains from new methods is due to improvement from the distillation mechanism or auxiliary improvement in the downstream model training. To overcome this, the paper introduces Rectified Decoupled Dataset Distillation (RD3) to systematically investigate how different evaluation settings affect test accuracy. RD3 find that previous observed performance increase of over 27% reduces to under 7% with the same evaluation setting.", "questions": "1. The paper argues \"to ensure fairness, all methods should adopt a unified training setting, regardless of their specific motivations\" but it is unclear from the paper why this matters or ensures fairness. Why does using the same evaluation procedure enforce fairness? For instance, method A, compared to method B, can benefit more in from training procedure X than Y. Is it fair to evaluate with procedure X or Y? To a partitioner, the will only care about the final performance on the downstream task. \n2. Figure 2 argues that different techniques leads to different performance impact by why is this the case? The accuracy gains from the technique isn't the same but the relatively ranking between the two method is consistent with the augmentations. Hence, the same conclusion can be drawn. \n3. Section 4.2 argues \"in comparison to marginal differences in test accuracy, computational efficiency should be considered a more critical criterion for evaluating different methods\". What is considered marginal? For example, RDED takes 1 hour to train and achieve 61.5% but D4M takes 52 hours to train and achieve 63.2%. To a practitioner, the 1-2% improvement in accuracy maybe well worth the 50x increase in training time. \n4. What is the total storage size (including soft labels) across the different methods? \n5. How does this differ from DD-Ranking?\n\nLi, Zekai, et al. \"Dd-ranking: Rethinking the evaluation of dataset distillation.\" arXiv preprint arXiv:2505.13300 (2025).", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper point out a concerning trend with current dataset distillation evaluation protocol that can misrepresent the capabilities of existing algorithm.\n2. The paper provides a rather thorough analysis towards understanding the capabilities of current dataset distillation methods, revealing the performance gap between the baseline SRe2L and RDED is a lot smaller than expected.\n3. To the best of my knowledge, this is the first work that provide comprehensive training time, generalization, and initialization analysis, which are important considerations for people that use these dataset distillation algorithm in practice.", "weaknesses": "1. The problem of different evaluation procedure exist is many subfields of machine learning and not unique to dataset distillation. More discussion on the unique challenges that exist in dataset distillation would strength the overall motivation of the paper. \n2. The presentation of many arguments in the paper can substantially be improved (see question 1-3).\n3. Minor: Figure 3 can substantially be improved. There are eight points on the graph that differs only in color. This makes it difficult to read and not color-blind friendly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h2h59vLyrA", "reviewer": "Reviewer_94Vn", "summary": "The paper proposes RD3, a standardized evaluation framework for decoupled dataset distillation. It argues and shows empirically that divergences in post-evaluation choices (batch size, LR schedule smoothing, data augmentations, label formulations, training epochs) drive a large share of performance differences across methods, not the synthesis procedure itself. The authors unify settings across optimization-based, selection-based, and generation-based approaches; evaluate on six datasets (CIFAR-10/100, TinyImageNet, ImageNet-1K, ImageNette/Woof); and test cross-architecture generalization (ResNet family, MobileNet-V2, EfficientNet-B0, Swin-T, ViT-B). They further emphasize efficiency (data generation time) and explore general techniques (alternative initialization, hybrid soft labels, MSE-GT loss) that materially shift outcomes under the same data.", "questions": "1.For MSE-GT, you adopt γ from G-VBSM. Could you provide a small γ sweep (e.g., {0, 0.01, 0.025, 0.05}) across two families to verify that conclusions don’t hinge on one borrowed value? \n\n2.In Appendix F, do you have analyses (e.g., attention maps, calibration, failure clusters) to explain the >40% ViT drop at IPC=10 and why it recovers at higher IPC? Any indications that label entropy or image diversity drives the effect?\n\n3.In Table1, you standardize on a single ResNet-18 teacher for soft labels. What happens if the teacher is ViT-B or an ensemble (distinct from hybrid labels used at relabel time)? I am curious about different teacher models' effects. (I checked the result in Appendix F, but it doesn’t test ViT-B or ensembles as the teacher)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Clear empirical finding with real stakes for the field.** Under a consistent setup, the headline gap between methods shrinks dramatically (27.3% → 6.7% on ImageNet-1K @ IPC=10), reframing prior “SOTA” claims as largely protocol effects.\n\n\n2.**Unified, well-documented evaluation protocol .** The paper spells out aligned epochs (400), batch size (BS=50 for R-18), smoothing cosine LR (ζ), and a fixed augmentation recipe, and shows incremental effects (Fig. 2). This is useful to both reproduce and compare. \n\n3.The paper does a comphrhensive analysis and isolates factors many works quietly vary: initialization can help/hurt (e.g., noise harms DWA/EDC; RDED init boosts several methods), and hybrid soft labels help at low IPC regardless of method family.1", "weaknesses": "1.The augmentation suite is taken from RDED and applied universally. That is reasonable for standardization, but it may advantage methods whose images/labels interact best with that augmentation mix. A short sensitivity analysis (swap in SRe2L/CDA augmentations) would address perceived bias. Similarly, fixing BS=50 and ζ (e.g., ζ=1 for ResNets) is pragmatic, yet different families could prefer distinct regimes. The appendix defends ζ choices, but a small grid over {BS, ζ} across families would strengthen neutrality claims.\n\n2.The efficiency plot excludes classifier training time for optimization/selection methods. I’d prefer an end-to-end accounting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0XmI7VLnB", "reviewer": "Reviewer_ihpb", "summary": "This paper revisits the evaluation of *decoupled dataset distillation (DDD)* methods and introduces Rectified Decoupled Dataset Distillation (RD3), a unified benchmark framework for fair and comprehensive comparison.\nThe authors identify that prior DDD papers (e.g., SRe2L, CDA, DWA, EDC, G-VBSM, D4M) have used inconsistent post-evaluation settings—varying batch sizes, loss functions, schedulers, soft label configurations, and augmentations—making their reported performance incomparable.\nRD3 standardizes all these evaluation aspects (batch size, loss, augmentations, optimization schedule, and model architectures), and re-evaluates the main DDD methods under identical conditions.\nThe results show that previously reported gaps of up to 27% shrink to ~6%, indicating that most gains in prior work were due to inconsistent settings rather than true algorithmic improvements.\nBeyond unification, RD3 also provides diagnostic analyses, new metrics (efficiency, cross-architecture generalization), and an open-source benchmark for reproducible comparison.", "questions": "1. Could the authors integrate **trajectory-matching-based distillation methods**, to strengthen coverage and demonstrate RD3’s scalability to newer paradigms?\n2. How feasible is it to extend RD3 to **non-image domains**, particularly graph data distillation (e.g., GCond[1], GEOM[2])? Would the same evaluation protocol and metrics apply?\n3. Are there plans to host RD3 as a public leaderboard or benchmark platform?\n\n[1] Graph condensation for graph neural networks. ICLR 2021.\n\n[2] Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching. ICML 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* **Timely and valuable contribution:** RD3 addresses a real and growing problem in the dataset distillation community: inconsistent evaluation practices that lead to misleading comparisons.\n* **Thorough analysis:** The paper systematically disentangles how batch size, loss function, learning rate scheduler, and soft-label design affect performance, providing empirical evidence that evaluation bias dominates algorithmic differences.\n* **Unified and transparent benchmark:** All post-evaluation protocols are standardized, with clear rationale (e.g., KL loss for soft-label consistency; batch size = 50 as a fair trade-off).\n* **Broader impact on reproducibility:** The open benchmark and diagnostic framework can serve as a long-term reference for fair evaluation of future DDD methods.\n* **Clarity:** The paper is well-structured and transparent about design decisions and their effects.", "weaknesses": "* **Limited coverage of cutting-edge methods:**\n  While RD3 re-evaluates several major DDD baselines, it currently does not include more *high-performing or near-lossless* distillation frameworks such as DATM and related trajectory-based optimization methods.\n  Extending RD3 to evaluate these *lossless distillation* regimes would significantly strengthen its comprehensiveness and demonstrate that the framework remains valid even for next-generation algorithms that achieve nearly full-data performance.\n\n* **Scope restricted to image datasets:**\n  The current benchmark focuses solely on vision data. Enabling RD3 to handle other modalities—such as **graph distillation**  or multimodal datasets—would greatly expand its applicability and impact.\n\n* **No new algorithmic contribution:**\n  The contribution is primarily evaluative rather than methodological, which is appropriate for a benchmark paper but may limit its novelty for readers seeking new distillation algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)