Elucidating the Design Space of Dataset Condensation
Authors: Shitong Shao
Zikai Zhou
Huanran Chen
Zhiqiang Shen
Venue: NeurIPS 2024 poster (2024)
Keywords: dataset condensation
efficient computer vision
design space
Primary area: machine_vision
TL;DR: We propose a comprehensive designing-centric framework that includes specific, effective strategies. These strategies establish a benchmark for both small and large-scale dataset condensation.

Abstract: Dataset condensation, a concept within $\textit{data-centric learning}$, aims to efficiently transfer critical attributes from an original dataset to a synthetic version, meanwhile maintaining both diversity and realism of syntheses. This approach can significantly improve model training efficiency and is also adaptable for multiple application areas. Previous methods in dataset condensation have faced several challenges: some incur high computational costs which limit scalability to larger datasets ($\textit{e.g.,}$ MTT, DREAM, and TESLA), while others are restricted to less optimal design spaces, which could hinder potential improvements, especially in smaller datasets ($\textit{e.g.,}$ SRe$^2$L, G-VBSM, and RDED). To address these limitations, we propose a comprehensive designing-centric framework that includes specific, effective strategies like implementing soft category-aware matching, adjusting the learning rate schedule and applying small batch-size. These strategies are grounded in both empirical evidence and theoretical backing. Our resulting approach, $\textbf{E}$lucidate $\textbf{D}$ataset $\textbf{C}$ondensation ($\textbf{EDC}$), establishes a benchmark for both small and large-scale dataset condensation. In our testing, EDC achieves state-of-the-art accuracy, reaching 48.6% on ImageNet-1k with a ResNet-18 model at an IPC of 10, which corresponds to a compression ratio of 0.78\%. This performance surpasses those of SRe$^2$L, G-VBSM, and RDED by margins of 27.3%, 17.2%, and 6.6%, respectively. Code is available at: https://github.com/shaoshitong/EDC.

Elucidating the Design Space of Dataset Condensation
Shitong Shao Zikai Zhout Huanran Chent!, Zhiqiang Shen
Mohamed bin Zayed University of AI, Tsinghua University
The Hong Kong University of Science and Technology (Guangzhou)
{1090784053ast, O com, huanran, chen@out com zhiqiang shen@mbzuai. ac. *; Corresponding author
Abstract
Dataset condensation, a concept within data-centric learning, aims to efficiently transfer critical attributes from original dataset to synthetic version, meanwhile maintaining both diversity and realism of syntheses. This approach can significantly improve model training efficiency and is also adaptable for multiple application areas. Previous methods in dataset condensation have faced several challenges: some incur high computational costs which limit scalability to larger datasets (e.g-, MTT, DREAM, and TESLA), while others are restricted to less optimal design spaces, which could hinder potential improvements, especially in smaller datasets (e.g., SRe²L, G-VBSM, and RDED). To address these limitations, we propose comprehensive designing-centrie framework that includes specific, effective strategies like implementing soft category-aware matching, adjusting the learning schedule and applying small batch-size. These strategies grounded in both empirical evidence and theoretical backing. Our resulting approach, Elucidate Dataset Condensation (EDC), establishes benchmark for both small and large- scale dataset condensation. In our testing, EDC achieves state-of-the-at accuracy, reaching 48.6% 88.6% on ImageNet- Ik with ResNet- 18 model at an IPC of 10, which corresponds to a compression ratio of 0.78%. This performance surpasses those of SRe²L, G- G-BM,, and RDED by margins of 27. 27.3%. 17.2%, and 6.6%, respectively.
Introduction
Dataset condensation, also known as dataset distillation, has emerged in response to the ever- increasing training demands of advanced deep learning models (He et al., 2016a,b; Brown et al., 2020). This task addresses the challenge of requiring massive amount of data to train high-precision models while also being bounded by resource constraints (Dosovitskiy al., 2020; Shao el., 2024). In the conventional setup of this problem, the original dataset acts as a "teacher", distilling and preserving essential information into a smaller, surrogate "student" dataset. The ultimate goal of this technique is to achieve comparable performance of models trained on the original and condensed datasets from scratch. This task has become popular in various downstream applications, including continual learning (Masarczyk and Tautkute, 2020; Sangermano et al., 2022; Zhao and Bilen, 2021), neural architecture search (Such et al., 2020; Zhao and Bilen, 2023; Zhao al., 2021), and training- free network slimming (Liu et al., 2017).
However, the common solution in traditional dataset distillation methods of bi-level optimiza- tion requires prohibitively expensive computation, which limits the practical usage, as prior works (Cazenavette al., 2022; Sajedi et al., 2023; Liu al., 2023a). This has become particularly when being applied to large-scale datasets like ImageNet-Ik (Russakovsky et al., 2015). In response, the uni-level optimization paradigm has gained significant attention an alternative solution, with recent contributions from the research community (Yin et al., 2023; Yin and Shen, 2024; Shao al., 2023) highlighting its applicability. These methods primarily leverage the rich
severe
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and extensive information from static, pre- ure-trained observer models, to facilitate more streamlined optimization process for synthesizing a condensed dataset without the need to adjust other parameters (e.g., those within the observer models). While uni-level optimization has demonstrated remarkable performance on large datasets, has yet to achieve the competitive accuracy levels seen with classical methods on small-scale datasets like CIFAR- 10/100 (Krizhevsky et al., 2009). Moreover, the recently proposed training-free method RDED (Sun et al., 2024) outperforms training- methods in efficiency and maintains effectiveness, yet it overlooks the potential information incompleteness due to the lack of optimization on syntheses. Also, some simple but promising skills (e.g,, smoothing learning rate schedule) that could enhance performance have not been well-explored in the exist- ing literature. We observe that a performance improvement of 16.2% in RDED comes from these techniques in this paper rather than the proposed data synthesis approach.
These drawbacks show the constraints of previous methods in several respects, highlighting the need for thorough investigation and assessment of potential limitations in prior frameworks. In contrast to earlier strategies that targeted one or a few specific improvements, our approach systematically examines all possible facets and integrates them into our comprehensive framework. To establish a strong framework, we carefully analyze all potential deficiencies in different stages of the data synthesis, soft label generation, and post-evaluation stages during dataset condensation, resulting in an extensive exploration of the design space on both large-scale and small-scale datasets. As a result, we introduce Elucidate Dataset Condensation (EDC), which includes range of concrete and effective enhancement skills for dataset condensation (refer to Fig. 1). For instance, soft category-awars matching ensures consistent category representation between the original and condensed data batches for more precise matching. Overall, EDC not only achieves state-of-the-art performance on CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet- 10, and ImageNet- Ik, using only half of the computational cost compared to the baseline G-VBSM, but it also provides in-depth both empirical and theoretical insights and explanations that affirm the soundness of our design decisions. Our code is available at: https: /tt//ggttttt /gttttttthb.tt com/ shaoshitong/EDDC
2 Dataset Condensation
Preliminary. Dataset condensation involves generating a synthetic dataset DS := fx y; " consisting of images X and labels y's designed to be as informative as the original dataset DT := {x7 . I which includes images XT and labels y", The synthetic dataset DS is substantially smaller in size than DT (DD1 < (DTD. The goal of this process to maintain the critical attributes of DJ to ensure robust or comparable performance during evaluations on real test protocol PD.
arg min E( (x. y, d²)], where 6* = arg min, - [(()(())) ). y.y?))
(I)
Here, ( x²)) represents the evaluation loss function, such as cross-entropy loss, which is parameterized by the neural network o² that has been optimized from the distilled dataset DS. The data synthesis process primarily determines the quality of the distilled datasets, which transfers desirable knowledge from DT to D' through various matching mechanisms, such as trajectory match- ing (Cazenavette et al., 2022), gradient matching (Zhao et al., 2021), distribution matching (Zhao and Bilen, 2023) and generalized matching (Shao et al., 2023).
Traditional dataset condensation algorithms, as referenced in studies such as (Wang et al., 2018; Cazenavette et al., 2022; Cui et al., 2023; Wang et al., 2022; Nguyen al., 2020), encounter computational challenges and are generally confined to small-scale datasets like CIFAR-10/100 (Krizhevsky et al., 2009), or larger datasets with limited class diversity, such as ImageNette (Cazenavette al., 2022) and ImageNet- 10 (Kim et al., 2022). The primary inefficiency of these methods stems from their reliance on a bi-level optimization framework, which involves alternating updates between the synthetic dataset and the observer model utilized for distillation. This approach not only heavily depends on the model's intrinsic ability but also limits the versatility of the distilled datasets in generalizing across different architectures. In contrast, the uni-level optimization strategy, noted for its efficiency and enhanced performance on the regular 224 224 scale of ImageNet- Ik in recent research (Yin et al., 2023; Shao et al., 2023; Yin and Shen, 2024), shows reduced effectiveness in smaller-scale datasets due to the massive optimization- iterations required the data synthesis process without a direct connection to actual data. Recent new methods in tranningffree distillation paradigms, such as in (Sun et al., 2024;
Small-scale vs. Large-scale Dataset
diaddensation/Distllltttin
2

---

Data Synthesis
Soft Label Generation
Post Evaluation
CONFIG
G-VBSM (heselinc)
CONPIG B
54.4
4.59
Icanca
Flarness Regularization
Small Batch Sale
Small Batch Size
CONFIG C
28.7
Radiile
Soft Category-
CONFIG D
59.5
Aware Matching
Smoothing LR Schedule
CONFIG
Real Image Initialization
CONEIG
Weak Angmentation
E.M.- MMA-based
Rackhoone Choice
Evaluation
CONPIG
1
5.11
Tine (baung
Aec.
Figure 1: Illustration of Elucidating Dataset Condensation (EDC). Left: The overall of our better design
choices in dataset condensation on ImageNet-Ik. Right: The evaluation performance and data synthesis required time different configurations on ResNet-18 with IPC 10. Our integral EDC refers CONFIG G.
Zhou al., 2023), offer advancements in efficiency. However, these methods compromise data privacy by sharing original data and do not leverage statistical information from observer models to enhance the capability of synthetic data, thereby restraining their potential in real environment.
Generalized Data Synthesis Paradigm. We consistently describe algorithms (Yin et al., 2023; Yin and Shen, 2024; Shao et al., 2023; Sun et al., 2024) that efficiently conduct data synthesis on ImageNet-Ik as "generalized data synthesis" as these methods are applicable for both small and large-scale datasets. This direction usually avoids the inefficient bi-level optimization and includes both image and label synthesis phases. Note that several recent works (Zhang et al., 2024a,b; Deng al., 2024), particularly DANCE (Zhang al., 2024a), can also effectively be applied to ImageNet- 1k, but these methods lack enhancements in soft label generation and post-evaluation. Specifically, generalized data synthesis involves first generating highly condensed images followed by acquiring soft labels through predictions from pre-trained model. The evaluation process resembles know ledge distillation (Hinton al., 2015), aiming transfer knowledge from a teacher a student model (Gou et al., 2021; Hinton et al., 2015). The primary distinction between the maining-depender (Yin et al., 2023; Yin and Shen, 2024; Shao et al., 2023) and training-free paradigm (Sun et al., 2024) centers on their approach to data synthesis. In detail, the training-dependen dependent paradigm employs Statistical Matching (SM) to extract pertinent information from the entire dataset.
Leya ||p(ulx*) p(u|x" lll + IIx(o²)x*) plo"1x" (xxx))aa Il,, s.t. Las Smatchs
S² arg min Egaya Sma ( xx)))
X88
where Smatch represents the extensive collection of statistical matching operators, which operate across variety of network architectures and layers described by (Shao al., 2023). Here, and o² are defined as the mean and variance, respectively. For more detailed theoretical insights, please refer to Definition 3.1. The training-fir ffree approach, discussed in (Sun et al., 2024; Zhou et al., 2023), employs a direct reconstruction method for the original dataset, aiming to generate simplified representations of images.
x
U S Xi
[x} concat(($)) C X H
where C denotes the number of classes, concat(-) represents the concatenation operator, X signifies the set of condensed images belonging to the i-th class, and X7T corresponds to the set of original images of the i-th class. is important to note that the default settings for N are and 4, as specified in the works (Zhou et al., 2023) and (Sun et al., 2024), respectively. Using one or more observer we then derive the soft labels ys from the condensed image set S
models, denoted as {Ø, )
(x
(4)
x
This plug-and-play component, as outlined in SR (Yin et al., 2023) and IDC (Kim et al., 2022), plays crucial role for enhancing the generalization ability of the distilled dataset DS.
3

---

Loss Laniscage (Inkk)
Placinss of Dadicis Stmnew
midddingg
desc
0.30
0.00
dess staare
nan
Les Landscape (est)
0.25
clao
ond
dast aan--rrae
diviil
0.5.
0.30
Closs
Barch fox
0.00
05
(b)
(c)
Figure 2: (a): Illustration of soft category- aware matching using Gaussian distribution R2 (b): The effect of employing smoothing LR schedule (@) on loss landscape sharpness reduction. (c) top: The role of flatness regularization
in reducing the Frobenius norm of the Hessian matrix driven by data synthesis
iteration. (c) bottom: Cosine similarity comparison between local gradients (obtained from original and distilled datasets via random batch selection) and the global gradient (obtained from gradient accumulation).
3 Improved Design Choices
Design choices in data synthesis, soft label generation, and post-evaluation significantly influence the generalization capabilities of condensed datasets. Effective strategies for small-scale datasets are well-explored, yer these approaches the limitations of existing algorithms' design choices on ImageNet-Ik, We then propose solutions, providing experimental results as shown Fig. 1. For most design choices, we offer both theoretical analysis and empirical insights to facilitate a thorough understanding, as detailed in Sec. 3.2.
less examined for large-scale datasets. We first delineate
3.1 Limitations of Prior Methods
Lacking Realism (solved by ticularly those employed for large-scale datasets, typically initiate the optimization process using Gaussian noise inputs (Yin al., 2023; Yin and Shen, 2024; Shao al., 2023). This initial choice complicates the optimization process and often results in the generation of synthetic images that do exhibit high levels of realism. The limitations in visualization associated with previous approaches are detailed in Appendix F.
condensation algorithms for datasets, par-
Coarse-grained Matching Mechanism (solved by 2 The Statistical Matching -SMedd pipeline (Yin et al., 2023; Yin and Shen, 2024; Shao et al., 2023) computes the global mean and vari- ance by aggregating samples across all categories and uses these statistical parameters for matching purposes. However, this strategy exhibits two critical drawbacks: does not account for the domain discrepancies among different categories, and it fails preserve the integrity of information across the original and condensed samples within each batch. These limitations result in gaasee matching approach that diminishes the accuracy of the matching process.
Overly Sharp Loss Landscape (sobved by and 2 ). The optimization objective C(0) can be expanded through second-order Taylor expansion as C(0)) (0 (+)"Vv (0- 0*), with an upper bound of L()*) | 1=10 1 upon model convergence (Chen al., 2024). However, earlier
condensation algorithms neglect to minimize the Frobenius norm of the Hessian matrix H to obtain flat loss landscape for enhancing its generalization capability through sharpness aware minimization theory (Foret et al., 2020; Chen et al., 2022). Please see Appendix C for more formal information.
Irrational Hyperparameter Settings (solved by @
and @ RDED (Sun et al., 2024) adopts smoothing LR schedule and (Liu et al., 2023b; Yin and Shen, 2024; Sun et al., 2024)

---

use a reduced batch size (O for post- cost-evaluation on the full 224x224 ImageNet- Ik. These changes, although critical, lack detailed explanations and impact assessments in the existing literature. Our empirical analysis highlights a remarkable impact on performance: absent these modifications, RDED achieves only 25. 25.8% accuracy on ResNet18 with IPC 10. With these modifications, however, accuracy jumps to 42.0%. In contrast, SRe²L. and G-BMMM do not incorporate such strategies in their experimental frameworks. This work aims to fill the gap by providing the first comprehensive empirical analysis and ablation study the effects of these and similar improvements in the field.
3.2 Our Solutions
To address these limitations described above, we explore the design space and elaborately present a range of optimal solutions at both empirical and theoretical levels, illustrated in Fig. 1.
Real Image Initialization
Intuitively, using real im-
Inkiotication
ages instead of Gaussian noise for data initialization during the data synthesis phase is a practical and effective strat- egy. As shown in Fig. 3, this method significantly improves the realism of the condensed dataset and simplifies the op- timization process, thus enhancing the synthesized dataset's s ability to generalize in post- post-evaluation tests. Additionally, incorporate considerations of information density and efficiency by employing a training- free condensed dataset (e.g., RDED) for initialization the start of the synthesis process. According to Theorem 3.1, based on optimal trans- port theory, the cost of transporting from a Gaussian distribution the original data distribution is higher than using the training-free condensed distribution as the initial reference. This advantage also allows us to reduce the number of iterations needed to achieve results to half of those required by our baseline G- G-VM model, significantly boosting synthesis efficiency.
Imauge
Gausiaan Initalization
Iter 1
Iter 20
Iter 1000
Figure 3: Comparison between real image initialization and random initialization.
Theorem 3.1. (proof in Appendix B. Considering samples X X.S. and XSS. Tandom from the original data, training-free condensed (e.g., RDED), and Gaussian distributions, respectively, let assume cost function defined in optimal transport theory that satisfies E[c(a- - b)] X (L[(La((a), Under this assumption, it follows that Elc(X xfree)) ) I Elc(X X )..
Soft Category-Awan Matching (θ). Previous dataset condensation methods (Yin et al., 2023; Yin and Shen, 2024; Shao al., 2023) based on the Statistical Matching (SM) framework have shown satisfactory results predominantly when the data follows a unimodal distribution (e.g., single Gaussian). This limitation is illustrated with a simple example in Fig. 2 (a). Typically, datasets consist of multiple classes with significant variations among their class distributions. Traditional SM-based methods compress data by collectively processing all samples, thus neglecting the differences between classes. As shown in the top part of Fig. 2 (a), this method enhances information density but also creates a big mismatch between the condensed source distribution X and the target distribution X². To tackle this problem, we propose the use of Gaussian Mixture Model (GMM) to effectively approximate any complex distribution. This solution is theoretically justifiable by the Tauberian Theorem under certain conditions (detailed proof is provided in Appendix B.2). In light of this, we define two specific approaches to Statistical Matching:
Sketch Definition 3.1. (formal definition in Appendix B.2) Given N random samples {a; LN fi=1 with an unknown distribution pmms( we define two forms to statistical matching. Form (1): involves synthesizing M distilled samples {3:) )M { where M & N, ensuring that the variances and means of both [ and 333 1. are consistent. Form (2): treats Pmix(z) as a GMM with C components. For random samples {xx M (E,,, N) within each component Cj, we synthesize M2 (E,M, M) M; M, Je-1. where Mj < Ng. to maintain the consistency of variances and means
distilled samples {9/
N,, between 1 and 1
In general, LR CDA, and G-VBSM are all categorized under Form (1), as shown in Fig. 2 (a) at the top, which leads to coarse-grained matching. According to Fig. 2 (a) at the bottom, transitioning to Form (2) is identified as practical and appropriate alternative. However, our empirical result indicates that exclusive reliance on Form (2) yields a synthesized dataset that lacks sufficient information density. Consequently, we propose a hybrid method that effectively integrates
5

---

Form (1) and Form (2) using weighted average, which we term soft category aware matching.
Syn = allp(u x=) - -ppppl ll2 + (1x()")x=)) - p(o?)x lll
#Form (1)
(5)
1 (1 a) pla) [1pta|x* .4) plula" aalla I 11x(0?) (a) p(o
allls]. ((o9)xT,c))ll]) #Form (2)
where C represents the total number of components, c, indicates the i-th component within a GMM, and a is coefficient for adjusting the balance. The modified loss function Lnyn is designed to effectively regulate the information density of X and to align the distribution of X with that of XT. Operationally, each category in the original dataset is mapped to distinct component in the
GMM framework. Particularly, when a 1, the sophisticated category-awarg matching described by f'yn in Eq. 5 simplifies to the basic statistical matching defined by Luya in Eq. 2.
Theorem 3.2. (proofs in Theorems B.5, B.7. B.8 and Corollary B.6) Given the original data distribution Pmix(z), and define condensed samples z and y in Form (1) and Form (2) with
their distributions characterized by P and Q. Subsequently, it follows that (i) E[xx] = Elyy (ii)
D[a] = D(y], (iii) H(P) - & [log(E[D(y']] + D(E - E[log(D(y'))
S H(Q) 1 H(P) +
: -Bly" (DD'" (Diy" (DD)""+"o"") and (iv) DxL |P| I Ei-ae BB-U( (i......... Div! and
Bly'|
[[v']
0.
We further analyze the properties of distributions P and Q as in Form (1) and Form (2). According to parts (i) and (ii) of Theorem 3.2, Q retains the same variance and mean as P. Regarding diversity, part (iii) of Theorem 3.2 states that the entropy H.)) of P and Q is equivalent, H(P) H(Q), provided the mean and variance of all components the GMM are uniform, suggesting a single Gaussian profile. Absent this condition, there is no guarantee that H(P) and H(Q) will consistently increase or decrease. These findings underscore the advantages of using GMM, especially when the initial data conforms an unimodal distribution, thus aligning the mean, variance, and entropy of distributions P and Q in the reduced dataset. Moreover, even in diverse scenarios, the mean, variance, and entropy of Q tend remain stable. Furthermore, when the original dataset exhibits a more complex bimodal distribution and the parameters of the Gaussian components are precisely estimated, utilizing GMM can effectively reduce the Kullback-Leibler divergence between the mixed original distribution pmix and Q to near zero. In contrast, the divergence DRi. [Pmix|| [P] always maintains a non-zero upper bound, noted in part (iv) of Theorem 3.2. Therefore, by modulating the weight a in Eq. 5, we can derive an optimally balanced solution that minimizes loss in data characteristics while maximizing fidelity between the synthesized and original distributions.
and EMA-based Evaluation ( e Choices and S are utilized to ensure flat loss landscapes during the stages of data synthesis and post-evaluation, respectively.
Flatness Regularization
During the data synthesis phase, the use of sharpness- aware minimization (SAM) algorithms is beneficial for reducing the sharpness of the loss landscape, as presented in prior research (Foret et al., 2020; Du et al., 2022: Bahri et al., 2021). Nonetheless, traditional SAM approaches, as detailed in Eq. 29 in the Appendix, generally double the computational load due to their two-stage parameter update process. This increase in computational demand is often impractical during data synthesis. Inspired by MESA (Du et al., 2022), which achieves sharpness-awar training without additional computational overhead through self-distillation, we introduce a lightweight flatness regularization approach for implementing SAM during data synthesis. This method utilizes teacher dataset, X maintained via exponential moving average (EMA). The newly formulated optimization goal aims to obtain a flat loss landscape in the following manner:
LER Eanns (X' ( AiSAA BA + - -
(6)
where 3 is the weighting coefficient, which is empirically to 0.99 in our experiments. The detailed derivation of Hq. 7 is in Appendix F., And the critical theoretical result is articulated as follows:
Theorem 3.3. (proof in Appendix E) The optimization objective LFR can ensure sharpness-aware minimization within p-ball for each point along straight path beiween X and 'EMA
This indicates that the primary optimization goal of LFR deviates somewhat from that of traditional SAM- algorithms, which are designed to achieve a flat loss landscape around S The constraint on flatness needs to ensure that the first-order term of the Taylor expansion equals zero, indicating normal model convergence. However, our exploratory experiments found that despite the good

---

ResNet-18
ResNoi-S0
ResNer-101
MohileNer-V2 V2
SRel1.
G-VBSM
RDED
EDC((Our)
G-VRSM
EDX (Ours)
RDED
(DCr(Our))
EDC (Ours)
229-00 04
32.6 0.1
80.6 0.4
26.1
20.2 + D.4
CIFAR-10
10
21.2
53.5 0.6
37.1
=.3 0.3
79.1 +
76.0 + 0.3
67.1 = 0.5
42.0 + 0.4
4.
44
62.1
87.0 0.1
0.0
85.8
70.8
20
05
3. 0.3
36.1::05
0.3
CIFAR-100
0.
59.5+04 04
426-02 022
63.7
62.1
61.7-011
44.3
49.5 0.3
65.0000 0.5
62.6-0.1
68.6 0.2
69.4 0.3
68.5 0.1
59.5 + D.1
9.7 - 014
9022+004
+5.9++022
38 0.1
4006 0.5
18.8 + D.1
Tiny-ImageNer
10
0.2
50.2 0.:.-.33 0.3
22.9 3.3
51.6=02
40.6 0.6
0.4
0:1
57.2
48.7
0.4
58.6
50.7
24.9 0.5
4:22::0::2
38.2 0.1
21.7 = 13
36.4 0.1
36,4 4: 0.3
ImageNet-10 10
10
53.3 0.1
63.4 1.2
6.4110.1
45.5 - 1.7
59.8 0.1
54.2 1 0.1 80.2 1 0.2
75.5 0.5
82.2 1 0.1
80.81002
71.4
022
80.8000 0.0
6.6
12.8
13.3
5.9 04
122=002
8.4 0.3
ImageNet-Ik
31.4 0.
4..0=0.1
35.4
48.3
51.7
46.8 1 0.2
51.8 4 0.4
56.5
$8.0 0.2
58.7
1 0.3
64.3
61.2 = 0.4
61.9 0.2
57.8 4 D.1
Table 1: Comparison with the SOTA baseline dataset condensation methods. SR and RDED utilize
ResNet-18 18 for data synthesis, whereas G-VBSM and EDC leverage various backbones for this purpose.
IPC
Method
R8sNet-18
ResNet-50
ResNet101
MobileNee-2
ElficiesNee-OO
DeiT-Tiny
Swin-Tiny
ComoNeu- Tiny
ShuiieNa-V2
RDEE
42.0
1944
EDC (Ouns)
48.6
54.1
$1.7
45.0
18.4
54.4
29.5
+2
6.6
8.1
344
10.6
8,3
4.4
9.1
6.1
10:4
RDED
45.6
57.5
58.0
41.3
48.]
22.1
41.6
54.0
20.7
20
EDC (Curs)
$2.0
58.2
60.0
48.6
55.6
26,0
49.6
614
33.0
+3
6,4
7,3
199
7,4
12.3
RDED
403
50.4
30.
253
EDC (Churs)
55.0
61.5
46.5
639
41.1
+&
5.1
2.1
2.2
89
160
11/4
18
17.6
RDED
$1.9
61.8
60.1
M0.3
56.
45.7
55.1
617
27.7
40
EDC (Ours)
56.4
62.2
62.3
$4.7
59.7
51,9
61.1
65.2
44.7
12
2,5
0.4
2.2
4.4
3.4
82
3.0
1.3
17.0
RDED
44.5
03.4
309
EDC (Ours)
58.0
55.0
66.6
1.5
10.5
1.2
14.8
Table Cross-acchitecture generalization comparison with different IPCs on ImageNet-1k. RDED refers to the latest SOTA method on ImageNet-1k 1k and + A stands for the improvement for each architecture.
performance EDC, the loss of statistical matching at the end of data synthesis still fluctuated significantly and did not reach zero. As result, we choose to apply flatness regularization exclusively to the logits of the observer model, since the cross-entropy loss for these can more straightforwardly reach zero
Cip DAL (ss(omttaax(((())))))))))))))))) | (/T))
EEA BMA (1-3)xx5
(7)
where softmax(-) and Φ represent the softmax operator, the temperature coefficient and the pre- trained observer model, respectively. As illustrated in Fig. 2 (c) top, it evident that L/R significantly lowers the Frobenius norm of the Hessian matrix relative to standard training, thus confirming its efficacy pushing flatter loss landscape.
In post-evaluation, we observe that method analogous to CER employing SAM does not lead to appreciable performance improvements. This result likely due to the limited sample size of the condensed dataset, which hinders the model's ability to fully converge post-training, thereby undermining the advantages of flatness regularization. Conversely, the integration of an EMA-updated model as the validated model noticeably stabilizes performance variations during evaluations. We term this strategy EMA-based evaluation and apply across all benchmark experiments.
Smoothing Learning Rate (LR) Schedule and Smaller Batch Size ① ). Here, we introduce two effective strategies for post-evaluation training. Firstly, it is crucial to clarify and distinguish between standard or conventional deep model training and post-evaluation in the context of dataset condensation. Specifically. (I) in dataset condensation, the limited number of samples in X results in fewer training iterations per epoch, typically leading to underfitting; and (2) the gradient of random batch from X aligns more closely with the global gradient than that from a random batch in TT. To support the latter observation, we utilize ResNet- 18 model with randomly initialized parameters to calculate the gradient of random batch and assess the cosine similarity with the global gradient of X²T After conducting over 100 iterations of this procedure, the average cosine similarity is consistently higher between S and the global gradient than with X7, indicating a greater similarity and reduced sensitivity to batch size fluctuations. Our findings further illustrate that the gradient from a random batch in S effectively approximates the global gradient, as shown in Fig. 2 (c) bottom. Given this, the inaccurate gradient direction problem introduced by the small batch
7

---

Design Choices
<
ResNet-18
ResNet-50 50
ResNet-101
Design Chuices
ResNet- 18
ResNet-50
RexNet- 101
CONFIC
1.0
42.0
RDED
25.8
CONFIG C
1.5
38.7
42.0
46.3
RDED+( HD
42.3
48.4
47.0
CONFIC C 2.0
38.8
47.9
G-VBSM+
34.4
CONFIG C 2.5
39.0
44.6
46.0
G-VBSM+
38.8
45.8
47.9
CONFIC C 3.0
38.8
46.2
G-VBSM
45.0
48.
Table 3: Ablation studies on ImageNet- 1k with IPC 10. Left: Explore the influence of the slowdown
coefficient with CONFIG C. Right: Evaluate the effectiveness of real image initialization smoothing
LR schedule and smaller batch size with Ç
Design Choices
Loss Type
Loss Weight
ResNet-18
ResNet-50
DenseNet-12 121
CONFIG C
1.5
38.7
42.0
40.6
CONFIG
LeR CIR
0.025
1.5
0.999
38.8
43.2
40.3
CONFIG D
0.25
1.5
0.999
37.9
43.5
40.3
CONFIG D
CAR LyR
2.5
1.5
0.999
31.7
37.0
32.9
CONFIG
0.25
1.5
0.99
39.0
43.3
40.2
CONFIG
0.25
1.5
0.99
39.5
44.1
41.9
CONFIG D
Cig
0.25
1.5
0.99
38.9
43.5
40.7
CONFIG D
vanilla SAM
0.25
1.5
38.8
44.0
41.2
Table 4: Ablation studies ImageNet-lk with IPC 10. Investigate the potential effects of several factors, including loss type, loss weight, and т, amid flatness regularization
size becomes less problematic. Instead, using small batch size effectively increases the number of iterations, thereby helping prevent model ceder-convergence
To optimize the training with condensed samples, we implement a smoothed LR schedule that moderates the learning rate reduction throughout the training duration. This approach helps avoid early convergence to suboptimal minima, thereby enhancing the model's generalization capabilities.
The mathematical formulation of this schedule is given by u(i) [too<(ir/ where i represents the current epoch, N the total number of epochs, p(i) is the learning rate for the i-th epoch, and S is the deceleration factor. Notably, a value of 1 corresponds to typical cosine learning rate schedule, whereas setting to 2 improves performance metrics from 34.4% to 38.7% and effectively moderates loss landscape sharpness during post-evaluation.
The principal role of these two design decisions is to address the flawed settings in the baseline G-VBSM. The key finding reveals that the minimum area threshold for cropping during data synthesis was overly restrictive, thereby diminishing the quality of the condensed dataset. To rectify this, we implement mild augmentations to increase this minimum cropping threshold, thereby improving the dataset condensation's 's ability to generalize. Additionally, streamlined AlexNet for generating soft labels on ImageNet- 1k, a change we refer to as an improved backbone selection. This modification maintains the performance without degradation. More details on the ablation studies for mild augmentation and improved backbone selection are in Appendix G.
Weak Augmentation
and Better Backbone Choice
substitute the computationally demanding EfficientNet-BOO with more
4 Experiments
To validate the effectiveness of our proposed EDC, conduct comparative experiments across various datasets, including ImageNet- (Russakovsky et al., 2015), ImageNet-10 (Kim et al., 2022), -mag-Im (Tavanaei, 2020), CIFAR-100 (Krizhevsky al., 2009), and CIFAR-10 (Krizhevsky et al., 2009). Additionally, we explore cross-architecture generalization and ablation studies on ImageNet-Ik: 1k. All experiments are conducted using 4x RTX 4090 GPUs. Due to space constraints, detailed descriptions of the hyperparameter settings, additional ablation studies, and visualizations of synthesized images are provided in the Appendix A. G, and H. respectively.
Network Architectures. Following prior dataset condensation work (Yin et al., 2023: Yin and Shen, 2024; Shao et al., 2023; Sun al., 2024), our comparison uses ResNet-(155 18, 50, 101) (He et al., 2016a) as our verified models. We also extend our evaluation to include MobileNet- V2 (Sandler et al., 2018) in Table and explore cross- aoos-architecture generalization further with recently advanced backbones such as DeiT-Tiny (Touvron et al., 2021) and Swin-Tiny (Liu et al., 2021) (detailed in Table 2).
Baselines. We compare our work with several recent state-of-the-arr methods, including SReP1. (Yin et al., 2023), G- G-VBSM (Shao et al., 2023), and RDED (Sun al., 2024) to assess broader practical

---

Weak Augmentation
EMA based Evaluation
Design Choices
(ca.c=(0.5.100
EMA Rate=0.99
ResNet-11
ResNett-0
ResNet-101
CONFIG F
0.00
2.0
46.2
53.2
49.5
CONELG
2.0
46.7
$3.7
49.4
CON
0.00
2.0
46.9
53.8
48.5
CONFEG F
0.25
2.0
X
X
46.7
53.4
50.6
CON F
0.25
20
46.8
53.6
30.8
CONFEG
2,0
47.1
53.7
48.2
CONFEE F
0.50
2.0
48.
$3.9
50.4
COMPIG F
0.50
2.0
48.4
53.9
52.7
CONFI
$1.7
CONEEG 0.75
2,0
46.1
52.7
51.0
CONAG F
0.75
2.0
46.9
52.8
51.6
CONFIC 0.75
2.0
47.0
53.2
49.3
Table 5: Ablation studies ImageNet-1k with IPC 10. Evaluate the effectiveness several design choices,
including soft category- matching weak augmentation (@) and EMA-base evaluation
that have omitted several traditional methods (Cazenavette et al., 2022; Liu et al., 2023a; Cui et al., 2023) from our analysis. This exclusion is due to their inadequate performance on the large-scale ImageNet-Ik and their lesser effectiveness when applied practical networks such as ResNet, MobileNet-V2, and Swin-Ting Tiny (Liu et al., 2021). For instance, the MTT method (Cazenavette al., 2022) encounters out-of-memory issue on ImageNet- Ik, and ResNet- 18 achieves only 46.4% accuracy on CIFAR-10 with IPC 10, which is significantly lower than the 79. % accuracy reported for our EDC in Table
impacts. It is important
4.1 Main Results
Experimental Comparison. Our integral EDC, represented CONFIG G in Fig. 1, provides versatile solution that outperforms other approaches across various dataset sizes. The results in Table affirm its ability to consistently deliver substantial performance gains and model architectures. Particularly notable is the performance leap in the highly compressed IPC 1 scenario using ResNet-18, where EDC markedly outperforms the latest state-of-the method, RDED. Performance rises from 22.9%. 1 11.0% 7.0%. 24.9%. and 6.6% to 32.6%, 39.7%, 39.2%, 45.2%, and 12.8% for CIFAR-10, CIFAR- 100, Tiny-ImageNet, ImageNet-10. and ImageNet- Ik, respectively. These improvements clearly highlight EDC's ss superior information encapsulation and enhanced generalization capability, attributed to the efficiently synthesized condensed dataset.
different IPCs, datasets,
Cross-Architee Arssttrchie Generalization. To verify the generalization ability of our condensed datasets, essential to assess their performance across various architectures such ResNet- 50, 101) (He et al., 201 MobileNet-V2 (Sandler et al., 2018), EfficientNer-BOO (Tan and Le, 2019), DeiT- Tiny (Touvron et al., 2021), Swin- Meiy-Tiny&& (Liu al., 2021), ConvNext- -inny (Liu al., 2022) and ShuffleNet-V2 (Zhang et al., 2018). The results of these evaluations are presented in Table 2. During cross-validation that includes all IPCs and the mentioned architectures, our EDC consistently achieves higher accuracy than RDED, demonstrating its strong generalization capabilities. Specifically, EDC surpasses RDED by significant margins of 8.2% and 14.42% on DeiT-Tiny and ShuffleNet-V V2, respectively.
Application. Our condensed dataset only serves as a versatile training resource but also enhances the adaptability of mod- els across various downstream tasks. We demonstrate its effectiveness by employing it in scenarios such as data-free network slimming (Liu et al., 2017) (w.r.t., param- eter pruning (Srinivas and Babu, 2015)) and glass-incremental continual learning (Prabhu et al., 2020) outlined in DM (Zhao and Bilen, 2023). Fig. 4 shows the wide applicability of our condensed dataset in both data-free
Dax Free Sitmmit
Contined Leaming
SR
30
Doo 2 Soneee (X)
EDC
1ny- Acuraay (2)
SiterL
G-VBSM
10
IFL 10
UPC 210
420
003
500
1000
Figure Application on ImageNet-Ik. effectiveness of data-free network slimming and continual learning using VGGI1-BN and ResNet- 18, respectively. network slimming and slass-incremental continual learning. It substantially outperforms SR and
evaluate
G-V G-VBSM, achieving significantly better results.

---

4.2
Ablation Studies
Smoothing LR Schedule @0) and Smaller Batch Size As shown in Table (left), these design choices, with zero additional computational cost, sufficiently enhance the performance of both G- VBSM and RDED. Furthermore, we investigate the influence of < within smoothing LR schedule in Table 3 (right), concluding that a smoothing learning rate decay is worthwhile for the condensed dataset's generalization ability and the optimal Ç is model-dependent.
Real Image Initialization
The results in Table demonstrate the effectiveness of flainess regular- ization, while requiring a well-designed setup. Specifically, attempting to minimize sharpness across all statistics (i.e., LFR) proves ineffective, instead, it is more effective to apply this regularization exclusively to the logit (i.e. F(R). Setting the loss weights 3 and T at 0.25, 0.99, and 4, respectively, yields the best accuracy of 39.5%, 44.1% %, and 45.9% for ResNet-18, ResNet-550, and DenseNet- 121. Moreover, our design of FR surpasses the performance of the vanilla SAM, while requiring only half the computational resources.
Flatness Regularization
Soft Category-Aware Matching Weak Augmentation
and EMA-based Evaluation
). Table 5 illustrates the effectiveness of weak augmentation and EMA-based evaluation, with EMA evaluation also playing crucial role in minimizing performance fluctuations during assessment. The evaluation soft eategory-awar matching primarily involves exploring the effect of parameter across the range [0, 1]. The results in Table 5 suggest that setting a to 0.5 yields the best results based on our empirical analysis. This finding not only confirms the utility of soft category- matching but also emphasizes the importance of ensuring that the condensed dataset maintains high level of information density and bears a distributional resemblance to the original dataset.
Conclusion
In this paper, we have conducted an extensive exploration and analysis of the design possibilities for scalable dataset condensation techniques. This comprehensive investigation helped us pinpoint a variety of effective and flexible design options, ultimately leading to the construction of novel framework, which we call EDC. We have extensively examined EDC across five different datasets, which vary in size and number of classes, effectively proving EDC's robustness and scalability. Our results suggest that previous dataset distillation methods have not yet reached their full potential, largely due to suboptimal design decisions. We aim for our findings to motivate further research into developing algorithms capable of efficiently managing datasets of diverse sizes, thus advancing the field of dataset condensation task.
References
K. He, X. Zhang, and Ren, "Deep residual learning image recognition." Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, Jun, 2016, pp. 770-778. 1, 8,
K. He, X. Zhang, S. Ren, and Sun, "Identity mappings deep residual networks," " in European Conference on Computer Vision. Amsterdam, North Holland, The Netherlands: Springer, Oct. 2016, pp. 630-645.
Brown, Mann, Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, 2020.
vol. 33, pp.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations. Event Virtual: OpenReview.nett May 2020. 1
S. Shao, Z. Shen, L. Gong, H. Chen, and X. Dai, "Precise knowledge transfer via flow matching." arXiv preprint arXiv:2402.0200 02012. 2024.
W. Masarezyk and I. Tautkute, "Reducing catastrophic forgetting with learning synthetic data," in Computer Vision and Pattern Recognition Workshops. Virtual Event: IEEE, Jun. 2020, pp. 252 253. 1
M. Sangermano, A. Carta, A. Cossu, and D. Bacciu, "Sample condensation in online continual learning." in International Joint Conference Neural Networks. Padua, Italy: IEEE, Jul. 2022, pp. 1-8.
10

---

B. Zhao and H. Bilen, "Dataset condensation with differentiable siamese augmentation." in International Conference on Machine Learning. M. Meila and T. Zhang, Eds., vol. 139. Virtual Event: PMLR, 2021, pp. 12 12674-12685.
F. P. Such, A. Rawal, Lehman, K. O. Stanley, and J. Clune, "Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data," in International Conference on Machine Learning, vol. 119. Virtual Event: PMLR, Jul. 2020. pp.
Zhao and Bilen, "Dataset condensation with distribution matching," Winter Conference on Applications of Computer Vision. Waikoloa, Hawaii: IEEE, Jan. 2023, pp. 5514-6523. 1, 1, 9, 20
Zhao, K. R. Mopuri, and H. Bilen, "Dataset condensation with gradient matching," in International Conference on Learning Representations. Virtual Event: OpenReview.net, May 2021. 2, 20
Liu, Li, Z. Shen, Huang, Yan, and C. Zhang, "Learning efficient convolutional networks through network slimming," in International Conference on Computer Vision. IEEE, 2017, pp. 2736-2744. 1,9
G. Cazenavette, Wang, A. Torralba, A. A. Efros, and J. Zhu, "Dataset distillation by matching training trajectories," in Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, Jun. 2022. 1, 2, 9, 20, 26
A. Sajedi, S. Khaki, E. Amjadian, L. Z. Liu, A. Lawryshyn, and K. N. Plataniotis, "Datadam: Efficient dataset distillation with attention matching." in International Conference on Computer Vision. Paris, France: IEEE, Oct. 2023, pp. 17097-17 107. 1, 3.
Y. Liu, J. Gu, K. Wang, Z. Zhu, W. Jiang, and Y. You, "DREAM: efficient dataset distillation by representative matching." arXiv preprint arXiv:2302. 144166 2023. 1.9
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein al., "Imagenet large scale visual recognition challenge." International Journal of Computer Vision, vol. 115, no. 3. pp. 211-252, 2015. 1,
Z. Yin, E. P. Xing, and Z. Shen, "Squeeze, recover and relabel: Dataset condensation imagenet scale from new perspective," in Neural Information Processing Systems. NeurIPS, 2023. 2, 3, 4, 5,
Z. Yin and Z. Shen, "Dataset distillation in large data era," 2024. [Online]. 1, 8, 23,
Available:
https:
Shao, Z. Yin, X. Zhang, and Z. Shen, "Generalized scrle data condensation via various backbone and statistical matching," " arXiv preprint arXiv:2311.17950 2023. 1, 5, 8. 16, 20,
A. Krizhevsky, G. Hinton et al., "Leerming multiple layers of features from tiny images," 2009.
P. Sun, B. Shi, D. Yu, and T. Lin, "On the diversity and realism of distilled dataset: An efficient datasct distillation paradigm." in Computer Vision and Pattern Recognition. IEEE, 2024, 3, 8, 16
T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros, "Dataset distillation," arXiv preprint arXiv:1811.10955 2018.
J. Cui, R. Wang, Si, and C. Hsich, "Scaling up dataset distillation imagenet- Ik with constant memory." International Conference on Machine Learning, vol. 202. Honolulu, Hawaii, USA: PMLR, 2023, pp. 6565-6590. 2,9
K. Wang, B. Zhao, Peng, Z. Zhu, S. Yang, S. Wang, G. Huang, H. Bilen, X. Wang, and You, "Cafc: Learning to condense dataset by aligning features," " in Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, Jun. 2022, pp. 12 196-112220 205. 2
T. Nguyen, Z. Chen, and J. Lee. "Dataset meta-learning from kernel ridge-regression." arXiv preprint arXiv:2011.0000 2020. 2, 26
J. Kim, Kim, S. J. Oh, S. Yun, H. Song. J. Jeong, Ha, and H. O. Song. "Dataset condensation via efficient synthetic-date parameterization" in International Conference on Machine Learning, vol. 162. Baltimore, Maryland, USA: PMLR, Jul. 2022, pp. 11 102-11 118. 2, 3,3,
D. Zhou, K. Wang, J. Gu, X. Peng, D. Lian, Y. Zhang, Y. You, and J. Feng, "Dataset quantization," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 17 205-17216 3, 16
H. Zhang, S. Li, F. Lin, W. Wang, Z. Qian, and S. Ge, "Dance: Dual-view distribution aligmment for dataset condensation," " arXiv preprint arXiv:2466.00 01063, 2024. 3, 34

---

H. Zhang, S. Li. P. Wang, D. Zeng, and S. Ge. "M3d: Dataset condensation by minimizing maximum mean discrepancy." in Proceedings of the AAAI Conference Artificial Intelligence, vol. 38, 2024, pp- 9314-9322. 34, 35
W. Deng, W. Li, T. Ding, L. Wang, H. Zhang, K. Huang. Huo, and Gao, "Exploiting inter-sample and inter-feature relations in dataset distillation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 17057-17 066. 3, 34, 35
G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," 2015. [Online). Available: https://rxiv.orr/abs/// 500p://bbss550023111
J. Gou, B. Yu. J. Maybank, and D. Tao, "Knowledge distillation: A survey," International Journal of Computer Vision, vol. 129, no. 6, pp. 1789-1819, 2021. 3
H. Chen, Y. Zhang, Y. Dong, and J. Zhu, "Rethinking model ensemble in transfer-based adversarial attacks," in International Conference Learning Representations. Vienna, Austria: OpenReview.net, May 2024. 4, 20,
21
Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, "Sharpness-aware minimization for efficiently improving generalization," " in International Conference on Learning Representations, 2020. 4, 6, 21, 22
H. Chen, S. Shao, Z. Wang. Z. Shang, J. Chen, X. Ji, and X. Wu, "Bootstrap generalization ability from loss landscape perspective," " in Eumpean Conference on Computer Vision. Springer, 2022, pp. 500-517. 4, 21, 25
H. Liu,, T. Xing. Li. V. Dalal, J. He. and H. Wang, "Dataset distillation via the wasserstein metric," arXiv preprint arXiv:2311.1853/ 2023. 4
J. Du, D. Zhou, J. Feng. V. Tan, and J. T. Zhou, "Sharpness-awaar training for free," in Advances in Neural Information Processing Systems, vol. 35. New Orleans, Louisiana, USA: NeurIPS, Dec. 2022, pp. 23 439- 23 451. 6, 21, 22, 23
D. Bahri, Mobahi, and Y. Tay. "Sharpness-awes aware minimization improves language model generalization." arXiv preprint arXiv:2110.0S53 2021. 6, 21
A. Tavanaei, "Embedded encoder-decode in convolutional networks towards explainable AL," vol. ahs/2000.00712 10712,, 2020. [Online]. Availahle: https://aniiv.or/abb//0000 06602 8
M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks"" in Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, Jun. 2018, pp. 4510-4520. 8,9
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. "Training data-efficient image transformers & distillation through attention," " in International Conference Machine Learning, M. Meila and T. Zhang, Eds., vol. 139. Virtual Event: PMLR, Jul. 2021, pp. 10347-10337. 8,
Liu, Lin, Cao, H. Hu, Y. Wei, Z. Zhang, Lin, and Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," " in International Conference on Computer Vision, 2021, pp. 10012- 10022. 8,
M. Tan and Q. Le. "Efficientnet: Rethinking model scaling for convolutional neural networks," in International conference machine learning. PMLR, 2019, pp. 6105-6114
Z. Liu, H. Mao, C.YY. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 11996-11 986.
X. Zhang. X. Zhou, M. Lin, and J. Sun, "Shufflenet: An extremely efficient convolutional neural network for mobile devices." " Computer Vision and Pattern Recognition, 2018 pp. 6848-6856.
Srinivas
Babu, "Data-free parameter pruning deep neural networks." arXiv preprint IrXiv:1507.000 2015.
A. Prabhu, P. Torr and P. K. Dokania, "Gdumb: A simple approach that questions our progress in continual learning." in European Conference on Computer Vision. Springer, Jan 2020, p. 524-540. 9
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high high-performanc deep learning library," in Neural Information Processing Systems, Vancouver, BC, Canada, Dec. 2019. 15
12

---

B. Ostle et al,, "Statistics in research." Statistics in research. no. 2nd Ed, 1963. 17
M. Zhou, Z. Yin, S. Shao, and Z. Shen, "Self-supervised dataset distillation: A good compression is all you need," arXiv preprint arXiv:2404.0 07776, 2024. 23
Y. Wu, J. Du, Liu, Y. Lin, W. Cheng, and W. Xu, "Dd-robustbench: An adversarial robustness benchmark for dataset distillation," arXiv preprint arXiv:2403.13322 2024. 33
H. Kim, "Torchttacks: pytorch repository for adversarial attacks," arXiv preprint 2.X0:20000011550,00000 2020. 33
13

---

Appendix
A
Implementation Details
Here, we complement both the hyperparameter settings and the backbone choices utilized for the comparison and ablation experiments in the main paper.
A.1
Hyperparameter Settings
(a) Data Synthesis
(b) Soft Label Generation and Post- Dost-Evaluation
Config
Value
Explanation
Config
Value
Explanation
Iteration
2000
NA f1. = (0.5, 0.9)
Epochs
300
NA
Optimizer
Adam
Optimizer
Adam
Leaming Rate
0.05
NA
Learning Rate
0.001
Only use le-4 Ior Swin-Tiny
Batch Size
80
NA
Batch Size
100
NA Control EMA-based
Initialized using images synthesized from RDED
Initialization
RDED
EMA Rate
0.99
Evaluation (=2
Control
Scheduler
Smoothing LR Schedule
a.
0.5, 0.99.
categury-aware matching and flatness regularization
Augmentation
RundomResiasdCreg RandomHorizontlllii
NA
Table 6: Hyperparameter setting on ImageNet-Ik.
(a) Data Synthesis
(b) Soft Label Generation and Post-Evve Dost-Evaluation
Config
Value
Explanation
Conlig
Value
Explanation
Iteration
2000
NA 0..5 NA NA
Epochs
Optimizer
Adam
81,8 82
Optimizer
Adam
NA
Learning Rate
0.001
Leaming Rate
0.05
Batch Size
50
NA Control EMA-based Evaluation
Batch Size
100
Initialized using images synthesized from RDED Control
EMA Rate
0.99
Initialization
RDED
Scheduler
Smoothing LR Schedule RandAugment RandomResizzdCrop RundoomoorzznttlFii
C (=2
a. B.
0.5. 0.99. 4
category-aware matching and flatness regularization
Augmentation
NA
Table 7: Hyperparameter setting on ImageNet-10.
(a) Data Synthesis
(b) Soft Label Generation and Post-Evaluation
Config
Value
Explanation
Config
Value
Explanation
Iteration
2000
NA
Epochs
300
Only uso 1000 for IPC
Optimizer
Adam
82 10.5, 0.9)
Optimizee
Adamw 0.001
NA
Learning Rate
0.05
Learning
Batch Size
100
NA Initialized using images from training dataset Control
Batch Size
100
Control EMA-based
EMA Rate
0.99
Initialization
Original Image
Scheduler
Smoothing LR Schedule
C-2
Rand Rand-ngment
o. B.
0.5, 0.99, 4
category-aware matching and flatness regularization
Augmentation
RandomResizeiKCCon
RandomHHorzzntalllgg
Table Hyperparameter setting ImageNe.
We detail the hyperparameter settings of EDC for various datasets, including ImageNet- 1k, ImageNet- 10, Tiny- Ting-Imag,Net, CIFAR- -00, and CIFAR-10, 10, in Tables 6, 7, 8, 9, and 10, respectively. For epochs, critical factor affecting computational cost, utilize strategies from SR G-VBSM, and RDED for ImageNet- kk and follow RDED for the other datasets. In the data synthesis phase, we reduce the iteration count of hyperparameters by half compared to those used in S and G-VBSM.
A.2 Network Architectures Different Datasets
This section outlines the specific configurations of the backbones employed the data synthesis and soft label generation phases, details of which are omitted from the main paper.
14

---

(a) Data Synthesis
(b) Soft Label Generation and Post-Eyaluation
Config
Explanation
Config
Value
Explamation
Iteration
2000
NA 10.5,
Epochs
1000
NA
Optimizer
Adam
Optimizer
AdamW
Learning Rate
0.05
Learning Rate
0,001
NA
Batch Size
100
NA Initialized using images from training dataset Control
Batch Size
Control EMA-based
Initialization
Original Image
EMA Race
0.99
Evaluation
Scheduler
Smoothing LR Schedule
<
RandAngment RundonResizedCryg RandemHootiznnalHHiip
a. F.
0.5. 0.99, 4
category- aware matching and flatness regularization
Augmentation
NA
Table 9: Hyperparameter setting on CIFAR-100.
(a) Data Synthesis
(b) Soft Label Generation and Post-Evaluation
Conlig
Value
Explanation
Conlig
Value
Explanation
Iteration
75
NA 81.02 (0.5,0.9) NA number of synthesized images Initialized using images from training dataset Control
Epochs
1000
NA
Optimizer
Adam
Optimizer
Adam
Learning Rate
0.05
Batch Size
Learning Rate
0.001
25
NA NA
Batch Size
All
Control EMA-based
Evaluation
Initialization
Original Image
Schoduler
MultSSteLLR
milestones- (00000000500000000005500
0.5
RandAuggent RandomReszzedCrrg
a, 3,
0.5. 0.99, 4
category-awee aware matching and flatness regularization
Augmentation
RundooHorizemallFig
Table 10: Hyperparameter setting CIFAR-10.
ImageNet-Ik. We utilize pre-trained models [ResNet-18, MobileNet-V2, ShuffleNet-V2 V2, EfficientNet-V2 AlexNet] from torchvision (Paszke et al., 2019) as observer models data synthesis. To reduce computational load, we exclude EfficientNet-V V2 from the soft label generation process, decision in line with our strategy of selecting more efficient backbones, a concept referred to as better backbone choice in the main paper. An extensive ablation analysis is available in Appendix G.
ImageNet-10. Prior to data synthesis, we train [ResNet-18, MobileNet-V2, ShuffleNet-V? EfficieetNet--2) from scratch for 20 epochs and save their respective checkpoints. Subsequently, these trained models are consistently employed for both data synthesis and soft label generation.
TityeemageNNt. We adopt the same backbone configurations G- VBSM, specifically utilizing [ResNet-18, 18, MobileNet- V2, ShuffleNet-V EfficientNet-V2 for both data synthesis and soft label generation. Each of these models has been trained the original dataset with 50 epochs.
CIFAR-10& For small-scale datasets, we enhance the baseline G- G-VBSM model by incorporating three additional lightweight backbones. Consequently, the backbones utilized for data synthesis and soft label generation comprise (ResNet-18, CenvNet-WIl 128, MobileNet-V2 V2, WRN-16- 2, ShuffleNet-V3 V2, ConvNet-D11 ConyNet-D22, ConvNet- To demonstrate the effectiveness of our approach, we conduct comparative experiments and present results in Table 11, which illustrates that G-VBSM achieves improved performance with this enhanced backbone configuration.
Verified Model
ResNet-18
AlexNet
VGGII-BN
CIFAR-10 (IPC 10)
100 backbones (MTT)
46.4
34,2
50.3
backbones (original setting G-VBSM)
53.5
31.7
55.2
backbones (new setting of G-VBSM)
58,9
36.2
58.0
Table Ablation studies on CIFAR-10 with IPC 10. With the remaining settings are same as those of G-VBSM, our new hackbone setting achieves better performance.
B Theoretical Derivations
Here, we give detailed statement of the definitions, assumptions, theorems, and corollaries relevant to this paper.
15

---

B.1 Random Initialization vs. Real Image Initialization
In the data synthesis phase, random initialization involves using Gaussian noise, while real im- age initialization uses condensed images derived from training-free algorithms, such as RDED. Specifically, we denote the datasets initialized via random and real image methods as x random and A respectively. For coupling (Arandom= xeal). where yandom ~ Trandom: X Treal
and satisfies p( rrandom Treal p( (Tandoo )P Treal), we have the mutual information (MI) between
Frandom and real is O, a.k.a., [Trandom:: Treat) O. By contrast, training- faining-free algorithms (Sun et al., 2024; Zhou al., 2023) synthesize the compressed data X (( via X satis-
fying p( ficel > 0. When the cost function IE[c(a bj] 1/T(Law La))), we have
Elc(22S real X S (eat X Xrandom))
Proof.
E B(c(Arrel Nire))
= k/ (Law((Ara)) Law( Law(Arre)) (Nirve)
k/DKI (p(Treal: Three) Dapeelllppp ( Treal) (tree))
k/ [H (Trea)
I7 (Treal [ine)]
I k/(H (Trea))
(8)
k/(H (Frea) (Treal Trandom)
k/ A[T(Law(( (Xrat) Law( (Xrantom)
BBcc(Xiaa Xeandoom)) )};
where k E IR+ denotes a constant. And (-L(-1l-) and (-) stand for Kullback- divergence and entropy, respectively.
From the theoretical perspective described, it becomes evident that initializing with real images enhances MI more significantly than random initialization between the distilled and the original datasets at the start of the data synthesis phase. This improvement substantially alleviates the challenges inherent in data synthesis. Furthermore, our exploratory experiments demonstrate that the generalized matching loss (Shao et al., 2023) for real image initialization remains consistently lower compared to that of random initialization throughout the data synthesis phase.
B.2 Theoretical Derivations of Soft Category-Awan Matching
Definition B.1. (Statistical Matching) Assume that we have N D-dimensional random samples fxr E RDYN with unknown distribution (m) define two forms of statistical matching for dataset distillation:
Form (I): Estimate the mean E[x] and variance DD [z]] of samples {x; E R Then, synthesize M
(M & N) distilled samples {y E R Ji=1
R such that the absolute differences benween the variances ([[x D(yll) and means [[[[²] - Elyll) of the original and distilled samples are S E.
Form (2): Consider Pmii to be linear combination of multiple subdistributions, expressed as (m) = Ic pppppcppppppc../pppppppppppppppp Jc,, where 0₁ denotes a component of the original distribution. Given Assumption B.4, we can treat Pmix(x) as a GMM, with each component p(x((;)) following a Gaussian distribution. For each component, estimate the mean E[æx) and variance D(z²) using N, samples {x) N- ensuring that EC N = -N. Subsequently, synthesize M (M < N) distilled samples across all components U where Σ5= M, = M. This process aims to ensure that for each component, the absolute differences between the variances (TD( - Diyy )) and means ([E[=") - Ely']) of the original and distilled samples <
Based Definition B.1, here provide several relevant theoretical conclusion.
Lemma B.2. Consider sample set S. where each sample X within $ belongs to RD, Assume any two variables x, and Vj in S satisfies p(z,, x.)) = p(x,)p(z,). This set S comprises disjoint subsets C, the intersection S, n 0 and the union Uh_ 1= S4 = S. Consequently, the expected value over the variance within the subsets, denoted as E3 ~ (8 Be) D. x [ is smaller than equal to the variance within the entire set, Dx~s(X)
(3 S2 ,e,...,,h,,, ensuring that for any 1 S
16

---

Proof.
E (6)....))) [X]
18 (e) (Bx (X X Ex [2]] Bxx-3mm (21)
= =Exeeld's o X1 Ex-s( (2) Ex Bx (X Bxnss
- -Beur Ex [2] o B [2]
-D Es (B Ex [2] Bx [2]
(9)
+Bx-s(A]] Bxxs([]
-Dews(A) P8 Re) Ex (X] Bxe-sen (2)
+ BB Ex [X] BB (S Bx (2]
Dx DS Sc) Ex [2]
s Dr-s(A].
Lemma B.3. Consider Gaussian Mixture Model (GMM) Pmix (2) comprising C components (i.e., Gau-Gaasiin distributions). These components are characterized by their means, variances, and weights, denoted as {u {o and {wi CC fim respectively. The mean E and variance D(a] of the distribution given by Eie = Walls and Σ - wips)) respectively (Ostle et al., 1963).
Proof.
(s--1))
E[a]
Wi
20
V
2/TT
i=1
1
(=-m
20
Wi
Waffi
D[a]
T
E[ar]
(10)
(x-ua)
2
20
E[x²]*
1m1
C
(x-
Wi
20
E[x]*
v V2T
wilke + of
Assumption B.4. For any distribution Q, there exists constant C enabling the approximation of Q by Gaussian Mixture Model P with C components. More generally, this is expressed as the existence of a C such that the distance between P and Q, denoted by the distance metric function E(P, (), is bounded above by an infinitesimal e.
Sketch Proof. The Fourier transform of Gaussian function does not possess true zeros, indicating that such a function, fxx), along with its shifted variant, f(x + a), densely populates the function space through the Tauberian Theorem. the context of L22, the space of all square-integrable functions, where Gaussian functions form subspace denoted as G, any linear functional defined on G-such as convolution operators- can be extended to all of L? through the application of the Theorem. This extension underscores the completeness of Gaussian Mixture Models (GMM) within L spaces.
The proof presents two primary limitations: firstly, it relies solely on shift, which allows argument to remain valid even when the variances of all components within GMM are identical (a relatively loose condition). Secondly, it imposes additional constraint by requiring that the coefficients w; and E, 4i in GMM. Accordingly, this study proposes, rather than empirically demonstrates, that GMM can approximate any specified distribution.
Remarks.
17

---

Theorem B.5. Given Assumption B.4 and Definition B.I, the variances and means of x and estimated through maximum likelihood, remain consistent across scenarios Form (1) and Form (2).
Proof. The maximum likelihood estimation mean E[a] and variance [2]] of samples {a within
Gaussian distribution are calculated باخ di and 2 (a, (a,=)) respectively. These estimations
enable us to characterize the distribution's ' behavior across different scenarios as follows:
ΣN N
Eis
Form (I): P(x) ~ N
N
Σe
En N,
N4
Form (2): Q(1) ~ Es E N,
N
N;
Intuitively, the distilled samples M will obey distributions P(x) and Q(y) in scenarios Form (!) and Form (2), respectively. Then, the difference of the means between Form (1) and Form (2) can be derived as
| 1 P())dd =((r)da)
N =
Σ
N
N Σ
k=
N
C 1=1
N2
N
(11)
0.
To further enhance explanation on proving the consistency of the variance, the setup introduces two sample sets, { and (x= j each drawn from their respective distributions, P(x)
and Qwy). After that, we can acquire:
D2y) D[xr Σ
N: 1=1 Σ, N, (Ely'12 DDy') +
Σ
N Σ, N
Ely')
# Lemma B.3
[[=] E| - ED "|
1
(12)
= [D[x) (DD -
[T[[[[]]]] -
# Lemma
0.
Corollary B.6. The mean and variance obtained from maximum likelihood for any split form {CI, .2,.....cc in Form (2) remain consistent.
Sketch Proof. According Theorem B.5 the mean and variance obtained from maximum likeli- hood for each split form in Form (2) remain consistent within Form (I), so that any split form (c1, c,....,cc) Form (2) remain consistent.
Theorem B.7. Based on Definition B.I, the entropy-pertaining to diversity-oe of the distributions characterized H(P) from Form (1) and H(Q) from Form (2).
which are estimated through maximum likelihood, exhibits the subsequent relation- ship: H(P)
[Iog[E[D[ [1] + [[[logDDw'']]]]]] S =H(Q) < H(P)
BG..N-TICCC holds if and only if both the variance and the mean of each component are consistent.
[Ely" "I''" DD'''+ D|vl
The two-sided equality (i.e., H(P) H(Q))
18

---

Proof.
Lowee bound:
B log( (a))) B[ log(Q(v)))
log (()))) / (og(P()))P(d)dyy
1 + (0g(2=
L. lug( 1 p(v')
1
lom(2xD(z1)
_ +
20100
di
p(v))
-20 dj)dy
2
v
12
22 |
(u-Blvin:2
(1-3la> 1
+
+
lTg
2019?1
-00
|dy
2
2
o
22 1ar1
v 12 y
1
(V-R 201021
(y- Ry-
(2x
+
+
Ellog(
]]
-2/0// !dy
2
e
V2
1
1
Cu Cu-E
lgg(2 +
+ B(s
log(
20(v
)(
20
)dy
2
12 19
v 2 To y
log (2 + TDyy'| + (F(y') 1)
log (og(2+D(xl)) + E(ss)-111C.c
20 [3y)
2 1
1
Dy''l + (TE | - -
2 log(2+D(u))
(
+ 2 B(s.)~
2
1
[Bly') Bly " (2
1
Dly
# Upper bound:
B log( (P())) - EI[ log(Q(u)))
loz(P(=))P()ddr>> / log( P(y()
1
1w- 102
P(z))) + log(El
201571
1)
lyy
/2nD(yp*]
[2=
(w-B 11
(og(P(x))P(zxddd) E 1 log( 22rD
1
-201y01
-20(1>) dyl
9
22r (Dr
log(2nD(xl) 2 log(2xD(v'I))
log(E|D(' D B[log(D(>>))]
(13)
Theorem B.8. Based on Definition B.1, 1, if the original distribution is pmico the Kullback-Leibler
divergence Dei [Q] has upper bound
E/y' and DKL. (pmxll P| 0. DT6T
Proof:
[P]]
N. N;
DRL.
Σ
No
N
N
-== 2
kml
#)
ΣN
ΣN
N₂
N;
N;
N
N
< Σ
N
DxL.
W
Σ k=l
Ruma
N;
|
Σ x;
ΣN
( -
2i
N
N.
N;
N
(14)
19

---

By applying the notations from Lemma B.3 for convenience, we obtain:
PPP
E/= W3 jug
+
oř]
Σ 23 + tojl
1E, w3Hr)²
$ Σ Wi
log
of [13 +oj]
20
Σ Wy
(E/= wjMl)
Σ W/ [13 + 031 (2 Wyll)²
$ log
Σ wi
Σ wi
of
2
Ee 4i1g
EE wMM4)²
wi ازز Wز
- وΣ w21e)²
$ log
+ Σ
wi
+ 2
Σ
o²
5 log
1 
Σ
wiw
Bly 12 T3 Dyy
(15)
When the sample size is sufficiently large, the original distribution aligns with Q. Consequently, we
obtain DEL Panix| P] < "; Be-et C) Dly"
Blr'J"
and establish that DRL (pmix ||Q] 0.
C Decoupled Optimization Objective of Dataset Condensation
In this section, we demonstrate that the training objective, as defined in Eq. 2, can be decoupled into two components fflatess and closeness using a second-orde Taylor expansion, under the assumption that Lyy E C (I) RR. We define the closest optimization point o, for X in relation the i-th matching operator (:) This framework can accommodate all matchings related to (.). including gradient Mattchng(ZZhao et al., 2021), trajectory matching (Cazenavette et al., 2022), distribution matching (Zhao and Bilen, 2023), and statistical matching (Shao et al., 2023). Consequently, we derive the dual decoupling of flatness and closeness as follows:
Sunth
LDD
E Regut-r) (or (a (X ,287')
(2. ,(²7) (X7)
Smanchh
Σ
P (o., xT ) (X - -.. Sap (0:, (0,, x²) + (x -o)" - o.)]
-o.))
-o.)²)
|Seaanch
[8unsti
(o., xx) ) (2'S - H' -o)]
|Boeatch
(16)
where H' refers to the Hessian matrix of Lign ( (T) at the closest optimization point o; Note that the optimization method for deep learning typically involves gradient descent-like approaches (e.g.. SGD and AdamW), the first-order derivative Vxs (o, x'T) can be directly discarded. After that, scanning the two terms in Eq. 16, the first one necessarily reaches optimal solution, while the second one allows us to obtain an upper definitive bound on the Hessian matrix and Jacobi matrix through Theorem 3.1 outlined in Chen et al. (2024). Here, we give special case under the &nrmr to discard the assumption that H' and S o,) are independent:
Theorem C.1. (improved from Theorem 3.1 in (Chen et al., 2024)) Baweel Σl (X' o;) H - oz) S |Smachl- B||[H'|| E ox][3], where TE[[[[]]]]] and IF [ 0i denote flatness and closeness, respectively.
(Souns/
20

---

Proof.
Ennhi
(XS -H -0,) Smasch
[ -o))lell -,)]la
# Hölder's inequality
Brutch
Bunad
1
Σ
11(xx oz)llel1H
# Definition of matrix norm
SSmatch
(macchl
|||||| ||22
o.1 S Smasch
[
o.l
(17) L
Actually, flatness be ensured by convergence in flat region through sharpness- aware mini- mization (SAM) theory (Foret et al., 2020; Bahri et al., 2021; Du et al., 2022; Chen et al., 2022). Specifically, body of work on SAM has established a connection between the Hessian matrix and the flatness of the loss landscape (i.e., the curvature of the loss trajectory), with a series of empirical studies demonstrating the theory's reliability. Meanwhile, the specific implementation of flatness is elaborated upon in Sec. E. By contrast, the concept of closeness was first introduced in Chen et al. (2024), where it is observed that utilizing more backbones for ensemble can result in smaller generalization error during the evaluation phase. In fact, closeness has been implicitly implemented since our baseline G-VBSM uses sequence optimization mechanism akin to the official implementation in Chen et al. (2024). Therefore, this paper will not elucidate on closeness and its specific implementation.
D Traditional Sharpness-Aware Minimization Optimization Approach
For the comprehensive of our paper, let us give brief yet formal description of sharpness-aware minimization (SAM). The applicable SAM algorithm was first proposed in Foret et al. (2020), which aims solve the following maximum minimization problem:
min max La(fote). # elkKlSS
(18)
where Lg (fa), e, p. and refer to the loss flattened region, and the model parameter, respectively. Let us define the final optimized model parameters 0*, then the optimization objective be rewritten as
[ En (ff(x;;) W:), the perturbation, the pre-defined
0" arg min Ra(f+) + La(fe), where Ra(f = max Ta(fet.) Ls(fo).
(19)
e:|eilSe
By expanding La La(fe+) 9 and by solving the classical dual norm problem, the first maximization objective can be solved as (In the special case of the (2-norm)
VoTa(fo)
arg max (s(fot)) =
(20)
(ela(fe)ll
cilel Se
The specific derivation as follows:
Proof. Subjecting Ls (s(++)) to Taylor expansion and retaining only the first-order derivatives:
Rs(fo)
Lo(fo) Ls(fo) + VoLoa(fo) Lo(fo) = Vooos(fo).
(21)
Then, get
= arg Is(fe=)) Ls(fe) = arg max
[ [Vlaa(fo)
(22)
<|k<||5
Next, we base our solution on the solution of the classical dual norm problem, where the above equation can be written as \eLs(f)))l Firstly, Hölder's incquality gives
Volalfo)
oe Vololfoo <
le VoTa(foll
(23)
< ||e VoLo((oo) S II S S
21

---

So. we just need to find a e that makes all the above inequality signs equal. Define m as
sign(VoLs(fa)) Vola 2- then we can rewritten Eq. 23 as
T Vols(fo) sign Seg
(24)
And we also get
||d
Σ
cl"
sign( Vola(()) (oLa((f)))
(25)
i=1
where 1/p + 1/9 substitute into (Voos(())
We choose a new e, defined y = Picllp which satisfies: Ilyllp = p. and
y VoLo((s) Σ 2 VoLs(fo) (olss(fo) [o]L(fo)]]
VoLo(So): = Tellp & VoLa(Jo)..
(26)
Due to ||e|b = ([VoLs((o)):1 and e" VoLs(fe) = ||VoLe(fo)) we can further derive and obtain that
P
Σ VoLol(fo) \Voos(fo)l g/ Velaa(fo) = PlVooa(foo)l
(27)
|elll
Therefore, can be rewritten as:
sign( V V=Ls((a)][V
sign( Vo VoLs Vod
y [sign (odLa
(28)
(ooLsfo)112
Vola(fo) q= = 2. = PVVooa(00
The above derivation is partly derived from Foret et al. (2020), to which we have added another part. To solve the SAM problem in deep learning (Foret al., 2020), had require two iterations to complete single SAM-based gradient update. Another pivotal aspect to note is that within the context of dataset condensation, transitions from representing the model parameter fo to denoting the synthesized dataset X
E Implementation of Flatness Regularization
As proved in Sec. D, the optimal solution e" is denoted Vola(fol P Analogously, in the dataset
condensation scenario, the joint optimization objective given by Σ Bunsil [C (X xxT)). There
Ex DlBarati (2) (x* ,xTT
exists an optimal c", which can be written as P "/s Ruuaa [c (x. .T)llaa"" approach of flatness regularization is shown below:
Thus, a dual-stage
Bua
Nu 8
V
V X
Smarii
[Cin (XSS XTT)1122
(29)
S-
S
(n xT)]
17
where 7) and X denote the learning rate and the synthesized dataset in the next iteration, re- spectively. However, this optimization approach significantly increases the computational burden, thus reducing its scalability. Enlightened by Du et al. (2022), we consider a single-stage opti- mization strategy implemented via exponential moving average (EMA). Given an EMA-updated synthesized dataset A = 3. + (1 3 where typically set to 0.99 in our exper- iments. The trajectories of the synthesized datasets updated via gradient descent (GD) and EMA
22

---

can be represented as { GD.. GD } and {dEMA. EMA . respectively. Assume that 8, = xx _ / Sign (X' X]] the j-th iteration, then f P + > B g; with the condition 1 Eÿو S N as outlined in Du et al. (2022). Consequently, we can provide the EMA-based SAM algorithm and applied backhone sequential optimization in dataset condensation follows:
LFR
Σ (X
Σ
[Cig OEMA)]
the j-th iteration.
(30)
In the vast majority of dataset distillation algorithms (Yin and Shen, 2024; Shao al., 2023; Zhou et al., 2024), the metric function used in matching is set to mean squared error (MSE) loss. Based on this phenomenon, we can rewrite Eq. 30 to Eq. 31, which guarantees flatness.
Buzsh
CSen [Sin (fla (EMa));
at the j-th iteration
GD
Breech
V PGD
Cisn Saallonn
X²) (.in::.)
Cagn CS (M.M,
Cayn (f% X²) ) Connllo -BT"BB( (T)
Bh,
Bruch
Cipo SognlOfi (0 XxT) Layn(0 + BB- -B--gg1, +..
1=1
j-2
+ Lis (0
gi-h Be, BE, x7) Sap (l
ge,
k=1
((3'-1p)]1 dgD Cig So (PED x 1|1 +...
+ (B) p)][ 6/ Et= as Age Cinn (O
gk, X7 [|2]
The solution of dual norm problem
[aazch
E (o0
Vo, Cipu (01, X7) Vo, Cig (0 (02, 207 12)].
" +,90
+ 85 m
(31)
Thus, we can further obtain SAM-like presentation.
Smatil
min
C (aa Ffxa)l,
at the j-th iteration
28
Imanda
min
(0.0 (2)) to0tt' (6--/11 Vo, (1,, 2/ Co X² [12]]
Sunal
min
max 1 100 + (1- (1-9)) S-201))) C )). -IIIISPP
(32) Consequently, optimizing Eq. 30 effectively addresses the SAM problem during the data synthesis phase, which results in flat loss landscape. Additionally. Eq. 32 presents a variant of the SAM algorithm that slightly differs from the traditional form. This variant is specifically designed to ensure sharpness-awes aware minimization within p-ball for each point along a straight path between &% and L
'Neglecting the learning rate for simplicity does not affect the derivation.
23

---

Visualization of Prior Dataset Condensation Methods
In Fig. 5, we present the visualization results of previous training- -erennneddd dataset condensation methods. These approaches, which optimize starting from Gaussian noise, tend to produce synthetic images that lack realism and fail to convey clear semantics to the naked eye.
SRe2L
CDA
G-VBSM
Figure 5: Visualization of the synthetic images of prior rraiing-ddepedeen dataset condensation methods.
G
More Ablation Experiments
In this section, we present series of ablation studies to further validate the design choices outlined the main paper.
G.1 Backbone Choices of Data Synthesis on ImageNet-Ik
Observer Model SauffleNet V2 WRN-40-2
Varified Model
ResNet-18
EfficiemNNet V2
Bo
AlexNet
Com-Next Tiny
DenseNet-12 121
ResNet 18
ReaNet--0
38.7
42.0
43.3
Table 12: Ablation studies on ImageNet-Ik with IPC 10. Verify the influence of backbone choices on data synthesis with CONFIG C CC = 1.5).
The results Table 12 demonstrate the significant impact of backbone architecture selection on the performance of dataset distillation. This study employs the optimal configuration, which includes ResNet- 18, MobileNet-Ve V2, EfficientNer-BB. ShuffleNet-V2 V2, and AlexNet.
G.2 Backbone Choices of Soft Label Generation on ImageNet-1k
Observer Model EllicientNet BO ShuflleNet-V2
Verified Model
ResNet-18
MobileNei-V2
AlleNNet
ResNet-18 18
ResNet-S0 50
ResNet-101
598
9.5
519
8.4
542
12.8
13.3
8,4
Table Ablation studies on ImageNet-Ik with IPC Verify influence backbone choice soft label generation with CONFIG G (C 2).
Our strategy better backbone choice, which focuses on utilizing lighter backbone combinations for soft label generation, significantly enhances the generalization capabilities of the condensed dataset. Empirical studies conducted with IPC I, and results detailed Table 13, show that optimal performance is achieved by using ResNet- 18, NobileNet-V2, EfficientNet-BB ShuffleNet-V2, and AlexNet for data synthesis. For soft label generation, the combination of ResNet-18 MobileNet-V2. ShuffleNet- V2, and AlexNet demonstrates most effecttvve

---

Learning Rate Schedules
0.0010
0.0008
0.0006
Learning
smoothing LR schedule (( = 1)
SSRS ({ 1)
0.0002
smoathing LR schedule ( 1.5)
SSRS (( 1.5)
smoothing schedule ([= 2.0)
0.0000
SSRS (2 2.0)
50
160
150
250
Step (epoch)
Figure 6: The visualization of SSRS and smoothing LR schedule.
G.3 Smoothing LR Schedule Analysis
Slowdown Coefficient
Config
1.0
2.5
3.0
CONFIG C 24.5
30.6
31.8
Table 14: Ablation studies on ImageNet- with IPC 10. Additional experimental result of the slowdown coefficient ( on the verified model MobileNet-V2 V2.
Verified ReaNet-50
Conlig
ReaNet-18
ResNet-101
CONFIG
53.5
CONFIG F
0.9975
47.4
$4.0
50.9
CONTIG CONFIG 0.997875
0.99775
47.3
53.7
50.3
53.8
50.7
Table 15: Ablation studies on TetgeNet-1k with IPC 10. Verify effectiveness ALRS post-cvaluation.
Due to space limitations in the main paper, the experimental results for MobileNet- V2, which are not included in Table 3 Left, are presented in Table 14. Additionally, we investigate Adaptive Learning Rate Scheduler (ALRS), an algorithm that adjusts the learning rate based on training loss. Although ALRS did not produce effective results, it provides valuable insights for future research. This scheduler was first introduced (Chen al., 2022) and is described as follows:
(Mg-l--s1 Shi and I.. Ta-1 |Sha [6a1
p(1) = ju(i 1)7
Here, represents the decay rate, L, is the training loss at the i-th iteration, and hy and h2 are the first and second thresholds, respectively, both set by default to 0.02. We list several values of y that demonstrate the best empirical performance in Table 15. These results allow us to conclude that our proposed smoothing LR schedule outperforms ALRS in the dataset condensation task.
Ultimately, we introduce a learning rate scheduler superior to the traditional smoothing LR schedule in scenarios with high IPC. This enhanced strategy, named early Smoothing Steep Learning Rate Schedule (SSRS). integrates the smoothing LR schedule with MultiSteLRR. It intentionally implements significant reduction in the learning rate during the final epochs of training to accelerate model convergence. The formal definition of SSRS is as follows:
++cos(ém (CN)
< 5N
(())
c (5x/66) (6N BAN-61 61)
i
5.
6
(33)
25

---

Config
Schedaler Type
ResNet-18
ResNet-50
Verified Model
MobileNet-V2
CONT1E G CONTII
smoothing LR schedale SSRS
56.4
62.2
62.3
54.7
57.4
63.0
63.6
56.5
Table 16: A Ablation studies ImageNet-Ik IK with IPC 40, Verify the effectiveness of SSRS in post-evaluation.
Note that the visualization of SSRS can found in Fig. 6. Meanwhile, the comparative experimental results of SSRS and the smoothing LR schedule are detailed in Table 16. Notably, SSRS enhances the verified model's performance without incurring additional overhead.
G.4 Understanding EMA-based Evaluation
CONFIG F
EMA Rate
0.99
0.999
0.9999
0.999945
Accuracy
0.45
Table 17: Ablation studies on ImageNet-1k with IPC 10. Verify effect of EMA Rate in EMA-based Evaluation.
The EMA Rate, a crucial hyperparameter governing the EMA update rate during post-evaluation, significantly influences the final results. Additional experimental outcomes, presented in Table 17, reveal that the EMA Rate 0.99 we adopt in the main paper yields optimal performance.
G.5 Ablation Studies CIFAR-10
This section details the process of deriving hyperparameter configurations for CIFAR-10 through exploratory studies. The demonstrated superiority of our EDC method over traditional approaches, as detailed in our main paper, suggests that conventional dataset condensation techniques like MTT (Cazenavette al., 2022) and KIP (Nguyen et al., 2020) are not the sole options for achieving superior performance on small-scale datasets.
Iteration
25
50
75
100
125
1000
Accuracy
42.1
42.4
42.7
42.5
42.3
41.8
Table Ablation studies on CIFAR-10 with IPC 10. We employ ResNet-18 18 exclusively for data synthesis
and soft label generation, examining the impact of itcration count during post-evaluation and adhering to RDED's consistent hy perparaaeter settings.
Data Synthesis
Soft Label Generation pre-train
Verified Model ResNet-101
pre-traie wlo pre-train
w/o pre-train
ResNet-18
ResNer-50
MobileNet-V2
/
77.7
73.0
68.2
38.2
X
60.5
56.3
52.2
39.9
X
60.0
56.1
50.7
39.0
70.9
61.4
38.2
Table 19: Ablation studies on CIFAR-10 with IPC 10. Hyperparameter settings follow those in Table 10,
excluding the scheduler and batch size. which are set to smoothing LR schedule (C = 2) and 50, respectively.
Verified Model
EMA Rate
Batch
ResNet-18
ResNet-50
ResNet-101
MobileNet-V2
0.99
38.2
0.999
50
13.1
11.8
11.6
11.2
0.9999
50
10.0
10.0
10.0
10.0
0.99
78.1
76.0
71.8
42.1
0.99
10
76.0
70.0
57.7
42.9
Table 20: Ablation studies CIFAR-10 with IPC 10. Explore the influence of EMA Rate and batch size in post-evaluation. Hyperparameter settings follow those Table 10, excluding scheduler, which are set to smoothing LR schedule (C = 2).
Our quantitative experiments, detailed in Table 18, pinpoint 75 iterations as the empirically optimal count. This finding highlights that, for smaller datasets with limited samples and fewer categories, fewer iterations are required to achieve superior results.
26

---

Verified
Scheduler
Option
ResNet-18
ResNet-50
ResNet-101
MobileNet-V2
Smoothing LR Schedule
78.1
76.0
71.8
42.4
Smoothing LR Schedule
77.3
75.0
68.5
41.1
MultiStepLR MultiStepLR
7 0.5, milestones=(8000000 00888000900,90000
79.1
76.0
67.1
42.0
0.25, milestenes=(8005900.95
77.7
75.8
67.0
40.3
Table 21: Ablation studies on CIFAR-10 with IPC 10. Explore the influence of various scheduler in post-
evaluation. Hyperparameter settings follow those in Table 10.
Subsequently, we evaluate the effectiveness of using pre-trained model on ImageNet- I for dataset condensation on CIFAR-10. Our study differentiates two training pipelines: the first involves 100 epochs of pre-training followed by 10 epochs of fine-tuning (denoted 'w/ pre-train') and the second comprises training from scratch for 10 epochs (denoted 'w/o pre-train'). The results, presented in Table 19, indicate that pre-training on ImageNet-Ik does not significantly enhance dataset distillation performance.
We further explore how batch size and EMA Rate affect the generalization abilities of the condensed dataset. Results in Table 20 show that reduced batch size of 25 enhances performance on CIFAR-10.
In our final set of experiments, in Table 21, MultiStepLR is superior for ResNet-18 and ResNet-50, whereas the smoothing LR schedule more effective for ResNet-101 and MobileNet-V2.
compare MultiStepL.R and smoothing LR schedules. As detailed
H Synthesized Image Visualization
The visualization of the condensed dataset is showcased across Figs. 7 to 11. Specifically, Figs. 7, CIFAR-100, and
10, and present the datasets synthesized from ImageNet- 1k, CIFAR-10, respectively.
I
Ethics Statement
Our research utilizes synthetic data to avoid the use of actual personal information, thereby addressing privacy and consent issues inherent in datasets with identifiable data. We generate synthetic data using a methodology that distills from real-world data but maintains no direct connection to individual identities. This method aligns with data protection laws and minimizes ethical risks related to confidentiality and data misuse. However, it is important to note that models trained on synthetic data may not achieve the same accuracy levels as those trained on the full original dataset.
J Limitations
The paper offers an extensive examination of the design space for dataset condensation, but it might still miss some potentially valuable strategies due the broad scope. Additionally, the IPC count grows, the performance of the described approach converges with that of the baseline RDED.
27

---

Figure 7: Synthetic data visualization on ImageNet- Ik randomly selected from EDC.
28

---

Figure& 8: Synthetic data visualization on ImageNet- 10 randomly selected from EDC.
29

---

LL
Figure 9: Synthetic data visualization on ImageNet randomly selected from EDC.
30

---

Figure 10: Synthetic data visualization on CIFAR-100 randomly selected from EDC.
31

---

Figure 11: Synthetic data visualization on CIFAR-10 10 randomly selected from EDC.
32

---

K Additional Experiments, Theories and Descriptions (Rebuttal Stage Supplement)
Here we add some experiments, theories and explanations that we think is necessary to add.
K.1 Scalability on ImageNet-21k
SRe"1.
CDA
RDED
EDC
Original Dataset
18.5
22.6
25.6
26.8
38.5
Table 22: Comparison of Different Methods on ImageNet-21k.
We conduct experiments on larger scale dataset ImageNet-21k-I with IPC 10. The results in Table 22 indicate that method outperforms the state-of-the-an method CDA (Yin and Shen, 2024) on this dataset, demonstrating that EDC can scale to larger datasets.
K.2
Complexity of Implementation
Configuration
GPU Memory (G/per GPU)
Time Spent (hours)
Top-1 Accuracy (%)
CONFIG A
4.616
9.77
31.4
CONFIG B
4.616
4.89
34.4
CONFIG C
4.616
4.89
38.7
CONFIG D
4.616
4.91
39.5
CONFIG E
4.697
4.91
46.2
CONFIG F
4,923
5.11
48.0
CONFIG
4,923
5.11
48.6
Table 23: Comparison of computational resources 4 RTX 4090.
Here we present Table complement the computational overhead in Fig. in the main paper. EDC is an efficient algorithm as it reduces the number of iterations by half. compared to the baseline G-VBSM. As illustrated in the table above, although transitioning from CONFIG A to CONFIG G adds small GPU memory overhead, it is minor compared to the reduction in time spent. Additionally, introducing EDC other tasks often requires significant effort for tuning syper-parameters or even redesigning statistical matching, which is a challenge EDC should address.
K.3 Robustness Evaluation
Attack Methods
MTT
SRe2L.
EDC (Ours)
Clean Accuracy
26.16
43.24
57.21
FGSM
1.82
5.73
12.39
PGD
0.41
2.70
10.71
CW
0.36
2.94
5.27
VMI
0.42
2.60
10.73
Jitter
0.40
2.72
10.64
AutoAttack
0.26
1.73
7.94
Table 24: Comparison DD-RobustBench.
We follow the pipeline in Wu et al. (2024) to evaluate the robustness of models trained on condensed datasets, utilizing the well-known adversarial attack library available at Kim (2020). As illustrared in Table 24. Our experiments are conducted on presented in the table above. Evidently, EDC demonstrates significantly higher robustness compared to other methods. We attribute this to improvements in post-evaluation techniques, such as EMA- based evaluation and smoothing LR schedule, which help reduce the sharpness of the loss landscape.
with IPC 50, with the test accuracy
K.4 Theoretical Explanation of Irrational Hyperparameter Setting (Sketch!!)
The smoothing LR schedule is designed address suboptimal solutions that arise due to the scarcity of sample sizes in condensed datasets. Additionally, the use of small batch size is implemented
33

---

where 0 stands for the model parameter. The right part of the penultimate row results from the loss Luyn. which ensures the consistency of p(-IX",,,, ..)) and p(- X cq). If the model initialization during training the same, the left part of the penultimate row is scalar and has little influence on the direction of the gradient. Since X" is the complete original dataset with global gradient, the because the gradient of the condensed dataset more closely resembles the global gradient of the original dataset, as illustrated at the bottom of Fig. Against the latter, we can propose a complete chain of theoretical derivation:
Leyn --Beoollpp .c) p(u|x", .i)ll
+ +Ilpo(o2| - 1(01 ,0)l22 # (Our statistical matching)
[ 10Lsym/Opol ( (0))(ppp--11')) c.)/8))de.
T. 180
[ ((po(u)Xx ppllll :4)] -p)oo (a))) c.)/000d
(34)
where ( (.) and p((X", .4) refer to Gaussian component in the Gaussian Mixture Model. Consider post-evaluation, We can derive the gradient of the MSE loss as:
yi 12//000 2R ([fo(a;)))))))))))))) -1)0))
= =BB,, ~x ([f((z)) - vi) /
(35)
(x3,
- wi)
(x)))0000 ,.)/00)de))
~ - (lf 1 /00
gradient of 8 approximates the global gradient of X thus enabling the use of small batch size.
K.S Additional Related Work
We additionally discuss the differences between published related papers (Sajedi et al., 2023; Zhang al., 2024b; Deng et al., 2024) and our work.
DataDAM (Sajedi et al., 2023) vs. EDC. Both DataDAM and EDC do not require model parameter updates during training. However, DataDAM struggles to generalize effectively to ImageNet- Ik because it relies on randomly initialized models for distribution matching. As noted in SRe"L, models trained for fewer than 50 epochs experience significant performance degradation. DataDAM does not explore the soft label generation and post-evaluation phases as EDC does, limiting its competitiveness.
DANCE (Zhang et al., 2024a) vs. EDC. DANCE is a DM-based algorithm that, unlike traditional distribution matching, does require model updates during data synthesis. Instead, it interpolates between pre-trained and randomly initialized models, using this interpolated model for distribution matching. Similarly, EDC also does need update the model parameters, but it uses a pre-trained model with a different architecture and does not incorporate random interpolation. The "random interpolation" technique was not adopted because it did not yield performance gains on ImageNet- Ik. Although DANCE considers both intra-class and inter-class perspectives, it limits inter-class analysis the logit level and intra-class analysis the feature map level. In contrast, EDC performs both intra-class and inter-class matching at the feature map level, where inter-class matching is crucial. To support this, last year, L focused solely on inter-class matching at the feature map level and still achieved state-of-the-an performance on ImageNet- k. EDC is the first dataset distillation algorithm to simultaneously improve data synthesis, soft label generation, and post-evaluation stages. In contrast, DANCE only addresses the data synthesis stage. While DANCE can be effectively applied to ImageNet-1k, 1k, the introduction of soft label generation and post-evaluation improvements essential for DANCE to achieve more competitive results.
34

---

M3D (Zhang et al., 2024b) vs. EDC. M3D is DM-based algorithm, but its data synthesis paradigm aligns with DataDAM by relying solely on randomly initialized models, which limits its generalization to ImageNet-1K, kk. M3D, similar to SRe²I., G-VBSM, and EDC, takes into account second-order information (variance), but this is not unique contribution of EDC. The key con- tributions of EDC in data synthesis are real image initialization, flatness regularization, and the consideration of both intra-class and inter-class matching.
Deng et al. (Deng et al., 2024) vs. EDC. Deng et al. (Deng et al., 2024) is DM-based algorithm, but its data synthesis paradigm consistent with M3D and DataDAM, as considers only randomly initialized models, which cannot be generalized to ImageNet- 1k. Deng et al. (Deng et al., 2024) considers both interclass and intraclass information, similar to EDC, However, while EDC obtains interclass information by traversing the entire training set, Deng et al. (Deng al., 2024) derives interclass information from only one batch, making its information richness inferior to that EDC. Deng et al. (Deng et al., 2024) only explores data synthesis and does not explore soft label generation or post-evaluation. Additionally, Deng et al. (Deng et al., 2024) only shares some similarity with Soft Category- Aware Matching among the 10 design choices in EDC.
K.6 Implementation of Cropping
The implementation this crop operation refers to torchvision, transforms wen.dodeeediieeeerrrrrr where the minimum area threshold is controlled by the parameter scale[O]. The default value is 0.08, meaning that the cropped image can be as small as 8% of the original image. Since 0,08 is too small for the model to extract complete semantic information during data synthesis, increasing the value to 0.5 resulted a significant performance gain.
K.7
Comprehensive Comparison Experiment
Dataset
IPC
MTT
TESLA
SRc²L
G-VBSM
CDA
WMDD
RDED
EDC (Ours)
22
3..6
CIFAR-10
10
46.1 # 1.4
48.9 2.2
2..2
23.5
3.3
79.1 0.3
4..5
59.2104
6..1
8.1
2.20.2
25.9
11.0
3..7
CIFAR-100
10
26.8 0.6
27.1 =
3,56
59.4
4..6
63.7 0.3
50
0.3
6.50
6.16
68.6
7.6
9.7
4..2
Imag-Im
10
4.1
4.29
51.2
50
28.0 0.3
4..1
47.6 0.3
48.7
5.54
58.2
57.2 0.2
2..9
45.2
ImageNet-10
10
0.1
63.4
50
0.5
8.12
0.3
6.6
12.820.1
ImageNet- Ik
10
17 1.3
0.6
3.4
0.2
42.0
48.6
50
2..
4.2
5.4
53.5
5..6
0.1
58.0
Table 25: Comparison with the SOTA baseline dataset condensation methods. MTT, TESLA, SRe"L,
CDA, WMDD and RDED utilize ResNet-18 for data synthesis, whereas G-VBSM and EDC leverage various backbones for this purpose.
Due to space constraints in the main paper and for aesthetic reasons, we have not fully presented the experimental results of other methods. However, since the benchmark for dataset distillation is uniform and rell-recognized, the performance of other algorithms can found their respective papers. We present the related experimental results of the popular convolutional architecture ResNet- 18 in Table 25.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the introduction and abstract, we comprehensive design framework for dataset condensation, incorporating specific and effective strategies supported by empirical evidence and theoretical foundations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer: [Yes]
Justification: Please Sec. B in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our supplemental materials contain the reproducible code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code has been provided in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Ceglli//guides/ooeSttteeee sosooodsosmmssiioooooooor for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: cttpp/cccppubllc/uuiicc /ateeee/ooodemmmicece ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details have been presented Appendix A.1. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
38

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments are conducted using 4x RTX 4090 GPUs, as detailed in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettpa://neurippee http:/pplll///tiaaaatiaaaeaaaaaa
Answer: [Yes]
Justification: Please Sec.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Sec. I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
39

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: There are no risk factors present here.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: our paper and accompanying code, we have carefully cited and credited the works of G-VBSM and RDED, which form the foundation of our implementation.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
40

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have attached code and user instructions in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not have any experiments or research relevant to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "n2sODNx3ve", "reviewer": "Reviewer_gMw3", "summary": "The authors address the limitations of previous methods, such as high computational costs and less optimal design spaces, by proposing a novel framework called Elucidate Dataset Condensation (EDC). EDC incorporates strategies like soft category-aware matching and a smoothing learning rate schedule, achieving state-of-the-art accuracy with a significant improvement over existing methods. The paper also provides empirical and theoretical insights into the design decisions made, demonstrating EDC's effectiveness across various datasets and model architectures.", "questions": "1. How does EDC perform when faced with adversarial examples or noisy data, and what measures are taken to ensure model robustness?\n2. Is there a plan to extend EDC to other domains beyond image recognition, and what challenges might arise in such extensions?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive Analysis: The authors systematically examine the design space, leading to a nuanced understanding of dataset condensation and the impact of various factors on performance.\n- State-of-the-Art Performance: The reported improvements in accuracy, especially on ImageNet-1k with a ResNet-18 model, are substantial and demonstrate a clear strength of the proposed method.", "weaknesses": "1. Scalability Concerns: While EDC shows promise, the paper might not fully address how the method scales with significantly larger datasets or higher dimensions of data.\n2. Potential Information Loss: The training-free distillation paradigm, while efficient, could potentially lead to information loss, which is not deeply explored in the paper.\n3. Complexity of Implementation: The paper could benefit from a more detailed discussion on the practical implementation of EDC, including computational resources and potential challenges.\n4. Generalization to Other Domains: The paper primarily focuses on image datasets; it is unclear how well EDC's strategies would generalize to other data domains, such as text or time", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our comprehensive analysis and SOTA performance, as well as the valuable suggestions for improvement. We hope our responses can address your concerns effectively.\n\n**Q1**: _Scalability Concerns_.\n\n| SRe$^2$L | CDA  | RDED | Ours | Original Dataset |\n|----------|:------:|:------:|:------:|:------:|\n| 18.5     | 22.6 | 25.6    | 26.8  | 38.5 |\n\n**A1**: Thanks for raising these concerns. We conduct experiments on a larger scale dataset ImageNet-21k-P with IPC 10. The results in the table above indicate that our method outperforms the state-of-the-art method CDA [1] on this dataset, demonstrating that EDC can scale to larger datasets.\n\n[1] Dataset Distillation in Large Data Era, 2024.\n\n**Q2**: _Potential Information Loss._\n\n**A2**: Thanks for pointing this out. RDED, as a training-free distillation paradigm, initially downsamples high-resolution images to obtain low-resolution images. Then, RDED concentrates the low-resolution images to produce condensed data. This paradigm inevitably loses some fine-grained information due to the downsampling operation.\n\nEDC compensates for the information loss of RDED by training-dependent data synthesis, i.e., the statistical information of the condensed dataset on different feature maps remains the same as the original dataset. This is our advantage over the training-free distillation paradigm.\n\n**Q3**: _Complexity of Implementation._\n\n| Configuration | GPU Memory (G/per GPU) | Time Spent (hours) | Top-1 Accuracy (%) |\n|---------------|:------------------------:|:--------------------:|:--------------------:|\n| CONFIG A      | 4.616                  | 9.77               | 31.4               |\n| CONFIG B      | 4.616                  | 4.89               | 34.4               |\n| CONFIG C      | 4.616                  | 4.89               | 38.7               |\n| CONFIG D      | 4.616                  | 4.91               | 39.5               |\n| CONFIG E      | 4.697                  | 4.91               | 46.2               |\n| CONFIG F      | 4.923                  | 5.11               | 48.0               |\n| CONFIG G      | 4.923                  | 5.11               | 48.6               |\n\n*Table: Comparison of computational resources on 4$\\times$RTX 4090.*\n\n\n**A3**: EDC is an efficient algorithm as it reduces the number of iterations by half, compared to the baseline G-VBSM. As illustrated in the table above, although transitioning from CONFIG A to CONFIG G adds small GPU memory overhead, it is minor compared to the reduction in time spent. Additionally, introducing EDC to other tasks often requires significant effort for tuning hyper-parameters or even redesigning statistical matching, which is a challenge EDC should address.\n\nWe will add the above table in the revised version. \n\n**Q4**: _Generalization to Other Domains._\n\n| Ratio (r) | Random | Herding | K-Center | GCOND-X | GCOND | Ours |\n|-----------|:--------:|:---------:|:----------:|:---------:|:-------:|:------:|\n| 1.3%      | 63.6   | 67.0    | 64.0     | 75.9    | 79.8  | 80.1 |\n| 2.6%      | 72.8   | 73.4    | 73.2     | 75.7    | 80.1  | 81.0 |\n| 5.2%      | 76.8   | 76.8    | 76.7     | 76.0    | 79.3  | 81.0 |\n\n*Table: EDC is performed with both SGC and GCN and evaluated using GCN on the Cora graph dataset.*\n\n\n**A4**: With careful redesign of statistical matching, EDC can be extended to other domains, such as graphs. We convert the graph data $G\\in \\mathbb{R}^{n\\times d}$ ($n$ is the number of nodes and $d$ is the feature length) into an image-like format. Specifically, we first derive ```M = G[None,... ,None].permute(0,2,1,3)```, then compute the feature map ```K = \\text{cosine}(M, M.permute(0,1,3,2))```, and finally perform a distillation paradigm for the dataset similar to that used in image data. We also introduce soft labels during post-evaluation. According to the table above, EDC performs well on the graph classification task, revealing that EDC can generalize to other domains.\n\n**Q5**: _How robust is EDC and how to ensure robustness_\n\n| Attack Methods/DD Methods | MTT   | SRe2L | EDC (Ours) |\n|----------------------|:-------:|:-------:|:------------:|\n| Clean Accuracy       | 26.16 | 43.24 | 57.21      |\n| FGSM                 | 1.82  | 5.73  | 12.39      |\n| PGD                  | 0.41  | 2.70  | 10.71      |\n| CW                   | 0.36  | 2.94  | 5.27       |\n| VMI                  | 0.42  | 2.60  | 10.73      |\n| Jitter               | 0.40  | 2.72  | 10.64      |\n| AutoAttack           | 0.26  | 1.73  | 7.94       |\n\n*Table: Comparison with baseline models with ResNet-18. The perturbation budget is set to $|\\epsilon|$ = 2/255.*\n\n\n**A5**: We follow the pipeline in [2] to evaluate the robustness of models trained on condensed datasets, utilizing the well-known adversarial attack library available at [3]. Our experiments are conducted on Tiny-ImageNet with IPC 50, with the test accuracy presented in the table above. Evidently, EDC demonstrates significantly higher robustness compared to other methods. We attribute this to improvements in post-evaluation techniques, such as EMA-based evaluation and smoothing LR schedule, which help reduce the sharpness of the loss landscape.\n\n[2] DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation, 2024.\n\n[3] https://github.com/Harry24k/adversarial-attacks-pytorch\n\n\n**Q6**: _How to extend EDC to other domains?_\n\n**A6**:  Thank you for your interesting question. As we replied in **A4**, introducing EDC to graph classification tasks is feasible. In the future, we expect to extend EDC to multimodal tasks. We argue that the biggest challenge in extending EDC is the inconsistent formats between different data, which necessitates redesigning statistical matching to make EDC adaptable to the target task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ms1dIFAOTy", "reviewer": "Reviewer_r3SQ", "summary": "This paper studies the combination of some techniques of data distillation (DD) in terms of data synthesis, soft label generation, and post-evaluation. The limitations of the existing methods, which are solved by these techniques, are provided. The extensive experiments verified the promising improvement of these techniques when using them with certain SOTA DD methods.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  The limitations of the existing methods, which are solved by these techniques, are provided.  \n2.  For some design choices, a theoretical analysis is conducted.\n3. The performance of the proposed method is very promising, and the ablation study is sound.", "weaknesses": "1.  The definition of generalized data synthesis is somewhat unclear. DM-based methods [1,2] also can efficiently conduct data synthesis on the ImageNet dataset. Statistical matching essentially is a distribution matching way using second-order information, and its similar way can be seen in certain DM-based methods [3,4]. Some discussions and comparisons about this are necessary.\n\n2. The solution for the limitation of irrational hyperparameter setting is very heuristic. Could the authors provide a theoretical analysis for it?\n\n3. Some statements lack clear evidence or explanation. For example, In Line 227, \"our findings unfortunately demonstrate that various SM-based loss functions do not converge to zero. This failure to converge contradicts the basic premise that the first-order term in the Taylor expansion should equal zero.\" In Line 263, \"smaller batch sizes\" helps prevent model under-convergence during post-evaluation. In Line 267, \"The key finding reveals that the minimum area threshold for cropping during data synthesis was too restrictive, thereby diminishing the quality of the condensed dataset\".\n\n[1] DataDAM: Efficient Dataset Distillation with Attention Matching. ICCV 2023\n\n[2] DANCE: Dual-View Distribution Alignment for Dataset Condensation. IJCAI 2024\n\n[3] M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy. AAAI 2024\n\n[4] Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your constructive comments and valuable suggestions, such as raising many unclear definition issues for us. Below, we make detailed clarifications to each question from this reviewer.\n\n**Q1**: _The definition of generalized data synthesis is somewhat unclear._\n\n**A1**: Thank you for raising these concerns. We adopted this definition to simplify the pipeline presentation of our work, but we did not anticipate that it would cause you distress.\n\n\"Generalized data synthesis\" begins with training one or more models on the original dataset to obtain global statistics. These statistics are then used in the data synthesis stage with Eq. 2 to synthesize the condensed dataset, and finally, Eq. 4 is applied to generate the soft label.\n\nAs mentioned in our original paper, \"generalized data synthesis\" avoids inefficient bi-level optimization. Although DM-based methods are also highly efficient, they still involve both inner and outer loops, and extending them to the full 224x224 ImageNet remains challenging in terms of performance and efficiency.\n\nStatistical matching is indeed formally similar to distribution matching, however, statistical matching is better suited to large datasets such as ImageNet-1k. One crucial reason is that statistical matching obtains global statistics across the comprehensive dataset by traversing it in advance, whereas distribution matching gathers local statistics during data synthesis, leading to inaccuracies within the local statistics. This is evidenced by the fact that the works you mentioned [3] and [4] did not experiment on the full 224x224 ImageNet.\n\nAdditionally, certain DM-based methods [3,4] differ from EDC. As shown in Eq. 5 and the experiments in the original paper, both Forms (1) and (2) are necessary, while [4] only applies to the case of $\\alpha=0$.\n\n**Q2**: _Provide a theoretical analysis for the solution of irrational hyperparameter setting._\n\n**A2**: Thank you for your insightful suggestion. The smoothing LR schedule is designed to address suboptimal solutions that arise due to the scarcity of sample sizes in condensed datasets. Additionally, the use of small batch size is implemented because the gradient of the condensed dataset more closely resembles the global gradient of the original dataset, as illustrated at the bottom of Fig. 2 (c).\n\nAgainst the latter, we can propose a complete chain of theoretical derivation:\n\n$L_{syn} = 𝔼_{c_i∼C} || p_θ(μ | X^S, c_i) - p(μ | X^T, c_i) ||_2 + || p_θ(σ^2 | X^S, c_i) - p(σ^2 | X^T, c_i) ||_2$ (Our statistical matching)\n\n$∂L_{syn} / ∂θ = ∫_{c_i} ( ∂L_{syn} / ∂p_θ(· | X^S, c_i) ) ( ∂p_θ(· | X^S, c_i) / ∂θ ) d_{c_i} $\n\n$≈ ∫_{c_i} ( [ p_θ(μ | X^S, c_i) - p(μ | X^T, c_i) ] + [ p_θ(σ^2 | X^S, c_i) - p(σ^2 | X^T, c_i) ] ) ( ∂p_θ(· | X^S, c_i) / ∂θ ) d_{c_i}$\n\nwhere $p_θ(· | X^S, c_i)$ and $p(· | X^T, c_i)$ refer to a Gaussian component in the Gaussian Mixture Model. Consider post-evaluation, We can derive the gradient of the MSE loss as:\n\n$∂ 𝔼_{x_i∼X^S} || f_θ(x_i) - y_i ||\\_2^2 / ∂θ   =  2𝔼_{x_i∼X^S} [ ( f_θ(x_i) - y_i ) ( ∂f_θ(x_i) / ∂θ ) ] $\n\n$ = 2𝔼_{x_i∼X^S} [ ( f_θ(x_i) - y_i ) ∫_{c_i} ( ∂f_θ(x_i) / ∂p_θ(· | X^S, c_i) ) ( ∂p_θ(· | X^S, c_i) / ∂θ ) d_{c_i} ] $\n\n$ ≈ 2𝔼_{(x_j, x_i) ∼ (X^S, X^T)} [ ( f_θ(x_j) - y_j ) ∫_{c_i} ( ∂f_θ(x_i) / ∂p_θ(· | X^T, c_i) ) ( ∂p_θ(· | X^T, c_i) / ∂θ ) d_{c_i} ] $\n\n$≈ ∂ 𝔼_{x_i∼X^T} || f_θ(x_i) - y_i ||_2^2 / ∂θ, $\n\nwhere θ stands for the model parameter. The right part of the penultimate row results from the loss $L_{syn}$, which ensures the consistency of $p_θ(· | X^T, c_i)$ and $p_θ(· | X^S, c_i)$. If the model initialization during training is the same, the left part of the penultimate row is a scalar and has little influence on the direction of the gradient. Since $X^T$ is the complete original dataset with a global gradient, the gradient of $X^S$ approximates the global gradient of $X^T$, thus enabling the use of small batch size.\n\n**Q3**: _Some statements lack clear evidence or explanation._\n\n**A3**: Thank you for your suggestions to help improve the quality of our manuscript. Here, we clarify the parts you pointed out as unclear, and we will double-check the full paper in future releases to make sure there are no relevant issues:\n\n*Line 227:* As shown in Line 122-127, the constraint on flatness needs to ensure that the first-order term $(\\theta-\\theta^*)^\\mathrm{T}\\nabla_\\theta{L}(\\theta^*)$ of the Taylor expansion equals zero, indicating normal model convergence. However, our exploratory experiments found that despite the good performance of EDC, the loss of statistical matching at the end of data synthesis still fluctuated significantly and did not reach zero. Therefore, we only enforced flatness on the logit level.\n\n*Line 263:* Since the gradient of the condensed dataset can approximate the global gradient of the original dataset, the inaccurate gradient direction problem introduced by the small batch size becomes less problematic. Instead, using a small batch size effectively increases the number of iterations, thereby helps prevent model under-convergence.\n\n*Line 267:* The implementation of this crop operation refers to `torchvision.transforms.RandomResizedCrop`, where the minimum area threshold is controlled by the parameter scale[0]. The default value is 0.08, meaning that the cropped image can be as small as 8% of the original image. Since 0.08 is too small for the model to extract complete semantic information during data synthesis, increasing the value to 0.5 resulted in a significant performance gain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V3jcAJHsff", "reviewer": "Reviewer_ZPCz", "summary": "This paper proposes a design framework to address the limitations of existing dataset condensation methods. Specifically, the authors have introduced some strategies, such as soft category-aware matching and learning rate scheduling. The authors have provided theoretical and empirical analysis of these strategies, proving the superiority of the proposed Elucidate Dataset Condensation (EDC) with extensive experiments.", "questions": "1. Since the structure of the paper is confusing, I am not really sure what the primary novelty of this work is. I wish the authors can clearly emphasize their motivations and novelties.\n2. It seems that the authors have discussed the limitations of previous works from two levels. In the introduction, the authors talked about effectiveness, accuracy, etc., while in Section 3, the authors talked about realism, matching mechanism, etc. Can the authors clearly state the relationship between these limitations?", "limitations": "From my point of view, the major limitation is the organization of the paper. I am not an expert in dataset distillation, but I try to understand the problems of previous works and the corresponding solutions in this paper. Unfortunately, I find it difficult to figure out these, though I think that this seems to be a solid work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It seems that this is a comprehensive work to address the existing problems in dataset condensation. I am not sure about this because the structure of the paper is a bit confusing, as I will note in the weaknesses of the paper.\n2. It seems that there is solid theoretical analysis in the paper. However, again, because of the confusing structure of the paper, I can hardly understand what these equations aim to prove.\n3. I acknowledge that the authors have conducted extensive experiments for evaluation and also put some effort into the visualization.", "weaknesses": "The major weakness is the lack of fluency in the paper, leading to the confusing structure of the paper. More explanations are as follows.\n1. In line 27-41, the authors have compared bi-level and uni-level optimization paradigms. However, I am not sure what “bi-level” and “uni-level” mean, maybe because I am not an expert in dataset condensation.\n2. The authors also mentioned the bi-level paradigm limits the effectiveness, but did not explain why.\n3. The motivation of the entire work is vague. Regarding the limitations of previous works, the authors have mentioned the effectiveness (line 27), accuracy (line 36), potential information loss (line 38). After reading these, I expect to see how the authors tackled these issues. However, later in Section 3, the authors talked about other limitations, involving realism (line 110), matching mechanism (line 115), loss landscape (line 122), and hyperparameter settings (line 128). And then the authors proposed strategies to address these limitations, which sounds reasonable. However, in the end of Section 3, the authors talked about augmentation and backbone choice, which aim to address other specific issues, making me confused again.\n4. Figure 1 mentions CONFIG A to G, but I have no idea what they mean.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and detailed suggestions on our paper's layout. We will accommodate all your comments in our revision.\n\n**Q1**: _The differences between “bi-level” and “uni-level”._\n\n**A1**: The main difference between “bi-level” and ”uni-level“ is that “bi-level” requires updating the dataset and the model alternately, whereas “uni-level” only updates the dataset. Here is our algorithmic process for “bi-level” that we taken from a well known survey [1]:\n\n**Input:** Original dataset $\\mathcal{T}$.\n\n**Output:** Synthetic dataset $\\mathcal{S}$.\n\n**Initialize** $\\mathcal{S}$.\n\n**While** not converge **do**\n\n&ensp;  Get a network $\\theta$.\n\n&ensp; Update $\\theta$ via $\\mathcal{S}$ or $\\mathcal{T}$ and cache it if necessary.\n\n&ensp; Update $\\mathcal{S}$ via $L(\\mathcal{S}, \\mathcal{T})$.  ($L$ denotes loss function)\n\n&ensp; **done**\n\n**Return** $\\mathcal{S}$\n\n[1] Dataset Distillation: A Comprehensive Review, 2023.\n\n**Q2**: _Explain the limitation of the bi-level paradigm._\n\n**A2**: Bi-level optimization in dataset distillation requires alternately updating the synthetic data and the model (e.g., trajectory and gradient matching). This alternating process hinders its application to large-scale datasets. In contrast, statistical matching only necessitates optimizing the synthetic data, making it more effectiveness and efficient than bi-level optimization.\n\n**Q3**: _The motivation of the entire work is vague._\n\n**A3**: Thank you for your concerns. The ``effectiveness (line 27), accuracy (line 36), potential information loss (line 38)'' you mention are issues that EDC addresses at a macro and general level. Specifically, we use statistical matching to ensure effectiveness, and a series of improvements presented in Fig. 1 to ensure the accuracy of EDC on both small- and large- scale dataset. We also compensate for the potential information loss caused by RDED through statistical matching.\n\nBy comparison, ``realism (line 110), matching mechanism (line 115), loss landscape (line 122), and hyperparameter settings (line 128)'' exemplify the limitations in previous work from details. We address these shortcomings to ultimately ensure effectiveness, accuracy, and compensation for information loss.\n\nThe augmentation and backbone choice is an improvement on the irrational hyperparameter settings (line 128) of past algorithms (i.e., G-VBSM, SRe$^2$L and CDA), and related experiments and discussions can be found in the Appendix (lines 493-499, 679-691).\n\n**Q4**: _The meaning of  CONFIG A to G._\n\n**A4**: Your question is critical. The logical presentation of our work is somewhat challenging because we delves deeply into analyzing the limitations of past algorithms and proposing 9 improvements. We borrowed the presentation style from ConvNet [2] and EDM [3], but it still caused confusion. Here, we describe the improvements included in CONFIG A to CONFIG G, and in the future we will provide this in the appendix:\n\nCONFIG A: G-VBSM (our baseline).\n\nCONFIG B: G-VBSM, real image initialization.\n\nCONFIG C: G-VBSM, real image initialization, smoothing LR schedule.\n\n$\\vdots$\n\nCONFIG F: G-VBSM, real image initialization, smoothing LR schedule, flatness regularization, small batch size, better backbone choice, soft category-aware matching.\n\nCONFIG G: G-VBSM, real image initialization, smoothing LR schedule, flatness regularization, small batch size, better backbone choice, soft category-aware matching, weak augmentation, ema-based evaluation.\n\n[2] A ConvNet for the 2020s, 2020.\n\n[3] Elucidating the Design Space of Diffusion-Based Generative Models, 2022.\n\n**Q5**: _The authors can clearly emphasize motivations and novelties._\n\n**A5**: Thanks for pointing this out. We are motivated by the discovery that (1) RDED causes information loss; (2) algorithms such as SRe$^2$L, G-VBSM, and CDA do not perform well on small-scale datasets; (3) traditional algorithms like MTT, Gradient Matching, and DataDAM do not scale well to large-scale datasets. Therefore, it is crucial to design a new algorithm to compensate for the information loss caused by RDED through a series of improvements, ensuring good generalization ability on both small and large-scale datasets.\n\nThe core novelty of our work is a comprehensive analysis of the limitations of past algorithms, followed by the proposed EDC that generalizes well to both small-scale (e.g., CIFAR-10/100) and large-scale (e.g., ImageNet-1k) datasets. No previous algorithm has performed well on both types of datasets. Up to now, EDC's performance remains the highest on CIFAR-10/100, Tiny-ImageNet, and ImageNet-1k, as you can confirm by comparing the performance of any paper in the field of dataset distillation.\n\nIn addition, this is the first time we focus simultaneously on data synthesis, soft label generation, and post-evaluation, whereas past algorithms generally focused only on data synthesis. Specifically for design choices, the core contributions of this paper include the real image initialization, soft category-aware matching, and the discussion and explanation of smoothing LR schedule and small batch size.\n\n**Q6**: _The relationship between two levels of limitation?_\n\n**A6**: As we replied in **A3**, ensuring “effectiveness, accuracy, etc.” is our overall goal, while “realism, matching mechanism, etc.” are the specific technical issues we need to address. The relationship between the two is that by guaranteeing realism and designing a more advanced matching mechanism (i.e., soft category-aware matching), we can achieve greater effectiveness and accuracy.\n\n**Q7**: _The major limitation is the organization of the paper, though I think that this seems to be a solid work._\n\n**A7**: Thank you for your efforts and acknowledgement. We apologize for any confusion caused by the organization of our lines. In the future, we will correlate the two levels of limitations more clearly in the main text and add ``Related Work Section'' in the appendix for additional clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gv8Dyq6eX7", "reviewer": "Reviewer_RdSF", "summary": "The paper explores scalable dataset condensation techniques, introducing Elucidate Dataset Condensation (EDC) which integrates multiple design strategies such as soft category-aware matching and learning rate adjustments. These methods achieve state-of-the-art accuracy across different datasets, demonstrating improved efficacy and efficiency over previous methods.", "questions": "See weaknesses part.", "limitations": "The limitations and societal impacts are not discussed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper conducts a thorough investigation into effective strategies for broadening the design space of dataset distillation while also reducing computational costs. These strategies are underpinned by solid theoretical support, enhancing the robustness of the proposed approaches.\n2. Empirical results demonstrate substantial improvements across various datasets and models, underscoring the practical efficacy and applicability of the proposed methods.", "weaknesses": "1. The comparison experiments presented are not sufficiently comprehensive. Given that the baseline method RDED (which represents the state-of-the-art with convolutional architecture) was used, it is essential to supplement the comparison experiments of the proposed method with other methods that also utilize convolutional architectures. This would provide a more thorough evaluation of the proposed approach against the full spectrum of existing techniques.\n2. The provided code cannot be executed as it lacks several necessary packages and related details. Detailed instructions, including a complete list of dependencies and environment setup guidelines, are essential to ensure the reproducibility of the results and to enable other researchers to effectively utilize and build upon this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and acknowledgement on the theoretical contributions of our work, and for sharing valuable suggestions. We hope we have addressed your concerns.\n\n**Q1:** _The comparison experiments presented are not sufficiently comprehensive._\n\n**A1:** Thank you for your valuable suggestions. Due to space constraints in the main paper and for aesthetic reasons, we have not fully presented the experimental results of other methods. However, since the benchmark for dataset distillation is uniform and well-recognized, the performance of other algorithms can be found in their respective papers. We present the related experimental results of the popular convolutional architecture ResNet-18 in the following table:\n\n| Dataset       | IPC | MTT           | TESLA         | SRe^2L         | G-VBSM        | CDA           | WMDD          | RDED           | EDC (Ours)      |\n|---------------|:-----:|:---------------:|:---------------:|:----------------:|:---------------:|:---------------:|:---------------:|:----------------:|:-----------------:|\n| CIFAR-10      | 1   | -             | -             | -              | -             | -             | -             | 22.9 ± 0.4     | **32.6 ± 0.1**  |\n|               | 10  | 46.1 ± 1.4    | 48.9 ± 2.2    | 27.2 ± 0.4     | 53.5 ± 0.6    | -             | -             | 37.1 ± 0.3     | **79.1 ± 0.3**  |\n|               | 50  | -             | -             | 47.5 ± 0.5     | 59.2 ± 0.4    | -             | -             | 62.1 ± 0.1     | **87.0 ± 0.1**  |\n| CIFAR-100     | 1   | -             | -             | 2.0 ± 0.2      | 25.9 ± 0.5    | -             | -             | 11.0 ± 0.3     | **39.7 ± 0.1**  |\n|               | 10  | 26.8 ± 0.6    | 27.1 ± 0.7    | 31.6 ± 0.5     | 59.5 ± 0.4    | -             | -             | 42.6 ± 0.2     | **63.7 ± 0.3**  |\n|               | 50  | -             | -             | 49.5 ± 0.3     | 65.0 ± 0.5    | -             | -             | 62.6 ± 0.1     | **68.6 ± 0.2**  |\n| Tiny-ImageNet | 1   | -             | -             | -              | -             | -             | 7.6 ± 0.2     | 9.7 ± 0.4      | **39.2 ± 0.4**  |\n|               | 10  | -             | -             | -              | -             | -             | 41.8 ± 0.1    | 41.9 ± 0.2     | **51.2 ± 0.5**  |\n|               | 50  | 28.0 ± 0.3    | -             | 41.1 ± 0.4     | 47.6 ± 0.3    | 48.7          | **59.4 ± 0.5**| 58.2 ± 0.1     | 57.2 ± 0.2      |\n| ImageNet-10   | 1   | -             | -             | -              | -             | -             | -             | 24.9 ± 0.5     | **45.2 ± 0.2**  |\n|               | 10  | -             | -             | -              | -             | -             | -             | 53.3 ± 0.1     | **63.4 ± 0.2**  |\n|               | 50  | -             | -             | -              | -             | -             | -             | 75.5 ± 0.5     | **82.2 ± 0.1**  |\n| ImageNet-1k   | 1   | -             | -             | -              | -             | -             | 3.2 ± 0.3     | 6.6 ± 0.2      | **12.8 ± 0.1**  |\n|               | 10  | -             | 17.8 ± 1.3    | 21.3 ± 0.6     | 31.4 ± 0.5    | -             | 38.2 ± 0.2    | 42.0 ± 0.1     | **48.6 ± 0.3**  |\n|               | 50  | -             | 27.9 ± 1.2    | 46.8 ± 0.2     | 51.8 ± 0.4    | 53.5          | 57.6 ± 0.5    | 56.5 ± 0.1     | **58.0 ± 0.2**  |\n\n\n**Q2:** _The provided code cannot be executed._\n\n**A2:** We apologize for any distress our oversight may have caused. We have shared the link anonymously: [EDC](https://drive.google.com/file/d/1jQxgR3rqd8V0J3316oqCSx4yhjTBB2XU/view?usp=sharing). Additionally, we have included instructions and pre-stored statistics in the new code to allow you to follow the steps and run it directly.\n\n**Q3:** _The limitations and societal impacts are not discussed in the paper._\n\n**A3:** Thank you for highlighting this issue. In the original paper, we included the limitations and societal impacts in the appendix. In future versions, we will place these sections after the conclusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions. We are pleased that our work received positive evaluations, with comments such as  "Comprehensive Analysis (gMw3)", "A theoretical analysis is conducted (r3SQ)", "Solid theoretical analysis (ZPCz), "conducts a thorough investigation into effective strategies for broadening the design space of dataset distillation while also reducing computational costs (RdSF)."

The reviewers also point out important suggestions that: 

1. The structure of the paper is a bit confusing. 

2. The definition of "generalized data synthesis" and some other parts of the paper are not clear. 

3. The submitted code does not execute properly. 

4. If EDC could scale to larger datasets and to other domains besides images.

Through this rebuttal, we aim to address unclear aspects of the presentation and typography, provide directly runnable code [EDC](https://drive.google.com/file/d/1jQxgR3rqd8V0J3316oqCSx4yhjTBB2XU/view) (a bit large because it contains pre-stored statistics ), and demonstrate EDC's ability to generalize over the ImageNet-21k-P (larger dataset) and Graph (other domains) datasets. Additionally, we will revise the manuscript by incorporating the detailed comments from the reviewers in our revision.

Meta-review: The paper studies dataset distillation and shows that with several proposed techniques, it is possible to achieve a remarkable 48.6% accuracy on ImageNet-1K with IPC=10, which substantially surpasses prior state-of-the-art results. While the degree of novelty may be somewhat limited, the reviewers unanimously commend the paper's thorough exploration of the shortcomings inherent in existing dataset distillation algorithms and its introduction of several effective methods to address these limitations. Therefore, we recommend acceptance of this paper.
Decision: Accept (poster)