Diverse Condensed Data Generation via Class Preserving Distribution Matching
Authors: Dandan Guo
Zhuo Li
He Zhao
Mingyuan Zhou
Hongyuan Zha
Venue: Accepted by TMLR ()

Abstract: Large-scale datasets for training many real-world machine learning models pose significant computational resource challenges. One approach to mitigate this is via data condensation, which aims at learning a small dataset but still sufficiently capturing the rich information in the original one. Most of existing approaches learn the condensed dataset and task-related model parameters (e.g., classifier) in a bi-level meta-learning way. The recently proposed distribution matching (DM), however, avoids the expensive bi-level optimization but ignores task-related models. This work proposes a novel class preserving DM framework consisting of two key components. The first one is responsible for capturing the original data distribution of each class based on energy distance, which can encourage the diversity in the generated synthetic data. The other is classifier-critic constraint, which forces the learned synthetic samples to fit pre-trained task-related models, such as an off-the-shelf classifier. Designing the optimization loss in this way, we can generate more diverse and class preserving distilled data without the bi-level optimization. Extensive experiments reveal that our method can produce more effective condensed data for downstream tasks with less training cost and can also be successfully applied to de-biased dataset condensation.

Published in Transactions on Machine Learning Research (July/2025,
Diverse Condensed Data Generation via Class Preserving Distribution Matching
Dandan Guo
quodandon@jlu.edu
School of Artificial Intelligence Jilin University
Zhuo Li
shuok3@likk.chhk edu.cc
Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen
He Zhao
he.zhao he::hhaoddttob.ccsiv
CSIRO's Data61, Australia
Mingyuan Zhou
miteguan.hoou@mooomls edu
The University of Texas at Austin, USA
Hongyuan Zha
zhaky edu.
The Chinese University of Hong Kong, Shenzhen
Reviewed on OpenReview: https: openreview. net/ per/forum? id- Q0ramDQYovu
Abstract
Large-scale datasets for training many real-world machine learning models pose significant computational resource challenges. One approach to mitigate this is via data condensation, which aims at learning small dataset but still sufficiently capturing the rich information in the original one. Most of the existing approaches learn the condensed dataset. and task- related model parameters (e.g., classifier) in bi-level meta-learning way. The recently proposed distribution matching (DM), however, avoids the expensive bi-level optimization but ignores task-related models. This work proposes a novel class preserving DM framework consisting of two key components. The first one is responsible for capturing the original data distribution of each class based on energy distance, which can encourage diversity in The generated synthetic data. The other is the classífier-critic constraint, which forces the learned synthetic samples to fit pre-trained task-related models, such as an off-the-shelf classifier. By designing the optimization loss in this way, we can generate more diverse and class-prest preespressrving distilled data without bi-level optinization. Extensive experiments reveal that our method can produce more effective condensed data for downstream tasks with less training cost and can also be successfully applied to de- de-biased dataset condensation
1 Introduction
Deep neural networks (DNNs) have demonstrated unprecodented results in many applications and come with a cost: the training of DNNs heavily relies on the sheer amount of data, sometimes up to tens of millions of samples. Although it becomes easier than ever to construct large scale datasets with advanced data collection and labeling tools, the rapidly growing size of datasets not only posts challenges to data storage and preprocessing, but also makes increasingly expensive to train a model on the given largo-scale
* *Corresponding author
Code is available httpp://githuss wttp:ttwww..coo///////tttt Bww.llaatttttttteee Dasaset__Condenattinn

---

Published in Transactions on Machine Learning Research (025)
dataset. More importantly, designing new deep learning models applying them to new tasks certainly require substantially more computations, as they involve to train multiple models on the same dataset for many times to verify the design choices, such as loss functions, architectures and hyperparameters (Cui ct al. 2022 Ying et al. 2019 Lee et al. 2022p. As a result, there is a strong demand for techniques that compress a large-scale dataset into a small subset of informative examples.
Numerous research endeavours have, therefore, focused on alleviating the cumbersome training process through constructing small training sets. One well-studied approach is referred to coresct or subset selection (Agarwal et al.) 2004 Sener & Savarese 2018), which chooses important data points for training based on some hcuristic critcria. However, most selection procedures incrementally and groedily select sam- ples, which are shortsighted and do not guarantee any optimal solution for the downstream tasks. Besides, the presence of representative samples is not guaranteed (Zhao & Bilen
2021 Zhao et al., 2021). Instead
of selecting from existing data points, recently, dataset condensation (DC) (Wang et al. 20118] Zhao et al.
2021
Ng guyen et al., 2021a b Zhao Bilen 2021 2023
Lee et al.,
2022 Cazenavette et al. 2022 Wang
et al.] 2022) has emerged as a competitive alternative with promising results, which aims to condense a
small training set S from a large-scale one T so that models trained on S can generalize to test data. Once the condensed dataset is learned, one can use it for various downstream classification applications, such as implementing neural architecture search (NAS) (Dong & Yang) 2020). Compared to performing NAS with the original large-scale dataset, using the small condensed dataset for capturing the rich information in the original one does save computational resources, which is the goal of data condensation.
Along this line, meta-learning- based strategy is proposed (Wang et al. 2018) followed by lots of data condensation methods. Among them, methods employing feature distribution matching (DM) (Zhao & Zhao et al. 2023) have gained prominence due to their ability to maintain competitive accuracy while requiring relatively low computational resources. Different from bi-level optimization based frameworks
Bilen
2023
(Zhao et al.
2021
Zhao & Bilen 2021 Lee et al., 2022; Cazenavette et al. 2022 Wang et al. 2022 Loo
ct al. 2022 2023; Feng et al.) 2023), DM-based techniques climinate the requirement for nested optimization.
In terms of DM-based methodss synthesized dataset is usually optimized by minimizing the distributing
distance between real dataset and synthetic synthetic dataset. Therefore, a key challenge in DM is how to design an effective metric for mcasuring the distributional distance. For example, Zhao & Bilen
2023 Zhao 2012 as the
et al. (023); Zhang et al. 20244 adopt Maximum Mean Discrepancy (MMD) (Gretton et al.
metrie; Yin et al. (2023) propose a matching metric based on batch normalization statistics; and Wang et al. (2025) introduce Neural Characteristic Function Discrepancy.
In this paper, we stress that a condensed dataset should preserve the class information of the original dataset and its samples should be diverse enough to represent the original dataset, so that a model trained on the condensed dataset is generalizable to test data. Taking the DM (Zhao & Bilen 2023) as the example, it generates condensed dataset independently for each class, without guarantecing that the synthetic samples actually belong to the class. In other words, the synthetic samples might ignore some discriminative features of their corresponding class, which will harm their availability in various downstream classification tasks. Moreover, the MMD distance used in DM ignores the diversity of the generated samples. As stated by Sun et al. (2024), a high-quality should cover a wide range of samples and labels, which is essential for robust learning and generalization. That is to say, diversity is a key point when we learning the synthetic set since we expect each condensed sample plays a different role for reducing redundancy. To this end, we propose a new distribution matching framework to satisfy above two factors simultaneously. To enhance the plese-igg property of the condensed datasets, inspired by the idea of plug-and-play generative models (Nguyen et al.) 2015; 2017), we propose to pre-train an off-the-shelf model to capture the class information in the training set and then use it to derive a classifier massifier-critic regularization on the synthetic dataset. Designed in this way, the learning of classification model and synthetic set is disentangled, which avoids the bi-level optimization. However, moving beyond DM, we can preserve the class information of the original dataset by learning the condensed samples from the view of the classifier-critic constraint, To encourage the synthetic samples to be more diverse, we introduce to minimize the energy distance (Székely et al. 2007 Rizzo & Székely 2016) between the synthetic samples and the real ones. In addition to making synth samples close to the real samples, energy distance enjoys an intrinsic mechanism to encourage the synthetic samples to be different from each other. Due to its flexibility, our proposed loss can be easily
2

---

Published in Transactions on Machine Learning Research (02l)
combined with most of existing distribution matching methods, which can further enhance the quality of synthetic datasets. In the comprehensive experiments, ours achieves the competing performance on standard dataset condensation tasks compared with other related methods. In addition, our proposed classifier-critic constraint enables our method to generate synthetic samples that preserve other necessary information from the original dataset, such as in the case of de-biased dataset condensation.
We summarize our contributions as follows: (1) We introduce energy distance to as the DM loss, for measuring the distributional distance between real and synthetic samples. (2) We propose class preserving distribution matching approach for optimizing the condensed dataset, where introduce a classifier-crite regularization and combine it with DM loss. (3) As plug-and-play matching loss, we instantiate how to combine ours with classical or recent popular data condensation methods. (4) We conduct extensive experiments under diverse scenarios and consider de- biased dataset condensation due to the flexibility of classifier-cittic constraint.
2 Related Work
2.1 Data Condensation Methods
Bi-level dataset distillation methods formulate dataset condensation as nested optimization problem, where the task model is trained in the inner loop and the synthetic data is optimized in the outer loop. Among them, a representative class of such methods is Meta-Model Matching, which directly optimizes the transferability or generalization performance of models trained on synthetic data. Typical approaches include Dataset Distillation (DD) (Wang et al. 2018), Kernel Inducing Points (KIP) (Nguyen et al. 2021b), FRePo (Zhou et al. 2022), , and LinBa (Deng & Russakovaky 2022 Gradient Matching aims to optimize the synthotic dataset by matching gradicnts between real and synthetic data, such as Dataset Condensation (DC) (Zhao et. al., 2021), Disferentiable Siamese Augmentation (DSA) (Zhao
Bi-level Optimization Methods.
& Bilen
Dataset Condensation (IDC) (Kim et al., 2022). Trajectory Matching matches the training trajectories of models trained on original and synthetic data in multiple steps, such as MTT |Cazenavette et al. 2022) and TESLA
2021), DC with contrastive signals (DCC) (Lee et al.) 2022) and Information-intern
(Cui et al.
2023 FTD (Du et al. 2023), ATT Liu et al.] (2024) etc. To improve efficiency, (Too et al.
2022 2023 introduce random feature approximations and convexified implicit gradients; (Feng et al. 2023
simplifies unrolled optimization to reduce computational cost while retaining effectiveness.
Distribution Matching Methods. Different from bi-level optimization methods, DM methods aim to align the statistical or structural distributions of real and synthetic data, without involving the nested model optimization. Some methods, including DD, Zhao & Bilen (2023), M3D (Zhang et al. 2024)and IDM (Zhao et al.] 2003),, aim to minimize the maximum mean discrepancy (MMD) between synthetic and real datasots. Some mothods also align fcature distributions without explicitly optimizing statistical distances, such as CAFE (Wang et al. 2022), Datadam (Sajedi et al. 2023), IID (Deng et al. 2024). Yin el al. (2023) propose to match batch normalization statistics and relabeling synthetic data using pretrained classifiers. Wang et al. (2025) reformulate distribution matching as minmax optimization problem and propose neural characteristic function discrepancy (NCFD) as the distance measure between datapoints.
Ours falls into the group of distribution matching. Tn contrast to existing DM methods, which either minimize MMD (e.g., in DD, M3D, IDM) or align feature distributions implicitly (e.g., CAFE, Datadam, IID), our method introduces a flexible and fundamental loss based on Energy Distance (ED). ED can simultancously encourage real-synthetic alignment and intra-synthetic diversity. To further enhance class separability and optimization stability, we incorporate a lightweight classifier loss as a regularization term. As an explicit loss, our proposed method is flexible, compatible with learnable feature mappings like NCFD, and can also be easily integrated as an auxiliary loss into broader optimization frameworks.
Diverse Dataset Condensation Methods. Recent works have highlighted the importance of diversity in dataset condensation to avoid mode collapse and improve generalization. MMDif (Gu el al. (024) formulates distillation as a minimax optimization over a diffusion- -nnppired potential, encouraging synthetic samples to spread across the data manifold efficiently. DSDM (Li ct al., 2024) introduces class- wisc semantic distribution matching to enhance diversity. It aligns real and synthetic feature distributions while promoting intra-class variation in the semantic space. RDED (Sun et al. 2024) balances sample diversity and realism via
3

---

Published in Transactions on Machine Learning Research (02l)
dual-objective loss, combining inter-sample repulsion with perceptual realism constraints from pretrained discriminators. Compared to these, our method promotes diversity via the repulsive term in the energy distance, while maintaining semantic consistency through classifier-guided constraint and the matching part in energy distance. Importantly, our design modular and can be flexibly combined with various objectives, without relying on generators or bi-level training.
2.2 Selection-based Methods
The classic tochnique to compress the training sct corcsct or subset selection (Agarwal al., 2004 Chen et al. 2010 Wei al. 2015). In addition to one naive method, which randomly picks data from the original dataset, most of these methods incrementally select important data points based some heuristic selection criteria. For example, (Sener & Savarosc 2018) select data points such that the largest distance between a data point and its nearest center is minimized. (Aljundi et al. 2019) use the parameter gradient as the feature to maximize the diversity of samples in the replay buffer. However, these heuristic selection criteria cannot ensure that the selected subset is optimal for the downstream task, especially for training DNNs. Besides, the information in the datasct usually uniformly distributed over all samples, thus finding such informative coreset. may not always be possible (Zhao & Bilen 2021 Zhao al. 2021
2.3 Plug and Play Generative models
Our work is also closely related to generative models such variational auto-encoder (Kingma & Welling, 2014) and generative adversarial networks (GANs) (Goodfellow et al. 2014). The difference is that image generation aims to generate real-looking and high-fidelh fidelityyyy images that can fool human eyes. As explored by (Zhao et al. 2021), the images produced by GANs have similar performances to those randomly selected real images and are usually weaker than the dataset condensation methods. Another related work is plug- and-play generative networks (PPGN) (Nguyen et al.] 2017; Graikos et al.) (022), which composes of 1) a generator for drawing wide range of image types, and 2) replaceable "condition" " network that tells the generator what to draw, similar to classifier-critic constraint. Despite of the recent community interest in PPGN, the possibility of the constraint in dalaset condensation has not been explored. Instead of producing "real-looking" samples under some constraints, we aim to generate condensed informative training set for the target task.
3 Preliminary
3.1 Energy distance
Energy distance is a statistical distance between probability distributions (Rizzo and Szekely ,2016). For two independent random vectors X and Y in Rd, the energy distance between them is defined as
eX, Y) 2F X YI| F - X'I F Y Y
where E means expectation operator, ||- is the Euclidean norm, X| < oo, E| < 00, X' and Y' denote the independent and identically distributed (iid) copy of X and Y. respectively.
A significant advantage of the energy distance is that e(X, Y)- 0 if and only if X and Y are identically distributed. Thus, the energy distance can be used for lesting of equal distributions multivariale goodness- of-fit measure. Another advantage of the energy distance is that it is distribution free. That is to say, the estimated value of energy distance does not depend on the distribution form of random vectors, although it can be represented as the form of characteristic function. Thcrefore, the energy distance can be estimated with the following surprisingly simple form. Denoting X1...., Xn, as the samples of X and y1...., Ynz as the samples of Y, we can estimate the energy distance as follows
BI
D2
B1
122
BJ D2
nn ( Y) = 717 i=1
x
yyll
X4
X,
Σ
y: Vy-yl yyll
(1)
7
76
j=1
4

---

Published in Transactions on Machine Learning Research (02))
3.2 Previous Works: Distribution Matching Methods for Dataset Condensation
Denote a large-scale dataset as T {a,, 343 the small (synthetic) dataset as S= { s/ ,, i=1' We can further represent the real set and syuthetic set in
[T] with [T] image and label pairs and C/ classes, and denote
class c as Te and Se: respectively. DM (Zhao & Bilen 2023) learn Se by minimizing the empirical estimate of MMD wetw the real and synthetic samples of the class c:
s. Σ fo (a.) - Seel fo (sy)
ITal
Pa
(2)
Tel
where fo is the embedding function parameterized with 0 sampled from Pe. |Tel and [Sel is the number of samples of To and Se. DM obtains the whole synthetic dataset S by summarizing Se in each class. Different from others (Zhao & Bilen 2021 Zhao et al. 2021 Wang et al. 2022). DM avoids the expensive bi-level optimization and second-order derivative with promising performance. However, DM ignores whether the synthetic set S. can actually be helpful to the classification of the class c when optimizing them. Moreover, DM ignores the diversity of distilled samples when using MMD and produces undesired performance with a slightly larger compression ratio.
4 Method
4.1 Proposed Method:
Class-preserving Distribution Matching
In this work, we explore novel efficient dataset condensation approach with the following intuition:
condensed synthetic set is desired if the generated samples can not only match the distribution of origi- nal training set but also can be confidently classified its corresponding labels by a well-trained classifier. Therefore, we introduce a distribution matching loss and a classifier-ciith constraint to learn the condensed dataset S, described below in detail.
4.2 Classifier-critic Constraint
Existing dataset condensation methods put their attention on the standard dataset condensation task, where they assume the original large-scale training set is high-quality without subpopulation shift or other prob- lems. However, datasets in real world are usually not perfect. Below, we first introduce the classifier-critic constraint for standard (commonly-used) dataset condensation and then explain how to design the corre- sponding classifier-critic constraint for de-biased dataset condensation.
Standard dataset condensation. Specifically, we can assume 96 is a classifier pre-trained on the training set T, e.g., based on the Cross-Entropy (CE) loss. Then we can use the given classifier Is to design classifier- critic regularization with CE loss when optimizing each synthetic samples s with label y, formulated as:
f= 1= log Pr(y s:
where Jø is frozen during the learning of S since it already has the acceptable classification ability. In addition to using a classifier pre-trained on T, we can also adopt existing available classifers, such as the powerful Contrastive Language" CImage Pre-training (CLIP) model (Radford et al., 2021). which efficiently learns visual concepts from natural language supervision based a sufficiently large datasct rather than T. This constraint forces the to-be-learned synthetic samples s being classified into its corresponding label y using the pre-trained classifier 9p. That is to say, consider distilling the discriminative information beneficial for the classification task into the synthetic set S.
De-biased dataset condensation. Subpopulation shift widely exists in many real-world applications. where subpopulations are seen but underrepresented in the training data (Han et al. 2022 Yao et al, 2022
such as fairness of machine learning and class imbalance (Hashimoto et al.) 2018 Japkowicz 20000. Then models may perform poorly when they falsely rely on the spurious correlation between the particular subpop- ulation and the label. Thus, a practical research is condensing training dataset with biased demographic
5

---

Published in Transactions on Machine Learning Research (02l)
subpopulations into small de-biased synthetic dataset. Here A denotes the number of spurious attributes like gender and ethnicity, where we can represent the attribute vector of synthetic sample s as a. Now we can pre-train two classifiers on original dataset T, where one is the label classifier 9d, for predicting class while other is spurious attribute classifier 9ø2- To enforce the presence of label and absence of the spurious altribule, we can define the classífier-critic as follows:
log Pr(y 8: 8: ) + log Pr(a 8; goz).
By minimizing the CE loss about label and maximizing the CE loss about the spurious attribute, we can encourage the synthetic image to be classified as its label and not be rocognized as its spurious attribute. It reduces the spurious correlations of label on subpopulation group in condensed samples.
4.3 Energy Distance between S. and To
Enforcing the matching between Se and Te is also important, otherwise we might encounter "spurious" examples that have high classification performance only under pre-trained classifier but ignore the structure in data, which may cause poor generalization performance when using S to train downstream classification lasks. Different from generative models that aim to generates real-looking images, datasel condensation aims to accurately match the distribution of the real training data with limited synthetic data. Despite DM learning S with MMD and proving its effectiveness, it may ignore the diversity in the synthesizod samples, which plays a key role in reducing the redundancy of condensed samples. To this end, we propose learn S by minimizing the energy distance between the real and synthetic data distributions for each class. We first introduce the definition of energy distance (Rizzo & Székely 2016). Denoting X and Y as two independent random vectors in Rd, their energy distance is defined as:
e(X, Y) = 2E X - YII - E - X'I -
(5)
where || indicates the Euclidean norm, X|| < oo, Yl < 0o, X' and Y' denote the independent and identically distributed (iid) copy of and y, respectively. A significant advantage of the energy distance is that s(X, Y) = O if and only if X and Y are identically distributed.
In this work, taking the class the example, we have samples æ and from Te and Se, respectively. Now the energy distance between To and Se can be estimated as follows:
ITe SS
Sel
Te TT
2
1
(To,
d (x, ==
(B) 84)
(x,,
(6)
ST
S.
1-=1
1-=1
where we compute the distance function in embodding spaces following DM Zhao & Bilen 2023), í.e. d (ar, 8,) = [[f((r;) - (a,)))l with fo denoting the feature extractor sampled from Pg. The main difference between Eq. ( in DM and our adopted energy distance the presence of repulsive term between generated data, d (s;; s which encourages the condensed samples to capture the full distribution and improve the representativeness.
Discussion about why adding the energy distance can help improve diversity: Recall that our adopted energy distance in Eq (5) is composed of Σnj=1 d(zi,,z), Σij=1 d(si, and Eij=1 d(z,, K;), where the third term can be ignored for only computing the distance between real samples, i.e., Iy and 2y. The first term Σnj=, d(x,, sy) aims to minimize the distance between real and synthotic samples, which the attractive term similar to the empirical estimate of MMD in Eq (1). The second term Σxj=r d(si,s) sj) of ED in Eq (5) aims to maximize the distance between the condensed data points, playing the role of repulsive term. Therefore, the main difference between the adopted ED in Eq (5) and the empirical estimate of MMD in Eq (1) is the repulsive term. And we had conducted the ablation study to compare the ED and MMD (i.e., DM) in Table 2 in experiments, where ED produces superior performance than that of DM. Especially, ED outperforms DM by large margins with larger IPCs. It is reasonable since ED not only minimizes the distance between synthetic data and real datasct, but also maximizes the distance between synthetic samples. Therefore, we stress that ED enjoys an intrinsic mechanism to encourage the synthetic samples be different from each other, improving the diversity of synthetic samples.
6

---

Published in Transactions on Machine Learning Research JJuly20233
Algorithm 1 Workflow of our method.
Training set T, synthetic set S randomly initialized from T with corresponding labels, off-the- shelf classifier J6: feature extractor fe, mumber of classes C, max-iter and learning rate for in maxiter do Randomly sample 0 from Po and compute
Require:
Randomly sample a minibatch B ~ To and S. ~ S for each class c in C; Compute L=Σe (s(BT (e)) [Pr( @:))))
Update S - VVsc; end for Output: a synthetic dalaset S
4.4 Training Loss
Recall that we aim to learn the class preserving and diverse condensed samples, which can be solved by introducing a classifier-critic constraint and the energy distance between Te and Se. Therefore, to learn the synthetic set S. of class for standard (or de-biasod) dataset condensation, we can jointly minimize the energy distance in (6) and the CE loss in (3) (or in Therefore, the optimization loss can be formulated follows:
min L = Ea P, Sc)l XE llog Pr(y s;
(7)
where the trade-off hyper-paramcter À maxiter-t with t denoting the t-th iteration and maxiter denoting maxiter
the maximum number of iterations. We summarize the training process of standard dataset condensation
in hm 1 which targets at image classification problems. Following Zhao & Bilen 2023
2021 we
initialize the synthetic images using randomly sampled real images with corresponding labels before training. At each training iteration, we can randomly sample embedding function # from Po and a mini-batch of real samples for each class. Our method enforces the synt hetic data to assimilate the task-related information by a very flexible classíficr-critic constraint paramcterized with already-learned Φ, avoiding a bi-level optimization and second-order derivative. It is different from DM which ignores classifier-elltted model when learning the synthesized dataset, i.e. without considering the classification performance of the synthetic samples; and also different from other related work (Zhao et al. 2021 Wang et al. 2022: Zhao & Bilen 2021) that usually couples the learning of the synthesized set with the learning of task-related model at each training iteration. Besides, it improves the diversity of the condensed samples by introducing the energy distance.
5 Combination between Ours and Baselines
As discussed above, our proposed loss is flexible for being an explicit loss. Therefore, can be easily combined with other methods. Below, we consider two specific examples.
5.1 Combining Characteristic Function Distance with Energy Distance in Ours
Characteristic Function Distance (CFD). To better compare probability distributions, we consider the characteristic function distance (CFD) proposed by Wang et al. (2025) as an alternative to traditional pairwise distances. Given random variable Rd,, its characteristic function is defined as the Fourier transform of its probability distribution:
(t Ex [et""]]
te Rdd
(8)
This representation uniquely determines the distribution and has been widely used in distribution testing and generative modeling. Given two samples x and T, the characteristic function distance is defined as:
(t, = -))())))))))) (1(()) -0x(0))))))))
- (1))
+x(()) (FT(1)
In practice, this integral can be estimated by Monte Carlo sampling over t, and each input is passed through learnable feature extractor fo, enabling learnable and expressive distance modeling.

---

Published in Transactions on Machine Learning Research (02l)
CFD-enhanced Energy Distance. We propose to incorporate CFD into the empirical energy distance structure. Recall the empirical energy distance in Eq. 66 replace the Euclidean distance d(::)) with our proposed CFD and can replace the first term in Eq. 7 with following equation:
(Te. = Ea-Te (T, So,
(10)
- Ss C (s, 8'; fe, Ea d'; Se, v)
where (x, for v) is the characteristic function distance computed in the feature space:
Cr = S. (Boot))( (.
1 (()((t)))
(t(m)(t)- (()())()) (Fr(t:
(11)
where projection parameter &o governs the distribution or transformation of frequency inpuls 4, which are treated as learnable or fixed random variables; please see more details in Wang et al. (2025). The formulation in Eq. 10) inherits the structural benefits of energy distance and characteristic function distance.
5.2 Ours Serving as a Regularization Term
The M3D method [Zhang et al. 2024 performs dataset condensation by minimizing the discrepancy between real and synthetic data distributions a reproducing kernel Hilbert space (RKHS). Tt introduces an efficient Maximum Mean Discrepancy (MMD) loss to match higher-order statistics without relying on a classifier. Given real data batch T, and synthetic data batch S. for class c, the M3D loss is defined as:
(Ta,
Σ k(Fo(z), + SS E.
Σ ((Jo(o), e
2
Σ rETc,
S(((I)
S
(12)
where -) is a universal kernel function (c.g., Gaussian RBF kernel k(z,x') exp(->Xl x' 12)); please find more details from Zhang et al. (024).. While effective for global distribution matching, M3D lacks explicit mechanisms for semantic preservation and intra-class diversity. To address these, our proposed loss in Eq. 77 can serve as the regularization term of M3D. The resulting overall objective is:
Lental = LMaD + aL,
(13)
where a is trade-off hyperparameter and we set a = 1. This formulation unifies model-agnostic distribution matching with semantic consistency and diversity control. Besides, ours can also be combined with SRe?L that minimizing the distance of batch normalization statistics between real and synthetic datasets (Yin et al. 2023). Similar to M3D+OURS, SURT is implemented by introducing our loss as the regularization loss of SR when optimizing the condensed images.
6 Experiments
6.1 Datasets and Implementation Details
Datasets. For standard dataset condensation, we consider two widely adopted image datasets, including CIFARIO and CIFAR100. CIFARIO (Krizhevsky et al., 2009) and CIFARI00 (Krizhevsky et al. 2009) consists of tiny colored natural images from 10 and 100 categories, respectively. Each datasct has 50K training images and 10K test images with the size of 32 32. Besides, following Yin et al.) (2023), we also evaluate our proposed method on large-scale TinylmageNet. TinylmageNet (Le & Yang. 2015 incorporates 200
classes derived from ImagoNet1K, with each class comprising 500 images processing resolution 64 x 64.
Implementation Details. Following previous work the commonly-used DSA augmentation (Zhao & Bilen 2021) and learn 1/10/ 50 image(s) per class as (IPC) synthetic sets for CIFAR- 10/100 100 using the ConvNet architocture. The ConvNet includes thrce repeated convolutional blocks, and each block involves a 128 128-kernel convolution layer, instance normalization layer (Uyanov et al. 2016 ReLU activation function (Nair & Hinton 2010) and average pooling. For large-scale
ZZhao & Bilen 2023 Wang et al. 2025 we adopt
8

---

Published in Transactions on Machine Learning Research (July/2025,
Table 1: Comparison with baseline models on CIEAR-10/100. We cite the results from
Wang et al.
(0025
Category
C-FAR-10
CIFAR-100
Method
1 (0.02%)
10 (0.2%)
50 (1%)
1 (0.2%)
10 (2%)
50 (10%)
Random
14.4
26.0
43.421
4.220.
14.610s
30.0ze4
Traditional
Herding
21.521a
31.6107
40.41os
8.410a
17.310a
33.71os
Forgetting
13.5
23.311.
23.31
1.51na
15.1
30.51ea
Diverse
DSDM
45.0 45.0=o.4
66.5+0.3
75.8+0.s
19.5+0a
46.2+0.s
54.1+0a
DC
28.320
44.9195
53.910.
12.810a
25.21m.1
DSA
28.8+0.7
52.1+ua
60.6+a.s
13.9+u.
32.3+u.4
42.8+04
DCC
32, 32.9-o.
49.4+a.
61.6+0.
13.3+ua
30.6+0.
40.0+o.
MTT
16.31os
65.31o.7
71.610a
24.31oa
10.1
17.7102
Bi-level
TESLA
48.51os
66. 66.11as
72.610.
24.8104
11.710.
FTD
16.81o.a
66.61..
73.810a
25.21m.a
13.41oa
48.51oa
ATT
48.3=1.
67.7=0.
74.5+0.
26.
44.0am.
51.2
FrePo
46.8 46.8
65.5
71.7
28.7
42.520.2
44.3
CAFE
30.3
16.31oo
55.51n.
12.91sa
27.81m.a
37.910a
IDM
45. 45.6
58.6+a.
67.5+0.2
20.1
45.1
50.0
DM
26.0.o.
18.9106
63.01o.
11.400a
29.71ma
43.61o4
OURS
29.3
53.4
65.91o.3
12.810a
33.410.a
15.21o.
Distribution matching
M3D
45.3
63.51sa
69.91s.
26.216.
12.410a
50.91o7
M3D+OURS
45.9
63.2
71.01o.
27.61n.
13.21o.4
52.51o.
NCFM (Reproduced)
17.01oa
69.91os
77.801..4
30.01aa
18.51o.
54.21oa
NCFM + OURS
47.210s
69.61o.4
77.910.
30.31oo
48.7_o.
54.61n.
Whole Dataset
84. S+0.1
56.2+0.
datasets, we employ ResNet18 as the backbone of TinylmageNet Yin al. (2023). To evaluate the synthesis- based methods, learn 5 synthetic datasets and train 20 randomly initialized classifiers on each synthetic datasct; then report the mcan and standard deviation of 100 test accuracies. We set maxiter 40K and learn model parameters & using the original training set, which adopts the same architecture and setting with the downstream classifier in the standard dataset condensation. Besides, we also utilize CLIF to represent off off-the-shelf model. We defer the lcarning rate, mini-batch size and training details for Φ to Appendix A
Baselines. We compare our method to (1) coreset selection methods, including Random and K-Center (Sener & Savarese 2018). (2) Bi-level optimization methods, including DC (Zhao al.] 2021). DSA (Zhao & Bilen 2021). DCC [Lee et al.
2022),
MTT (Cazenavette al. 2022). TESLA (Cui et al. 2023 FTD
Du et al. 2023). ATT Liu et al.
20024
(3) Distribution matching methods, including DM (Zhao & Bilen
2023)
CAFE (Wang et al. 2022).
IDM
Zhao et al.) (023)., M3D (Zhang et al., 2ad SRe"T. (Yin
(4) Diverse condensation methods, including DSDM (Li et al., DSA augmentation is enabled during evaluation for all base methods to make a fair comparison. For the implementation details of the baselines, please refer to dataset condensation Benchmark (Cui et al. (022) for more details. Due the ficxibility of ours, we can also combine with other methods, where we use their official code. Here, we consider adding our proposed loss with the matching loss in M3D and SR denoted as OURS+M33 and OURS+ respectively, where set the coefficient for two losses as 1 without exhausted adjustment. Bosides, we can also use characteristic function to compute the wisn distance in energy distance, denoted as OURS+NCFM.
et al. 2023), NCFD (Wang et al.
2025).
2024). Following Cui et al. (2022).
6.2 Experimental Results on Standard Dataset Condensation
Results on CIFAR-10/100. We adopt common evaluation method for dataset condensation methods,
i.e. measuring the test accuracy of the neural networks trained on the condensed data. Here we report the performance comparison with different IPCs in Table | on CIFAR-10/100. While DSA and CAFE can condense more data-efficient samples when IPC-1, our method outperforms them at IPC-10/50. A
hhtps///ghhubbcmo//ppaa///C1

---

Published in Transactions on Machine Learning Research (02l)
similar phenomenon has been observed by DM. The possible reason is that DSA and CAFE on bigger synthetic data use limited iterations in the inner-loop model optimization to ensure scalability, making them less accurate. Despite both DM and ours adopting the distribution matching and avoiding the bi-level optimization, ours significantly outperforms DM on various scttings. It is not surprising since we introduce the classifier-critic regularization and energy distance, whose eveeetiveness will be further explored in ablation studies. Considering our proposed method is flexible, we can also combine ours with statc-of-the-art (SOTA) methods optimize the synthetic samples. For example, we can take M3D and NCFM as two examples and explore where ours can be combined with them. We find that combining ours can usually obtain performance gains in most settings. The possible reason for the improvement is that ours loss enforces the synthetic dataset to match the real large-scale dataset in the embedding space with energy distance and classifier-eritic constraint, which might be complementary to the representation matching in M3D and the characteristic function in NCFM. It demonstrates the desired generalization and flexibility of ours.
Table 2: Results of our method on Tiny-ImageNer MTT (ConvNetd) DM(ConvNett) Ours(ConNNet)) (ReeNee))))
Dataset
IPC
SRe'L + OURS(Re
50
28.04 0.3
22.72 0.3
28.1 0.2
41.1 + 0.4
44.28 0.56
Tiny-ImageNet
100
49.7 1 0.3
50.16 1 0.21
Results on Immg- We also evaluate ours on the large-scale datasets, where we consider Tiny- ImageNet following L by Yin et al. (2023) and adopt the same backbone. Since ours is so flexible that it can be combined with other methods, we consider combine it with SRe*L, denoted as SUURL. As shown in Table 2 we can find that introducing ours can obtain consistent boosts on performance. That is to say, matching energy distance between real and synthetic datasots in ours complementary to matching the statistics between real and synthetic datasets. It demonstrates that ours still effective even in largo-scale dataset with more diverse classes and larger backbone, proving its generalization and effectiveness.
Computational Efficiency. Considering the importance of training cost when optimizing the con- densed dataset, without loss of generality, we evaluated the training speed and GPU memory required by our method and several strong baselines across different datasets under identical distillation set- tings. As reported in Table 3 although trajectory matching based methods usually achieve better re- sults than distribution matching. they encounter out of memory (OOM) issucs at IPC = 50. Be- sides, ours introduce the modest increase in computation time than NCFM, which is acceptable for maintaining competitive efficiency, especially compared with bi-level trajectory matching methods. This indicates that our method strikes a practical balance between effectiveness and computational cost.
Table 3: Comparison of training speed (s/iter) and peak GPU memory (GB) on CIFAR-100 with a sin- gle NVIDIA A100 S0G. 0OM indicates oft-of-memory cases.
Learning Larger Synthetic Scts. Prior works mainly evaluate the condensed dataset of sizes up to 50 TPCs, such as 1% compression ratio for CI- FAR1O, which is rather extreme case. To get more informative subset in most practical ap- plications, we further explore the performance of condensation algorithms under larger compression ratios on CIFARIO, where we set IPC up to 1,000. As shown in Figure I the performance gain of ours over baselines obvious for IPCs larger than 100 and less than 500. When IPC > 500, the per- formance gap between data condensation methods
Speed [s/iter)
CPU Memory (GB)
Resource
IPC 10
IPC 50
IPC 10
IPC 50
MTT
1.92
00M
61.6
00M
FTD
1.68
0OM
61.4
OOM
TESLA
5.71
28.24
10.3
44.2
NCFM
1.33
1.36
1.6
2.0
NCTM+OURS
2.15
2.68
2.1
3.3
and the random baseline narrows, where ours still outperforms other competitors. Especially, our method needs only about 500 images for each class to reach the performance of basclines trained for IPC = 1000. All methods achieve similar performance to the random baseline when IPC=1000. The possible reason is that randomly selecting more samples will approach the whole dataset, which we consider as the upper-bound. In summary, our method can be successfully applied into realistic settings, where the synthetic data of differ- ent classes can be learned independently and in parallel for avoiding bi-level optimization and second-order derivative.
10

---

Published in Transactions on Machine Learning Research (02l)
Table 4: Ablation study on CIFARIO, where we only perform 1PC=10,50,100 for diversity- based loss due to
its degraded performance.
Method
10
50
100
200
300
DM DM KL DM + CLIP DM + Classifier
47.6440.55
61.99:10.33
65.12::..0
69.15:10.17
09.36:t0.35
48.43:1.228
62.39:0.28
66.3710.18
70.29:0.31
30.56
50.25=10223
63.07=0.05
66.30::0.15
71.26::0.26
71.83+00.42
50.34+0.01
64.79:+0.18
09.4440.22
72.45=00.33
73.54+0032
Diversity
31.02 + 0.74
41.06 0.29
55.61 + 0.37
Divensity + Classifier
39.91 + 0.51
48.58+ 0.48
62.98 + 0.33
ED ED + KL
50.71+0.30
63.30+0.33
67.88+00.25
71.92+0.31
03.85+00.44
51.12-+0.25
15.19-0.11
68.43-+0.02
72.88+0.25
74.01+00.77
ED + CLIP
50.34:0.22
64.002:0.24
69.01:1.22
7..27+0.63
75.11+0.33
ED Classifier
51.34110224
65.92:10.30
69.47=0.30
73.7310.21
76.02=0.26
6.3 Ablation Studies
Energy distance. To illustrate the effectiveness of our intro- duced energy distance, we compare it with another distribution matching method, i.e. DM (Zhao & Bilen 2023 Here we dis- card our proposed constraint for a fair comparison and denote ED as our docayed variant. As listed in Table ED produces superior performance than that of DM. Especially, ED outper- forms DM by large margins with larger IPCs. It is reasonable since ED not only minimizes the distance between synthetic data and whole-dataset, but also maximizes the diversity of synthetic samples. Besides, we also discard the first term (i.e., matching term between real dataset and synthetic dataset) in energy distance and we report their performance for IPC=10, 50,100. We can find that discarding the matching part can re- duce the performance a lot. This is because that the matching term guides the synthetic samples toward the real data distri- bution. When discarding the matching term, the resultant loss only pushes the synthetic samples to move away from each other in each class (via the repulsion term), without providing direction toward the target distribution. Those observations suggest that using the energy distance to match fcature distributions of the synthetic and original training images is effective dataset condensation.
Random
KCCenter
Text
DC
DM
OURS
Image Per Class (IPC)
Performance comparison CIFAR-10 under different IPCs using Ran- dom, K-Center, DC, DSA, DM and Ours.
Figure 1:
Classifier-critic constraint. To evaluate the effectiveness of our proposed classifier-cittic constraint for training the synthetic images, consider different ways to build the pre-trained classifier, including tradi- tional classifier trained on original training set and CL.IP Besides, we also implement the constraint following knowledge distillation (Hinton et al.) by minimizing the KL divergence KL (p B (B) where we keep the mini-batch size for real data in accord with the size of synthetic set. As listed in Table 4 adopting the CLIP, which trained on a public dataset instead of the target large-scale training set, is inferior to using the traditional classifier. The reason behind this might be that the traditional classifier is optimized by the original training set. However, replacing the traditional classifier with CLIP only reduce the performance little. Compared with ignoring the classification constraint, introducing CLIP to implement the classifier- critic result in remarkable performance improvement when IPC is equal greater than 50, where DM+CLIP is better than DM, ED+CLIP is better than ED, and ED+CLIP still outperforms DM+CLIP. Thus, even using existing available classifiers trained on different dalaset, such as CLIP model, ours can still achieve a desired performance. Besides, KL inferior to our method. The possible reason is that KL provides the soft label prodicated by classifier 9g on the real datasct as the target of the synthetic set but we use the ground-truth label. To sum up, introducing KL, CLIP or traditional classifier as the constraint usually result in remarkable performance improvement. Thus, claim that the classifier-critic con-
11

---

Published in Transactions on Machine Learning Research (ull)
straint synthetic images is key-point in data condensation settings; this highlights the contribution and originality of our method.
Table 5: Ablation study of hyperparameter À on CIFAR1O with IPC=10 and IPC=50.
A
10
0.1
0.01
macitur-t maxiter
IPC=10
51.26:10.38
50.3011.55
50.80-10.48
50.17=0.26 63.59-0.22
53.42.1..17 65.92-0.30
50.71=0.30
TPC=50
06.72-0.24
65.48+0.49
63.90+0.17
63.30+0.33
Hyperparameter X. The hyperparameter À in Eq. 7 indicates the importance of the log-likelihood of the task-related information in synthetic set, i.e. the weight of task-related loss. To explore the effect of À in our method, we design ablation study on CIFAR10 with IPC=10/50. As listed in Table 5 small X, such as 0.01, achieves slightly better or even worse performance than abandoning the task-related loss i.e.A = 0. Both settings of = 10 and À = maxiter-t can produce a better performance than others. Since we aim to use a much more general hyperparameter rather than searching perfect À for each dataset with each IPC jadedly, we set À = maxiter= That is to say, we wish learn synthetic set with more emphasis on p(C]S) during the carly training stage and pay more attention on p(S) during the late training stage. Our proposed method achieves desired performance in various tasks using this hyperparameter varying with the training iterations.
maxiter
maxiter
The efficiency of the CE loss during the train- ing. Recall that the coefficient of the additional CE loss is set to be gradually reduced during the train- ing. To further explore the efficiency of the CE loss during the whole training process, we compare the test performance learned by ours and DM with vary- ing number of training iterations, where we consider ours with CE loss and ours without CE loss. As shown in Figure [ we evaluate the performance of the condensed samples at each 2000 training itera- tions. We can find that ours with CE loss performs better than DM in the whole training stage. Be- sides, ours with CE loss outperforms ours without CE loss all the time. Therefore, CE loss is not only beneficial in the later phase of the training but also useful in the beginning phase.
DM OURS CE
64
DURS w/b OF
62-
Text accuracy
58
56-
2000 4000
6000
8000
10000 12000 14000 16000 18000 200n0 Training ikeraton
Figure 2: The test performance of ours (w/ CE loss or w/o CE loss) and DM on CIFAR10 (IPC=50) with the varying number of training iterations.
6.4 Transferability
Table 6: Testing accuracy of different methods on ConvNet versus transferred to other architectures, where 10/-10 The "Transfer" column records the average results on MLP, ResNet18, ResNet152 and VIT.
Dataset
CIFAR-10
CIFAR-100
TayllnggeNet
Method
IPC=50
IPC=10 10
IPC=50
IPC=10
IPC=50
IPC=10
ConwNet
Trausfer
CourNel
Trausfer
ConvNNe
Truudee
Net
Trazefer
CouvNNe
Transfer
ComcNel
Transer
Randoa
50.35
36.39
31.00
25.16
34.06
23.12
18.61
11.31
18.62
8.85
6.88
3.53
K-Center
66.00
41.19
38.64
265.1
15:31
3.42
DC
56.81
$1.12
50.99
32.22
30.56
14.95
25.
11.95
12.66
1.62
12.83
3.71
DSA
60:25
52.96
16.77
16.34
6.75
DM
61.09
40.04
47.64
30.06
42.12
25.22
29.23
13.59
22,76
9.75
13.51
4.08
OURS
65.92
41.55
53.42
31.74
44.90
26.83
32.80
16.22
28.10
23.24
16.41
7.13
Following CCui et al. 2022), we explore the transfer-
ability of different. condensation methods on the 3 datasets using 5 architectures, including ConvNet, MLP; ResNet18 (He et al. 2016), ResNet152 (He et al.) 2016) and ViT (Dosoviskkiy et al. 2020). More details about architectures and transfer results on each architecture can be found in the Appendix B. As shown
Table 6 although the performance of all methods drops when transferring other architectures, ours still produces a desired balance between the seen ConvNet and unseen architectures. When IPC=10 on CIFAR10
12

---

Published in Transactions on Machine Learning Research /025)
and IPC=50 on CIFAR100, ours is slightly worse than DC DSA. In addition to mentionen setings, ours achieves the better performance over the seen ConvNet and better-averaged genera lizatoizttiion performance over the unseen architectures. Remarkably, when LPC=50 on TinylmageNet, the performance gain of ours over DSA is about 14%. The possible reason is that learning the synthetic data from TinyImageNet is very challenging and ours extract more class-relevant information with less architectural inductive bias than other baselines; see the visualization below. Thus, our proposed method is reliable and generalizable whose condensed images using one architecture can be effectively used to train other unseen ones.
6.5 Neural Architecture Search (NAS)
NAS aims to automatically search for a top archi- tecture from a vast search space, which typically re- quires expensive training of numerous architectures multiple limes on the whole training set and pick- ing the best performing ones on validation set. It is beneficial that applying the smaller condensed dataset to NAS (Zhao et al.] 2021). Following the implementations of :Cui et al. 2022,, we randomly
Table 7: We implement NAS on CIFAR10 using NAS- Bench-201. The correlation of the original dataset is lower than 1.0 as we use a small architecture and per- form ranking based on the validation set.
Random
K-Center
DC
D8A
DM
OURS
Whole
Correlation 1
-0.06
0.11
-0.10
-0.37
-0.37
-0.17
0.7487
Top 1 (96)
91.9
91.78
86.44
73.54
92.16
02.47
90.5
sample 100 networks from NAS-Beuch-201 (Dong & Yang, 2020), which contains the ground-truth perfor- mance of 15,625 networks. We reduce the number of repeated blocks from 15 3 during the search phase, making the size of condensed dataset adapt the networks. All models are trained on CIFAR10 and IPC=50 for 50 epochs under 5 random seeds and ranked according to their average accuracy on a held-out validation set of 10k images. We consider two metrics 1) Spearman's 's rank correlation coefficient between the ranking of models trained on condensed dataset and the original dataset; 2) The ground-truth performance of the best architecture trained on the condensed dataset (Top 1). As we can see in Table 7 all methods produce negative correlation between the performance on condensed and full dataset except for K-Center, indicating the difficulty of utilizing the condensed dataset to guide model designs. Besides, ours still achieves rel- atively better trade-off between correlation and ground-truth performance than other methods. It verifies that our condensed images can be used to efficiently train multiple networks to identify the best network for saving computation resources.
6.6 Continual Learning (CL)
CL (Kirkpatrick et al. 2017 aims to address the catastrophic forgetting problem when the model learns soquentially from stream of tasks. Due to its highly condensed nature, distilled samples have been success- fully applied to the CL. We evaluate the effectiveness of the condensed dataset in CL. Following (Zhao & Bilen 2023), we set up the baseline based on GDumb (Prabhu et al. 2020) which stores class- balanced train- ing samples in memory greedily and trains a model from scratch on the latest memory only. We randomly and evenly split the 100 classes from CIFAR100 dataset into 5 steps, i.e. 20 classes per step respectively. The memory budget is 20 images/class for all seen classes. We compare our method with other baselines. Figure g shows that the proposed method is superior to other baselines. It indicates that method can pro- ducc a better and more informative condensed set for training models than those produced by competitors. Similarly, the results of 10 step class-incremental learning are shown in Figure 4
6.7 Visualization of Data Distribution
Considering data condensation methods synthesize small dataset, natural question is whether it intro- duces any bias into the data distribution. Therefore, we visualize the data distribution of the real dataset and the synthetic datasets by different methods in Figure 5 for CIFARIO with IPC-50. Specifically, we use the ResNet18 pretrained on the real dataset to extract features and visualize the features with T-SNE (Van der Maaten Hinton| 2008), where we plot the samples from the first class, second class and fourth class. We find that the synthetic images learned by baselines are usually biased to real image distribution, especially for DC and CAFF. However, ours can obtain more evenly distributed samples and cover the real
13

---

Published in Transactions on Machine Learning Research (225)
Test Accuracy (%)
70
DSA
RANDOM
Test Accuracy (%)
70
RANDOM
DSA
DM
DM
60
OURS
OURS
50
40
40
30
40
60
100
10
20 30
40
50
60
70
80
90
100
5-step: Number of classes
10-Step: Number of classes
Figure 3: Test accuracy of 5 steps continual learn- ing with condensed samples on CIFAR100 with IPC-20 for Random, DM, DSA and ours, respec- tively.
Figure 4: Test accuracy of 10 steps continual learn- ing with condensed samples on CIFAR100 with IPC-20 for Random, DM, DSA and ours, respec- tively.
Figure Visualization of synthetic dataset distribution (red points) and original dataset (blue points) on CIFARIO with IPC=50, where the first row is the first class, and the second row is the second class and the third row is the fourth class.
image distribution in a better way. thus proves the effectiveness of ours in above- ment lownssteaa tasks.
6.8 Debiased Dataset Condensation
Table 8: Performance on CelebA with different IPCs. dataset are 94.53 + 0.19 and 93.85 + 0.27, respectively.
The average and worst performance on the whole
Method
10
50
100
200
300
Avg.
Avg.
Arg.
Avg
Argg
Worst
RANDOM
80.30110.2
76.18-0.29
96.6820.13
77. 11.1111.11
53. $3.4510.23
81.69.10.5
88.06.20008
$1.66..0 .6
88. $8.29.20.20
$6.37.10 12
30.74::0.40
88.64400.2
ED
86.1811117
69. 1.11
06.86110.75
81.68.0336
58 18.80111.07
88.3820.88
90.61120.56
89.60100.3
90.72-0.05
$9.56.10 90
86.76-11.49
87.04:20.50
90.30:50.31
80.66:40.25
91.00-00.32
89.84.10.11
OURS
86.06:11.55
76.78.:1.68
87.85:10.06
46.48.10.48
89.54-00.56
58.36:10.28
90.71:10.65
89.71100.6
91.48.10.38
90.74110.09
14

---

Published in Transactions on Machine Learning Research (lul)
We examine whether ours can be used to compress biased training set into small de-biased synthetic dataset. Here we adopt the CelebA face dataset (Liu et. al. 2015) that includes 40 attributes. We use the hair color (blond, not - blond) as the target and gender (male, female) as the spurious attribute, where the label is spurionsly correlated with gender. We now have 4 groups: (blond, male), (not blond, femalc), (blond, female), (not. blond, male). To reduce the computational resource, we resize the original image into 64 x 64 and keep the proportions of groups in original dataset to randomly drop some samples, resulting in 10000 training examples with 85 in the smallest group (blond-haired males). See Appendix C for more details. We adopt the official test split to evaluate the methods and report the average and worst-group performance. We follow (Graikos et al.) 2022) that uses the pretrained ResNet18 face attribute classifier on CelebA to realize both Φ1 and Φ2. We consider random selection, DM, ED, and ED+label as our baselines, where ED+label adds the minimization about the CE loss of label in the distilled image to the ED. Ours further adds the maximization about the CE loss of spurious attribute to the ED+label. As listed in Table ours can generally achieve the better worst-group and average performance than its basclines, indicating the effectiveness of introduced constraints in de-biased dataset condensation. Besides, adding the constraint about the label usually leads to improved average performance. It is reasonable since ED+label pays more attention to all groups. These observations claim that introducing classifier-ciitie constraint about the specific task is effective in dataset condensation and can provide more applications. We also perform a data condensation on point cloud classification task using our proposed method and report the results in Appendix to validate the effectiveness of ours on different modalities.
6.9 Visualization of Condensed Dataset
Figure 6: Visualization of synthetic images on CIFARIO with IPC-50. Please zoom in for a better display.
We visualize the synthetic images on CIFAR10 with IPC=50 in Figure D Duc the limitod space, we defer the generated synthetic sets of CIFARIO with IPC=10 in Figure 7 of Appendix and the generated synthetic sets of CelebA with IPC=10 in Figure g of Appendix, respectively. With the development of IPC, it is easy to find that the synthetic images generated by our method is visually similar to original real CIFAR10 images but looks more dlassrreprseentatiie. Taking the Figure 6 (the visualization of synthetic images CIFAR10 with IPC=50) as the example, the 50 images per class results are diverse which cover the main variations, such as the various airplanes in the first row ("airplane" class). For CelebA, we consider the debiased dataset condensation, where the hair color (blond or not-blond) is the label and the gender (male or female) is the spurious attributes. That is to say, we aim to obtain a condensed face dataset with two classes, where one class is composed of blond hair face images and another is composed of non-blond hair face images and both of them can not keep the gender information in the images soon possible. As shown in Figure S of Appendix, we can see that the condensed can meet the expected constraints to a great extent.
7 Results on Other Modalities
To validate the effectiveness of ours different modalities, perform a data condensation point cloud classification task using our proposed method. Here we use preprocessed modelnet dataset (Wu et al. 2015), which contains 3981 training and 400 testing samples from 10 categories and each sample is com- posed of 10K points (x:y; z-coordinates:::GGa. fewer points (c.g., 2048) for each object. We compare random sampling and DM, our proposod ED, and ED+classifier to achieve fewer points. We randomly sample 100 training samples and condense each sample with 10K 3D points (x:y; z-coordinates) into synthetic sample with 64 3D points. We then utilize condensed
Researchers usually utilize random subsampling to achieve
15

---

Published in Transactions on Machine Learning Research (02l)
datasets to train the pointnet-+ for point cloud classification task, where set epoch=50. We adopt common evaluation method for dataset condensation methods, i.e. measuring the test accuracy of the neural networks trained on the condensed data, where we use all test samples and report instance accuracy (IA) and class accuracy (CA). The test performance is summarized in Table g Compared with random sampling and DM, ours can generate more data-efficient samples, indicating the effectiveness of our proposed data condensation method in 3D.
Table 9: The accuracy comparison of TA and CA for point cloud classification.
Method
Random
ED
DM
ED+cclssifeer
IA
51.92
62.87
62.20
67.24
CA
53.43
65.01
64.28
68.93
8
Conclusion
This work aims to compress the original training dataset into a condensed set via paaas--reerrinng distribution matching. To this end, we design the optimization method of synthetic samples from two key points. One is responsible for capturing the original data dist ributiio, where we introduce energy distance to increase the diversity. The other aims to introduce the task-related information, such as classification, into the learning of synthetic samples without a bi-level optimization. We introduce classifier-critic constraint based an off- the-shelf pre-trained classifier to implement the second goal. By minimizing the energy distance between real and synthetic samples and the classifier-critic constraint, we learn a diverse and class-preserving synthetic set with less training cost, avoiding an expensive bi-level optimization. Extensive experiments demonstrate the effectiveness of our proposed method in commonly used downstream tasks and the de-biased dataset condensation task.
9 Acknowledgment
This work was primarily conducted while Dandan Guo was a postdoctoral researcher The Chinese Uni- versity of Hong Kong, Shenzhen. The work of Dandan Guo was supported by the National Natural Science Foundation of China (NSFC) under Grant 62306125, This work of Hongyuan Zha supported by the National Natural Science Foundation of China (NSFC) under Grant 72495131.
References
Pankaj K Agarwal, Sariel Har-Peled, and Kasturi R Varadarajan. Approximating extent measures of points. Journal of the ACM (JACM), 51(4):606 635, 2004.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. volume 32, 2019.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distilla- tion by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Yutian Chen, Max Welling, and Alexander J. Smola. Super-samples from kernel herding. In UAI 2010, Proceedings of the Twenty-Sinth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, July 8-11, 2010, pp. 109-116. AUAI Press, 2010.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation benchmark. In Thitty-sizth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference Machine Learning, pp. PMT.R, 2023.
16

---

Published in Transactions on Machine Learning Research (02l)
Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang. Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pettern Recognition, pp. 17057 17066, 2024.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. Advances in Newral Information Processing Systems, 35:34391-34400 2022.
Xuanyi Dong and Yi Yang. Nas-bench-22 2011 Extending the scope of reproducible neural architecture search. arXiv preprint ar Xxxi:2001.00322 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Rep- resenlations, 2020.
Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2023.
Yunzhen Feng, Ramakrishna Vedantam, and Julia Kempe. Embarassingly simple dataset distillation. 2023.
Ian J. Goodfellow, Jean Ponget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farie Fardey Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models Slug-and--lay priors. In Advances in Neural Information Processing Systems, 2022.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723 773, 2012.
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 15793 15803, 2024.
Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopulation shift via uncertainty-aware mixup. In Advances in Neural Information Processing Systems, 2022.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang, Fairness without demo- graphics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929-1938.
PMLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network.
Nathalie Japkowicz. The class imbalance problem: Significance and strategies. In Proc. of the Intj l Conf. on Artificial Intelligence, volume 56, pp. 111-117. Citescer, 2000.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference Machine Learning, pp. 11102-11118. PMLR, 2022.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -3526, 2017.
17

---

Published in Transactions on Machine Learning Research (02))
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenel visual recognition challenge. CS 231N, 7(7):3 2015.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, volume 162, pp. 22352-12364. PMLR, 2022.
Hongeheng Li, Yucan Zhou, Xiaoyan Gu, Bo Li, and Weiping Wang. Diversified semantic distribution matching for dataset distillation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7542-7550, 2024.
Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, and Martin Schulz. Dataset distillation by automatic training trajectories. In European Conference on Computer Vision, pp. 334-334- Springer, 2024.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730- 2015.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 35:13877 13891, 2022.
Noel Loo, Ramin Hasami, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In International Conference on Machine Learning, pp. 22649 226744 PMLR, 2023.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are casily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional itcrative generation of images in latent space. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 4467-4477, 2017.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridgo-regression. Tn International Conference on Learning Representations, 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lec. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5198, 2021b.
Ameya Prabhu, Philip HS Torr, and Puneet. K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pp. 524 540. Springer, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International Conference Machine Learning, pp. 8748 8763. PMLR, 2021.
Maria L Rizzo and Gábor J Szćkely. Energy distance. wiley interdlsciplinary reviews: Computational statistics, 8(1):27-38, 2016.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos Platan- iotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVE /CVF International Conference on Computer Vision, pp. 17097-17107, 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations. OpenReview.net, 2018.
18

---

Published in Transactions on Machine Learning Research (2ly)
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9390 9399, 2024.
Gábor J Székely, Maria L Rizzo, and Nail K Bakirov. Measuring and testing dependence by correlation of distances. The annals of statistics, 35(6):2769-2 2794, 2007.
Dmitry Uyyano, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint 1r7:::107..08022, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning condense datasct by aligning features. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 12196-12200 2022.
Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang, Dataset distillation with neural characteristic function: A minmax perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2025.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. abs/18 0bs//181..1999, 2018.
CoRR,
Kai Wei, Rishabh Tyer, and Jeff Bilmes. Submodularity data subset selection and active learning. Tn International conference on machine learning, pp. 1954-1963. PMLR, 2015.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1922-1920, 2013.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. arXiv preprint arXiw:2201.002: 2022.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. Advances in Newral Information Processing Systems, 36:73582-73603, 2023.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reprodueible neural architecture search. In International Conference on Machine Learning, pp. 7105-7114. PMLR, 2019.
Hansong Zhang, Shikun Li, Pengiu Wang, Dan Zeng, and Shiming Ge. M3d: Dataset condensation by minimizing maximum mean discrepancy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 9314 2314-9322, 2024.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pp. 12674-12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Datasct condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514 14-26623, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. International Conference on Learning Representations, 2021.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset con- densation. In Proceedings of the IEEE/CVE /CF Conference on Computer Vision and Pottern Recognition, pp. 7856-7865, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiw:2206.00) 00719, 2022.
19

---

Published in Transactions on Machine Learning Research (22l)
A Detailed Implementation Settings
We list our used learning rate lor each dalaset with varying IPCs Table 10 and the mini-batch size of the real dataset in Table.
We also report the pretraining details for the model parameters on different datasets in Table 12 For standard dataset condensation task, indicates classifier and its training details can be found in 12 where we adopt the same architecture and settings with the downstream classification task. For the di- biased dataset condenstaion task, @ represents a face attribute classifier, we use well pretrained model following GGrikoo et al. 2022).
Table 10:
Learning rate on CIFAR-10, CIFAR-100, TinylmageNet and CelebA with different IPCs. To
obtain the results in Figure
we set the learning rate as 6 with LPC 2 300 on CIFAR-10.
IPC
CIFAR-10
CIFAR-100
TinyImageNet
CelebA
0.1
0.1
0.2
10
0.1
0.1
0.2
10
50
0.1
0.1
0.2
10
100
1
10
200
10
300
10
Table 11: Mini-batch size of real datasct CIFAR-10, CIFAR-100 and TinyImageNet with different IPCs.
IPC
Dataset
10
50
100
200
300
400
500
600
700
800
900
1000
CIFAR-10
512
512
512
512
512
512
512
512
256
256
64
64
64
CIFAR-100
512
512
512
NityImageNet
512
CelebA
512
512
512
512
512
Table 12: Training details when & indicates classifier on CIFAR-10, CIFAR-100 and TinyImageNel,, where BS denotes Batch Size, E denotes Epoch, IR denotes Learning rate, Optim denotes Optimizer, Mome denotes Momentum and WD denotes Weight Decay.
Data
Model
BS
LR
Scheduler
Optim
Mome
WD
CIFARIO
ConvNet
256
1000
0.01
STEPLR
SGD
0.9
5c-4
CIFAR100
ConvNet
256
1000
0.01
STEPLR
SGD
0.9
5c-4
Net
ConvNetD4
32
1000
0.01
STEPLR
SGD
0.9
5e-4
CelebA
ResNet18
256
1000
0.01
STEPLR
SGD
0.9
5e-4
B More Details about Transfer Experiments
Following (Cui et al. 2022). the evaluated network architectures are as follows:
ConvNet: A standard convolutional network which is classical in vision tasks, is used to extract features from the syuthetic and original datasets in previous works (Zhao & Bilen 2023 The structure includes three 3x3 convolutional layers followed by 2x2 avgpooling and instance normalization with hidden widths of 128 on CIFAR-10 and CIFAR-100. For Tiny Imagenet, a four layers network called ConvNett4 is used for better performance and ability in extracting features following (Cui et al. 2022). There are 0.32M trainable parameters in ConvNet and 0.45M in ConyNetD4, correspondingly.
MLP: A simple fully connected network with 3 layers and 128 neurons in each layer is considered to evaluate methods. There 0.41M trainable parameters in MT.P.
20

---

Published in Transactions on Machine Learning Research (025)
ResNet18/ResNetl33 Standrad architecture includes 4/50 residual blocks, resepectively. A ReLU acti- vation layer and batch normalization layer in each block follow two convolution layers. There are 11.17M trainable parameters in ResNet18 and 58.16M in ResNet152.
VIT: Vision Transformer is a new and competitive model architecture used in image processing, which is based on transformer structure. There are 10M trainable parameters in our VIT.
Previously, we show the transfer results in Table. with IPC-50. We further report the detailed transfer results on each architecture in Table. 14 for easier references.
C Details about the De-biased Dataset Condensation
We extract features and evaluate the synthetic images in ResNet where we the same architecture in transfer experiments, Refer to Table. 13 for details about dataset.
Table 13: Number of samples in different groups in training dataset and test dataset CelebA
Group
Blond Male
blond Male
Blond Famale
Not blond Famale
All
Original training dataset
1387
66874
22880
71629
162770
Our training dataset
85
4101
1405
4401
10000
Test dataset
180
2480
7536
9767
19963
Visualization of the Learned Synthetic Samples
We visualize the generated synthetic sets CIFAR10 with IPC=10 in Figurd respectively.
We visualize the generated synthetic sets of CelebA with IPC-10 in Figur respectively.
We can observe that the synthetic images can capture diverse appearances in the categories. Besides, the synthetic images in each class are less redundant.
21

---

Published in Transactions on Machine Learning Research (02l)
Figure 7: Visualization of synthetic images CIFAR10 with IPC=10.
Figure 8: Visualization of synthetic images on CelebA with IPC=10. Please zoom in for better display.
22

---

Published in Transactions on Machine Learning Research (ull)
Table 14: Comprehensive transferability results of different methods tested on ConvNet, MLP, ResNet18, ResNet152 and ViT with IPC 1, 10 and 50. We report testing accuracy.
Network ResNet18
Dataset
Method
TPC
ConvNet
MLP
ResNet152
VIT
15.40:10.28
14.37:10.38
16.56:10.46
14.19:10.99 12.15::180
Random
10
31.0010.48
25.08.10.27
20.52.10.87
15.84.1.911
26.21
50.55++0.32 50
25.21+0.44
47.26+0.27
23 36+2.31
39.73+0.52
1
25. 15.1620044 45
24.01-0.32
25.9910.57
1466111.0 30
21.541..55
K- K-Center
10
41.499-0773
32.92-0.38
40.0820.88
19.35=0.71
31.95+0.67
50
56.00:1..22
40.61-10.34
52.69
27.84:11007
44.65+10.39
29.34++0.37
29.02+0.52
27.43+0.71
15.31+00.36
28.14+1.11
DC
10
50.99++0.62
34.06+0.40
13.96+1.37
16.51+0.89
34.36+0.35
50
56.81:0044
31.6320.55
15.9411441
17.9811.06
30.14:0.51
CIEAR-10
1
27.76=0.47
25.01=0.77
25.59+0.56
15.12+0.65
23.70=0.20
DSA
10
52.96.10.411
34.49
42.11100.56
16.10.11.03
31.8810.35
50
60.28+00.37
41.01+0.36
49.52+0.72
19.65+1.16
43.30+0.43
26.45++0.39
10.02+0.555
20.64+0.47
04.09+00.58
20.47+0.46
DM
10
47 07.64:5..55
04.4010.30
38.21+1005
15.60+1.61
34.37 49
50
01.99:10.33
40.49:10.38
52.76:1044
21.67:+1.34
43.22:10377
1
CAFE
10
50.68++0.17
35.94+0.65
42.40+0.88
25.51+0.41
35.22+00.16
50
61.9711..2
24.41:0.117
26.1220.77
14.78:10.5
25.1040.37
1
29.30:20.33
24.58
24.56+0.34
13.60#00211
20.74+0.30
OURS
10
53.42.10.14
34.94
40.65.10.14
17.2010.33
34.17.0.116
65.92+0.24 50
41.40=0.67
55.49+0.21
23.12+0.42
46.20+0.29
5.30-0.23
4.2740009
4.3620.15
1.7340.12
4.5:10.15
Random
18.6410.25 10
10:20+0.18
15.7740.24
6.1940.46
14.0740.21
50
34.66:10.41
16.30
30.23:10.61
18.55.1..22 1.29
26.90
1
10.89-10.17
7.9610017
8.75-0.43
2.2210.19
7.8110.13
K-Center
10
25.04+0.30
13.92+0.20
22.18+0.59
7.14+0.79
17.98+00.44
50
38.6410.43
19.3240.30
34.00110.51
21.25:11:46
30.1210.65
13.66:00.00 29
9.78++0.27
9.71=0.46
2.67=0.16
9.274+0.14
DC
10
28.42.10.22 29
12.36
17.94.10.59
5.2811.05
12:2210.17
50
30.56++0.56
13.29++0.30
17.64+0.31
11.36+00.95
17.51+0.15
CIFAR-100
13.73+00.45
10.56+0.22
9.95-0.55
2.95+0.44
9.48+00227
DSA
10
32.23:10.35
16.1740.20
86.00.4443 43
5.45+1.04
19.61:0011
50
43.13:10.33
21.42:10.31
34.34:10.44
20.79:11.76
31.8910.49
1
11.20:10.20 27
8.17.10.21
5.3610.31
2.1110.113
4.50.10.26
DM
10
29.23++0.20
14.68++0.18
18.72+0.49
3.91+0.73
17.06+00.25
50
42.32.0.35 .37
20.1410.24
33.3410.40
17.20-2-41
30.11100.2
1
12.80:10.20
6.71+0.116
5.76=0.11
2.26+0.32
7.28:0019
OURS
10
33.39.10.29
16.06
22.58.10.33
6.73100.7
19.51
50
45.19++0.18
19.63+0.17
38.54+0.66
18.44+00.36
30.70+0.21
1.65+0.11
1.37++0.08
1.27-0.08
0.63+0000
1.71+0.03
Random
10
6 6.8820.25
3.12
3.34-0.16
1.0120.15
6 6.63+0021
50
18.62:10.22
5.28:10.2
10.35:10.33
2.90
16.87:10.20
3.03-10.12
2.5310.13
2.29-0.10
0.8210.10
2.2710.02
CeCneer
10
11.38++0.26
4.73++0.08
5.46=0.24
1.550.21
9.92++0.34
50
22.02.00 40
5.9910.17
13.5110.34
3.9110.49
19.70104
1
6.27=0.10
2.67+0.11
3.17=0.21
0.90=0.14
2.00+0.12
DC
10
12.83.10.14
4.12.10.11
5.44
1.2410.18
4.17.10.10
TingImageNet
50
12 60+00.36
3.81+0.17
7.03=0.21
2.39+0.21
5.22+0.23
5.67+0.14 14
3.90++0.16
3.20+0.13
0.84+0.12
3.17+0.03
DSA
10
16.34:1.21
6.31+0021
7.60-0.36
1.9010.21
11.1740.11
50
25.31:10.22
6.72
13.36:10.40
3.78
19.87-10.44
1
3.8210.21
3.11100110
1.79-0.17
0.8910.07
3.2110.07
DM
13.51+0.31
4.24+H011
3.57=0.20
1.06+0.19
7.46+0.20
50
22.76110.28
5.7411027
11.07:1.399
3.3310.46
18.8840036
1
4.41+0.37
2,20=50.17
1.640.007
0.81+0.16
3.74+0.11
OURS
10
16.14.10.15
3.511
8.98
6.28
9.60 0..01
50
28.10++0.21
25.04=0.12
22.44+0.31
23.74+0.42
21.72+0..7
23


Reviews:
[{"review_id": "CkkDC3lNNe", "reviewer": "Reviewer_gXRh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper covers various experiments on the application of the condensed data.\n\n2. The method is well-motivated.\n\n### Weakness\n\n1. The reviewer's primary concern is the lack of comparison to related work and, therefore, limited new insights for the dataset condensation community. The baselines compared in this work (DC, DSA, DM, and CAFE) were mostly published before 2023. After these initial works in dataset condensation, various papers published in 2023 pushed the frontiers of dataset condensation: including 1) Those on the line of bi-level optimization. Cazenavette et al., 2022 proposed a trajectory matching method and achieved performance better than all the numbers in the \"OURS\" column reported in the paper. Follow-up work aims to alleviate the computational cost with approximation (Loo et al. 2022, Loo et al. 2023) and simplified unrolling (Feng et al. 2023). It is not yet established that the proposed method could achieve better performance with the same or less computational cost compared to these papers. 2) Those on the line of distribution matching. Yin et al. 2023 proposed a similar pipeline of first training a classifier, then learning the condensed images by matching the mean and standard deviation of the normalization layer (thus including the diversity part of the energy function), and finally relabeling them with the classifier (thus minimizing the classification loss). This paper scaled up dataset condensation from small datasets like CIFAR10 or CIFAR100 for the first time to ImageNet-1K and ResNet 50, while this paper is still training with ConvNet and with Tiny-ImageNet being the largest dataset used. Yin et al. 2023 should be a necessary comparison. The reviewer did not follow the literature since these works and there could be various follow-ups, and none of these works are cited in this paper. With these highly-cited prior works that have scaled up performance, the current experiments seem providing limited new information to the community.\n\n2. The ablation study: The current ablation study only experiments with changing the objective from DM to ED and including the classifier as regularization. Since the ED objective consists of two parts – a matching part between distributions X and Y, and the distances between samples in dataset Y, which helps preserve diversity – it would be helpful to understand which of them helps improve performance from DM to ED. How would the method perform with only the diversity term?\n\n3. Table 8: Why is there no result with other dataset condensation methods?\n\n4. It is recommended to include the objective for the \"OURS+M3D\" method in the main paper for completeness.\n\n\n-----\n\n### Reference\n\nLoo, Noel, et al. \"Efficient dataset distillation using random feature approximation.\" Advances in Neural Information Processing Systems 35 (2022): 13877-13891.\n\nLoo, Noel, et al. \"Dataset distillation with convexified implicit gradients.\" International Conference on Machine Learning. PMLR, 2023.\n\nFeng, Yunzhen, Shanmukha Ramakrishna Vedantam, and Julia Kempe. \"Embarrassingly Simple Dataset Distillation.\" The Twelfth International Conference on Learning Representations. 2023.\n\nYin, Zeyuan, Eric Xing, and Zhiqiang Shen. \"Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective.\" Advances in Neural Information Processing Systems 36 (2023): 73582-73603.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new dataset condensation method based on existing distribution matching methods. It combines an energy distance of the data with a pre-trained classifier constraint. It shows improved performance compared to other methods including DC, DSA, DM, and CAFE. The paper further demonstrates its effectiveness regarding training cost and its performance for debiased dataset condensation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please address Weakness #1.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9iZSmsWH8q", "reviewer": "Reviewer_AXfZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The proposed method is conceptually simple and demonstrates strong performance within a specific compression range, particularly between 10 to 800 images per class (IPC) on the CIFAR-10 dataset.\n\n**Weakness:**\n- The primary limitation of the method is its lack of robustness in extreme compression scenarios (i.e., IPC < 10 or IPC > 1000). Although it performs well within the mid-range and the authors argue that this range is more relevant for practical applications, this claim is debatable. A more practical question would be: What is the minimum compression ratio at which the method maintains nearly lossless accuracy (e.g., within 1 percentage point of the full dataset accuracy)? The proposed method offers no clear advantage in this regard, as its performance converges to that of the random baseline beyond 1000 IPC on CIFAR-10.\n\n- In terms of training cost, the method does not Pareto-dominate the DM baseline. This raises additional concerns about its efficiency and practical applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new dataset condensation (or distillation) method that falls into the category of distribution matching. This line of work is cheaper than the line of work that matches gradient to condense the dataset, as it avoids bi-level optimization and the second-order derivative during the optimization process. Compared to previous distribution matching work (DM), the paper introduces two main changes:\n\n- Classifier-critic: They add a loss term that forces the synthetic sample to be classified to its corresponding label by the classifier while not relying on spurious attribute.\n- Energy distance: Adds repulsive term between the generated data so that condensed samples preserve some diversity internally.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In Table 1, the CIFAR-100 results incorrectly bold the OURS+M3D method, even though IDM achieves higher accuracy.\n- If the authors intend to emphasize the efficiency of their method, it would be more convincing to include experiments using larger models (e.g., ResNet-50) or more challenging datasets (e.g., ImageNet instead of TinyImageNet). Such results would better support claims of scalability and practical relevance.", "broader_impact_concerns": "I do not have concerns regarding ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zGFiQ5u6Fc", "reviewer": "Reviewer_om5U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and easy to follow.\n* The paper has relatively solid experiments to show the benefit of the proposed method.\n* The paper has good visualization.\n* The proposed method is practical and easy to incorporate into the existing method.\n\nWeaknesses:\n* The related work and baseline methods in the experiments do not include all the important methods. Some more recent methods are not incorporated.\n* The bi-level optimization has be dealt with by some existing works, like using multiple early-stopping models to reduce the computation. I was wondering what the advantages of the proposed method are. Also, it would be good to add some computational comparisons in the experiments.\n* The related work regarding diverse data condensation is insufficient.\n* Apart from the debiased quality, could the method also introduce other good data quality?\n* The proposed method seems to simply combine two methods and lacks novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the data condensation problem and proposes a method to provide a condensed dataset with both improved diversity and good distribution matching to the original large-scale dataset. Existing methods either lack good diversity or require bi-level optimization. The paper uses a class-preserving distribution matching approach and energy-distance-based distribution matching to solve the limitations. Meanwhile, other data quality metrics, such as debiased data, can also be enhanced by the proposed method. Experiments are provided to demonstrate its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* It would be good to include a more comprehensive discussion of the related work and experimental comparison with the baseline method.\n* It would be good to have more discussion about the computational benefit to support the claim at the beginning.\n* It would be good to add more discussion about the unique method design and the contribution over existing methods.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)