Group Distributionally Robust Dataset Distillation with Risk Minimization
Authors: Saeed Vahidian
Mingyu Wang
Jianyang Gu
Vyacheslav Kungurtsev
Wei Jiang
Yiran Chen
Venue: ICLR 2025 Poster (2025)
Keywords: dataset distillation
distributional robustness
generalization
Primary area: optimization

Abstract: Dataset distillation (DD) has emerged as a widely adopted technique for crafting a synthetic dataset that captures the essential information of a training dataset, facilitating the training of accurate neural models. Its applications span various domains, including transfer learning, federated learning, and neural architecture search. The most popular methods for constructing the synthetic data rely on matching the convergence properties of training the model with the synthetic dataset and the training dataset. However, using the empirical loss as the criterion must be thought of as auxiliary in the same sense that the training set is an approximate substitute for the population distribution, and the latter is the data of interest. Yet despite its popularity, an aspect that remains unexplored is the relationship of DD to its generalization, particularly across uncommon subgroups. That is, how can we ensure that a model trained on the synthetic dataset performs well when faced with samples from regions with low population density? Here, the representativeness and coverage of the dataset become salient over the guaranteed training error at inference. Drawing inspiration from distributionally robust optimization, we introduce an algorithm that combines clustering with the minimization of a risk measure on the loss to conduct DD. We provide a theoretical rationale for our approach and demonstrate its effective generalization and robustness across subgroups through numerical experiments.

Published as a conference paper at ICLR 2025
GROUP DISTRIBUTIONALIY ROBUST DATASET DISTIL- LATION WITH RISK MINIMIZATION
Saeed Vahidian 1 Mingyu Wang**
Jianyang Gu
Vyacheslav Kungurtsey
Wei Jiang"
Yiran Chen'
1 Duke University 2 Zhejiang University
The Ohio State University
4 Czech Technical University
ABSTRACT
Dataset distillation (DD) has emerged as a widely adopted technique for crafting a synthetic dataset that captures the essential information of training dataset, facili- tating the training of accurate neural models. Its applications span various domains, including transfer learning, federated learning, and neural architecture search. The most popular methods for constructing the synthetic data rely on matching the convergence properties of training the model with the synthetic dataset and the training dataset. However, using the empirical loss as the criterion must be thought of as auxiliary the same sense that the training set an approximate substitute for the population distribution, and the latter is the data of interest. Yet despite its popularity, an aspect that remains unexplored is the relationship DD to its generalization, particularly across uncommon subgroups. That is, how we ensure that a model trained on the synthetic dataset performs well when faced with samples from regions with low population density? Here, the representativeness and coverage of the dataset become salient over the guaranteed training error at inference. Drawing inspiration from distributionally robust optimization, we intro- duce an algorithm that combines clustering with the minimization of risk measure on the loss conduct DD. We provide a theoretical rationale for our approach and demonstrate its effective generalization and robustness across subgroups through numerical experiments.
INTRODUCTION
Dataset distillation (DD) is a burgeoning area of interest, involving the creation of synthetic dataset significantly smaller than the real training set yet demonstrating comparable performance on model (Wang et al.] 2018 Sachdeva & McAuley 2023). This practice has gained prominence in various computation-sen sensiiiieeeeeeevveeeeeeee applications, providing a valuable means to efficiently construct accurate models (Gu et al. 2023b Medvedev & D yakonov 2021 Xiong et al. 2023). The standard optimization objectives that are used to steer the construction of the synthetic data typically aim to foster either distributional similarity the training set (Zhao & Bilen 2023a Zhao al. (023) or similar stochastic gradient descent (SGD) training dynamics as the original dataset (Zhao et al. 202T
Cazenavette et al. (002)). Notably, recent literature suggests that the latter category has proven more successful (Kim et al. [2022] Cazenavette et al. (023).. Intuitively, this success can be attributed to the rationale that, with considerably fewer samples, prioritizing the most relevant information for training and model building becomes more judicious.
In this paper, we aim to address two important practical concerns in DD training. First, it essential to note that the synthetic dataset might be applied across a wide range of potential circumstances with distinctions from the training phase. Consequently, models trained on the synthetic dataset must exhibit low out-of-distribution error and strong generalization performance. This means the synthetic dataset should be designed have certain higher-order probabilistic properties, particularly in relation to the model and loss functions. In addition, shifts in circumstances over time (some latent exogenous variables, formally) mean that the population distribution on which the model trained
"qqual contribution.

---

Published as a conference paper at ICLR 2025
Figure 1: A visual representation of the robust inference task involves the partial partitioning of the population distribution, that is X' across subgroups {c:). A classifier is considered ro- bust when it demonstrates high performance across the subgroups. As practical hypothet- ical example of online learning, at a particular time, a steady stream of samples from X' X'(c3 may appear to the classifier. Note that in this case the region of sample space defined by this subgroup is geometrically small, and we can consider that it has low overall prior den- sity. If this subgroup's 's behavior is particularly anomalous, a model and any associated distilled dataset trained only on minimizing empirical risk may perform poorly on this subgroup.
1. 6.y-
XM
may resemble a clustered sample, i.e., subset of the population. Therefore, it imperative that the synthetic dataset ensures good coverage across the support of the sample space.
To this end, we propose to enhance the generalization performance and group coverage properties of the distilled dataset using concepts and methods in the field of Distributionally Robust Optimization (DRO) (Lin et al. 2022] Zeng Lam 2022] Vilouras et al., 2023). Within DRO, one solves bilevel optimization problem where the objective is to minimize the loss over data distribution, subject to the constraint that this distribution is a specific probabilistic distance away from the population
distribution:
min max. F(O, Q). Q = (Q: ((Q)))))) S A
AER QEQ
where F is the loss function, 0 is the parameter, Q is the distributional uncertainty set, D is an f-divergence and P is the population distribution.
We now present the two desiderata of DD to which we aim to apply the concepts of DRO. First, our goal is to construct a synthetic dataset that is quality representative of the underlying population distribution. Thus, the consideration of generalization is paramount. Xu & Mannor 2012) present arguments indicating the theoretical equivalence of testing to training error as robustness to pertur- bations of the data distribution, and the generalization accuracy. Second, conventional maximum likelihood classifier inherently assigns higher weights to sample ranges with larger prior distributions than to those with lower overall probability density. This inherent bias poses fundamental risk of underperformance in small population subgroups. Furthermore, in online inference, it is common observe alterations in the overall prior distribution, particularly in terms of subgroups. DRO has been conjectured and observed to mitigate this issue, both generically (Duchi & Namkoong 2021 and with explicit quantification of subgroups in the distributional uncertainty set (Oren et al. 2019 where subgroups of topics are considered for training language models.
Moreover, this permits flexibility as far as easily modularized methods to specific concerns. Indeed, domain expertise in regards to the properties of the population distribution has been shown to assist in generalization- (Hu et al. 2018) indicates that a more precisely defined distributional uncertainty set, focusing on specific collections of probability distributions from subsamples of the population rather than encompassing all possibilities
ball, yields improved generalization performance.
In formal sense, we define that each subsample involved in inference task is drawn from union of closed, convex, connected subsets of the population distribution. An intuitive representation of the envisioned scenario is illustrated in Figure
Inspired by these concepts, this paper introduces a Double-DRO-based DD procedure designed specifically to address these two concerns. Fundamentally we employ DRO at two levels to promote different advantages that we shall witness in the experimental results. 1) We leverage the loss across clusters of the latent variables to ensure group robustness. 2) We approximately solve DRO problem internally in order to promote better within- ghoupp generalization, using a risk measure as a proxy for DRO solution.
2

---

Published as a conference paper at ICLR 2025
Our paper continues below as follows. Next, present algorithm, analyze the optimization problem defining the notion of generalization of interest, and argue how our method effectively targets this criterion in Section We give the mathematical analysis in Section B In Section 4 we present numerical results validating the superior overall and subgroup generalization performance on Related works
standard machine learning benchmarks. Finally, we conclude this work in Section 5 are reviewed in the Appendix in Section
2
ALGORITHM
In this Section we describe our main procedure, in which we incorporate two techniques informed by DRO in order to improve both generic generalization performance and group distributional coverage.
Let S {.s, ysb be the synthetic distilled dataset, where ([æs]e, [ysk) with 1, 1,... [S] is an individual input and label pair. Let T {IT, yr) be the real training set where ([rT) i, (url.)) an individual input and label pair with [7] >> |S]. The target dataset distillation is to optimize the synthetic dataset S so that network with parameter & can achieve similar performance compared with that attained on T. For simplicity, we define the input and label pair as = (z, y). And F(0; 2) is the full loss function to optimize parameter 0 on data which is typically cross entropy loss in classification tasks.
j 1,... [T]
Matching- based based dataset distillation methods usually involve imitating certain training characteristics of the real training set, such as training gradients, feature distribution, training trajectory, etc (Zhao) al. 202T Zhao & Bilen 2023a; Cazenavette et al. 2022). Take gradient matching as an example. At each iteration, the training gradient is extracted for the synthetic data and the real data, based on the same network. The gradient difference is set as the objective of optimizing synthetic data, Considering that diverse supervision gradients from different training stages should be provided to enhance the consistency throughout model training process, the network simultaneously updated with the synthetic data optimization. Previous methods utilize the same loss function F(0; 2) to update the network. However, in this work, we propose to incorporate DRO at this stage for more robust network optimization as well as distillation supervision.
Specifically, we wish to solve a DRO locally in order to promote solutions that are particularly suitable for generalization. However, as the dimensionality and desired training size in our settings of interest present intractability for DRO, instead of solving the full DRO we use a proxy in the form of Conditional Value at Risk (CVaR), which we define below. In the theoretical analysis, we describe the known correspondence between CVaR and DRO.
In algorithm, each iteration begins with subsampling the training dataset T T and then clustering them based on their nearest synthetic data point. That is for all i E [[S]] the set c, contains the elements of T that closer to 2; than to any other point in S. The following criterion optimization problem is solved for 0, obtaining the set of parameters that solves for minimax loss across the subgroups defined by the clusters {
min Σ CVaR[F((O + (Var|(((000)
By considering the clusters C; separately, especially including the max term in the objective, we robustify the performance with respect to the different clusters. The synthetic dataset is meant to represent the training set, and with clustering, we ensure the entire support of the training distribution is represented by some sample z; S. This yields effort ensure the group distributional robustness. We intend that both the worst case and the average performance across the sub-groups are minimized.
In addition, to promote generalization broadly speaking, note that rather than the expectation, we use risk measure, as far statistically agglomerating the loss over the training data. We set a to be some tail probability. Risk measures enable one to minimize one-sided tail behavior. The operator denoting the Conditional Value at Risk, CVaR, is defined, with respect to the empirical distribution of data points within C,;
1 1 al Eneo F(0: fal
(00;:](0;
(3)
+Sa(a- - |o Enjec, 1[((0 < 1n))}
3

---

Published as a conference paper at ICLR 2025
Algorithm Robust Dataset Distillation
Input: Real training set T, synthetic set S, network with parameter 0, distilling objective T(S).
Execute:
While not converged:
Subsample the training set T CT.
Cluster 7 by distillation set, i.e. define, & [[71]:
((+)) = arg min, || - [s].]] and
C, & T : C(+c) = i}.
Solve the optimization problem in equation 2 obtain 6*, Optimize synthetic with L(S) based on the optimized parameter 6*,
with the quantity fo defined as, with |cs| denoting the size of the training set in cluster C,,
Sa = min SER:
Σ ELER,
(4)
C2
The choice of a CVaR weighing of the loss informed by DRO-associated theoretical work aims to improve test error accuracy rather than merely minimizing training error. Additionally, by clustering and weighting the balanced loss, Group DRO is applied across the connected components of the sample space distribution, which defines the population.
After updating the network with Group DRO, the synthetic dataset S will be optimized based on matching metrics, denoted by L(S), where existing matching-basee DD methods can be broadly plugged This makes the algorithm modular any choice of DD procedure. The algorithm is shown in Alg. [ As L(S) involves a risk function of the loss F(0; 2). averaged across a set of partitions of S, the supervision helps enhance the distributional robustness of the distilling process. Thereby, the distilled dataset can obtain superior generalization performance across different domain shifts, as well as better group coverage for more practical usefulness. We proceed to iterate between an update the parameter based on solving equation 2 and update to the synthetic dataset S with our tailored DD procedure until the procedure reaches a fixed point, wherein the two do not significantly change.
ANALYSIS
In this Section we describe the formal optimization problem being solved as well as its convergence and statistical properties. The focus will be on justifying why we expect the procedures defined the Algorithm to improve (group) generalization, through foundational results in DRO theory.
Let Q be partition of the population distribution D. As such, we can describe the problem of group coverage at the point of inference as yielding comparable performance across elements of Q. To this end, we denote the support the population distribution as supp (D) and we assume that compact. Let us define,
(Q)), with Q; supp (2)), ÚQ, - D, S. (())) <
(5)
i=1
for some > & >>> 0 and every Q: is itself a union of compact, convex, and connected sets. Here integration with respect to the Lebesgue measure. Recall that we denote the synthetic dataset by S with [S] =
Note the assumption that D is finite. In addition, assume certain regularity of the loss function. Formally, we state the following:
Assumption 3.1.
The population distribution D has bounded support, i..., X(D) < co. The loss F is Lipschitz continuously differentiable with respect to the first argument (the parameters) and continuous with respect to the second argument (the input features and labels). Finally, the partitions probabilistically small, i.e., there exists Pqp such that P E Qil 2 Pro for all
4

---

Published as a conference paper at ICLR 2025
Additionally, let's explore the asymptotic learning regime when examining the problem, where the entire population set could be sampled in the limit.
Assumption 3.2. For all Q E Q, consider the asymptotic online learning limit,
lim sup (supp( () Tr) supp( Q)
(6)
t-+00
where Te is the training set sampled at iteration t.
Informally, group coverage corresponds to enforcing adequate performance for prediction using the trained model regardless of what portion of the population set is taken. Essentially, at partic- ular future instant, the learner could be expected to classify or predict a quantity for some small subpopulation cohort that may appear as subsample at the time of online inference.
Let us formally articulate the optimization problem interest, as outlined the Introduction, as follows
min max E Q) 8* arg ming F).. QEQ
Let us briefly consider the standard circumstance by which Algorithm converges, By considering each iteration as two players' best response to cooperative Nash game, we can ensure convergence asymptotically of iteratively solving the two optimization problems, as given in [ass (1953).
Theorem 3.3. Under the circumstance by which the Morse-Saard condition holds (Souček & Souček 1972) and so the optimal sei {S*(0), (0)), (S)} is compuct (possibly finite) for all 0, S, then Algo-
rithm 2 converges fixed point of equation
remark on the time complexity of the Algorithm. Using CVaR instead of a sample average only takes constant multiple of operations the subsample. Clustering itself can be done polynomial in the number of samples. Since the properties of the model and loss function, that is, nonconvex and nonsmooth, fundamentally unchanged, there is no change the iteration complexity.
Now we will discuss generalization and DRO. Recall that there are two layers of generalization: on the one hand, we are solving problem robust with respect to the choice of Q and on the other hand, we are considering the population error rather than an empirical loss.
We posit that the computation of gradient estimate employs conventional Stochastic Approx- imation procedure to address some bilevel optimization problem for S. Thus we do not address the convergence guarantees (that is, asymptotic stationarity and convergence rate) of the training procedure itself but study the properties of the associated optimization problems and their solutions.
Accordingly, our emphasis is on scrutinizing the properties associated with the criterion,
min max IEq [F(,, 2)].
0 QeQ
that ultimately used to steer the synthetic dataset, as the DD task is finding a dataset on whose associated O-minimizer also minimizes this quantity. In the analysis, we shall argue about the validity of these criteria as far ensuring the generalization as well as group robustness properties of the solution of equation 7
To begin with, we rewrite equation 7 as:
min max maxored
E[F 2') s.t. 0t E arg ming F))
S QEQ
(9)
@ = (C' IQI' QN) S)
where r >> some bound and we replace the inner problem to be evaluated on the entire training dataset with data-driven DRO. In the appendix we present several results in the literature that indicate how the DRO to an empirical risk minimization problem exhibits generalization guarantees and hence is valid auxiliary criterion for minimizing equation 7
3.1 LARGE DEVIATIONS AND SOLVING THE BILEVEL DRO
To justify the clustering and risk measure minimization algorithm as an appropriate procedure for solving equation 9 apply some theoretical analysis on the relationship between DRO and
5

---

Published as a conference paper at ICLR 2025
Large Deviations Principles (LDP). LDPs (e.g., Deuschel & Stroock (2001)) define an exponential asymptotic decay the measure of the tails of an empirical distribution with respect to a sample size.
The particular, Large Deviations Principle (LDP) capturing out-of-sample disappointment, which is of interest to us, is defined as follows:
-r, lim sup N log @ (F(O > ((O" 2N)) -T, VQ E Q N
(10)
for some O. This states that for all partitions Q of the population space partition Q. there is an exponential asymptotic decay the probability of one-sided sample error (i.e., samples size N from the population IP 2) relative the computed loss on the synthetic and training data points ON. as the number of training data points grows asymptotically.
Now, we present the following Proposition, whose proof is in the appendix,
Proposition 3.4. Algorithm equation 2 satisfies, under Assumptions 3.7 and 3.2 the LDP equation 10
Next, we relate Algorithm equation together with 3.1 to the DRO problem equation 9
3.2 LARGE DEVIATIONS AND DISTRIBUTIONALLY ROBUST OPTIMIZATION
Applying Assumption 3.2 the LDP, and the continuity of with respect to the data, Assumption 3.]
(see also the proof of Lemma I as well as Example 3 in Duchi & Namkoong (2021) we can bound the quantity:
P(F(O", .)) > "(0",SUU" QN)
(11)
for q Q', with D(Q,, Q') S with sufficient step-size.
This implies that one can bound the objective value of the DRO problem of interest equation g
min max max (IF(0",2) 2')] s.t. 0* E arg ming F(),S)
S QeQ
(12)
o = (0' 1("' Sr)
ie., there is some small C > 0 such that,
(0"0",s)~-0 Ō(0, 8) c
(13)
where O and Ô refer the optimal value of the data driven DRO equation 9 and the approximation given by Algorithm and
3.3 DISTRIBUTIONALLY ROBUST OPTIMIZATION AND SUBGROUP COVERAGE
Let us return to equation
min max [F 2) s.t. 0" E argming F(0,S) QEQ
We have established that Algorithm approximately solves DRO which approximately bounds the population loss. Now consider the outer DRO itself. The same theory regarding DRO and LDPs now be applied, but Indeed the data being sampled are simply Q. E Q that is, some i.i.d. selection w over the finite of partitions. Since the entire subpopulation taken, each estimator is constructed with the full population error, and w.p. the entire set Q is sampled for finite N.
yield a stronger result, since we evaluating the full (test) loss.
We directly apply Theorem in the work of Van Parys et al. 2222 to deduce that the result satisfies:
lim sup log p00 N
(F()"QQw)> > ((0"8)) < -
(14)
Thus by the Kolmogorov 0-1 principle we have that,
F(0', Q) < (((0,S))
for all Q. This guarantees the quality of S in ensuring group robust guarantees on the loss.

---

Published as a conference paper at ICLR 2025
Table 1: Top-1 test accuracy on robustness testing sets. "Gain" denotes the performance gain of applying our proposed method. Except for absolute values, the relative fluctuation compared with the standard case also reported as subscripts. The experiments are conducted under the IPC setting of 10. R. indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked bold.
Dataset
Sctting
Random
IDC
IDC
Gain
GLaD
GLaD
Gain
Standard
67.5g0.5
68.610.3
26.720.6
50.21..3
Cluster-min
31.420.9
63.3 14.2
65.0.0.6 Ja.s
1.700.
40,2 1a.
46.7.2.9 43.5
6.5 13.0
CIFAR-10
Noise
3..4
57.221-1,10.
59.41..
19.2
2.
44.1
12.5
49.5
10.7
54
f1.a
Blur
29.4 18.1
20.520.7138.1
22
36.920.6
19.5
39.0.0.5111.2
2.1
11.4
Invert
9.541.0
25.640.6.44.9
26.540.6142.1
0.
10.6411,88.1
130++1.2137.2
24 41.1
Standard
06.9
72.8g0.0
74.610.0
1.8
50.920.9
15.2
4.3
Cluster-min
31.2++1.0
61,44100,11.4
65.7
48.9
4.3,2.6
34940.9.10.0
47.1+1.2
48.1
12.2p7.9
ImageNet-10
Noisc
42.6:..8
65.8.0.8 17.0
68.820.9
Ja.a
3.0
48.62..7 12.3
53.9
41.3
5.3 11.0
Blur
1
71.90.9 10.9
74.++1.1
20.5
2.240.4
47.9+0.9
13.0
54.6
41.6
6.7 11.4
Invert
21.010.6
27.8.0.7.44.0
30.3.0.
2.5...7
17.041.0a5..9
21.6:0.7138.6
10.3
Table Top-1 test accuracy on standard testing sets. indicates the result is reported based on our runs. R indicates the proposed robust dataset distillation method applied on the baseline. The better results between baseline and the proposed method are marked in bold.
Dataset
IPC
Random
DSA
DM
KIP
IDC
IDC
GLaD
GLaD
1
14.6
27.5
24.2
57.3
68.5
28.9
32.5
35.7 +0.3
SVHN
10
35.1
79.2
72.0
75.0
87,5
88.1
68.220.4
72.5
50
70.9
84,4
84.3
80.5
90. 20.1
90.8=0.4
71.8
76.620.3
1
14.4
28.7
26.0
49.9
50.640.4
51.3
28.0.o.a
29.2
CIFAR-10
10
37.2
52,1
53.8
62.7
67.5
68.620.2
46.720.o!
50.2
50
56.5
60.6
65.6
68.6
74.
75.3
59.920.9
62.5g0.7
1
23.21
30.61
30.2
54.4
58.2
33.5
36.410.8
ImageNet- 10
10
46.9
52.7
52.3
72.820.6
74.620.9
50.921.0
55.2
NUMERICAL RESULTS
In this section we present numerical results to validate the efficacy of the proposed robust dataset distillation method. Implementation details are listed in Sec. H
Results on Robustness Settings We first show the notable advantage offered by our proposed method that is the robustness against various domain shifts. This property is assessed through multiple protocols. Firstly, suggested before, present validation results different partitions of the testing set. A clustering process is conducted to divide the original testing set into multiple sub-sets. We test the performance on each of them and report the worst accuracy among the sub-sets to demonstrate the robustness of distilled data, denoted as "Cluster-min" in Tab. In the sub-scripts, the performance drop compared with the standard case is reported. Several key observations emerge from the experiment results. (1) Compared with random images, the Cluster-min accuracy of baseline DD methods exhibits improvement alongside the standard performance. It suggests that by condensing the knowledge from original data into informative distilled samples, DD methods contribute to enhanced data robustness. (2) Compared with CIFAR-10 10, the performance gap between the standard case and the worst sub-cluster on ImageNet-10 10 is more pronounced. This discrepancy can be attributed to a higher incidence of ID-unrelated interruptions in ImageNet-10. 10, resulting in larger domain-shifts between sub-clusters and the original distribution. This finding aligns with the observation in Tab, 2 (3) With our proposed robust method applied, not only is the cluster-min performance improved, but the performance drop from the standard case is also significantly mitigated compared with baselines. It suggests exceptional overall generalization and robustness conferred by our method.
Furthermore, provide testing results in Tab. truncated testing sets, simulating scenarios where the testing set exhibits more substantial domain shifts compared with the training data. We employ three data truncation means, including the addition of Gaussian noise, application of blur effects, and
7

---

Published as a conference paper at ICLR 2025
Table 3: Top-1 test accuracy on subpopulation shift benchmarks MetaShift and ImageNetBG. All the results are reported based on the average of 5 runs. R. indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.
MetaShift
ImageNetBG
Metric
GLaD
GLaD
GLaD
GLaD
Average Accuracy
58.6
62.2
41.7
45.5
Worst-group Accuracy
51.3 7.3)
57.0 5.2)
32.2+1.e( 9.5)
38.6 6.9)
Table 4: (a) Cross-architecture generalization performance comparison. The experiment is conducted on ImageNet- 10 under 10 IPC settings. The distilling architecture for IDC is ResNet- 10, while for GLaD is ConvNet-5, R indicates the proposed robust dataset distillation method applied. (b) Ablation study on CVaR loss. The experiment is conducted on CIFAR-10 and ImageNet-A under 10 IPC. The baseline model is GLaD. "CE" denotes cross entropy loss, and "CL-min" refers to the worst sub-cluster accuracy. "aCVaR" and "mCVaR" refer to average and maximum CVaR loss, respectively. The best results are marked in bold.
(a)
(b)
Method
Architecture
Loss
CIFAR 10
ImageNet-AA CL-min
Conv
Res10
Res18
VII
VGGI1
aCVaR mCVaR
Ace
CL-min
IDC
21.92..8
72.
7..8
58,241.2 64.52..6
46.720.6
40.2g0.s
53.920.6
40.5..7
IDC
72.6...6
74.620.7
72.7.0.8
56441.
65.620.5
49.120.7
45.3g1.0
56.40.
43.920.
GLaD
48.2_0.7
50.920.
51.210.4
36.810.a
44.221.0
47.920.8
44.210.9
56.1
43.720.a
GLaD R
51.920.7
55.2g0.0
53.610.5
39.20.00
46.3
50.2-0.6 46.7+0.s
57.5+0.7
45.8
inversion of image colors. The conclusions drawn from truncated testing sets align with those from partitioned testing sets. While dataset distillation generally contributes to improved data robustness, the introduction of CVaR loss further amplifies the trend. In this analysis with a higher resolution to capture more details, we observe that truncation on ImageNet has a smaller impact compared with CIFAR- 10. Generally, the improvement by our proposed method truncated performance is also larger than that on standard testing sets, especially on ImageNet-10 The increased advantage on robustness settings further validates that the proposed method not only elevates overall accuracy but significantly fortifies the robustness of distilled data.
Subpopulation Shift Experiments In addition to the previous standard classification tasks, we also extend the method to subpopulation shift benchmarks MetaShift and ImageNetBG (Yang et al.) 2023 Liang& & Zou 2022 Xiao et al., (022). The two benchmarks consider the spurious correlation and attribute generalization problems, respectively. We GlaD the baseline method to distill 50 images for each class. Subsequently, we conduct the standard evaluation as in the other datasets, with the results reported in T The proposed robust dataset distillation algorithm not only improves the average accuracy, but also yields a smaller worst-group accuracy margin compared with the baseline method. The results further suggest that RDD can enhance the robustness of distilled data for datasets with subpopulation shift issues.
Results on Standard Benchmark We then evaluate our proposed method on standard testing sets, including SVHN (Sermanet et al. 2012), CIFAR-IO (Krizhevsky al., 2009). ImageNet- 10 (Deng et al. 2009), and Tiny-ImageNe
(Deng et al. 2009) in Table The ImageNet- 10 split follows the confguration outlined in IDC (Kim et al. (022). We use IDC (Kim et al.) 2022) and GLaD (Cazenavette et al. (023) as baselines in this section, representing distilling at the pixel level and the latent level, respectively. Additionally, the validation results are also compared with DSA,
DM, and KIP (Zhao & Bilen 2021 2023b Nguyen et al. 2021).
The incorporation of CVaR loss consistently enhances performance across all scenarios. Notably, under the IPC of 10, facilitated by the supervision from segregated sub-clusters, our proposed method demonstrates the most substantial performance improvement over baselines. In the case of 1 IPC, where only single synthetic sample is available for sub-cluster construction, the optimization comes back to a DRO problem. And minimizing CVaR still captures tail risk and helps improve the validation performance, despite the absence of guidance from multiple sub-clusters. Our proposed
8

---

Published as a conference paper at ICLR 2025
Table 5: Top-1 robustness evaluation on IDM (CIFAR-10) and GLaD (TinylmagNeet).
Accusaa 44
GL00 Strnter
Method
Setting
IDM
IDM
GLaD
GLaD
Acc
67.
00.1
24.9
26.8+0.66
C1
Chaizenni
Cluster-min
20.3
63.2
g0.5
22.640.6
Noise
61.8=0.5
62.8
22.1
24.4
02
64
6a
10
Blur
54.5
16.2
20.8
23.2
CVaR Ratio
Invert
20.3
21.5
10.2 11.7
Figure 2: Analysis CVaR ratio
method is especially effective on ImageNet-10, characterized by finer class divisions and more substantial intra-class variation compared with CIFAR-10. On ImageNet-10 we achieve an average top-1 accuracy gain of 3.1%, further elevating the state-of-the-an baseline in DD methods. More results presented in the Appendix Section
Cross-architecture Generalization In addition to enhanced robustness against domain-shifted data, the incorporation of CVaR loss also yields better cross-architecture generalization capabilities. We assess multiple network architectures including ConvNet-5, ResNet, ViT (Dosovitskiy et al. 2020) and VGGI1 (Simonyan & Zisserman 2014), and compare the performance with and without our proposed robust method in Table 4a Despite the different distilling architectures employed in IDC and GLaD, both methods achieve their highest accuracy on ResNet-10. Notably, the proposed robust distillation method consistently enhances performance across all architectures, showcasing remarkable cross-architecture generalization capabilities.
Ablation Study We conduct ablation study on the incorporation of CVaR loss in Table 4b In addition to CIFAR-10, 10, we also report results on the ImageNet-A sub-set according to the setting in the work of Cazenavette et al. (2023). Both accuracy the standard testing set and the worst sub-set accuracy "Cluster-min min" are presented. Our focus is primarily on two aspects of utilizing the CVaR loss, i.e. the maximum value and average value of CVaR losses across all sub-clusters. The CVaR loss operates as a complement to the standard cross-entropy optimization. Hence the baseline case, involving only cross entropy loss, mirrors the performance of GLaD. Compared with the maximum value, the average CVaR loss proves more effective in enhancing the validation performance when applied independently. While both average and maximum CVaR loss yield considerable improvement over the baseline, their combined application further fortifies the robustness of the distilled data, which is selected as our eventual implementation. Besides, cross entropy loss still employed the implementation for a stable optimization.
Experiments on Distribution Matching addition to methods constrained by gradient matching. the proposed robust DD can also be plugged into DD methods with other matching metrics. We evaluate the efficacy with IDM (Zhao et al.) 2023) as the baseline, where distribution similarity is used as the matching metric. Similar the integration in gradient matching, the CVaR loss is adopted during model training phases. As shown in the first two columns in Table our proposed robust optimization achieves improvement across all metrics. The results demonstrate the possibility of applying RDD to broader dataset distillation methods for robustness enhancement.
Scalability We also scale up the proposed robust optimization to TinyImageNet, which contains 100 classes, and hence is more challenging compared with other benchmarks. The experiments are conducted on GLaD, and the results are shown in the last two columns of Table When the class number to be optimized is increased, the proposed RDD method provides consistent improvement on the robustness of the distilled data by enhancing model training phases.
Parameter Analysis An analysis is conducted to evaluate the influence of different CVaR ratio a choices of equation 3 in Figure 2 We vary the a value from 0.2 to 1.0 to explore different ratios the standard testing set
of data for calculating CVaR loss. Both the validation performance and the worst seb-set accuracy (Cluster-min) reach their peak at a=0.8. Including all the samples (o=1.0) introduces certain interruptions for the optimization due to large loss values and results

---

Published as a conference paper at ICLR 2025
in slight performance drop. On the other hand, considering only small portion of samples for CVaR loss loses essential information, leading to performance degradation. Notably, even the worst performance obtained at c=0.22 is significantly higher than the GLaD bascline, particularly in terms of the Cluster-min metric. This observation strongly supports the effectiveness of our proposed robust dataset distillation method.
GLaD
GLaD
Ours
T-SNE distribution visualization of original samples (blue dots) and synthesized sam- ples (orange dots) on ImageNet bonnet class.
Figure 3:
Figure 4: Synthesized sample visualization com- parison between GLaD and our proposed method. The samples are initialized identically.
Visualization To explicitly illustrate the impact of CVaR loss on the distillation results, visualize the sample distribution comparison in Figure 3 In the feature space, samples optimized by GLaD tend form small clusters, while the introduction of robust optimization leads more evenly distributed distilled dataset. Note that there is no constraint on the feature distribution applied during the distilling process. The proposed robust optimization involves loss calculation on different data partitions, contributing to better coverage over the original data distribution. The more even distribution observed further affirms the effectiveness of our proposed robust distillation method.
Additionally, we compare the synthesized samples of the same ImageNet bonnet class in the pixel space between the baseline GLaD and our proposed method in Figure 4 The images are initialized with the same original samples for better comparison. Remarkably, the additional CVaR loss introduces more irregular shapes into the image during optimization. These irregular shapes weaken specific features present in each image while introducing common features of the corresponding class, leading to a more even sample distribution in the latent space.
CONCLUSION
This paper explores the intricate relationship between DD and its generalization, with a particular focus on performance across uncommon subgroups, especially in regions with low population density. To address this, we introduce an algorithm inspired by distributionally robust optimization, employing clustering and risk minimization to enhance DD. Our theoretical framework, supported by empirical evidence, demonstrates the effectiveness, generalization, and robustness of our approach across diverse subgroups. By prioritizing representativeness and coverage over training error guarantees, the method offers promising avenue for enhaneing the models trained on synthetic datascts in real-world scenarios, paving the way for enhanced applications of DD a variety of settings.
ACKNOWLEDGMENTS
The paper is supported by NSF 2112562, ARO W911NF-23-2-0224, and Czech National Science Foundation Project 24-11664S
Reprodueibility Statement We have provided detailed instructions to help reproduce the experi- mental results of this work. In Section. G we provide the statistics of the datasets used for evaluating the proposed method. In Section. H we provide implementation details on the baseline methods and the dyper-paraametrr settings of our method. The evaluation metric design of the domain- shift setting is also included. Additionally, we have attached the adopted source code in the supplementary material, which will further help understand the proposed method.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Haoyue Bai, Yifei Ming. Julian Katz-Samueks and Yixuan Li. Hypo:
Hyperspherical out-of- distribution generalization. The Twelfth International Conference on Learning Representations, pp. 1-27, 2024.
Albert Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg, A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22(2):507-560, 2022.
David B Brown. Large deviations bounds for estimating conditional value-at-risk. Operations Research Letters, 35(6):722-755) 2007.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, pp. 4750-4759, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. General- izing dataset distillation via deep generative prior. In CVPR, pp. 3739-3748, 2023.
Xuxi Chen, Yu Yang, Zhangyang Wang, and Baharan Mirzasoleiman. Data distillation be like vodka: Distilling more times for better quality, In The Twelfth International Conference on Learning Representations, 2024. URL https:: http://pppeeeviieeeeeee net/ wtt.:eeeooooommoleodoeeeiennennn
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jean- Daan-Domiiique Deuschel and Daniel W Stroock. Large deviations, volume 342. American Mathe- matical Soc., 2001.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' A'mour Dan Moldovan, et On robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16458-16468, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Jiawei Du, Yidi Jiang, Vincent Y.F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation. CVPR, pp. 3749-3758, 2023.
John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distribu- tionally robust optimization. The Annals of Statistics, 49(3):1378-144 1406, 2021.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4367-4375. 2018.
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. arXiv preprint arX1v:2311.1552,, 2023a.
Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memory-restricted online continual learning, arXiv preprint arXiv:2305. 16666553 2023b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029-2037. PMLR, 2018
11

---

Published as a conference paper at ICLR 2025
Zeyi Huang. Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-doman generalization. In Computer vision- vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part II 16, pp. 124-140. Springer, 2020.
Yuqi Jia, Saeed Vahidian. Jingwei Sun, Jianyi Zhang, Vyacheslav Kungurtsev. Neil Zhenqiang Gong, and Yiran Chen. Unlocking the potential of federated learning: The symphony of dataset distillation via deep generative latents. CoRR, abs/2312.01553. 2023. doi: 10.48500ARXX.3312..5777 URL wttpthttp///oooooo org/ 48550/ /t..5/////iii..m 2312.
Yang Jiao, Kai Yang, and Dongjin Song. Federated distributionally robust optimization with non- convex objectives: Algorithm and analysis. arXiv preprint arXiv:2307. 14364, 2023.
Mohsen Joneidi, Saeed Vahidian, Ashkan Esmaeili, Weijia Wang. Nazanin Rahnavard, Bill Lin, and Mubarak Shah. Select to better learn: Fast and accurate deep learning using data selection from nonlinear manifolds. Computer Vision and Patern Recognition, CVPR 2020. IEEE Conference on, 2020.
Haun-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1 11102-11118, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5400-5409, 2018.
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations, pp. 1-19, 2022.
Fengming Lin, Xiaolei Fang, and Zheming Gao. Distributionally robust optimization: A review on theory and applications. Numerical Algebra, Control and Optimization, 12(1):159-212, 2022.
Ping Liu, Xin Yu, and Joey Tianyi Zhou. Meta Knowledge Condensation for Federated Learning. In ICLR, 2022a.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. NeurlPS, 35:1 1501110-1,,,, 2022b.
Yanqing Liu, Jianyang Gu, Kai Wang. Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In ICCV, 2023.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06655. 2023.
Dmitry Medvedev and Alexander D' yakoonv.. Learning generate synthetic training data using gradient matching and implicit differentiation. In Proceedings of the International Conference on Analysis of Images, Social Networks and Texts (AIST), pp. 138 138-150, 2021.
Zakaria Mhammedi, Benjamin Guedj, and Robert C Williamson. Pac-bayesian bound for the conditional value at risk. Advances in Neural Information Processing Systems, 33:17919-17930, 2020.
John Nash. Two-person cooperative games. Econometrica: Journal of the Econometric Society, pp. 128-140, 1953.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 5186-5198, 2021.
12

---

Published as a conference paper at ICLR 2025
Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCNIIP),, pp. 4227-4237, 2019.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. Transactions on Machine Learning Research, 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. DataDAM: Efficient Dataset Distillation with Attention Matching. In ICCV, 2023.
Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st international conference on pattern recognition (ICPR2012), pp. 3288-3291. IEEE, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Jiří Souček and Vladimír Souček. Morse-sard theorem for -nalytia functions. Commentationes Mathematicae Universitatis Carolinae, 13(1):45-51, 1972.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data. In ICMI., pp. 9206-9216, 2020.
Saeed Vahidian, Baharan Mirzasoleiman, and Alexander Cloninger. Coresets for estimating means and mean square error with limited greedy samples. In Proceedings of the Thirty-Sixte Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, volume 124, pp. 350-359, 2020.
Bart PG Van Parys, Peyman Mohajerin Esfahani, and Daniel Kuhn. From data to decisions: Distribu- tionally robust optimization is optimal. Management Science, 676)):337773402, 2021.
Konstantinos Vilouras, Xiao Liu, Pedro Sanchez, Alison Q'Neeil and Sotirios A Tsaftaris. Group distributionally robust knowledge distillation. In International Workshop on Machine Learning in Medical Imaging, pp. 234-242. Springer, 2023.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv preprint 2xXi:2303300407,, 2023a.
Serena Wang. Harikrishna Narasimhan, Yichen Zhou, Sara Hooker, Michal Lukasik, and Aditya Kr- ishna Menon. Robust distillation for worst-class performance: on the interplay between teacher and student objectives. In Uncertainty in Artificial Intelligence, pp. 2237-2247. PMLR, 2023b.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 181 1.0599 2018.
Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma. Sparse parameterization for epitomic dataset distillation. In NewrlPS, volume 36, 2024.
Xindi Wu, Zhiwei Deng, and Olga Russakovsky. Multimodal dataset distillation for image-text retrieval. arXiv preprint arXiv:2308. 07545, 2023.
Yihan WU, Xinda Li, Florian Kerschbaum, Heng Huang, and Hongyang Zhang. Towards robust dataset learning. arXiv preprint arXiv:2211. 2022.
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations, 2021. https: http:///ppppppp/e////tp/ee http://rweeeeeeeeeeoooooo///////
13

---

Published as a conference paper at ICLR 2025
Yuanhao Xiong, Ruochen Wang, Minhao Cheng. Felix Yu, and Cho-Jui Hsich. FedDM: Iterative Distribution Matching for Communication--fiiiii Federated Learning. In CVPR, pp. 16323- 16332, 2023.
Huan Xu and Shie Mannor. Robustness and generalization. Machine learning, 86:391-423, 2012.
Eric Xue, Yijiang Li, Haoyang Liu, Yifan Shen, and Haohan Wang. Towards adversarially robust dataset distillation by curvature regularization. arXiv preprint arXiv:2403. 10045, 2024.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzych Ghassemi. Change is hard: A closer look subpopulation shift. International Conference on Machine Learming, pp. 39584-39622. PMLR, 2023.
Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Yibo Zeng and Henry Lam. Generalization bounds with minimal dependency on hypothesis class via distributionally robust optimization. Advances in Neural Information Prucessing Systems, 35: 77576-27599 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 12674-12685, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, pp. 6514- 6523, 2023a.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6514-6523, 2023b.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved Distribution Matching for Dataset Condensation. CVPR, pp. 7856-7865, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. NeurIPS, 35:9813-98227 2022.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
The appendix is organized as follows: Sec. A provides more detailed theoretical analysis. Sec. C presents the proof for proposition 3.4 Sec. and Sec. contaiin the dataset details and implemen- tation details, respectively. Sec. Doffers more experimental results of the proposed robust dataset distillation method. Section [ discusses the broader impact; and finally, Section F presents more sample visualization of the robust dataset distillation method.
A THEORETICAL DETAILS
A. DRO AND GENERALIZATION
Here we present several results from the literature that indicate that a DRO on an empirical risk mini- mization problem exhibits generalization guarantees, i.e., approximately minimizes the population (or expected test dataset) loss. Consider equation
9
repeated here
min max
Q'e
B[F(0",
9 eeQ
s.t.
0" E Erg S)
Q IQ' sr).
1. Theorem in the work of Xu & Mannor (2012) provides bound for the test enror at the DRO solution.
2. Theorem 3.1 the work of Zeng Lam 2022) presents a probability that
E [F Q)] - B[F opt: Q)] 6 as function of where we denote the DRO and the exact minimum, respectively.
A.2 LARGE DEVIATIONS AND DATA DRIVEN DRO
The analysis of the theoretical convergence and robustness properties of method will rely significantly on the theoretical foundations of data driven DRO established in the work of Van Parys al. (202I). To this end, we review few pertinent definitions. A predictor is a function 12 X + R that defines a model as applied to a data distribution. A data driven predictor uses an empirical distribution of samples Py in the prediction.
The sample average predictor is given as
c0, Pp)
Σ F(0.E))
Let 0 arg min 6(0, IP) be the data driven predictor.
ordering = is introduced to rank the set of predictors, with
(ē, 01) = (82, 02) and only if è (0 (IP', PP) s (2(() v0, PEP
The Distributionally Robust Predictor is defined as,
P.; = sup (c(0, D, (P) <
PEP
where (A, B) is the mutual information of random variables A and B.
They define the optimization problem
min (2. 8) (8,6)
(15)
lim Σ & 1og 1o:0 (c(0((r,))) -(0,Pr)) s -r, VPEP E
s.t.
T
where the minimum the lower bound with respect to the ordering - of all feasible (LDP satisfying) predictors.
15

---

Published as a conference paper at ICLR 2025
Van Parys et al. (2021) prove that the distributionally robust predictor solves equation 13 in Theorem 4, for discrete distributions, and Theorem 6, for continuous ones.
We are constructing synthetic dataset, while simultancously considering training subsamples to form the empirical measure at each itcration, suggesting that the data-driven framework can fit the problem of interest. Formally, consider solving optimization problems defined set of measures as decision variables,
min max QEQ
E(F 2)
s.t.
0* E arg ming F(O,S)
by solving the data driven DRO approximation for the test error,
min max S eeQ
B[F(0", 2))
s.t.
0" E arg ming F(0,5)
(16)
2 (0' TQ r)
Note that the form is the nested DRO problem described earlier. At the upper layer, there is an uncertainty set regarding the choice of @ Q targeting group robustness. There is the data-driven DRO in the lower level, which is an algorithmic approximation of the population risk minimization with established accuracy guarantees.
B GENERALIZE TO BROADER DATASET DISTILLATION SCENARIOS
Algorithm 2 Zero-order Dataset Distillation
Input: Initial synthetic set S, distilling objective L(S)
Execute:
For epoch E = 11 Compute with en (O, T):
g(S) ~ VL(S) = M
1
Σ
LSS + ovi) - L(S)
(17)
Set stepsize (diminishing stepsize rule, stepwise decay). For instance:
0.1/vI+ E
(18)
Set - sg
Existing dataset distillation methods adopt differentiable matching metrics to optimize the synthetic data S. However, there can be circumstances where the gradient of the objective cannot be directly acquired. In this case, dataset distillation can still be performed by zero order approximation of the gradient. Starting with some initial S and proceed as in Algorithm 2, standard gradient descent approach can be applied, using a zero order approximation of the gradient of concerning the dataset S, that is gS) VL(S), computed only using evaluations of L(S).
We can use any method from Berahas et al. (2022) to obtain gradient estimate of g R Vsl(S), here the featured procedure is Gaussian smoothing. The proposed robust dataset distillation method is still able to be applied towards L(S) for more robust optimization.
C PROOF OF PROPOSITION 3.4
Consider the quantity:
N log poe (P) ((
(19)
Now recall the expression for CVaR:
16

---

Published as a conference paper at ICLR 2025
Table 6: Top-I test accuracy on CUB-200 transferred from ImageNet subsets. All the results are reported based on the average of 5 runs. R indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked bold.
Transfer Learning
GLaD
GLaD
IeageNet-50 CUB-200
49.2
54.6
Table Top-1 test accuracy on more challenging test on the syn data trained model. All the results reported based on the average of 5 runs. R indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.
Group
GL.aD
GLaDD
ImageNet-Il 10 ImageNet-A
56.0+1.1
59.6
ImageNet-Il 10 ImageNet-B
43.0
47.2
CVaR[F(0:ca] = - E F(0, fol a
+fa(a- - 1
[o: Ezree 1[F(0; s sal]
and
fa min E R:
Σ
1[F(0: $ fal
Step 1: CVaR itself satisfies a LDP (Brown 2007; Mhammedi et al. 2020 With these results, we can relate the empirical to the population CVaR.
Step 2: Notice that as long as a < 1. then minimizing the CVaR will also minimize equation 19
Step 3: There is at least one estimator satisfying the LDP, namely, the DRO. Thus minimizing the empirical CVaR, by the feasibility the LDP, must result in an estimator that satisfies the LDP.
D MORE EXPERIMENTS AND ANALYSIS
D.1 MORE CHALLENGING TESTING SCENARIOS
In Section
demonstrate that the proposed robust dataset distillation method is able enhance the robustness of the generated data on shifted or truncated testing sets of the same classes. this section, we evaluate RDD on more challenging testing cases.
4
Downstream Fine-tuning The robustness of neural network can be tested in transfer learning scenarios (Djolonga et al. 202T). Accordingly, we conduct the experiment of transferring the model trained distilled data to downstream tasks. More concretely, the model is first pre-traie tre-trained distilled data from the 50-class subset ImageNet (Cazenavette et al.) 2023). Then is fine-tuned CUB-201 --0- Wah et al. (2011). We use the top-1 accuracy CUB-200 to evaluate the transferabilitty of the distilled data, and the results are shown in Table The results suggest that the data distilled with RDD applied significantly enhances the transferability of the trained model.
One-shot Direct Transfer A more challenging case would be directly applying the model trained on the distilled data on completely different classes. in the original model, a classifier involved to predict the classification probability. directly applying the classifier to different classes would be infeasible. Thus, we adopt one-shot retrieval-style evaluation approach instead. In this approach, the model is initially trained using the distilled data of 10 classes and subsequently tested another set of 10 classes. During testing, the corresponding images are passed through the backbone to obtain embedded features, with which similarity matrix is computed. We use each sample query, and check whether the most similar sample among the remaining samples belongs to the same class and report the top-1 accuracy.
17

---

Published as a conference paper at ICLR 2025
Table 8: Top-1 test accuracy on standard and robustness lesting sets. All the results are reported based on the average of 5 runs. The experiment is conducted with DREAM and PDD on the CIFAR-16 10 IPC-50 setting. indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.
Method
Acc
Cluster-min
Noise
Blur
Invert
PDD
67.910.2
63.9
58.2
48.9 18.9
25.7
PDD
68.710.6
65.1
59.4
50.6
26.7
DREAM
69.4
64.7.0.6
18.8
50.1
25.7
DREAM
69.7
65.5
59.8
50.7
26.9
Table Top-1 test accuracy on different cluster numbers. All the results are reported based on the average of 5 runs. The experiment is conducted with GLaD on CIFAR-I0 under the IPC of 50. R indicates the proposed robust dataset distillation method applied. The best results marked bold.
Cluster
Acc
Cluster-min
Noise
Blur
Invert
5
61.640.8
$0.1
54.6
40.3
17.5
10
62.5
56.6+1.0
55.7
41.120.6
18.0
20
60.9
54.3
52.8
39.8
17.3
30
59.9
53.2
51.5
38.9
17.141.
The results are presented in Table 7 For these two groups, the model is firstly trained on distilled ImageNet-10 and then tested on ImageNet-A and ImageNet-B. respectively. The subset split of ImageNet-A and ImageNet-B follows the setting in GLaD (Cazenavette et al. 2023). With the robust optimization applied, the distilled data also shows better robustness on unseen classes, demonstrating the efficacy the proposed method enhancing the generalizability the distilled data.
D.2 APPLICATION ON MORE DD METHOODS
In addition to IDC (Kim et al. 2022) and GLaD Cazzenavette et al. 2023). we further conduct experiments on CIFAR-10 10 with DREAM Liu et 2023) and PDD (Chen al. 2024) baselines in Table & Similar to the implementation for IDC. we use the extra CVaR criterion during the model updating. By the application of robust optimization, the reported metries have been improved. especially on the domain-shifted settings. This result demonstrates the generality of the proposed framework across DD methods.
D.3 EFFICIENCY EVALUATION
As the proposed robust dataset distillation method involves extra CVaR loss calculation during the model updating, the efficiency issue might be a concern. Accordingly, we record the required time to complete the extra robust optimization on IDC. The baseline requires 70s to finish a loop, while CVaR loss calculation takes up extra 30s. The robust optimization takes up less than 50% original calculation time. The extra time is not overwhelming, but brings significant improvements on the robustness of the distilled data.
D.4 CHOICE OF THE NUMBER OF CLUSTERS
The CVaR loss calculation involves separating the training set into several sub-sets. Different settings of sub-set division can have influence on the CVaR optimization effects. In this work, the choice of the number of clusters is primarily based on common settings observed in DD benchmarks. Typically. our method evaluated across different manges-per-class (IPC) settings, such as 10, and 50. For an IPC below 10, we simply use the IPC as the number of clusters, and treat the synthetic samples the cluster center. However, when dealing with larger IPC, too many clusters might potentially result in insufficient samples in each cluster for CVaR loss calculation. To address this concern, conduct parameter analysis on the number of clusters specifically under an IPC of 50, where the number of clusters varies from 5 30. Synthetic samples of the same number randomly selected serve
18

---

Published as a conference paper at ICLR 2025
Table 10: Top-1 test accuracy on different mini-batch sizes, All the results are reported based on the average of 5 runs. The experiment is conducted with GLaD on CIFAR-10 under the IPC of 10. R indicates the proposed robust dataset distillation method applied. The best results are marked bold.
Mini-batch Size
Acc
Cluster-min
64
46.9
40.3
128
20.
45.1
256
50.2
46.7=0.9
512
45.7
Table 11:
Ablation study on the initialization of synthetic samples. All the results are reported based on the average of runs. R indicates the proposed robust dataset distillation method applied. "Cluster" means that the synthetic samples are initialized with clustering centers.
Dataset
IPC
GLaD
GLaD
GLaD +Cluste
28. 28.04..8
29.2
29.4
CIFAR-10
10
46.710.6
50.2+0.5
50.3
33.5
36.4
36.5
ImageNet-10 10
50.9
55.2
55.0
as cluster centers, and sub-sets are accordingly separated based these centers. The experiment is conducted with GLaD on CIFAR-10, and the results are shown Table
When the number of clusters increases over IPC, as the total training sample number is fixed, the sample number belonging to each sub-set would decrease. And the insufficient samples from each cluster cause sub-optimal CVaR optimization, leading to a performance drop. By empirical observation, we fix the number clusters to 10 for large-IPC settings.
D.5 CHOICE OF MINI-BATCH SIZE
As the group DRO does require abundant samples for precise calculation, we set the sample number in mini-batch as 256 in experiments. We further conduct analysis the parameter setting, and the results are listed in Table TO (on CIFAR-10). When there are limited samples in a mini-batch, the performance will be largely influenced. Applying mini-batch size around 256 brings mild influence on the validation performance. The slight performance drop when the mini-batch size is set 512 might be due less diversity between different mini-batches.
D.6 CHOICE OF INITIALIZATION
As the clusters are separated based on the synthetic samples, different initialization of synthetic samples can influence the effects of CVaR loss calculation. In our experiments, we follow the baseline setting, which random initialization with real samples for both IDC (Kim et 2022) and GLaD CCzenavette et al., 2023). We accordingly conduct an ablation study to evaluate the robustness RDD on the initialization. In addition to random sampling, we also test initializing the synthetic samples with clustering centers, which leads to more even distribution. The results are shown in Table TT where clustering initialization yields similar results to random initialization. Although random initialization cannot guarantee even distribution at the beginning, during the subsequent optimization process, the algorithm still robust enough to handle different initializations and provide stable performance improvement over the baseline.
D.7 EVALUATION ON DOMAIN GENERALIZATION BASELINES
The proposed robust dataset distillation method mainly focuses enhancing the robustness of distilled datasets. The method can also be combined with other domain generalization methods to further enhance the robustness of model training. We have included the three generalization methods MMD (Li et al. 2018). RSC (Huang et al. 2020). and HYPO (Bai et al. 2024) Table serve as training pipelines. The results suggest that the one hand, RDD consistently provides performance
19

---

Published as a conference paper at ICLR 2025
Table 12: Evaluation of the proposed robust dataset distillation on other domain generalization methods with GLaD. All the results are reported based on the average of 5 runs. Ω indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.
CIFAR-10
ImageNet-10
Method
GLaD
GLaD
GLaD
GLaD
Baseline
46.7:0.6
50.2
50.9_1.0
55,2
MMD
47.1
51.3
51.8_1.3
56.5
RSC
47.9
52.5
52.4=1.1
56.8
HYPO
49.0
53.2
53.6_1.2
58.1
Table 13: Validation accuracy more ImageNet sub-sets in comparison with GLaD. The data is distilled with ConvNet-5 and evaluated with ResNet-10 under the IPC of 10. R indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.
Acc
Cluster-min
Dataset
GLaD
GLaD
GLaD
GLaD
ImageNet-A
53.9
57.5
20.5
45.8
ImageNet-B
50.3
53.820.8
42.9
47.0
ImageNet-C
49.2+0.8
51.3
28.2+
32.8 #0.6
ImageNet-D
39. +
40.9
27.0.
31.3
ImageNet-E
38.9
41.1
25.8
30.0
improvement different training pipelines. On the other hand, the combination of RDD and other domain generalization methods can further improve the results over the baseline.
D.8 RESULTS ON MORE IMAGENET SUB-SETS
GLaD (Cazenavette et al. 2023 designs experiments on multiple ImageNet sub-sets. We further provide results these sub-sets comparison with GLaD in Table The sub-set division remains consistent with that in the work of Cazenavette et al. (2023). Our proposed robust DD method consistently outperforms the baseline, particularly in terms of the worst accuracy across clustered testing sub-sets. This observation underscores the stability and versatility of the proposed method.
E RELATED WORK
Dataset Distillation Dataset distillation (DD) seeks distill the richness of extensive datasets into
compact sets of synthetic images that closely mimic training performance (Sachdeva & McAAleyy
2023). These condensed images prove invaluable for various tasks, including continual learning (Gu
et al.
2023b), federated learning (Liu et al.) 2022a Jia et al. 2023), neural architecture search (Such
al. 2020; Medvedev D' yakonov [2021], and semi-supervised learning (Vahidian al. 2020
Joneidi et al. (2020). Existing DD methodologies can be broadly categorized into livee optimization
and training metric matching approaches. Bi-level optimization integrates meta-learning into the direct optimizatino target (Zhou) cct al. 2022; Loo et al. 2023). Conversely, metric matching techniques refine synthetic images by
surrogate image update process with the validation performance
aligning with training gradients (Kim et al. 2022 Liu et al.
2023). feature distribution (Sajedi et al.)
2023 Zhao et al. 2023), or training trajectories [Wu et al.
2023
Du et al.
2023) compared to the
original images. Data parametrization (Kim et al. 2022
Liu al. 2022b
Wei et al. 2024) and
generative prior (Cazenavette et al., 2023 Gu et al.
2023a
Wang et al. 2023a are also considered
for more efficient DI) method construction.
Robustness in Dataset Distillation To the best of our knowledge. this is the first work subgroup accuracy specifically, or even DRO generally, for Dataset Distillation. We consider a few classes of related works. In the context of DD, adversarial robustness is another popular notion of robustness,
20

---

Published as a conference paper at ICLR 2025
Table 14: The configurations of different datasets
Training Dataset
Class
Images per class
Resolution
SVHN
10
6000
32x32
CIFARIO
10
5000
32x32
ImageNet-10
10
1200
128x128
ImageNet subsets
10
1200
128x128
which simply minimization with respect to the worst possible sample in the support of some perturbation the data sample, rather than the distribution. Adversarial Robustness is more conservative than DRO, however, in some security-sensitivy circumstances, this is a more appropriate framework (Wu al. 2022 Xue al. 2024). In the field of knowledge distillation, wherein more parsimonious model, rather than dataset, is of interest, group DRO has been considered in the work of Vilouras al. (2023) and Wang et al. (2023b).
The concept of group DRO is considered and an implementation thereof is demonstrated in the work of (Sagawa al. (2019), complemented with regularization strategy that appears to assist in performance for small population subgroups. The coverage of disparate data distributions is also concern in Non-IID federated learning. [īio al. (2023) present convergence theory for DRO in federated setting.
F VISUALIZATION OF SYNTHESIZED SAMPLES
We provide the visualization of synthesized samples in different datasets in Figure 5 to Figure Each row represents a class.
10 10
0
0
10
0
0
0
0
21
2
2
2
2
2
33
3
3
3
3
315
4
4
4
1
4
5 5
5
6
5
5
6
6
6
6
6
6
6
7
7
7
07
7
17
37 7 8
8 8
8
18
8 8
8
8
9
9
9 9
919
9
919
Figure 5: Synthesized samples of CIFAR-10 (left) and SVHN (right).
G DATASET STATISTICS
We evaluate our method on the following datasets:
SVHN Yuvall 2011) dataset for digits recognition cropped from pictures of house number plates that is widely used for validating image recognition models. It includes 600,000 32x32 RGB images of printed digits ranging from 0 to 9. SVHN comprises three subsets: training set, testing set, and extra set of 530,000 less challenging images that can aid in the training process. SVHN dataset is released with a CCO:Public Domain license.
21

---

Published as a conference paper at ICLR 2025
Figure 6: Synthesized samples of ImageNet-10 (left) and ImageNet-A (right).
Figure 7: Synthesized samples of ImageNet-B (left) and ImageNet-C (right).
CIFAR-10 (Krizhevsky et al., 2009) is subset of the Tiny Images dataset and consists of 60000 32x32 color images. The images are labeled with one of 10 mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. There 6000 images per class with 5000 training and 1000 testing images per class. CIFAR-10 dataset is released with an MIT license.
ImageNet-10 and ImageNet subsets is the subset of ImageNet-IK IK (Deng et al. 2009) containing 10 classes, where each class has approximately I, 200 images with a resolution of 128 128. The individual configurations of these datasets shown in Table 14 No license is specified for ImageNet.
H IMPLEMENTATION DETAILS
The proposed method can applied to various popular dataset distillation frameworks. this paper, perform experiments pixel-level distillation method IDC Kim et al.) 2022) and latent-level method GLaD (Cazenavette et al.) 2023) to substantiate the consistent efficacy our method. The experiments are conducted on popular dataset distillation benchmarks, namely SVHN, CIFAR-10,
22

---

Published as a conference paper at ICLR 2025
Figure 8: Synthesized samples of ImageNet-D (left) and ImageNet-E (right).
and ImageNet (uvva 2011 Krizhevsky et al. 2009 Deng et al. 2009). The images of SVHN and
CIFAR-10 are resized to 32 32, while those from ImageNet-Il 10 resized to 128 128, representing diverse resolution scenarios. The split of ImageNet- 10 subset follows Kim et al. (2022)
The CVaR loss and the Cluster-min metric caleulation both involve clustering. For the CVaR loss, Euclidean distance adopted to evaluate the sample relationships. The real samples are assigned to the synthetic sample with the smallest distance. Due to the CVaR loss calculation involving number of samples, the mini-batch size during model updating is increased to 256. In cases where the IPC setting is less than 10, the cluster number in Eq. 2 is set equal to IPC. For larger IPCs, the cluster number is fixed at 10, with 10 random synthesized samples chosen as the clustering centers. The ratio in CVaR loss is set to 0.8. For the Cluster- metric calculation, we first apply the standard MMaans algorithm to partition the original test set into 10 subsets. Specifically, as outlined in Algorithm the subsampling and clustering processes are performed within each class. For the cluster-min metric calculation, we execute the clustering in the RGB space. This method ensures that clusters are formed based on common features shared by these samples rather than random selection, resulting in clusters that include samples from different classes. We have also verified that each cluster contains samples from all classes.
ample
For the Cluster-min metric calculation, standard KMeans algorithm is conducted to separate the original testing set into 10 sub-sets. To clarify, According to Algorithm 1, the subsampling and clustering processes
based on each class. As for the cluster-min metric calculation, the clustering in the RGB space. This approach ensures that clusters are formed based on common features rather than random selection and include samples from different classes. We have examined the cluster division to make sure that each cluster contains samples from all classes.
conduct
All the experiments conducted on a single 24G RTX 4090 GPU.
For fair comparison, the experiment settings are generally kept the same as in the original papers. Detailed explanations are listed below for both baselines.
H.1 IDC DETAILS
A multi-formation operation is proposed in IDC to increase the information contained in each sample. The multi-forrmtiinn factor is set as on CIFAR-10 and 3 on ImageNet, which is kept the same as the original implementation.
On SVHN and CIFAR-10 datasets, a 3-layer ConvNet GGiaris & Komodakis 2018) is employed for distillation, while on ImageNet, Re (He al. 2016) is utilized. After distillation, we conduct
23

---

Published as a conference paper at ICLR 2025
validation procedures on ConvNet-3 for 32x datasets and ResNet-10 for ImageNet, ensuring fair and consistent basis for comparison.
H.2 GLAD DETAILS
There are multiple different matching metrics presented in GLaD. Here adopt gradient matching (DC in GLaD paper) in our experiments for two primary reasons. Firstly, gradient matching is deemed be more practical when compared with alternative metrics. Secondly, gradient matching incorpo- rates model updating, providing a convenient avenue for embedding the proposed distributionally robust optimization method.
On SVHN and CIFAR-10 datasets, similar to IDC, a 3-layer ConvNet (Gidaris & Komodakis, 2018) employed for distillation. 5-layer ConvNet is adopted for ImageNet, which is kept the same as in the original paper. For better comparison, adopt the validation protocol in IDC as it yields better performance. The results of baseline GLaD are also re- ee-produced with the same protocol.
H.3 DISTILLING ITERATION
Initially, the iteration randomly initializes network according to the architecture setting for each baseline. Then the synthetic images are updated to match the gradients from the network. The network supplying the gradient for the images is updated following the image update. During the network updates, the robust objective proposed in this paper is employed for training. The network training is restricted to an early stage, using only 4,000 images for IDC and 1,000 images for GLaDD, 100 steps of synthetic image update together with the network training forms an iteration for IDC, and 2 steps for GLaD. For CIFAR-10 and ImageNet subsets, we adopt 2,000 and 500 iterations to complete the distilling process, respectively.
I BROADER IMPACTS
The primary objective of dataset distillation is to alleviate the storage and computational resource demands associated with training deep neural networks. This need becomes particularly pronounced in the era of foundational models. Dataset distillation endeavors expedite environmental sustainability efforts. Our proposed method, viewed from this perspective, markedly diminishes the resources needed for the distillation process. We aspire draw attention practical dataset distillation methods within the computer vision community, thereby fostering the sustainable development of society. Further, this work does involve direct ethical concerns. Our experiments utilize publicly available datasets, namely ImageNet, SVHN, and CIFAR-10, In forthcoming research, we are committed to addressing issues related generation bias and diversity when constructing small surrogate datasets.
24


Reviews:
[{"review_id": "NXbClM6FxP", "reviewer": "Reviewer_DaAx", "summary": "This paper proposes an algorithm for dataset distillation by incorporating distributionally robust optimization into it. There is theoretical justification and empirical validation of the proposed algorithm.", "questions": "In Line 017, what does \"targeting the training dataset\" mean?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "There are both theoretical and experimental demonstrations of the effectiveness of the algorithm.", "weaknesses": "1. There seems to be a mismatch between the motivation and the experiments. The motivation emphasizes regions with low population density, which usually correspond to the worst-group performance in subpopulation shift [1]. However, the main experiments related to distribution shift are conducted on test sets with perturbations or the worst group induced by an additional clustering process. It would be better to conduct more experiments on subpopulation datasets included in [1]. \n2. The introduction has too many paragraphs, which makes the logic of the introduction tedious with poor readability. \n\nSome minor issues:\n\n- In Line 049, \"some technique\". \n- In Line 129 and 131, \"Algorithm\" \"Numerical Results\" their first letters do not need to be capitalized. \n- In the last paragraph of introduction, Section 3 is not mentioned.\n\n[1] Yang, Yuzhe, et al. \"Change is Hard: A Closer Look at Subpopulation Shift.\" *International Conference on Machine Learning*. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1LGsJODM4y", "reviewer": "Reviewer_p8Rd", "summary": "The work proposes a robust dataset distillation approach that incorporates distributional robust optimization (DRO) to enhance generalization and performance across subgroups. This method combines clustering with risk-minimized loss to conduct dataset distillation. By prioritizing representativeness and coverage over training error guarantees, the approach enhances the models trained on synthetic datasets for real-world scenarios. Both theoretical analysis and empirical validation on multiple standard benchmarks are provided, demonstrating the effectiveness of the proposed approach.", "questions": "Please refer to the detailed suggestions provided in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes applying distributional robust optimization to dataset distillation, providing a reasonable approach to enhance generalization.\n\n2. Drawing on distributional robust optimization theory, this work establishes a theoretical foundation to support the proposed approach to dataset distillation.\n\n3. The paper is well-structured, with clear algorithm block and effective visualizations that enhance the presentation of the work.", "weaknesses": "1. The empirical experiments focus primarily on CIFAR-10, ImageNet-10. Extending the evaluation to larger, real-world datasets with a greater number of classes would better demonstrate the effectiveness of proposed approach in generalization and robustness under real-world conditions.\n\n2. Comparing the proposed approach with additional baseline method addressing generalization and robustness would provide a more comprehensive assessment of the proposed approach, including comparisons with baseline methods such as [1, 2, 3].\n\n3. It would be valuable to visualize the robust inference tasks using real-world data rather than illustrative visuals, which could provide more insight and observations in real-world scenarios.\n\nReference:\n\n[1] Domain generalization with adversarial feature learning. CVPR 2018.\n\n[2] Self-challenging Improves Cross-Domain Generalization. ECCV 2020.\n\n[3] HYPO: Hyperspherical Out-of-Distribution Generalization. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cr5Ux7AQB1", "reviewer": "Reviewer_1jHv", "summary": "This paper proposes a new data distillation approach emphasizing the subgroup coverage and generalization ability of the models trained on the synthetic dataset. This paper provides a theoretical analysis rooted in Distributionally Robust Optimization (DRO) and verifies the effectiveness of the proposed method with various experiments.", "questions": "I have stated many of my suggestions and concerns in the weaknesses section. Below, I have further questions with some minor issues.\n\n* While theoretical analyses have been provided, I wonder whether the proposed method would affect the convergence rate and make it more difficult to find an optimal solution for the proposed objective. I would appreciate an empirical time complexity analysis regarding the proposed method.\n\n* In Eq.10, what does the *N* represent? I did not see any introduction of the *N*.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper considers the group coverage and generalization ability of the synthetic dataset in Data Distillation(DD), which is interesting and novel.\n\n* The introduced algorithm is clear and the theoretical analysis seems solid.\n\n* The numerical results do show the effectiveness of the proposed algorithm. Meanwhile, the figures (Fig.3, 4) seem to indicate that the proposed method improves group coverage.", "weaknesses": "* While the paper emphasizes group coverage and generalization in data distillation, the experiments are mainly conducted on IID datasets. More experimental results in scenarios with distribution shift between training and testing sets (such as long-tail classification, subpopulation shift, domain adaptation, domain generalization, etc) can further validate the improvement in group coverage and generalization ability,\n\n* According to Algorithm 1, the initialization of the synthetic dataset seems very important because it involves how the training data samples are clustered into subgroups. It may require further ablation study to verify the stability of the proposed algorithm.\n\n* The proposed method seems like a plug-in as it only modifies the objective for data distillation and could be combined with any data distillation methods. A more sophisticated comparison between the proposed method with other objectives regarding training gradients, feature distributions, and training trajectory could help readers better understand the improvement of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vnPkOadMKs", "reviewer": "Reviewer_gBwM", "summary": "This paper proposes a robust dataset distillation method that enhances generalization across underrepresented subgroups by using a double-layer distributionally robust optimization (DRO) approach. Traditional dataset distillation often compresses training data into synthetic sets by matching training characteristics but may struggle with subgroup generalization. To improve this, the authors cluster the training data around synthetic data points and apply a Conditional Value at Risk (CVaR) loss to minimize worst-case subgroup errors, making the synthetic dataset more representative of diverse data distributions. Experimental results show that this method significantly improves robustness under domain shifts, outperforming baseline methods on benchmarks like CIFAR-10 and ImageNet-10, particularly in challenging conditions like noise and blurring.", "questions": "Please see weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- As far as I know, this paper firstly addresses a major limitation in standard dataset distillation (underrepresented or rare subgroups) by applying a double-layer distributionally robust optimization (DRO) framework. This approach ensures the synthetic dataset better represents the full diversity of the data, reducing performance drops across different data subgroups. \n\n- This paper provides not only empirical evidence of the proposed method's robustness against domain shifts such as noise, blur, and inversion, but also a theoretical analysis of the effectiveness of CVaR in the loss function to enhance robustness.\n\n- The proposed method is designed to be adaptable and can integrate with various existing distillation techniques, such as gradient or distribution matching. This modularity makes it compatible with a range of distillation methods, which are still actively developing and evolving.", "weaknesses": "- The proposed DRO approach, particularly with clustering and CVaR, may introduce significant computational overhead. This added complexity could be amplified for large-scale datasets due to the clustering of training (real) data. However, the paper does not discuss the computational overhead of the proposed method.\n\n- Although the method shows promising results on benchmarks like CIFAR-10 and ImageNet-10, the experiments are limited to controlled domain shifts (e.g., noise, blur). Testing under more realistic settings, such as in transfer learning, would further validate its robustness and practical relevance. For example, one could (1) train neural networks on a synthetic dataset distilled from a coarse-grained dataset (e.g., ImageNet) and (2) fine-tune and evaluate them on a fine-grained dataset (e.g., Birds 200). This setup would better illustrate the method's effectiveness in addressing the challenges posed by rare subgroups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)