Group Distributionally Robust Dataset Distillation with Risk Minimization

## Abstract

Abstract Dataset distillation (DD) has emerged as a widely adopted technique for crafting a synthetic dataset that captures the essential information of a training dataset, facilitating the training of accurate neural models. Its applications span various domains, including transfer learning, federated learning, and neural architecture search. The most popular methods for constructing the synthetic data
rely on matching the convergence properties of training the model with the synthetic dataset and the training dataset. However, targeting the training dataset must be thought of as auxiliary in the same sense that the training set is an approximate substitute for the population
distribution, and the latter is the data of interest. Yet despite its popularity, an aspect that remains unexplored is the relationship of DD to its generalization, particularly across uncommon subgroups. That is, how can we ensure that a model trained on the synthetic dataset performs well when faced with samples from regions with low population density? Here, the representativeness and coverage of the dataset become salient over the guaranteed training error at inference. Drawing inspiration from distributionally robust optimization, we introduce an algorithm that combines clustering with the minimization of a risk measure on the loss to conduct DD. We provide a theoretical rationale for our approach and demonstrate its effective generalization and robustness across subgroups through numerical experiments. Source Code .

## 1 Introduction

Dataset distillation (DD) is a burgeoning area of interest, involving the creation of a synthetic dataset significantly smaller than a training set yet demonstrating comparable performance on a model. This practice has gained prominence in various computation-sensitive applications, providing a valuable means to efficiently construct accurate models. The standard optimization objectives that are used to steer the construction of the synthetic data typically aim to foster either distributional similarity to the training set or exhibit similar stochastic gradient descent (SGD) training dynamics as the original dataset. Notably, recent literature suggests that the latter category has proven more successful. Intuitively, this success can be attributed to the rationale that, with considerably fewer samples, prioritizing the most relevant information for training and model building becomes more judicious.

However, in practical terms, executing the same training task with a reduced computational load is not universally applicable, nor is it the standard application of DD. Instead, the usual scenario involves re-training and inference on a synthetic dataset under circumstances that are *similar but perturbed* to the original task of interest. To this end, we aim to execute DD in a manner that specifically addresses the challenges posed by distributional shifts. Additionally, in the context of online inference, it is common to observe alterations in the overall prior distribution of samples, particularly in terms of subgroups. In practice, within a stream of samples, there are often clumps belonging to specific subgroups, collectively representing a small portion of the sample space. The ability of DD to facilitate the construction of a model that performs effectively across time varying distributional shifts and subgroup appearance is a noteworthy practical concern. This aspect has yet to be prioritized in DD literature, as per our current understanding.

When evaluating test accuracy, a commonly utilized algorithmic framework is Distributionally Robust Optimization (DRO). Within DRO, the approach involves solving a bilevel optimization problem where the objective is to minimize the loss over a data distribution, subject to the constraint that this distribution is a specific probabilistic distance away from the population distribution:

$$ $\min\limits_{\theta}\max\limits_{\mathcal{Q}\in\mathbb{Q}},\,F(\theta,\mathcal {Q}),\,\mathbb{Q}=\{\mathcal{Q}:\mathcal{D}(\mathcal{Q}||\mathcal{P})\leq\Delta\}$ (1) $$

where $F$ is the loss function, $\theta$ are the parameters, $\mathbb{Q}$ is the distributional uncertainty set, $\mathcal{D}$ is an f-divergence and $\mathcal{P}$ is the population distiribution.

Intuitively, there are two considerations through which DRO can be understood to be the appropriate framework for addressing the generalization concerns discussed above. First, one can consider that a test and training distribution are both subsamples from the same population distribution, and so there must be a significant resemblance between the two. Indeed, the work (Xu & Mannor, ) presented arguments indicating theoretical equivalence of testing to training error as robustness to perturbations of the data distribution, and the generalization accuracy.

Second, a conventional least likelihood classifier inherently assigns greater weight to samples with larger prior distributions compared to those with lower overall probability density. This inherent bias poses a fundamental risk of underperformance in small subgroups within the population. DRO has been conjectured and observed to mitigate this, both generically (Duchi & Namkoong, ) and with explicit quantification of subgroups in the distributional uncertainty set (Oren et al., ).

Nonetheless, the conventional formulation of DRO, as depicted in (1), has been revealed to be excessively conservative, effectively optimizing towards the exact training distribution and hindering practical generalization (Hu et al., ). Research indicates that a more precisely defined distributional uncertainty set, focusing on specific collections of probability distributions from subsamples of the population rather than encompassing all possibilities in a ball, yields improved generalization performance. An illustrative example of this refined approach to DRO can be found in (Oren et al., ), where subgroups of topics are considered for training language models.

In a formal sense, we define that each subsample involved in an inference task is drawn from a union of closed, convex, connected subsets of the population distribution. An intuitive representation of the envisioned scenario is illustrated in Figure 1.

Figure: Figure 1: A visual representation of the robust inference task involves the partial partitioning of the population distribution across subgroups. A classifier is considered robust when it demonstrates high performance regardless of the specific group’s data it is conditioned on. For instance, at a particular time, a steady stream of samples from $\mathcal{X}|c_{3}$ may appear to the classifier, while it may be that the region of sample space defined by this subgroup is both geometrically small (as in the Figure) as well as having low overall prior density. If this subgroup’s behavior is particularly anomalous, a model trained only on minimizing empirical risk may perform poorly on this subgroup, and a synthetic dataset aiming to ensure similar training will poorly represent $\mathcal{X}|c_{3}$.
Refer to caption: extracted/5461728/Images/robustdd.png

Inspired by these concepts, this paper introduces a DD procedure designed specifically for this objective. We synchronize the dataset construction with the loss measured by robust criteria. To circumvent the challenges posed by the intractable bilevel DRO, we employ risk measures applied to the data loss as the criteria during training. At each iteration, a zero-order gradient estimate of the synthetic dataset, considering the sampled data’s performance on our robust criteria, is calculated. The bilevel DD problem is then minimized by a sequence of these iterations.

This presents an important shift in perspective with respect to dataset distillation. Standard consideration of DD broadly considers each data point independently, rather than their interrelationships. This is unfortunate, as the small size of the synthetic dataset suggests that there is both tractability in computing collective valuations of the data, and necessity in ensuring parsimony with minimal redundancy in the information in the dataset. In the context of deploying memory-sensitive AI pipelines for DD, one can anticipate the implementation of time-ordered stratified sampling across subgroups and adaptations to address general distributional shifts. Giving thoughtful attention to the dataset’s coverage across the actual support can markedly enhance the practical utility of DD.

Our paper continues below as follows. In Section 2 we review related works. Next, we present our Algorithm, we analyze the optimization problem defining the notion of generalization of interest and argue how our method effectively targets this criterion in Section 3. In Section 4 we present Numerical Results validating the superior overall and subgroup generalization performance on standard machine learning benchmarks. Finally, we conclude this work in Section 5.

## 2 Related Work

### 2.1 Dataset Distillation

Dataset distillation (DD) seeks to distill the richness of extensive datasets into compact sets of synthetic images that closely mimic training performance. These condensed images prove invaluable for various tasks, including continual learning (Gu et al., ), federated learning (Liu et al., ; Jia et al., ), neural architecture search (Such et al., ), and semi-supervised learning (Vahidian et al., ; Joneidi et al., ). Existing DD methodologies can be broadly categorized into bi-level optimization and training metric matching approaches. Bi-level optimization integrates meta-learning into the surrogate image update process (Zhou et al., ; Loo et al., ). Conversely, metric matching techniques refine synthetic images by aligning with training gradients (Kim et al., ; Liu et al., ), feature distribution (Sajedi et al., ; Zhao et al., ), or training trajectories (Wu et al., ; Du et al., ) compared to the original images.
There are also some methods involving generative prior for DD (Cazenavette et al., ; Wang et al., ; Gu et al., ).

### 2.2 Robustness in Dataset Distillation

To the best of our knowledge, this is the first work on subgroup accuracy specifically, or even DRO generally, for Dataset Distillation. We consider a few classes of related works.
In the context of DD, adversarial robustness is another popular notion of robustness, which is simply minimization with respect to the worst possible sample in the support of some perturbation on the data sample, rather than the distribution. Adversarial Robustness is more conservative than DRO, however, in some security-sensitive circumstances, this is the more appropriate framework (Wu et al., ).
In the field of knowledge distillation, wherein a more parsimonious model, rather than dataset, is of interest, group DRO has been considered in (Vilouras et al., ) and (Wang et al., ).

Group DRO is considered and demonstrated effectiveness for in (Sagawa et al., ), complemented with a regularization strategy that appears to assist in performance for small population subgroups. The coverage of disparate data distributions is also a concern in Non-IID federated learning. The work (Jiao et al., ) presents convergence theory for DRO in a federated setting.

## 3 Algorithm

Let $\mathcal{S}=\{x_{D},y_{D}\}$ be the synthetic distilled dataset, where $([x_{D}]_{i},[y_{D}]_{i})$, with $i=1,...,n_{D}$ are an individual input and label. Let $\mathcal{T}=\{x_{T},y_{T}\}$ be the training set where $([x_{T}]_{i},[y_{T}]_{j})$, $j=1,...,n_{T}$ are an individual input and label with $n_{T}\gg n_{D}$. Let $F(\theta;x,y)$ be the full loss function with parameter vector $\theta$ and data $x$ and $y$.

The synthetic dataset $\mathcal{S}$ will be optimized based on an objective function that includes a risk function of the loss, averaged across a set of partitions of $\mathcal{S}$. We denote the loss function $L(\mathcal{S})$. Consider Algorithm 1. This procedure encapsulates one function evaluation of a loss with respect to the distilled data $\mathcal{S}$. Each iteration begins with subsampling the training dataset $\bar{\mathcal{T}}\subset\mathcal{T}$ and then clustering them based on their nearest synthetic data point. That is for all $i\in[|\mathcal{S}|]$ the set $\mathbf{c}_{i}$ contains the elements of $\bar{\mathcal{T}}$ that are closer to $\mathcal{S}_{i}$ than to any other point in $\mathcal{S}$.

The following criterion optimization problem is solved for $\theta$, obtaining the set of parameters that
solves for minimax loss across the subgroups defined by the clusters $\{\mathbf{c}_{i}\}$

$$ $\begin{array}[]{l}\min\limits_{\theta}\left[\frac{1}{|\mathcal{S}|}\sum\limits _{i}\mathop{CVaR}{[F(\theta;\mathbf{c}_{i})]}+\max\limits_{i}\mathop{CVaR}{[F( \theta;\mathbf{c}_{i})]}\right].\end{array}$ $$

Note that rather than the expectation, we use a risk measure. We set $\alpha$ to be some tail probability. Risk measures enable one to minimize one sided tail behavior. The operator denoting the Conditional Value at Risk, $\mathop{CVaR}$, is defined, with respect to the empirical distribution of data points within $\mathbf{c}_{i}$:

$$ $\begin{array}[]{l}\mathop{CVaR}[F(\theta;c_{i})]:=-\frac{1}{\alpha}\left\{ \frac{1}{|c_{i}|}\sum_{z_{t}\in c_{i}}F(\theta;z_{t})1[F(\theta;z_{t})\leq f_{ \alpha}]\right.\\ \qquad\qquad\left.+f_{\alpha}(\alpha-\frac{1}{|c_{i}|}\sum_{z_{t}\in c_{i}}1[F (\theta;z_{t})\leq f_{\alpha}]\right\}\end{array}$ $$

with the quantity $f_{\alpha}$ defined as,

$$ $f_{\alpha}:=\min\{f\in\mathbb{R}:\frac{1}{|c_{i}|}\sum_{z_{t}\in c_{i}}1[F( \theta;z_{t})\leq f_{\alpha}]\geq\alpha$ $$

Figure: Algorithm 1 Evaluation of the Loss on the Distilled Dataset

This procedure constitutes one (noisy, due to the training subsampling) function evaluation for the dataset $\mathcal{S}$. Consider that we want to optimize $\mathcal{S}$ so as to minimize this criterion $L(\mathcal{S})$ as a function of $\mathcal{S}$.
Assume that you start with some initial $\mathcal{S}$ and proceed as in Algorithm 2. To perform the full optimization, we apply a standard gradient descent approach, using some zero order approximation of the gradient of $L$ concerning the dataset $\mathcal{S}$, that is $g(\mathcal{S})\approx\nabla L(\mathcal{S})$, computed only using evaluations of $L(\mathcal{S})$.

Figure: Algorithm 2 Training the Distilled Dataset

We can use any method from (Berahas et al., ) to obtain a gradient estimate of $g\approx\nabla_{\mathcal{S}}L(\mathcal{S})$, here the featured procedure is Gaussian smoothing.

### 3.1 Analysis

Let $\mathcal{Q}$ be a *partition* of the population distribution $\mathcal{D}$. As such, we can describe the problem of group coverage at the point of inference as yielding comparable performance across elements of $\mathcal{Q}$. To this end, we denote the support of the population distribution as $D=\text{supp}\left(\mathcal{D}\right)$ and we assume that it is compact. Let us define,

$$ $\mathbb{Q}=\{\mathcal{Q}_{i}\},\,\text{ with }Q_{i}=\text{supp}(\mathcal{Q}_{i }),\,\bigcup_{i=1}^{q}Q_{i}=D,\,\underline{s}\leq\lambda(Q_{i})\leq\bar{s}$ $$

for some $\bar{s}>\underline{s}>0$ and every $Q_{i}$ is itself a union of compact, convex, and connected sets. Here $\lambda$ is integration with respect to the Lebesgue measure. Recall that we denote the synthetic dataset by $\mathcal{S}$ with $|\mathcal{S}|=S$.

Note the assumption that $\mathcal{D}$ is finite. In addition, we assume a certain regularity of the loss function. Formally, we state the following:

###### Assumption 3.1 .

The population distribution $\mathcal{D}$ has bounded support, i.e., $\lambda(D)<\infty$. The loss $F$ is Lipschitz continuously differentiable with respect to the first argument (the parameters) and continuous with respect to the second argument (the input features and labels).

Additionally, let’s explore the asymptotic learning regime when examining the problem, where the entire population set could be sampled in the limit.

###### Assumption 3.2 .

For all $\mathcal{Q}\in\mathbb{Q}$, consider the asymptotic online learning limit,

$$ $\lim\sup\limits_{t\to\infty}(\text{supp}(\mathcal{Q})\cap\bar{\mathcal{T}}_{t} )=\text{supp}(\mathcal{Q})$ $$

where $\bar{\mathcal{T}}_{t}$ is the training set sampled at iteration $t$.

Informally, group coverage corresponds to enforcing adequate performance for prediction using the trained model regardless of what portion of the population set is taken. Essentially, at a particular future instant, the learner could be expected to classify or predict a quantity for some small subpopulation cohort that may appear as a subsample at the time of online inference.

Let us formally articulate the optimization problem of interest, as outlined in the Introduction, as follows

$$ $\begin{array}[]{rl}\min\limits_{\mathcal{S}}\max\limits_{\mathcal{Q}\in\mathbb {Q}}&\mathbb{E}[F(\theta^{*},\mathcal{Q})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,\mathcal{S}).\end{array}$ (3) $$

Notice that there are two layers of generalization: on the one hand, we are solving a problem robust with resepect to the choice of $\mathcal{Q}\in\mathbb{Q}$, and on the other hand, we are considering the population error rather than an empirical loss.

We posit that the computation of a gradient estimate employs a conventional Stochastic Approximation procedure to address *some* bilevel optimization problem for $\mathcal{S}$. Thus we do not address the convergence guarantees (that is, asymptotic stationarity and convergence rate) of the training procedure itself but study the properties of the associated optimization problems and their solutions.

Accordingly, our emphasis is on scrutinizing the properties associated with the criterion,

$$ $\min\limits_{\theta}\max\limits_{\mathcal{Q}\in\mathbb{Q}}\mathbb{E}_{Q}[F( \theta,\mathcal{Q})].$ (4) $$

that is ultimately used to steer the synthetic dataset, as
the DD task is finding a dataset on whose associated $\theta$-minimizer also minimizes this quantity. In the analysis, we shall present:

(1) A justification for why our Algorithm approximately solves optimization problem (3).

(2) The discussion of the group robustness properties associated with the solution of (3).

To begin with, we rewrite (3) as:

$$ $\begin{array}[]{rl}\min\limits_{\mathcal{S}}\max\limits_{\mathcal{Q}\in\mathbb {Q}}\max_{\mathcal{Q}^{\prime}\in\bar{\mathcal{Q}}}&\mathbb{E}[F(\theta^{*}, \mathcal{Q}^{\prime})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,\mathcal{S})\\ &\bar{\mathcal{Q}}=\left\{\mathcal{Q}^{\prime}:I(\mathcal{Q}^{\prime},\mathcal {S}\cup\mathcal{Q}_{N})\leq r\right\}\end{array}$ (5) $$

where $r>0$ is some bound and we replace the inner problem to be evaluated on the entire training dataset with a data-driven DRO. In the appendix
we present several results in the literature that indicate how the DRO to an empirical risk minimization problem exhibits generalization guarantees and hence is a valid auxiliary criterion for minimizing (3).

###### Assumption 3.1 .

###### Assumption 3.2 .

### 3.2 Large Deviations and Solving the bilevel DRO

To justify the clustering and risk measure minimization algorithm as an appropriate procedure for solving (5), we apply
some theoretical analysis on the relationship between DRO and Large Deviations Principles (LDP). LDPs (e.g. (Deuschel & Stroock, ))
define an exponential asymptotic decay of the measure of the tails of an empirical distribution with respect to a sample size.

The particular, Large Deviations Principle (LDP) capturing out-of-sample disappointment, which is of interest to us, is defined as follows:

$$ $\begin{array}[]{l}\lim\sup\limits_{N\to\infty}\frac{1}{N}\log\mathbb{P}_{ \mathcal{Q}}^{\infty}\left(F(\theta^{*}(\mathcal{S}),\mathbb{P}_{\mathcal{Q}}) >F(\theta^{*}(\mathcal{S}),\mathcal{S}\cup\mathcal{Q}_{N})\right)\\ \qquad\leq-r,\,\forall\mathcal{Q}\in\mathbb{Q}\end{array}$ (6) $$

for some $r>0$. This states that for all partitions $\mathcal{Q}$ of the population space partition $\mathbb{Q}$, there is an exponential asymptotic decay of the probability of one-sided sample error (i.e., samples from the population $\mathbb{P}_{\mathcal{Q}}$) relative to the computed loss on the synthetic and training data points $\mathcal{Q}_{N}$, as the number of training data points grows asymptotically.

Now, we present the following Proposition, whose proof is in the appendix,

###### Proposition 3.3 .

Algorithm (2) satisfies, under Assumptions 3.1 and 3.2, the LDP (7).

Next, we relate Algorithm (2) together with 3.1 to the DRO problem (5).

###### Proposition 3.3 .

### 3.3 Large Deviations and Distributionally Robust Optimization

Applying Assumption 3.2, the LDP, and the continuity of $F$ with respect to the data, Assumption 3.1 (see also the proof of Lemma 1 as well as Example 3 in (Duchi & Namkoong, )) we can bound the quantity:

$$ $\mathbb{P}(F(\theta^{*},q)>F(\theta^{*},\mathcal{S}\cup\mathcal{Q}_{N}))$ $$

for $q\in\mathcal{Q}^{\prime}$, with $\mathcal{D}(\mathcal{Q}_{N},\mathcal{Q}^{\prime})\leq r$ with a sufficient step-size.

This implies that one can bound the objective value of the DRO problem of interest (5):

$$ $\begin{array}[]{rl}\min\limits_{\mathcal{S}}\max\limits_{\mathcal{Q}\in\mathbb {Q}}\max_{\mathcal{Q}^{\prime}\in\bar{\mathcal{Q}}}&\mathbb{E}[F(\theta^{*}, \mathcal{Q}^{\prime})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,\mathcal{S})\\ &\bar{\mathcal{Q}}=\left\{\mathcal{Q}^{\prime}:I(\mathcal{Q}^{\prime},\mathcal {S}\cup\mathcal{Q}_{N})\leq r\right\}\end{array}$ $$

i.e., there is some small $C>0$ such that,

$$ $|O(\theta^{*},S^{*})-\hat{O}(\hat{\theta},\hat{S})|\leq C$ $$

where $O$ and $\hat{O}$ refer to the optimal value of the data driven DRO (5) and the approximation given by Algorithm 1 and 2.

**Table 1: Top-1 test accuracy on standard testing sets. ${}^{\dagger}$ indicates the result is reported based on our runs. All the results are reported based on the average of 5 runs. ${}^{\mathcal{R}}$ indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.**
| Dataset | IPC | Random | DSA | DM | KIP | IDC | IDC${}^{\mathcal{R}}$ | GLaD | GLaD${}^{\mathcal{R}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SVHN | 1 | 14.6 | 27.5 | 24.2 | 57.3 | 68.5 | 68.9${}_{\pm 0.4}$ | 32.5${}^{\dagger}$ | 35.7${}_{\pm 0.3}$ |
| 10 | 35.1 | 79.2 | 72.0 | 75.0 | 87.5 | 88.1${}_{\pm 0.3}$ | 68.2${}^{\dagger}$ | 72.5${}_{\pm 0.4}$ |  |
| 50 | 70.9 | 84.4 | 84.3 | 80.5 | 90.1 | 90.8${}_{\pm 0.4}$ | 71.8${}^{\dagger}$ | 76.6${}_{\pm 0.3}$ |  |
| CIFAR-10 | 1 | 14.4 | 28.7 | 26.0 | 49.9 | 50.6${}^{\dagger}$ | 51.3${}_{\pm 0.3}$ | 28.0${}^{\dagger}$ | 29.2${}_{\pm 0.8}$ |
| 10 | 37.2 | 52.1 | 53.8 | 62.7 | 67.5 | 68.6${}_{\pm 0.2}$ | 46.7${}^{\dagger}$ | 50.2${}_{\pm 0.5}$ |  |
| 50 | 56.5 | 60.6 | 65.6 | 68.6 | 74.5 | 75.3${}_{\pm 0.5}$ | 59.9${}^{\dagger}$ | 62.5${}_{\pm 0.7}$ |  |
| ImageNet-10 | 1 | 23.2${}^{\dagger}$ | 30.6${}^{\dagger}$ | 30.2${}^{\dagger}$ | - | 54.4${}^{\dagger}$ | 58.2${}_{\pm 1.2}$ | 33.5${}^{\dagger}$ | 36.4${}_{\pm 0.8}$ |
| 10 | 46.9 | 52.7 | 52.3 | - | 72.8 | 74.6${}_{\pm 0.9}$ | 50.9${}^{\dagger}$ | 55.2${}_{\pm 1.1}$ |  |

### 3.4 Distributionally Robust Optimization and Subgroup Coverage

Let us return to (3):

$$ $\begin{array}[]{rl}\min\limits_{\mathcal{S}}\max\limits_{\mathcal{Q}\in\mathbb {Q}}&\mathbb{E}[F(\theta^{*},\mathcal{Q})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,\mathcal{S})\end{array}$ $$

We have established that our Algorithm approximately solves a DRO which approximately bounds the population loss.
Now consider the outer DRO itself. The same theory regarding DRO and LDPs can now be applied, but now to yield
a stronger result, since we are evaluating the full (test) loss. Indeed the data being sampled are simply $\mathcal{Q}_{\omega}\in\mathbb{Q}$,
that is, some i.i.d. selection $\omega$ over the finite set of partitions. Since the entire subpopulation is taken, each estimator is constructed with the full population error, and w.p. 1 the entire set $\mathbb{Q}$ is sampled for finite $N$.

We can directly apply Theorem 7 of  (Van Parys et al., ) to deduce that the result satisfies:

$$ $\begin{array}[]{l}\lim\sup\limits_{N\to\infty}\frac{1}{N}\log\mathbb{P}_{ \omega}^{\infty}\left(F(\theta^{*},Q_{\omega})>F(\theta^{*},\mathcal{S})\right )\leq-r.\end{array}$ (7) $$

Thus by the Kolmogorov 0-1 principle we have that,

$$ $F(\theta^{*},\mathcal{Q})\leq F(\theta^{*},\mathcal{S}),$ $$

for all $\mathcal{Q}$. This guarantees the quality of $\mathcal{S}$ in ensuring group robust guarantees on the loss.

## 4 Numerical Results

**Table 2: Top-1 test accuracy on robustness testing sets. “Gain” denotes the performance gain of applying our proposed method. Except for absolute values, the relative fluctuation compared with the standard case is also reported as subscripts. The experiments are conducted under the IPC setting of 10. ${}^{\mathcal{R}}$ indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.**
| Dataset | Setting | Random | IDC | IDC${}^{\mathcal{R}}$ | Gain | GLaD | GLaD${}^{\mathcal{R}}$ | Gain |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | Standard | 37.2 | 67.5 | 68.6 | 1.1 | 46.7 | 50.2 | 3.5 |
| Cluster-min | 31.4 | 63.3${}_{\downarrow 4.2}$ | 65.0${}_{\downarrow 3.6}$ | 1.7${}_{\uparrow 0.6}$ | 40.2${}_{\downarrow 6.5}$ | 46.7${}_{\downarrow 3.5}$ | 6.5${}_{\uparrow 3.0}$ |  |
| Noise | 35.4 | 57.2${}_{\downarrow 10.3}$ | 59.4${}_{\downarrow 9.2}$ | 2.2${}_{\uparrow 1.1}$ | 44.1${}_{\downarrow 2.5}$ | 49.5${}_{\downarrow 0.7}$ | 5.4${}_{\uparrow 1.8}$ |  |
| Blur | 29.4 | 48.3${}_{\downarrow 19.2}$ | 50.5${}_{\downarrow 18.1}$ | 2.2${}_{\uparrow 1.1}$ | 36.9${}_{\downarrow 9.8}$ | 39.0${}_{\downarrow 11.2}$ | 2.1${}_{\downarrow 1.4}$ |  |
| Invert | 9.5 | 25.6${}_{\downarrow 41.9}$ | 26.5${}_{\downarrow 42.1}$ | 0.9${}_{\downarrow 0.2}$ | 10.6${}_{\downarrow 36.1}$ | 13.0${}_{\downarrow 37.2}$ | 2.4${}_{\downarrow 1.1}$ |  |
| ImageNet-10 | Standard | 46.9 | 72.8 | 74.6 | 1.8 | 50.9 | 55.2 | 4.3 |
| Cluster-min | 31.2 | 61.4${}_{\downarrow 11.4}$ | 65.7${}_{\downarrow 8.9}$ | 4.3${}_{\uparrow 2.5}$ | 34.9${}_{\downarrow 16.0}$ | 47.1${}_{\downarrow 8.1}$ | 12.2${}_{\uparrow 7.9}$ |  |
| Noise | 42.6 | 65.8${}_{\downarrow 7.0}$ | 68.8${}_{\downarrow 5.8}$ | 3.0${}_{\uparrow 1.2}$ | 48.6${}_{\downarrow 2.3}$ | 53.9${}_{\downarrow 1.3}$ | 5.3${}_{\uparrow 1.0}$ |  |
| Blur | 45.5 | 71.9${}_{\downarrow 0.9}$ | 74.1${}_{\downarrow 0.5}$ | 2.2${}_{\uparrow 0.4}$ | 47.9${}_{\downarrow 3.0}$ | 54.6${}_{\downarrow 1.6}$ | 6.7${}_{\uparrow 1.4}$ |  |
| Invert | 21.0 | 27.8${}_{\downarrow 45.0}$ | 30.3${}_{\downarrow 44.3}$ | 2.5${}_{\uparrow 0.7}$ | 17.0${}_{\downarrow 33.9}$ | 21.6${}_{\downarrow 33.6}$ | 4.6${}_{\uparrow 0.3}$ |  |

### 4.1 Implementation Details

The proposed method can be applied to various popular dataset distillation frameworks.
In this paper, we perform experiments on pixel-level distillation method IDC (Kim et al., ) and latent-level method GLaD (Cazenavette et al., ) to substantiate the consistent efficacy of our method.
The experiments are conducted on popular dataset distillation benchmarks, namely SVHN, CIFAR-10 and ImageNet (Yuval, ; Krizhevsky et al., ; Deng et al., ).
The images of SVHN and CIFAR-10 are resized to 32$\times$32, while those from ImageNet-10 are resized to 128$\times$128, representing diverse resolution scenarios.
During the distilling process, we generally adhere to the hyperparameter settings of the original methods.
Due to the CVaR loss calculation involving an ample number of samples, the mini-batch size during model updating is increased to 256.
In cases where the IPC setting is less than 10, the cluster number in Eq. 2 is set equal to IPC.
For larger IPCs, the cluster number is fixed at 10, with 10 random synthesized samples chosen as the clustering centers.
The ratio $\alpha$ in CVaR loss is set to 0.8.
The validation protocol aligns with the configurations in (Kim et al., ), while the results of GLaD are all re-obtained following the same protocol.

### 4.2 Results on Standard Benchmark

We first evaluate our proposed method on standard testing sets, including SVHN, CIFAR-10, and ImageNet-10 in Tab. 1.
The ImageNet-10 split follows the configuration outlined in IDC (Kim et al., ).
Except for IDC and GLaD, we also compare the validation results with DSA, DM, and KIP (Zhao & Bilen, , ; Nguyen et al., ).
To fully demonstrate the performance of the compared methods, we implement distillation using the architectures specified in the original papers.
For both IDC and GLaD on SVHN and CIFAR-10, a 3-layer ConvNet (Gidaris & Komodakis, ) is employed. On ImageNet, ResNet-10 (He et al., ) is utilized for IDC, while a 5-layer ConvNet is used for GLaD. After distillation, we conduct validation procedures on ConvNet-3 for 32$\times$32 datasets and ResNet-10 for ImageNet following the IDC protocol, ensuring a fair and consistent basis for comparison.

The incorporation of CVaR loss consistently enhances performance across all scenarios.
Notably, under the IPC of 10, facilitated by the supervision from segregated sub-clusters, our proposed method demonstrates the most substantial performance improvement over baselines.
In the case of 1 IPC, where only a single synthetic sample is available for sub-cluster construction, the introduction of additional CVaR loss still yields a significant performance boost, despite the absence of guidance from multiple sub-clusters.
Our proposed method is especially effective on ImageNet-10, characterized by finer class divisions and greater intra-class variation compared with CIFAR-10.
On ImageNet-10 we achieve an average top-1 accuracy gain of 3.1%, further elevating the state-of-the-art baseline in DD methods.

### 4.3 Results on Robustness Settings

In addition to standard validation scenarios, a notable advantage offered by our proposed method is the robustness against various domain shifts.
This property is assessed through multiple protocols.
Firstly, as suggested before, we present validation results on different partitions of the testing set.
The worst accuracy among the clustered testing sub-sets is reported to demonstrate the robustness of distilled data, denoted as “Cluster-min” in Tab. 2.
In the sub-scripts, we report the performance drop compared with the standard case.
Several key observations emerge from the experiment results.
(1) In comparison to random images, the Cluster-min accuracy of baseline DD methods exhibits improvement alongside the standard performance.
It suggests that by condensing the knowledge from original data into informative distilled samples, DD methods contribute to enhanced data robustness.
(2) Compared with CIFAR-10, the performance gap between the standard case and the worst sub-cluster on ImageNet-10 is more pronounced.
This discrepancy can be attributed to a higher incidence of ID-unrelated interruptions in ImageNet-10, resulting in larger domain-shifts between sub-clusters and the original distribution.
This finding aligns with the observation in Tab. 1.
(3) With our proposed robust method applied, not only is the cluster-min performance improved, but the performance drop from the standard case is also significantly mitigated compared with baselines.
It suggests exceptional overall generalization and robustness conferred by our method.

**Table 3: Cross-architecture generalization performance comparison. The experiment is conducted on ImageNet-10 under 10 IPC settings. The distilling architecture for IDC is ResNet-10, while for GLaD is ConvNet-5. ${}^{\mathcal{R}}$ indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.**
| Method | Architecture |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Conv | Res10 | Res18 | ViT | VGG11 |  |
| IDC | 71.9 | 72.8 | 70.8 | 55.2 | 64.5 |
| IDC${}^{\mathcal{R}}$ | 72.6 | 74.6 | 72.7 | 56.4 | 65.6 |
| GLaD | 48.2 | 50.9 | 51.2 | 36.8 | 44.2 |
| GLaD${}^{\mathcal{R}}$ | 51.9 | 55.2 | 53.6 | 39.2 | 46.3 |

Furthermore, we provide testing results in Tab. 2 on truncated testing sets, simulating scenarios where the testing set exhibits more substantial domain shifts compared with the training data.
We employ three data truncation means, including the addition of Gaussian noise, application of blur effects, and inversion of image colors.
The conclusions drawn from truncated testing sets align with those from partitioned testing sets.
While dataset distillation generally contributes to improved data robustness, the introduction of CVaR loss further amplifies the trend.
In this analysis with a higher resolution to capture more details, we observe that truncation on ImageNet has a smaller impact compared with CIFAR-10.
Generally, the improvement by our proposed method on truncated performance is also larger than that on standard testing sets, especially on ImageNet-10.
The increased advantage on robustness settings further validates that the proposed method not only elevates overall accuracy but significantly fortifies the robustness of distilled data.

**Table 4: Ablation study on CVaR loss. The experiment is conducted on CIFAR-10 and ImageNet-A under 10 IPC. The baseline model is GLaD. “CE” denotes cross entropy loss, and “CL” refers to cluster accuracy. The best results are marked in bold.**
| Loss | CIFAR-10 | ImageNet-A |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| CE | CVaR | CVaR | Acc | CL | Acc | CL |
| -avg | -max | -min | -min |  |  |  |
| ✓ | - | - | 46.7 | 40.2 | 53.9 | 40.5 |
| ✓ | ✓ | - | 49.1 | 45.3 | 56.4 | 43.9 |
| ✓ | - | ✓ | 47.9 | 44.2 | 56.1 | 43.7 |
| ✓ | ✓ | ✓ | 50.2 | 46.7 | 57.5 | 45.8 |

**Table 5: Validation accuracy on more ImageNet sub-sets in comparison with GLaD. The data is distilled with ConvNet-5 and evaluated with ResNet-10 with an IPC of 10. ${}^{\mathcal{R}}$ indicates the proposed robust dataset distillation method applied. The better results between baseline and the proposed method are marked in bold.**
| Dataset | Acc | Cluster-min |  |  |
| --- | --- | --- | --- | --- |
| GLaD | GLaD${}^{\mathcal{R}}$ | GLaD | GLaD${}^{\mathcal{R}}$ |  |
| ImageNet-A | 53.9 | 57.5 | 40.5 | 45.8 |
| ImageNet-B | 50.3 | 53.8 | 42.9 | 47.0 |
| ImageNet-C | 49.2 | 51.3 | 28.2 | 32.8 |
| ImageNet-D | 39.1 | 40.9 | 27.0 | 31.3 |
| ImageNet-E | 38.9 | 41.1 | 25.8 | 30.0 |

### 4.4 Cross-architecture Generalization

In addition to enhanced robustness against domain-shifted data, the incorporation of CVaR loss also yields better cross-architecture generalization capabilities.
We assess multiple network architectures including ConvNet-5, ResNet, ViT (Dosovitskiy et al., ) and VGG11 (Simonyan & Zisserman, ), and compare the performance with and without our proposed method in Tab. 3.
Despite the distinct distilling architectures employed in IDC and GLaD, both achieve their highest accuracy on ResNet-10.
Notably, the proposed robust distillation method consistently enhances performance across all architectures, showcasing remarkable cross-architecture generalization capabilities.

### 4.5 Ablation Study

We conduct an ablation study on the incorporation of CVaR loss in Tab. 4.
In addition to CIFAR-10, we also report results on the ImageNet-A sub-set (Cazenavette et al., ).
Both accuracy on the standard testing set and the minimum cluster accuracy are presented.

Our focus is primarily on two aspects of utilizing the CVaR loss, i.e. the maximum value and average value of CVaR losses across all sub-clusters.
The CVaR loss operates as a complement to the standard cross-entropy optimization.
Hence the baseline case, involving only cross entropy loss, mirrors the performance of GLaD.
Compared with the maximum value, the average CVaR loss proves more effective in enhancing the validation performance when applied independently.
While both average and maximum CVaR loss yield considerable improvement over the baseline, their combined application further fortifies the robustness of the distilled data, which is selected as our eventual implementation.

### 4.6 Results on More ImageNet Sub-sets

We further provide additional results on ImageNet sub-sets in comparison with GLaD in Tab. 5.
The sub-set division remains consistent with that in (Cazenavette et al., ).
Our proposed robust DD method consistently outperforms the baseline, particularly in terms of the worst accuracy across clustered testing sub-sets.
This observation underscores the stability and versatility of the proposed method.

Figure: Figure 2: T-SNE distribution visualization of original samples (blue dots) and synthesized samples (orange dots) on ImageNet bonnet class.
Refer to caption: extracted/5461728/Images/tsne.png

Figure: Figure 3: Synthesized sample visualization comparison between GLaD and our proposed method. The samples are initialized with the same original images and updated with different methods.
Refer to caption: extracted/5461728/Images/sample.png

### 4.7 Visualization

To explicitly illustrate the impact of CVaR loss on the distillation results, we visualize the sample distribution comparison in Fig. 2.
In the feature space, samples optimized by GLaD tend to form small clusters, while the introduction of robust optimization leads to a more evenly distributed distilled dataset.
Note that there is no constraint on the feature distribution applied during the distilling process.
The proposed robust optimization involves loss calculation on different data partitions, contributing to better coverage over the original data distribution.
The more even distribution observed further affirms the effectiveness of our proposed robust distillation method.

Additionally, we compare the synthesized samples of the same ImageNet bonnet class in the pixel space between the baseline GLaD and our proposed method in Fig. 3.
The images are initialized with the same original samples for better comparison.
Remarkably, the additional CVaR loss introduces more irregular shapes into the image during optimization.
These irregular shapes weaken specific features present in each image while introducing common features of the corresponding class, leading to a more even sample distribution in the latent space.

Figure: Figure 4: Parameter analysis on the CVaR ratio $\alpha$.
Refer to caption: extracted/5461728/Images/alpha.png

### 4.8 Parameter Analysis

An analysis is conducted to evaluate the influence of different CVaR ratio $\alpha$ choices in Fig. 4.
We vary the $\alpha$ value from 0.2 to 1.0 to explore different ratios of data for calculating CVaR loss.
Both the validation performance on the standard testing set and the minimum cluster accuracy reach their peak at $\alpha$=0.8.
Including all the samples ($\alpha$=1.0) introduces certain interruptions for the optimization due to large loss values and results in a slight performance drop.
On the other hand, considering only a small portion of samples for CVaR loss loses essential information, leading to performance degradation.
Notably, even the worst performance obtained at $\alpha$=0.2 is significantly higher than the GLaD baseline, particularly in terms of the minimum cluster accuracy.
This observation strongly supports the effectiveness of our proposed robust dataset distillation method.

## 5 Conclusion

This paper explored the intricate relationship between DD and its generalization, with a particular focus on performance across uncommon subgroups, especially in regions with low population density. While existing methods prioritize matching convergence properties with the training dataset, we emphasize the importance of considering the broader population distribution. To address this, we introduced an algorithm inspired by distributionally robust optimization, employing clustering and risk minimization to enhance DD. Our theoretical framework, supported by empirical evidence, demonstrates the effectiveness, generalization, and robustness of our approach across diverse subgroups. By prioritizing representativeness and coverage over training error guarantees, our methodology offers a promising avenue for improving the performance of models trained on synthetic datasets in real-world scenarios, paving the way for enhanced applications of dataset distillation in a variety of settings.

## Appendix A Appendix

### A.1 Theoretical Details

#### A.1.1 DRO and Generalization

Here we present several results from the literature that indicate that a DRO on an empirical risk minimization problem
exhibits generalization guarantees, i.e., approximately minimizes the population (or expected test dataset) loss.
Consider (5), repeated here

$$ $\begin{array}[]{rl}\min\limits_{S}\max\limits_{\mathcal{Q}\in\mathbb{Q}}\max_{ \mathcal{Q}^{\prime}\in\bar{\mathcal{Q}}}&\mathbb{E}[F(\theta^{*},\mathcal{Q}^ {\prime})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,S)\\ &\bar{\mathcal{Q}}=\left\{\mathcal{Q}^{\prime}:I(\mathcal{Q}^{\prime},\mathcal {S}\cup\mathcal{Q}_{N})\leq r\right\}.\end{array}$ $$

- 1.
Theorem 3 in (Xu & Mannor, ) provides a bound for the test error at the DRO solution.
- 2.
Theorem 3.1 in  (Zeng & Lam, ) presents a probability that $|\mathbb{E}[F(\theta^{*}_{DRO},\mathcal{Q})]-\mathbb{E}[F(\theta^{*}_{opt},
\mathcal{Q})]|\leq\epsilon$ as a function of $\epsilon$, where we denote the DRO and the exact minimum, respectively.

#### A.1.2 Large Deviations and Data Driven DRO

The analysis of the theoretical convergence and robustness properties of our method will rely
significantly on the theoretical foundations of data driven DRO established in (Van Parys et al., ).
To this end, we review a few pertinent definitions. A *predictor* is a function $c:\mathbb{R}^{d}\times\Xi\to\mathbb{R}$
that defines a model as applied to a data distribution. A *data driven predictor* uses
an empirical distribution of samples $\hat{\mathbb{P}}_{T}$ in the prediction.

The *sample average predictor* is given as

$$ $c(\theta,\hat{\mathbb{P}}_{T})=\frac{1}{T}\sum\limits_{t=1}^{T}F(\theta,\xi_{t})$ $$

Let $\hat{\theta}=\arg\min\hat{c}(\theta,\mathbb{P})$ be the data driven predictor.

An ordering $\preceq$ is introduced to rank the set of predictors, with

$$ $(\hat{c}_{1},\hat{\theta}_{1})\preceq(\hat{c}_{2},\hat{\theta}_{2})\text{ if and only if }\hat{c}_{1}(\hat{\theta}_{1}(\mathbb{P}^{\prime}),\mathbb{P}^{ \prime})\leq\hat{c}_{2}(\hat{\theta}_{2}(\mathbb{P}^{\prime}),\mathbb{P}^{ \prime}),\,\forall\theta,\,\mathbb{P}^{\prime}\in\mathcal{P}$ $$

The *Distributionally Robust Predictor* is one defined as,

$$ $\hat{c}_{r}(\theta,\mathbb{P};)=\sup\limits_{\mathbb{P}\in\mathcal{P}}\left\{c (\theta,\mathbb{P}):\mathcal{D}_{I}(\mathbb{P}^{\prime},\mathbb{P})\leq r\right\}$ $$

where $\mathcal{D}_{I}(A,B)$ is the mutual information of random variables $A$ and $B$.

They define the optimization problem

$$ $\begin{array}[]{rl}\min\limits_{(\hat{c},\hat{\theta})}&(\hat{c},\hat{\theta}) \\ \text{s.t. }&\lim\sum\limits_{T\to\infty}\frac{1}{T}\log\mathbb{P}^{\infty} \left(c(\hat{\theta}(\hat{\mathbb{P}}_{T},\mathbb{P})>\hat{c}(\hat{\theta}, \hat{\mathbb{P}}_{T})\right)\leq-r,\,\forall\mathbb{P}\in\mathcal{P}\end{array}$ (8) $$

where the minimum is the lower bound with respect to the ordering $\preceq$ of all feasible (LDP satisfying) predictors.

In (Van Parys et al., ), they prove that the distributionally robust predictor solves (8) in Theorem 4, for discrete distributions, and Theorem 6, for continuous ones.

We are constructing a synthetic dataset, while simultaneously considering training subsamples to form the empirical
measure at each iteration, suggesting that the data-driven framework can fit the problem of interest. Formally, we consider solving optimization problems defined on a set of measures as decision variables,

$$ $\begin{array}[]{rl}\min\limits_{S}\max\limits_{\mathcal{Q}\in\mathbb{Q}}& \mathbb{E}[F(\theta^{*},\mathcal{Q})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,S)\end{array}$ $$

by solving the data driven DRO approximation for the test error,

$$ $\begin{array}[]{rl}\min\limits_{S}\max\limits_{\mathcal{Q}\in\mathbb{Q}}\max_{ \mathcal{Q}^{\prime}\in\bar{\mathcal{Q}}}&\mathbb{E}[F(\theta^{*},\mathcal{Q}^ {\prime})]\\ \text{s.t. }&\theta^{*}\in\arg\min_{\theta}F(\theta,S)\\ &\bar{\mathcal{Q}}=\left\{\mathcal{Q}^{\prime}:I(\mathcal{Q}^{\prime},S\cup \mathcal{Q}_{N})\leq r\right\}\end{array}$ (9) $$

Note that the form is the nested DRO problem described earlier. At the upper layer, there is an uncertainty set regarding the choice
of $\mathcal{Q}\in\mathbb{Q}$, targeting group robustness. There is the data-driven DRO in the lower level, which is an algorithmic approximation of the population risk minimization with established accuracy guarantees.

### A.2 Proof of Proposition 3.3

Consider the quantity:

$$ $\frac{1}{N}\log\mathbb{P}_{\mathcal{Q}}^{\infty}\left(F(\theta^{*}(\mathcal{S} ),\mathbb{P}_{\mathcal{Q}})>F(\theta^{*}(\mathcal{S}),\mathcal{S}\cup\mathcal{ Q}_{N})\right)$ (10) $$

Now recall the expression for CVaR:

$$ $\begin{array}[]{l}\mathop{CVaR}[F(\theta;c_{i})]=-\frac{1}{\alpha}\left\{\frac {1}{|c_{i}|}\sum_{z_{t}\in c_{i}}F(\theta;z_{t})1[F(\theta;z_{t})\leq f_{ \alpha}]\right.\\ \qquad\qquad\left.+f_{\alpha}(\alpha-\frac{1}{|c_{i}|}\sum_{z_{t}\in c_{i}}1[F (\theta;z_{t})\leq f_{\alpha}]\right\}\end{array}$ $$

and

$$ $f_{\alpha}=\min\{f\in\mathbb{R}:\frac{1}{|c_{i}|}\sum_{z_{t}\in c_{i}}1[F( \theta;z_{t})\leq f_{\alpha}]\geq\alpha$ $$

Step 1: CVaR itself satisfies a LDP (Brown, ; Mhammedi et al., ). With these results, we can relate the empirical to the population CVaR.

Step 2: Notice that as long as $\alpha<\frac{1}{2}$, then minimizing the CVaR will also minimize (10).

Step 3: There is at least one estimator satisfying the LDP, namely, the DRO. Thus minimizing the empirical CVaR, by the feasibility of the LDP, must result in an estimator that satisfies the LDP.

## Appendix B Visualization of Synthesized Samples

We provide the visualization of synthesized samples in different datasets in Fig. 5 to Fig. 8.
Each row represents a class.

Figure: Figure 5: Synthesized samples of CIFAR-10 (left) and SVHN (right).
Refer to caption: extracted/5461728/Images/sample-cifar-svhn.png

Figure: Figure 6: Synthesized samples of ImageNet-10 (left) and ImageNet-A (right).
Refer to caption: extracted/5461728/Images/sample-im10-a.png

Figure: Figure 7: Synthesized samples of ImageNet-B (left) and ImageNet-C (right).
Refer to caption: extracted/5461728/Images/sample-imb-c.png

Figure: Figure 8: Synthesized samples of ImageNet-D (left) and ImageNet-E (right).
Refer to caption: extracted/5461728/Images/sample-imd-e.png

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: In Line 017, what does "targeting the training dataset" mean?
soundness: 2
strengths: There are both theoretical and experimental demonstrations of the effectiveness of the algorithm.
summary: This paper proposes an algorithm for dataset distillation by incorporating distributionally robust optimization into it. There is theoretical justification and empirical validation of the proposed algorithm.
weaknesses: 1. There seems to be a mismatch between the motivation and the experiments. The motivation emphasizes regions with low population density, which usually correspond to the worst-group performance in subpopulation shift [1]. However, the main experiments related to distribution shift are conducted on test sets with perturbations or the worst group induced by an additional clustering process. It would be better to conduct more experiments on subpopulation datasets included in [1]. 
2. The introduction has too many paragraphs, which makes the logic of the introduction tedious with poor readability. 

Some minor issues:

- In Line 049, "some technique". 
- In Line 129 and 131, "Algorithm" "Numerical Results" their first letters do not need to be capitalized. 
- In the last paragraph of introduction, Section 3 is not mentioned.

[1] Yang, Yuzhe, et al. "Change is Hard: A Closer Look at Subpopulation Shift." *International Conference on Machine Learning*. PMLR, 2023.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the detailed suggestions provided in the weaknesses section.
soundness: 3
strengths: 1. The paper proposes applying distributional robust optimization to dataset distillation, providing a reasonable approach to enhance generalization.

2. Drawing on distributional robust optimization theory, this work establishes a theoretical foundation to support the proposed approach to dataset distillation.

3. The paper is well-structured, with clear algorithm block and effective visualizations that enhance the presentation of the work.
summary: The work proposes a robust dataset distillation approach that incorporates distributional robust optimization (DRO) to enhance generalization and performance across subgroups. This method combines clustering with risk-minimized loss to conduct dataset distillation. By prioritizing representativeness and coverage over training error guarantees, the approach enhances the models trained on synthetic datasets for real-world scenarios. Both theoretical analysis and empirical validation on multiple standard benchmarks are provided, demonstrating the effectiveness of the proposed approach.
weaknesses: 1. The empirical experiments focus primarily on CIFAR-10, ImageNet-10. Extending the evaluation to larger, real-world datasets with a greater number of classes would better demonstrate the effectiveness of proposed approach in generalization and robustness under real-world conditions.

2. Comparing the proposed approach with additional baseline method addressing generalization and robustness would provide a more comprehensive assessment of the proposed approach, including comparisons with baseline methods such as [1, 2, 3].

3. It would be valuable to visualize the robust inference tasks using real-world data rather than illustrative visuals, which could provide more insight and observations in real-world scenarios.

Reference:

[1] Domain generalization with adversarial feature learning. CVPR 2018.

[2] Self-challenging Improves Cross-Domain Generalization. ECCV 2020.

[3] HYPO: Hyperspherical Out-of-Distribution Generalization. ICLR 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I have stated many of my suggestions and concerns in the weaknesses section. Below, I have further questions with some minor issues.

* While theoretical analyses have been provided, I wonder whether the proposed method would affect the convergence rate and make it more difficult to find an optimal solution for the proposed objective. I would appreciate an empirical time complexity analysis regarding the proposed method.

* In Eq.10, what does the *N* represent? I did not see any introduction of the *N*.
soundness: 3
strengths: * This paper considers the group coverage and generalization ability of the synthetic dataset in Data Distillation(DD), which is interesting and novel.

* The introduced algorithm is clear and the theoretical analysis seems solid.

* The numerical results do show the effectiveness of the proposed algorithm. Meanwhile, the figures (Fig.3, 4) seem to indicate that the proposed method improves group coverage.
summary: This paper proposes a new data distillation approach emphasizing the subgroup coverage and generalization ability of the models trained on the synthetic dataset. This paper provides a theoretical analysis rooted in Distributionally Robust Optimization (DRO) and verifies the effectiveness of the proposed method with various experiments.
weaknesses: * While the paper emphasizes group coverage and generalization in data distillation, the experiments are mainly conducted on IID datasets. More experimental results in scenarios with distribution shift between training and testing sets (such as long-tail classification, subpopulation shift, domain adaptation, domain generalization, etc) can further validate the improvement in group coverage and generalization ability,

* According to Algorithm 1, the initialization of the synthetic dataset seems very important because it involves how the training data samples are clustered into subgroups. It may require further ablation study to verify the stability of the proposed algorithm.

* The proposed method seems like a plug-in as it only modifies the objective for data distillation and could be combined with any data distillation methods. A more sophisticated comparison between the proposed method with other objectives regarding training gradients, feature distributions, and training trajectory could help readers better understand the improvement of the proposed method.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weaknesses section.
soundness: 3
strengths: - As far as I know, this paper firstly addresses a major limitation in standard dataset distillation (underrepresented or rare subgroups) by applying a double-layer distributionally robust optimization (DRO) framework. This approach ensures the synthetic dataset better represents the full diversity of the data, reducing performance drops across different data subgroups. 

- This paper provides not only empirical evidence of the proposed method's robustness against domain shifts such as noise, blur, and inversion, but also a theoretical analysis of the effectiveness of CVaR in the loss function to enhance robustness.

- The proposed method is designed to be adaptable and can integrate with various existing distillation techniques, such as gradient or distribution matching. This modularity makes it compatible with a range of distillation methods, which are still actively developing and evolving.
summary: This paper proposes a robust dataset distillation method that enhances generalization across underrepresented subgroups by using a double-layer distributionally robust optimization (DRO) approach. Traditional dataset distillation often compresses training data into synthetic sets by matching training characteristics but may struggle with subgroup generalization. To improve this, the authors cluster the training data around synthetic data points and apply a Conditional Value at Risk (CVaR) loss to minimize worst-case subgroup errors, making the synthetic dataset more representative of diverse data distributions. Experimental results show that this method significantly improves robustness under domain shifts, outperforming baseline methods on benchmarks like CIFAR-10 and ImageNet-10, particularly in challenging conditions like noise and blurring.
weaknesses: - The proposed DRO approach, particularly with clustering and CVaR, may introduce significant computational overhead. This added complexity could be amplified for large-scale datasets due to the clustering of training (real) data. However, the paper does not discuss the computational overhead of the proposed method.

- Although the method shows promising results on benchmarks like CIFAR-10 and ImageNet-10, the experiments are limited to controlled domain shifts (e.g., noise, blur). Testing under more realistic settings, such as in transfer learning, would further validate its robustness and practical relevance. For example, one could (1) train neural networks on a synthetic dataset distilled from a coarse-grained dataset (e.g., ImageNet) and (2) fine-tune and evaluate them on a fine-grained dataset (e.g., Birds 200). This setup would better illustrate the method's effectiveness in addressing the challenges posed by rare subgroups.

Meta-review: The submission develops a novel formulation of dataset distillation that makes use of ideas from distributionally robust optimisation. A method based on segmenting the input space into different subgroups and ensuring that a distilled dataset leads to a model that has good worst-case performance on these subgroups is provided. The reviewers agree that this direction is interesting, novel, and well-executed. However, all reviewers had some concerns about the particular datasets distribution shifts used in the experimental validation of the new method.

I agree with the consensus the reviewers have come to; despite some issues about realism of the experimental evaluation, the paper makes a good contribution and should be accepted.

Decision: Accept (Poster)