Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator

## Abstract

Abstract Dataset distillation has emerged as a technique aiming to condense informative features from large, natural datasets into a compact and synthetic form.
While recent advancements have refined this technique, its performance is bottlenecked by the prevailing class-specific synthesis paradigm.
Under this paradigm, synthetic data is optimized exclusively for a pre-assigned one-hot label, creating an implicit class barrier in feature condensation.
This leads to inefficient utilization of the distillation budget and oversight of inter-class feature distributions, which ultimately limits the effectiveness and efficiency, as demonstrated in our analysis.
To overcome these constraints, this paper presents the Inter-class Feature Compensator (INFER), an innovative distillation approach that transcends the class-specific data-label framework widely utilized in current dataset distillation methods. Specifically,
INFER leverages a Universal Feature Compensator (UFC) to enhance feature integration across classes, enabling the generation of multiple additional synthetic instances from a single UFC input.
This significantly improves the efficiency of the distillation budget.
Moreover, INFER enriches inter-class interactions during the distillation, thereby enhancing the effectiveness and generalizability of the distilled data.
By allowing for the linear interpolation of labels similar to those in the original dataset, INFER meticulously optimizes the synthetic data and dramatically reduces the size of soft labels in the synthetic dataset to almost zero, establishing a new benchmark for efficiency and effectiveness in dataset distillation.

## 1 Introduction

Figure: Figure 1: Performance vs. compression ratio of SOTA dataset distillation methods (G-VBSM [], SRe2L[], MTT []) on three benchmarks. Performance is measured as the Top-1 accuracy of ResNet-18 (ConvNet128 for MTT) on the corresponding validation set, trained from scratch using synthetic datasets. The compression ratio, which includes the additional soft labels, is the proportion of the distilled dataset size to the original dataset size. The star indicates the optimal performance.

The remarkable success of Deep Neural Networks (DNNs)  [, , , ] in recent years can largely be attributed to their capability to extract complex and representative features from vast amounts of real-world data [, ].
However, the extensive data requirements for training DNNs pose significant challenges.
These challenges include not only the time-consuming training process [, , ], but also the substantial costs associated with data storage and computational resources [, ].

In response to the rapid growth of computational and storage demands in training DNNs, dataset distillation [, , ] has emerged as an effective solution.
Dataset distillation condenses essential features from extensive datasets into a compact, synthetic form, allowing models to maintain comparable performance levels with fewer resources []. Recent advancements in dataset distillation encompass techniques such as gradient matching [, , , ], trajectory matching [, , , ], data factorization [, , , ], and kernel ridge regression [, , ]. These approaches have significantly enhanced dataset distillation by compressing dense knowledge into single data instances.
Despite the diversity of these methods, most of them adhere to a uniform paradigm: each synthetic data instance is class-specific and optimized exclusively for a pre-assigned one-hot label.

While this “one label per instance” paradigm aligns with the traditional data-label pair structure of original datasets, it presupposes that the most effective encapsulation of a dataset’s knowledge can be achieved through individual instances representing discrete class identities. When the amount of synthetic data is limited, this class-specific paradigm benefits distillation by encouraging synthetic data to condense the most distinctive features of each class. However, as more synthetic instances are assigned to the same class, they tend to capture significant but similar features rather than diversifying to include unique, rarer features. This phenomenon, known as “Inefficient Utilization of the Distillation Budget”, leads to feature duplication [, , ], thereby limiting the potential for creating a richer, more comprehensive synthetic representation.

Figure: Figure 2: Left: Overview of dataset distillation paradigms. The first illustrates the traditional “one instance for one classes” approach, where each instance is optimized exclusively for its pre-assigned label, creating implicit class barriers. The second illustrates our INFER method, designed for “one instance for ALL classes” distillation. Right: t-SNE visualization of the decision boundaries between the traditional approaches (i.e., SRe2L []) and our INFER approach. We randomly select seven classes from CIFAR-100 dataset for the visualization. INFER forms thin and clear decision boundaries among classes, in contrast to the chaotic decision boundaries of the traditional approach.

Another critical downside of the class-specific synthesis paradigm is the oversight of inter-class features. By focusing on distinctive class-specific characteristics under the “one label per instance” approach, implicit “Class Barriers” are created between classes. These class barriers prevent the synthetic data instances from capturing inter-class features that bridge different classes in the original dataset. This oversight, termed “Oversight of Inter-Class Features”, inhibits the formation of thin and clear decision boundaries among classes, which are essential for models to generalize well across complex scenarios. We demonstrate the visulization of frormed decision boundaries in Figure 2.

Recognizing the aforementioned limitations, we introduce a novel paradigm for dataset distillation, termed the Inter-class Feature compEnsatoR (INFER).
Unlike traditional methods that follow the “one instance for one class” paradigm and generate separate synthetic instances for each class, INFER pioneers a “one instance for ALL classes” paradigm by introducing a Universal Feature Compensator (UFC).
The UFC, designed to reflect the general representativeness across all classes of the original dataset, depreciates the importance of pre-assigned labels.
This feature enables UFC to compensate for inter-class features while INFER randomly incorporates a few natural data instances to enhance intra-class features.
Notably, INFER integrates one UFC with multiple natural data instances from different classes through a simple additive process without auxiliary generator networks [], allowing the generation of multiple synthetic instances from a single input.
This “one instance for ALL classes” paradigm significantly enhances the efficiency of the distillation budget.

Furthermore, we have meticulously designed the optimization of UFCs to encompass inter-class features.
This optimization makes synthetic instances generated from UFCs compatible with MixUp data augmentation, which promotes inter-class interactions and aids in forming thin, clear decision boundaries among classes.
Prior works applying MixUp to synthetic data [] involve dynamic generating and storing an extensive amount of soft labels, which can increase storage requirements up to 30-fold.
INFER, however, dramatically eliminates the need for such extensive soft label storage by elegantly adopting the linear interpolation of labels used with natural datasets, decreasing the storage requirement by $99.3\%$.
This enhancement not only preserves the distillation budget but also streamlines the entire training process, underscoring INFER’s efficiency and effectiveness. Notably, INFER achieves 62.5% accuracy on ImageNet-1k with a ResNet-50 model, training solely on the synthetic dataset, which is only 4.04% the size of the original ImageNet-1k. This performance, which does not require additional models or dynamically generated soft labels, outperforms existing approaches in both accuracy and compression ratio.

Our contribution can be summarized as follows:

- •
We rethink the prevailing “one label per instance” paradigm that exclusively optimizes each synthetic data instance for a specific class. Through empirically analysis, we identify and address its two main limitations: inefficient utilization of the distillation budget and oversight of inter-class features.
- •
To overcome these issues, we introduce a new paradigm INFER, for “one instance for all classes” dataset distillation. Our INFER incorporates a novel Universal Feature Compensator (UFC) to efficiently condense and integrate features across multiple classes. Extensive experiments across CIFAR, tiny-ImageNet and ImageNet-1k datasets demonstrate the state-of-the-art performance of INFER.

## 2 Preliminaries and Related Works

The precursor to dataset distillation in condensing of datasets is coreset selection [, , , , ]. This method involves selecting a coreset of the original dataset that ideally contains the entire representativeness of the population. However, this approach encouters a significant performance drop when compression ratio(^1^11 Compression Ratio (CR) = synthetic dataset size / original dataset size []. A lower ratio indicates a more condensed dataset.) is small. A plausible explanation for this could be the low density of representativeness within natural data instances. Therefore, dataset distillation seeks to synthesize data instances with densely packed features. We begin with a brief formulation of dataset distillation.

Problem formulation. We are given a natural and large dataset ${\mathcal{T}}=\{({\bm{x}}_{i},{\bm{y}}_{i})\}_{i=1}^{|{\mathcal{T}}|}$, where each element ${\bm{x}}_{i}\in{\mathbb{R}}^{d}$ is drawn i.i.d. from a natural distribution ${\mathcal{D}}$, and the class labels ${\bm{y}}_{i}\in{\mathcal{Y}}=\{0,1,\ldots,C-1\}$ with $C$ representing the number of classes.
Dataset distillation aims to synthesize a tiny dataset ${\mathcal{S}}=\{({\bm{s}}_{i},{\bm{y}}_{i})\}_{i=1}^{|{\mathcal{S}}|}$, where $s_{i}\in{\mathbb{R}}^{d}$ and $y_{i}\in{\mathcal{Y}}$, to server as an approximate solution to the following optimization problem:

$$ $\displaystyle{\mathcal{S}}=\operatorname*{arg\,min}_{{\mathcal{S}}\subset{ \mathbb{R}}^{d}\times{\mathcal{Y}}}\mathop{\mathbb{E}}_{({\bm{x}},{\bm{y}}) \sim{\mathcal{D}}}\left[\ell\left(f_{\theta_{{\mathcal{S}}}},{\bm{x}},{\bm{y}} \right)\right],$ (2) $$

where $\theta_{\mathcal{S}}$ represents the converged weights trained with $\mathcal{S}$, and $\ell$ is the loss function. The class labels $y_{i}$ are typically pre-assigned one-hot labels [, , , , , ] to encourage $\mathbf{s}_{i}$ to exhibit more distinct, class-specific features.

Pioneering this approach, Wang et al. [] is the first to propose DD, a method that optimizes ${\mathcal{S}}$ directly after substituting ${\mathcal{D}}$ with ${\mathcal{T}}$ in Equation 2. Due to the limited guidance, this approach often leads to suboptimal performance, prompting the development of gradient-matching methods [, , , ]. These methods improve supervision by aligning the model’s gradients. The success of gradient-matching has inspired further research into matching the trajectory of gradients [, , , ], yielding even better performance. Despite these advancements, most current dataset distillation methods still primarily focus on generating class-specific synthetic instances.
This ongoing adherence to the class-specific paradigm not only constrains the efficiency of the distillation budget but also results in the neglect of critical inter-class features.
These limitations drive our investigation into a new paradigm, aimed at developing a more efficient and effective dataset distillation solution.

Distillation Budget Consistency. As our INFER model depreciates the class-specific paradigm, it becomes necessary to establish clear criteria for maintaining consistency in the distillation budget. Traditional methods adopt Images Per Class (IPC) as described by [, , ], such that $|{\mathcal{S}}|=\texttt{ipc}\times C$. This approach provides a uniform criterion across various datasets. Therefore, we continue to employ IPC as the criterion to measure the distillation budget.

However, IPC does not account for the auxiliary generator [, ] or additional soft labels [] that are used to enhance the performance of a synthetic dataset, resulting in an asymmetric advantage compared to methods that solely utilize the data-label pair. Therefore, we also employ the compression ratio, as described by [], to measure the distillation budget. The total bit count of any auxiliary modules will be considered part of the synthetic dataset. To compute the compression ratio, we divide the total bit count of the synthetic dataset, including these auxiliary modules, by that of the original dataset, i.e., $\texttt{CR}={\mbox{synthetic dataset size}}/{\mbox{original dataset size}}$.

## 3 Methodology

In this section, we introduce our novel distillation paradigm, INFER. We begin by detailing the limitations associated with the class-specific distillation paradigm, highlighting inefficiencies and oversight of inter-class feature distributions. Following this, we describe the Universal Feature Compensator (UFC), the cornerstone of our methodology, designed to integrate inter-class features. Lastly, we discuss our approach to augmenting synthetic datasets, which is aimed at facilitating the formation of thin and clear decision boundaries among classes.

Figure: Algorithm 1 Distillation on synthetic dataset via Inter-class Feature Compensator (INFER)

### 3.1 Limitations in Class-Specific Distillation

Wang et al. [] established the general approach to solve ${\mathcal{S}}$ as outlined in Equation 2:
Each synthetic data instance ${\bm{s}}_{i}$ is assigned a one-hot label $y_{i}$ and optimized to capture intra-class features by minimizing $\sum_{({\bm{x}},{\bm{y}})\in{\mathcal{T}}}\ell\left(f_{\theta_{{\mathcal{S}}}}
,{\bm{x}},{\bm{y}}\right)$.
Unlike Equation 2, ${\mathcal{D}}$ is replaced by ${\mathcal{T}}$, given that ${\mathcal{D}}$ is inaccessible and ${\mathcal{T}}\sim{\mathcal{D}}$.
Initially, this class-specific design achieved progress in the early stages.
However, as dataset distillation research has evolved, two major limitations become prominent, compelling a rethinking of this design.

Inefficient utilization of the distillation budget. Recent advancements in dataset distillation [, , ] have enabled individual synthetic data instances to capture more features specific to a class, particularly notable in highly compressed scenarios where $\texttt{ipc}=1$ [, ]. However, as ipc increases, additional synthetic data instances tend to capture distinctive but duplicated intra-class features, leading to redundancy within the synthetic dataset.
This redundancy explains the marginal performance gains observed in less compressed scenarios ($\texttt{ipc}=50$). SeqMatch [] and DATM [] addressed this redundancy by dividing the synthetic data into several subsets optimized diversely. We validate this hypothesis through experiments shown in Figure 4(a). Ideally, newly optimized synthetic data instances should capture rare and diversifying features that complement the distinctive class-specific features.

Oversight of inter-class features. The prevalent focus on optimizing synthetic instances for the most distinctive intra-class features often leads to the oversight of inter-class features.

Figure: Figure 3: Illustration of the integration process between Universal Feature Compensators (UFCs) and natural data instances as described in Equation 3. The integration is performed through a simple addition process. Consequently, only the sets $\mathcal{S}=(\mathcal{P}^{k},\mathcal{U}^{k})$ need to be stored as the synthetic dataset. The synthetic dataset $\tilde{\mathcal{S}}^{k}$ is generated on-the-fly during training.

This oversight significantly limits the synthetic data’s ability to represent the feature distributions that span across different classes, which is crucial for complex classification tasks.
Consequently, the potential for these synthetic datasets to support the training of models that can generalize across varied scenarios may be significantly hampered.
As depicted in Figure 2, this issue is evidenced by the chaotic decision boundaries formed by the “one label per instance” synthetic dataset, which neglects the inter-class features necessary for forming thin and clear decision boundaries.

### 3.2 Universal Feature Compensator: Breaking Class Barriers

The Universal Feature Compensators (UFCs), denoted as ${\mathcal{U}}=\{{\bm{u}}_{i}\}_{i=1}^{|{\mathcal{U}}|}$, forms the core of our novel INFER paradigm, designed specifically to address the inefficiencies and oversight inherent in the class-specific distillation approach.

Design and Functionality. The primary objective of designing the UFC is to enable the generation of multiple synthetic instances from a single compensator. To achieve this, our INFER divides the base synthetic dataset $\mathcal{S}$ into $K$ subsets, such that $\mathcal{S}=\mathcal{S}^{1}\cup\mathcal{S}^{2}\cdots\mathcal{S}^{K}$. Each base subset $\mathcal{S}^{k}$ consists of a pair $(\mathcal{P}^{k},\mathcal{U}^{k})$, where $\mathcal{U}^{k}$ represents the set of UFCs, and $\mathcal{P}^{k}\subset\mathcal{T}$ contains natural instances to be integrated with UFCs. The generation process, as shown in Figure 3, integrates UFCs with natural data instances as follows:

$$ $\tilde{{\mathcal{S}}}^{k}=\{(\tilde{{\bm{s}}}_{i},\tilde{{\bm{y}}}_{i})\mid \tilde{{\bm{s}}}_{i}={\bm{x}}_{i}+{\bm{u}}_{j},\text{ for each }{\bm{x}}_{i} \in{\mathcal{P}}^{k}\text{ and each }{\bm{u}}_{j}\in{\mathcal{U}}^{k}\},$ (3) $$

where $\tilde{\mathcal{S}}^{k}$ is the generated synthetic dataset used for actual model training. Assuming that $|\mathcal{U}^{k}|=M$ where $M$ represents the number of architectures used to optimize ${\mathcal{U}}$, each $\mathcal{S}^{k}$ can be multiplied approximately $M$ times by INFER. We repeat the generation process described above $K$ times, once for each subset of $\mathcal{S}$. This structured approach allows INFER to utilize the distillation budget approximately $\bm{M}$ times more efficiently compared to the class-specific paradigm.

Optimization.
Under the INFER paradigm, each $\mathcal{P}^{k}$ contains exactly one instance for each class, randomly selected from the original dataset $\mathcal{T}$. Consequently, $\mathcal{P}^{k}$ effectively captures the intra-class features with minimal duplication. In contrast, the UFCs are designed to optimize the capture of inter-class features. Therefore, the elements in $\mathcal{P}^{k}$ uniformly cover each class, allowing the element $\mathcal{U}^{k}$ to focus on capturing inter-class features by solving the follow minimization problem:

$$ $\displaystyle\operatorname*{arg\,min}_{{\bm{u}}_{j}\in{\mathbb{R}}^{d}}\sum_{( {\bm{x}}_{i},y_{i})\in{\mathcal{P}}^{k}}\left[\ell\left(f_{\theta_{{\mathcal{T }}}},{\bm{x}}_{i}+{\bm{u}}_{j},y_{i}\right)+\alpha\mathcal{L}_{\mathrm{BN}} \left(f_{\theta_{{\mathcal{T}}}},{\bm{x}}_{i}+{\bm{u}}_{j}\right)\right],\quad \mbox{where}$ (4) $\displaystyle\mathcal{L}_{\mathrm{BN}}\left(f_{\theta_{{\mathcal{T}}}},{\bm{x} }_{i}+{\bm{u}}_{j}\right)=\sum\nolimits_{l}\|\mu_{l}({\mathbb{S}})-\mu_{l}({ \mathcal{T}})\|_{2}+\sum\nolimits_{l}\|\sigma^{2}_{l}({\mathbb{S}})-\sigma^{2} _{l}({\mathcal{T}})\|_{2},$ $$

we define ${\mathbb{S}}=\{(\tilde{{\bm{s}}}_{i},y_{i})\mid\tilde{{\bm{s}}}_{i}={\bm{x}}_{
i}+{\bm{u}}_{j},\text{ for each }({\bm{x}}_{i},y_{i})\in{\mathcal{P}}^{k}\}$, which is the generated synthetic dataset by integrating ${\bm{u}}_{j}$ with the corresponding ${\mathcal{P}}^{k}$. $\mathcal{L}_{\mathrm{BN}}$ is the BN loss inspired by [] to regularize the values generated $\tilde{{\bm{s}}}_{i}$ to fall within the same normalization distribution as those in ${\mathcal{T}}$. Intuitively, ${\bm{u}}_{i}$ serves as the feature compensator for the natural instance set $\mathcal{S}$, carrying universal inter-class features that are beneficial for classification.

### 3.3 Enhancing Synthetic Data with Inter-Class Augmentation

Although UFCs are encouraged to encapsulate more inter-class features, their limited size, especially compared to the size of the original dataset ${\mathcal{T}}$, restricts the breadth of features crucial for forming the decision boundaries of neural networks. Therefore, we also leverage MixUP [] as a technique to enhance inter-class augmentation.

Data augmentation [] has been well-developed and proven effective in training neural networks with natural datasets. However, the advancements in data augmentation do not generalize well to synthetic datasets generated through dataset distillation. DSA [] designed an adapted augmentation method specialized for dataset distillation, but it is not as effective as standard data augmentation methods. SRe2L [] applies MixUp [] to the synthetic dataset, achieving superior performance across many datasets. Unfortunately, the cost of applying MixUp in SRe2L is expensive, due to the massive volume of soft labels. The soft labels are dynamically generated for each augmented instance in each validation epoch, resulting enormous storage size. For example, the synthetic dataset generated by SRe2L for ImageNet-1k requires 0.7GB to store sythetic images, but requires additional 25.9GB to store the soft labels.

Motivated by this, we aim to apply MixUp to our INFER model in the same way it is used in natural datasets, which we refer to as “static” soft labels. The linear interpolation of labels can be represented mathematically as:

$$ $f_{\theta_{\mathcal{T}}}[\lambda\tilde{{\bm{s}}}_{i}+(1-\lambda)\tilde{{\bm{s} }}_{j}]\approx[\lambda f_{\theta_{\mathcal{T}}}(\tilde{{\bm{s}}}_{i})+(1- \lambda)f_{\theta_{\mathcal{T}}}(\tilde{{\bm{s}}}_{j})],\forall\tilde{{\bm{s}} }_{i},\tilde{{\bm{s}}}_{j}\in\tilde{{\mathcal{S}}},$ (5) $$

where $f_{\theta_{\mathcal{T}}}(\cdot)$ represents the logits output, $\lambda\sim\mbox{Beta}(\beta,\beta)$, and $\beta>0$. To achieve this, we propose three improvements in INFER: (a) We use $\mathcal{P}^{k}$, a subset of natural datasets, for integration with UFCs because natural instances inherently follow the linear interpolation of labels. (b) We make $\mathcal{P}^{k}$ to span across all the classes, rather than limiting it to instances within the same class. As such, the optimized UFCs, $\mathcal{U}$, which are integrated with $\mathcal{P}^{k}$ for optimization, also embody the characteristic of linear label interpolation. (c) We employ $M$ neural networks with various architectures ($M=|{\mathcal{U}}^{k}|$), to relabel the generated synthetic data instance $\tilde{{\bm{s}}}_{i}$ through averaging, i.e.,

$$ $\tilde{{\bm{y}}}_{i}=\frac{1}{M}\sum_{m}f^{m}_{\theta_{\mathcal{T}}}(\tilde{{ \bm{s}}}_{i}).$ (6) $$

By doing so, INFER does not requires the dynamic soft labels for any combinations of $[\lambda\tilde{{\bm{s}}}_{i}+(1-\lambda)\tilde{{\bm{s}}}_{j}]$, but only the static soft labels of $\tilde{{\bm{s}}}_{i},\tilde{{\bm{s}}}_{j}$, which reduces the size of soft labels by up to 99.9%. More details on synthesizing ${\mathcal{S}}$ can be found in Algorithm 1.

## 4 Experiments

To evaluate the effectiveness of our proposed INFER distillation paradigm, we conducted a series of experiments across multiple benchmark datasets and compared our results with several state-of-the-art approaches. In this section, we provide details on the experimental setup, the datasets used, and the results obtained. We summarize our main results in Table 1 and Table 2. Following this, we perform ablation studies to examine the impact of individual components of our method. Finally, we discuss the limitations of INFER and outline directions for future work. We conduct our experiments on two Nvidia 3090 GPUs and one Tesla A-100 GPU.

**Table 1: Comparison with SOTAs on CIFAR-10/100 and Tiny-ImageNet. Except for SRe2L [], G-VBSM [], and our INFER, all other methods use ConvNet128 for synthesis. The distilled synthetic datasets are then evaluated on ConvNet128 and ResNet18. “INFER+Dyn” denotes the application of INFER using dynamically generated soft labels, as described in []. The best performers in each setting are highlighted in red.**
|  |  | CIFAR-10 | CIFAR-100 | Tiny-ImageNet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ipc | $10$ | $50$ | $10$ | $50$ | $100$ | $10$ | $50$ |
|  | Random | $\taformat{31.0}{0.5}$ | $\taformat{50.6}{0.3}$ | $\taformat{14.6}{0.5}$ | $\taformat{33.4}{0.4}$ | $\taformat{42.8}{0.3}$ | $\taformat{5.0}{0.2}$ | $\taformat{15.0}{0.4}$ |
| \rotatebox90ConvNet128 | DC [] | $\taformat{44.9}{0.5}$ | $\taformat{53.9}{0.5}$ | $\taformat{25.2}{0.3}$ | - | - | - | - |
| KIP [] | $\taformat{62.7}{0.3}$ | $\taformat{68.6}{0.2}$ | $\taformat{28.3}{0.1}$ | - | - | - | - |  |
| RFAD [] | $\bltaformat{66.3}{0.5}$ | $\taformat{71.1}{0.4}$ | $\taformat{33.0}{0.3}$ | - | - | - | - |  |
| MTT [] | $\taformat{65.4}{0.7}$ | $\taformat{71.6}{0.2}$ | $\taformat{39.7}{0.4}$ | $\taformat{47.7}{0.2}$ | $\taformat{49.2}{0.4}$ | $\taformat{23.2}{0.2}$ | $\taformat{28.0}{0.3}$ |  |
| G-VBSM [] | $\taformat{46.5}{0.7}$ | $\taformat{54.3}{0.3}$ | $\taformat{38.7}{0.2}$ | $\taformat{45.7}{0.4}$ | - | - | - |  |
| INFER | $\taformat{33.7}{0.8}$ | $\taformat{54.1}{0.2}$ | $\bltaformat{42.9}{0.7}$ | $\bltaformat{55.1}{0.4}$ | $\bltaformat{56.7}{0.5}$ | $\bltaformat{26.2}{0.1}$ | $\bltaformat{34.7}{0.1}$ |  |
| INFER+Dyn | $\taformat{30.1}{0.8}$ | $\taformat{52.4}{0.7}$ | $\taformat{37.2}{0.3}$ | $\taformat{50.7}{0.3}$ | $\taformat{53.4}{0.2}$ | $\taformat{24.9}{0.3}$ | $\taformat{33.9}{0.6}$ |  |
| \rotatebox90ResNet18 | Random | $\taformat{29.6}{0.9}$ | $\taformat{36.7}{1.7}$ | $\taformat{15.8}{0.2}$ | $\taformat{32.0}{0.0}$ | $\taformat{47.5}{0.0}$ |  | $\taformat{17.7}{0.0}$ |
| SRe2L [] | $\taformat{27.2}{0.5}$ | $\taformat{47.5}{0.6}$ | $\taformat{31.6}{0.5}$ | $\taformat{49.5}{0.3}$ | - | - | $\taformat{41.1}{0.4}$ |  |
| G-VBSM [] | $\taformat{53.5}{0.6}$ | $\taformat{59.2}{0.4}$ | $\bltaformat{59.5}{0.4}$ | $\taformat{65.0}{0.5}$ | - | - | $\taformat{47.6}{0.5}$ |  |
| INFER | $\bltaformat{33.5}{0.3}$ | $\taformat{59.2}{0.8}$ | $\taformat{50.2}{0.2}$ | $\taformat{65.1}{0.1}$ | $\taformat{68.6}{0.4}$ | $\taformat{38.6}{0.4}$ | $\bltaformat{55.0}{0.1}$ |  |
| INFER+Dyn | $\taformat{30.7}{0.3}$ | $\bltaformat{60.7}{0.9}$ | $\taformat{53.4}{0.6}$ | $\bltaformat{68.9}{0.1}$ | $\bltaformat{73.3}{0.2}$ | $\bltaformat{41.0}{0.4}$ | $\taformat{54.6}{0.4}$ |  |

**Table 2: Comparison with SOTAs on ImageNet-1K. SRe2L [], G-VBSM [], and our INFER use ResNet18 for distillation. The distilled synthetic datasets are then evaluated on ResNet18, 50, and 101. “INFER+Dyn” denotes the application of INFER using dynamically generated soft labels, as described in []. The best performer is highlighted in red.**
|  | Compression Ratio | ResNet18 | ResNet-50 | ResNet-101 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ipc | $10$ | $50$ | $10$ | $50$ | $10$ | $50$ | $10$ | $50$ |
| Random | 0.78% | 3.90% | \taformat10.50.4 | \taformat31.40.3 | \taformat9.30.3 | $\taformat{31.5}{0.2}$ | \taformat10.00.4 | $\taformat{33.1}{0.1}$ |
| INFER | 0.81% | 4.04% | $\bltaformat{37.0}{0.5}$ | $\taformat{54.3}{0.4}$ | $\bltaformat{39.3}{0.3}$ | $\taformat{62.5}{0.2}$ | $\taformat{37.0}{0.4}$ | $\bltaformat{63.0}{0.4}$ |
| SRe2L [] | 4.53% | 22.67% | $\taformat{21.3}{0.6}$ | $\taformat{46.8}{0.2}$ | $\taformat{28.4}{0.1}$ | $\taformat{55.6}{0.3}$ | $\taformat{30.9}{0.1}$ | $\taformat{60.8}{0.5}$ |
| G-VBSM [] | 4.53% | 22.67% | $\taformat{31.4}{0.5}$ | $\taformat{51.8}{0.4}$ | $\taformat{35.4}{0.8}$ | $\taformat{58.7}{0.3}$ | $\taformat{38.2}{0.4}$ | $\taformat{61.0}{0.4}$ |
| INFER+Dyn | 4.53% | 22.67% | $\taformat{36.3}{0.3}$ | $\bltaformat{55.6}{0.2}$ | $\taformat{38.3}{0.5}$ | $\bltaformat{63.4}{0.3}$ | $\bltaformat{38.9}{0.5}$ | $\taformat{60.7}{0.1}$ |

### 4.1 Experimental Setup

Baselines and datasets. We conduct the comparison with several representative distillation methods, including DC [], DSA [], KIP [], RFAD [], MTT [], FTD [], G-VBSM [] and SRe2L []. This evaluation is performed on four popular classification benchmarks, including CIFAR-10/100 [], Tiny-ImageNet [], and ImageNet-1K [].

Implementation details. Our INFER use $k=4$, i.e., four different architectures for optimizaing UFCs, including ResNet18 [], MobileNetv2 [], EfficientNetB0 [], and ShuffleNetv2 []. When distilling ImagNet-1K, only the first three ($k=3$) are involed. More details in implementing INFER is introduced in Algorithm 2 and Appendix C.

Consistant Distillation Budget. As we stated in Section 2, the baselines SRe2L [] and G-VBSM [] use the dynamic generated soft labels from a teacher model in each validation epoch for enhanced performance. However, these additional dynamic soft labels are not considered in the “image per class” distillation budget. Therefore, we also adopt the compression ratio ($\texttt{CR}={\mbox{synthetic dataset size}}/{\mbox{original dataset size}}$ []) for consistant distillation budget. As the size of the soft labels is proportional to the number of validation epochs, we only report the CR in ImageNet-1k dataset, as shown in Table 2.

Our INFER also employ the soft labels as shown in Equation 6. However, INFER only stores one soft label for one instance, which equals one epoch dynamic soft labels. We term it as static soft labels in contrast to the dynamic soft labels generated across every validation epochs. Therefore, INFER reduces the size of soft labels by up to 99.3% in ImageNet-1k dataset (from 300 epoch to 1 epoch). We also implement the dynamic soft labels under INFER, denoted as “INFER+Dyn”. As increasing the validation epoch can significantly improve the performance, but it incurs a larger size of soft labels. To ensure fair comparison, we reduce the validation epoch to $\frac{1}{4}$, as INFER generates four-fold synthetic instances by integrating UFC (The numer is 3 in ImageNet-1k). Lastly, we also average the number of UFCs into image per class (ipc) for fair comparison.

### 4.2 Main Results

Performance results on CIFAR-10/100 and Tiny-ImageNet are reported in Table 1.
Our INFER method demonstrates significantly superior performance on CIFAR-100 and Tiny-ImageNet. For instance, with an ipc budget of 50, ResNet18 trained on the distilled CIFAR-100 achieves classification accuracies of 68.9% and 65.1%, surpassing SRe2L [] by 19.4% and 15.6%, respectively. Additionally, INFER outperform the SOTA competitor, G-VBSM [], by 7.4% on Tiny-ImageNet with ipc=50. Notably, although our INFER reduces 99.3% dynamic soft labels, it still outperforms the methods [, ] employing dynamic soft labels in each validation epoch.

Table 2 provides a performance comparison with SRe2L [], the first method to extend dataset distillation to large-scale datasets, specifically ImageNet-1K. Both our method and SRe2L utilize ResNet18 for distillation and are evaluated on ResNet18, ResNet50, and ResNet101 architectures.
Our INFER method outperforms SRe2L across multiple evaluation architectures. For instance, INFER achieves a performance gain of 7.5%, requiring only 4.04% the size of original datasets when training ResNet18 using $\texttt{ipc}=50$.

**Table 3: Cross-architecture performance of distilled dataset. Here, the synthetic CIFAR-100 datasets are evaluated by training ResNet-50, ResNet-101 [], MobileNetV2 [], EfficientNetB0 [], and ShuffleNetV2 [] from scratch. The best performers in each setting are highlighted in red.**
|  | $\texttt{ipc}=10$ | $\texttt{ipc}=50$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Networks | SRe2L [] | INFER | INFER+Dyn | SRe2L | INFER | INFER+Dyn |
| ResNet50 | $\taformat{22.4}{1.3}$ | $\taformat{49.3}{0.6}$ | $\bltaformat{52.3}{0.4}$ | $\taformat{52.8}{0.7}$ | $\taformat{65.6}{0.3}$ | $\bltaformat{70.0}{0.2}$ |
| MobileNetV2 | $\taformat{16.1}{0.5}$ | $\bltaformat{50.8}{0.3}$ | $\taformat{43.3}{0.7}$ | $\taformat{43.2}{0.2}$ | $\taformat{66.9}{0.3}$ | $\bltaformat{67.2}{0.2}$ |
| EfficientNetB0 | $\taformat{11.1}{0.3}$ | $\bltaformat{37.9}{0.3}$ | $\taformat{34.9}{0.8}$ | $\taformat{24.9}{1.7}$ | $\taformat{60.5}{0.3}$ | $\bltaformat{62.7}{0.4}$ |
| ShuffleNetV2 | $\taformat{11.8}{0.7}$ | $\bltaformat{42.4}{0.5}$ | $\taformat{30.9}{0.5}$ | $\taformat{27.5}{1.1}$ | $\bltaformat{64.5}{0.6}$ | $\taformat{62.1}{0.2}$ |

### 4.3 Discussions

Feature Duplication Study. We verify our hypothesis that synthetic data instances tend to capture distinctive yet duplicated intra-class features under the traditional class-specific distillation paradigm. We measure feature duplication by averaging the cosine similarities between each pair of synthetic data instances within the same class. Our experimental results, as shown in Figure 4(a), support our hypothesis: the class-specific approach SRe2L exhibits higher feature duplication as the number of ipc increases. Conversely, our method, INFER, achieves improved feature uniqueness, more closely resembling natural datasets.

Visualization on Decision Boundaries. To verify our hypothesis regarding the“oversight of inter-class features” in the traditional class-specific distillation paradigm, we visualize the decision boundaries of ResNet-18 models trained with synthetic datasets generated by SRe2L and our INFER, respectively. We randomly select seven classes from the CIFAR-100 dataset and use the t-SNE approach for visualization. As illustrated in Figure 2, INFER forms thin and clear decision boundaries between classes, in contrast to the chaotic decision boundaries produced by the traditional approach. Additionally, we visualized the 3D loss landscape in pixel spaces of the decision boundaries in Figure 5, which further supports our hypothesis from a different perspective.

Compensator Generation. We examine the effectiveness of the proposed UFC. As shown in Figure 4(b), regardless of the ipc setting and the labeling strategy, our UFC significantly enhances the quanlity of the distilled dataset. For example, the Top-1 classification accuracy of the model trained with dynamic labels is improved by $35.0\%$ with $\texttt{ipc}=10$. We also study the influence of network architectures participating compensator generation. According to the results provided in Table 4, with more networks ensembled, there is a sustained improvement in performance.

Figure: Figure 4: Left: The change in feature duplication with the increase of ipc. To measure the level of feature duplication, we employ the averaged cosine similarities between each pair of synthetic data instances within the same class. Therefore, a greater value represents higher feature duplication, as SRe2L [] shows. In contrast, our INFER obtains a lower feature duplication, which is closer to the level observed in natural datasets. Right: The ablation study of UFC. The first two groups are under the $\texttt{ipc}=10$ setting, while the other two are under $\texttt{ipc}=50$. The purple annotations indicate the performance gains contributed by our UFC.

Cross-Architecture Generalization. Table 3 presents the performance evaluation of synthetic CIFAR-100 dataset across different architectures, trained from scratch.
When ipc=10 , our INFER+Dyn and INFER methods outperform SRe2L across all architectures. For instance, INFER+Dyn achieves an accuracy of 52.3% on ResNet50, significantly higher than the 22.4% achieved by SRe2L.
For ipc=50, the performance advantage of INFER+Dyn and INFER remains evident. INFER+Dyn reaches an accuracy of 70.0% on ResNet50, far surpassing SRe2L. Our INFER shows a well generalization abilities across different architectures.

Figure: Figure 5: Visualizations of loss landscapes in pixel space on CIFAR-100 datasets. The optimal decision boundary is supposed to have a rapid change in cross-entropy loss at the edge, indicating a clear and distinctive decision boundary. Left: A distinctive decision boundary trained on the original dataset ${\mathcal{T}}$. Middle: A less distinctive decision boundary trained on the synthetic dataset of outstanding class-specific approach SRe2L. Right: An improved decision boundary trained on the synthetic dataset of INFER.

**Table 4: Ensemble of architectures for UFC generation. “R”, “M”, “E”, and “S” represent ResNet18 [], MobileNetv2 [], EfficientNetB0 [], and ShuffleNetV2 []. ✔ indicates the network architectures participating in UFC generation. $\uparrow$ denotes the performance gain contributed by the current ensembles compared with the baseline (only ResNet18).**
|  | $\texttt{ipc}=10$ | $\texttt{ipc}=50$ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R | M | E | S | INFER | $\uparrow$ | INFER+Dyn | $\uparrow$ | INFER | $\uparrow$ | INFER+Dyn | $\uparrow$ |
| ✔ |  |  |  | $\taformat{45.0}{0.3}$ | + 0.0 | $\taformat{38.1}{1.7}$ | + 0.0 | $\taformat{62.6}{0.2}$ | + 0.0 | $\taformat{65.3}{0.2}$ | + 0.0 |
| ✔ | ✔ |  |  | $\taformat{48.4}{0.6}$ | + 3.4 | ${46.2}$ | + 8.1 | $\taformat{64.2}{0.1}$ | + 1.6 | $\taformat{67.3}{0.1}$ | + 2.0 |
| ✔ | ✔ | ✔ |  | $\taformat{49.6}{0.5}$ | + 4.6 | $\taformat{50.6}{0.7}$ | + 12.5 | $\taformat{64.9}{0.2}$ | + 2.3 | $\taformat{68.3}{0.2}$ | + 3.0 |
| ✔ | ✔ | ✔ | ✔ | $\taformat{50.2}{0.2}$ | + 5.2 | $\taformat{53.4}{0.6}$ | + 16.3 | $\taformat{65.1}{0.1}$ | + 2.5 | $\taformat{68.9}{0.1}$ | + 3.6 |

Limitations and Future Work.
Despite the promising results of our INFER distillation paradigm, there are two limitations that should be acknowledged. First, the UFC may not generalize well to datasets with more than 1,000 classes (e.g., ImageNet-21k []), as it is designed to reflect the general representativeness across all classes. With an increased number of classes, optimizing the UFC becomes potentially more challenging and may struggle to converge. This limitation suggests the potential for future exploration into clustering classes with common inter-class features for each UFC, thereby simplifying the optimization process and improving scalability.

Second, our INFER currently employs a two-level structure in extracting knowledge from the original datasets.
While the natural instances focus on condensing intra-class features, the UFC uniquely emphasizes the inclusion of inter-class features to enhance the dataset’s representational breadth.
It not only allows the UFC to strategically enrich the dataset by emphasizing inter-class features, but also opens opportunities to harness the unique synergies between intra-class and inter-class features.
Future work could explore more integrated approaches that simultaneously optimize for both types of features (e.g. integrating dataset pruning techniques), potentially enhancing the overall representativeness and performance of the synthetic data.

## 5 Conclusion

In this work, we rethink the current “one class per instance” paradigm in dataset distillation and identified its limitations, including inefficient utilization of the distillation budget and oversight of inter-class features. These issues arise as distillation techniques advance, leading to synthetic data that often captures duplicated class-specific features. To address these limitations, we introduce a novel paradigm INFER that employs a Universal Feature Compensator for “one instance for all classes” distillation. Our experimental results demonstrate that INFER improves the efficiency and effectiveness of dataset distillation, achieving state-of-the-art results in several datasets, reducing resource requirements while maintaining high performance. Future work will focus on scaling INFER for extremely large dataset and exploring its application in various real-world scenarios.

## Appendix A Training with Synthetic Dataset

Figure: Algorithm 2 Training with synthetic dataset via Inter-class Feature Compensator (INFER)

## Appendix B More Related Works

The goal of dataset distillation is to create a condensed dataset that, despite its significantly reduced scale, maintains comparable performance to the original dataset.
This concept was first introduced by Wang et al. [] as a bi-level optimization problem.
Building on this foundational work, recent advancements have broadened the range of techniques available for effectively and efficiently condensing representative knowledge into compact synthetic datasets.
Techniques such as gradient matching [, , , ] optimize synthetic data to emulate the weight parameter gradients of the original dataset, while trajectory matching [, , , ] aims to replicate gradient trajectories to tighten synthesis constraints, showcasing the focus on effectiveness.
Additionally, factorized methods [, , , ] use specialized decoders to generate highly informative images from condensed features, enhancing both the utility and efficiency of distilled datasets.
Another strategy, distribution matching [, , , , ], optimizes synthetic data to align its feature distribution with that of the original dataset on a class-wise basis.
Although its efficiency has significantly improved, the performance of this method may still lag behind those using gradient or trajectory matching.
Despite these innovations, most methods continue to grapple with balancing final accuracy and computational efficiency, presenting challenges for their application to large-scale and real-world datasets.

To further enhance dataset distillation for large datasets like ImageNet-1K [], researchers have developed various innovative strategies to overcome inherent challenges.
Cui et al. [] introduced unrolled gradient computation with constant memory usage to manage computational demands efficiently.
Following this, Yin et al. [] developed the SRe2L framework, which decouples the bi-level optimization of models and synthetic data during training to accommodate varying dataset scales.
Recognized for its excellent performance and adaptability, this framework has spurred further research.
Further, Yin et al. proposed the Curriculum Data Augmentation (CDA) [], a method that enhances accuracy without substantial increases in computational costs.
Based on SRe2L, Zhou et al. [] developed SC-DD, a Self-supervised Compression framework for dataset distillation that enhances the compression and recovery of diverse information, leveraging the potential of large pretrained models.
Additionally, Xuel et al. [] focused on improving the robustness of distilled datasets by incorporating regularization during the squeezing stage of the SRe2L process.
In Generalized Various Backbone and Statistical Matching (G-VBSM) [], Shao et al. introduced a "local-match-global" matching technique based on SRe2L, which yields more precise and effective results, producing a synthetic dataset with richer information and enhanced generalization capabilities.
Most recently, the Curriculum Dataset Distillation (CUDD) [] was introduced, employing a strategic, curriculum-based approach to distillation that balances scalability and efficiency.
This framework systematically distills synthetic images, progressing from simpler to more complex tasks.

## Appendix C Training Recipes

In this section, we provide the implementation details for further reproduction. The hyperparameter settings for the experimental datasets, including CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K, are shown in Tables 5, 6, and 7. These settings generally adhere to SRe2L []. The only modification is that we proportionally reduce the validation epoch number for the dynamic version to ensure fair comparison. All other crucial hyperparameters remain unchanged.

**Table 5: Training recipe of CIFAR-10/100.**
|  | Optimizer | Learning<br>Rate | Batch<br>Size | Epoch/Iteration | Augmentation | Architectures |
| --- | --- | --- | --- | --- | --- | --- |
| Synthesis | Adam<br>$\{\beta_{1},\beta_{2}\}=\{0.5,0.9\}$ | 0.25<br>cosine decay | 100 | Iteration:1000 | - | {ResNet18, MobileNetv2,<br>EfficientNetB0, ShuffleNetV2} |
| Validation | AdamW<br>weight decay = 0.01 | 0.001<br>cosine decay | 64 | Epoch:400 | RandomCrop<br>RandomHorizontalFlip<br>MixUp | {ResNet18, MobileNetv2,<br>EfficientNetB0, ShuffleNetV2} |
| Validation+dyn | AdamW<br>weight decay = 0.01 | 0.001<br>cosine decay | 64 | Epoch:80 | RandomCrop<br>RandomHorizontalFlip<br>MixUp | {ResNet18, MobileNetv2,<br>EfficientNetB0, ShuffleNetV2} |

**Table 6: Training recipe of Tiny-ImageNet.**
|  | Optimizer | Learning<br>Rate | Batch<br>Size | Epoch/Iteration | Augmentation | Architectures |
| --- | --- | --- | --- | --- | --- | --- |
| Synthesis | Adam<br>$\{\beta_{1},\beta_{2}\}=\{0.5,0.9\}$ | 0.1<br>cosine decay | 200 | Iteration:2000 | RandomResizedCrop<br>RandomHorizontalFlip | {ResNet18, MobileNetv2,<br>EfficientNetB0, ShuffleNetV2} |
| Validation | SGD<br>weight decay = 0.9 | 0.2<br>cosine decay | 64 | Epoch:200 | RandomResizedCrop<br>RandomHorizontalFlip<br>MixUp | {ResNet18, MobileNetv2,<br>EfficientNetB0, ShuffleNetV2} |
| Validation+dyn | SGD<br>weight decay = 0.9 | 0.2<br>cosine decay | 64 | Epoch:50 | RandomResizedCrop<br>RandomHorizontalFlip<br>MixUp | {ResNet18, MobileNetv2,<br>EfficientNetB0, ShuffleNetV2} |

**Table 7: Training recipe of ImageNet-1K.**
|  | Optimizer | Learning<br>Rate | Batch<br>Size | Epoch/Iteration | Augmentation | Architectures |
| --- | --- | --- | --- | --- | --- | --- |
| Synthesis | Adam<br>$\{\beta_{1},\beta_{2}\}=\{0.5,0.9\}$ | 0.25<br>cosine decay | 1000 | Iteration:2000 | RandomResizedCrop<br>RandomHorizontalFlip | {ResNet18, MobileNetv2,<br>EfficientNetB0} |
| Validation | AdamW<br>weight decay = 0.01 | 0.001<br>cosine decay | 32 | Epoch:300 | RandomResizedCrop<br>RandomHorizontalFlip<br>MixUp | {ResNet18, MobileNetv2,<br>EfficientNetB0} |
| Validation+dyn | AdamW<br>weight decay = 0.01 | 0.001<br>cosine decay | 32 | Epoch:75 | RandomResizedCrop<br>RandomHorizontalFlip<br>MixUp | {ResNet18, MobileNetv2,<br>EfficientNetB0} |

## Appendix D More Experiments

Soft-label Generation. To align with the multi-model aided compensator generation, we employ multiple networks for soft label generation. Table 8 presents the model performance trained with soft labels generated by various architecture ensembles. Notably, our INFER paradigm demonstrates resilience to changes in the number of labeling networks. ResNet18 and MobileNetv2 contribute significantly to performance enhancement, while EfficientNetB0 and ShuffleNetv2 exhibit marginal impact, occasionally leading to negative outcomes (e.g., $\texttt{ipc}=50$ for INFER+Dyn).

More Comparisons with SOTA Method SRe2L. Here we offer more comparison with the SOTA method
As illustrated in Figure 6, the x-axis represents the cross-entropy loss calculated by the pretrained model, reflecting the difficulty of samples. It is evident from the graph that the superiority of our INFER model over SRe2L remains consistent across varying sample difficulties.
Figure 6 shows the Kullback-Leibler divergence (KLD) between dynamic labels and static labels. A smaller KLD indicates better adaptability of our method to static labeling, making high compression rates practical. Conversely, SRe2L relies heavily on memory-consuming dynamic labeling. This explains why our method can achieve high compression rates while still delivering satisfactory performance.

**Table 8: Ensemble of architectures for soft label generation. “R”, “M”, “E”, and “S” represent ResNet18 [], MobileNetv2 [], EfficientNetB0 [], and ShuffleNetV2 []. ✔ indicates the network architectures participating in soft labels generation. $\uparrow$ denotes the performance gain contributed by the current ensembles compared with the baseline (only ResNet18).**
|  | $\texttt{ipc}=10$ | $\texttt{ipc}=50$ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R | M | E | S | INFER | $\uparrow$ | INFER+Dyn | $\uparrow$ | INFER | $\uparrow$ | INFER+Dyn | $\uparrow$ |
| ✔ |  |  |  | $\taformat{41.3}{0.2}$ | + 0.0 | $\taformat{46.4}{0.2}$ | + 0.0 | $\taformat{61.6}{0.4}$ | + 0.0 | $\taformat{69.0}{0.1}$ | + 0.0 |
| ✔ | ✔ |  |  | $\taformat{48.6}{0.4}$ | + 7.3 | $\taformat{53.2}{1.0}$ | + 6.8 | $\taformat{65.3}{0.3}$ | + 3.7 | $\taformat{70.2}{0.3}$ | + 1.2 |
| ✔ | ✔ | ✔ |  | $\taformat{50.0}{0.4}$ | + 8.7 | $\taformat{53.0}{0.6}$ | + 6.6 | $\taformat{65.0}{0.3}$ | + 3.4 | $\taformat{69.0}{0.1}$ | + 0.0 |
| ✔ | ✔ | ✔ | ✔ | $\taformat{50.2}{0.2}$ | + 8.9 | $\taformat{53.4}{0.6}$ | + 7.0 | $\taformat{65.1}{0.1}$ | + 3.5 | $\taformat{68.9}{0.1}$ | - 0.1 |

Figure: Figure 6: Performance on the validation set. The bars represent the number of samples correctly classified by SRe2L and our INFER, in each CE loss interval.

Figure: Figure 7: Kullback-Leibler divergence (KLD) between dynamic labels and static labels.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The authors claim that the existing dataset distillation methods focus on optimizing synthetic data exclusively for a pre-assigned one-hot label. However, RDED [1] introduces a non-optimization method that changes this paradigm. It is recommended that the authors analyze and compare their method with RDED.

2. In Table 1, lines 416-417, the authors claim that static labels perform better on small networks, while dynamic labels excel on large networks. However, in Table 2, static labels sometimes outperform dynamic labels on large networks and datasets, particularly when 
IPC=50 using ResNet-101. Why is this the case?

3. Additionally, in Table 4, the performance comparison between static and dynamic labels is inconsistent. Sometimes static labels are superior, other times dynamic labels are. What factors determine the choice between static and dynamic labels in real-world applications?

4. Analyzing computational cost is crucial in dataset distillation. It would be beneficial to compare both computation cost and peak memory usage against existing methods. 

[1] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
soundness: 2
strengths: 1. It analyses the limitations of the current optimization-based dataset distillation methods.
2. The proposed method can significantly reduce the storage.
3. The paper is well-written.
summary: This paper first identifies two limitations of the current dataset distillation methods: inefficient utilization of the distillation budget and oversight of inter-class features. Based on this, this paper proposes to leverage a Universal Feature Compensator to enhance feature integration across classes.
Experiments demonstrate that the proposed INFER improves the storage efficiency and effectiveness of dataset distillation.
weaknesses: 1. The authors claim that the existing dataset distillation methods focus on optimizing synthetic data exclusively for a pre-assigned one-hot label. However, RDED [1] introduces a non-optimization method that changes this paradigm. It is recommended that the authors analyze and compare their method with RDED.

2. In Table 1, lines 416-417, the authors claim that static labels perform better on small networks, while dynamic labels excel on large networks. However, in Table 2, static labels sometimes outperform dynamic labels on large networks and datasets, particularly when 
IPC=50 using ResNet-101. Why is this the case?

3. Additionally, in Table 4, the performance comparison between static and dynamic labels is inconsistent. Sometimes static labels are superior, other times dynamic labels are. What factors determine the choice between static and dynamic labels in real-world applications?

4. Analyzing computational cost is crucial in dataset distillation. It would be beneficial to compare both computation cost and peak memory usage against existing methods. 

[1] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How to calculate the utilized budget of INFER? According to Algorithm 1, it seems that $P^k$, $U^k$, and $Y^k$ are stored in budget after training. $P^k$ consists of instances equal to the number of classes $C$ (Line 263), and $U^k$ consists of $M$ instances (Line 269). Since there are a total of $K=\text{IPC}$ (Line 261) sets of $P^k$ and $U^k$, the total number of instances in $P$ and $U$ amounts to $\text{IPC} \times (C + M)$, which already exceeds the budget of $\text{IPC}×C$ used by conventional methods. If my understanding is incorrect, please clarify the budget utilized by INFER.

2. Is there a reason for learning $U^k$ independently for each $k$? Learning $U^k$ independently for each $k$ aims to capture the inter-class features of $P^k$. This approach seems to capture only local inter-class features, which may be suboptimal. Introducing a single $U$ could potentially be beneficial for obtaining global inter-class features. Are there experimental results using a single $U$?

3. In Algorithm 1, all $u_j \in U^k$ are initialized with zero vector and independently optimized with same ingredient. This suggests a risk that all instances of $u_j$ may converge to the same optimal value. I am curious about how different values of the optimized $u_j$ actually are. Additionally, I am interested in the rationale for initializing with a zero vector rather than a random noise vector, as well as the resulting performance when using the latter.

4. In general, data factorization methods generate more instance than IPC under same budget. Given this characteristic, feature duplication may present a greater limitation in data factorization methods. Does feature duplication occur in data factorization as well? If it does, is there an improvement in performance when combining INFER with data factorization methodologies?
soundness: 3
strengths: 1. This manuscript is well written and easy to follow along with their reasoning.
2. The idea of UFC is simple and easy to adapt.
3. The motivations of this work, inefficient budget utilization and oversight of inter-class features, are straightforward and novel. The proposed method aligns with these motivations.
summary: In general, dataset distillation methods employ a class-specific synthesis paradigm, each synthetic instance has one label. This work suggests two limitations that these "one label per instance" approaches do not utilize the budget efficiently and fails to adequately reflect inter-class features. To overcome these limitations, this work introduces a new approach, Inter-class Feature Compensator (INFER), for “one instance for all classes” framework by utilizing Universal Feature Compensator (UFC). Experimental results show the effectiveness of the proposed framework.
weaknesses: 1. The calculation of utilized budget of INFER is unclear. Some components in INFER seem to be consist of several vectors which have same size with original natural instance. However, in the manuscript, there is explicit description of utilized budget of INFER. Therefore, it is essential to clearly specify how the budget used by INFER is calculated and how it is set to ensure a fair comparison with prior studies.

2. There is insufficient baseline used for performance comparison. Although many dataset distillation methods are recently proposed to enhance the performance and budget efficiency, this work only compares INFER to some (possibly outdated) studies. Additionally, comparing INFER’s performance with recent methods raised doubts about its effectiveness. For example, DATM [1], which also address the feature redundancy as the manuscript said (Line 238-240), achieves higher performance than INFER in many settings: 47.2% (IPC=10), 57.5% (IPC=100) for CIFAR-100 and 31.1% (IPC=10). 39.7% (IPC=50) for Tiny-ImageNet.

3. One of the component in INFER, $P^k$, is consist of real instances. Storing real instances can be a drawback in tasks where privacy issues are crucial (e.g., continual learning). Continual learning is one of the primary applications of dataset distillation.

[1] Towards lossless dataset distillation via difficulty-aligned trajectory matching

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: e. I have a question regarding the IPC used in the experiments, which refers to the number of images per class. If the authors apply the augmentation approach described in Equation 2, the final number of distilled images per class becomes IPC \times \left(1 + \frac{M}{{class number}}\right) . This seems to differ slightly from the IPC used in other comparison methods. While this difference might have minimal impact on large datasets with many classes, it could be more influential for smaller datasets. Have the authors accounted for this adjustment to ensure a fair comparison?
soundness: 3
strengths: a. The authors present an innovative analysis of existing class-specific dataset distillation methods. This work distinguishes itself from prior research by proposing the Universal Feature Compensator (UFC), which enhances inter-class feature integration, resulting in a more efficient dataset distillation framework. Relevant related works are appropriately cited. b. The submission is technically sound, with thorough evaluations and comparisons. c. The paper is clearly written and well-organized, making the content easy to follow and understand. d. This work addresses two key limitations of class-specific dataset distillation methods and achieves SOTA performance. Moreover, the proposed static labelling strategy reduces storage requirements by 99.3%.
summary: This paper presents a novel dataset distillation (DD) method, INFER (Inter-Class Feature Compensator), which explores the potential of leveraging inter-class features to significantly enhance distillation performance. In addition, it addresses a critical challenge faced by decoupled DD methods (e.g., SRe2L), where soft labels require nearly 30 times more storage than images. To overcome this limitation, the proposed method introduces a static labelling strategy that not only retains competitive performance but also dramatically reduces storage requirements, making it a more efficient and scalable solution.
weaknesses: a. The compensator is denoted by $\mu$ in Fig. 2, but by $u$ in Fig. 3. Maintaining consistent notation across the figures would improve clarity and help avoid potential confusion.
b. Some results fall short of SOTA benchmarks, and it would be helpful if the authors provided a discussion of potential reasons for this discrepancy.
c. The current evaluation only focuses on same-architecture distillation performance for ConvNet and ResNet18 on CIFAR and tiny-ImageNet. A broader cross-architecture evaluation would offer better insights into generalization. 
d. While the static labelling approach effectively reduces storage requirements and, in some cases, even outperforms the dynamic method, the paper does not explore whether these benefits extend to other decoupled DD methods, such as SRe2L.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: please see the weakness part.
soundness: 3
strengths: - The analysis of the two limitations in class-specific dataset distillation is insightful.
- The proposed Inter-class Feature Compensator is novel and improves the utilization of the distillation budget by leveraging inter-class features.
- To assess the size of the final distilled dataset, the authors introduce a compression ratio, in addition to the commonly used IPC metric, providing a fairer comparison with methods that utilize soft labels.
- The paper is well-written and well-structured.
- It offers extensive validation experiments across multiple datasets and architectures, reinforcing the reliability of the proposed approach.
summary: The authors highlight that the “one label per instance” paradigm reduces intra-class diversity and amplifies inter-class differences within the distilled dataset. These limitations hinder the efficient use of the distillation budget and lead to poorly defined decision boundaries. To overcome these challenges, the authors propose a novel “one instance for all classes” approach, incorporating a Universal Feature Compensator (UFC). This method enhances class overlaps and ambiguities, improving representation quality and achieving state-of-the-art performance across a variety of datasets and architectures.
weaknesses: - [1] explores inter-class and inter-class relations in dataset distillation, introducing a class centralization constraint to tighten within-class clustering and improve class discrimination. While both methods report significant performance improvements, the conclusions appear to contrast with those of this paper. Could the authors elaborate on this discrepancy?
- [2] also employs multiple backbone architectures for dataset distillation. It would be helpful if the authors discuss how their approach aligns with or differs from this.
- Since the optimizable parameter is not the entire distillation budget but only the M compensators, will this speed up the distillation or synthesis phase?
- How did the authors accurately calculate the compression ratio in Table 2? Additionally, how did the author estimate the size of soft labels?
- Does the compensator consume the IPC budget in the experiments? If yes, according to the design of compensator, the IPC values should not be integers.
- Including comparisons with recent state-of-the-art dataset distillation methods extended from SRe2L [3, 4] would provide a more comprehensive evaluation.
- There are some typos; for example, in line 401, “Our INFER also employ the soft labels” should be corrected to “Our INFER also employs the soft labels.”
[1] Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation. CVPR 2024.
[2] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching. CVPR 2024
[3] Dataset Distillation in Large Data Era
[4] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm. CVPR 2024

Meta-review: This work proposes a new framework for dataset distillation that can better utilise the distillation budget and consider inter-class feature distributions. The idea is to move from the traditional class-specific data-label framework and develop a one-instance-for-all-class approach. The key components consist of the universal feature compensator and the enhancement of synthetic data with inter-class augmentation. Experimental study is conducted on benchmark datasets to verify the efficacy of the proposed method. 

Reviewers comment that the proposed method can significantly reduce the storage, the paper is well written, the work is technically sound with thorough evaluations, and the proposed inter-class feature compensator is novel, and so on. Meanwhile, the reviewers raise concerns related to the comparison with more methods, the evaluation of computational cost, the fairness of experimental study, the privacy issue related to the use of real instances, the effectiveness under high compression ratios, the cross-architecture evaluation, the generality of the proposed idea, and the clarification of details. The authors provide a rebuttal of high quality. The responses to the comments of three reviewers address most or all of the concerns and lead to the increase of score. The authors’ responses during the multiple rounds of discussion with Reviewer RT9a are effective. During the further discussion between reviewers, AC highlights two concerns for them to check, including 1) the fairness of the experimental comparison (raised primarily by Reviewer RT9a), and 2) the effectiveness and consistency of the proposed idea under high compression ratios (raised primarily by Reviewer hymx). Reviewer aHjp replies that the two concerns have been well addressed during the rebuttal period. The final ratings are 3 (Reviewer RT9a), 5 (Reviewer hymx), 6 (Reviewer enyK) and 8 (Reviewer aHjp) .   

AC has carefully reviewed the submission, along with the reviews, the rebuttals, the discussions, and the message provided by the authors. AC concurs with the reviewers regarding the novelty and efficiency of the proposed method and are satisfied with the authors' rebuttal in addressing the major concerns. Taking all the factors into account, AC recommends to accept this work and suggests the authors to further strengthen this work by properly incorporating the responses in the rebuttal.

Decision: Accept (Poster)