Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator
Authors: Xin Zhang
Jiawei Du
Ping Liu
Joey Tianyi Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset distillation
inter-class feature compensator (INFER)
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: This paper presents Break Class Barriers (BCB), using the Inter-class Feature Compensator (INFER) to overcome class-specific limitations, greatly improving the efficiency and effectiveness of dataset distillation.

Abstract: Dataset distillation has emerged as a technique aiming to condense informative features from large, natural datasets into a compact and synthetic form. While recent advancements have refined this technique, its performance is bottlenecked by the prevailing class-specific synthesis paradigm. Under this paradigm, synthetic data is optimized exclusively for a pre-assigned one-hot label, creating an implicit class barrier in feature condensation. This leads to inefficient utilization of the distillation budget and oversight of inter-class feature distributions, which ultimately limits the effectiveness and efficiency, as demonstrated in our analysis.
To overcome these constraints, this paper presents the Inter-class Feature Compensator (INFER), an innovative distillation approach that transcends the class-specific data-label framework widely utilized in current dataset distillation methods. Specifically, INFER leverages a Universal Feature Compensator (UFC) to enhance feature integration across classes, enabling the generation of multiple additional synthetic instances from a single UFC input. This significantly improves the efficiency of the distillation budget.
Moreover, INFER enriches inter-class interactions during the distillation, thereby enhancing the effectiveness and generalizability of the distilled data. By allowing for the linear interpolation of labels similar to those in the original dataset, INFER meticulously optimizes the synthetic data and dramatically reduces the size of soft labels in the synthetic dataset to almost zero, establishing a new benchmark for efficiency and effectiveness in dataset distillation. In practice, INFER demonstrates state-of-the-art performance across benchmark datasets. For instance, in the $\texttt{ipc} = 50$ setting on ImageNet-1k with the same compression level, it outperforms SRe2L by 34.5\% using ResNet18. Codes are available at https://github.com/zhangxin-xd/UFC.

Published as a conference paper at ICLR 2025
BREAKING CLASS BARRIERS: EFFICIENT DATASET DISTILLATION VIA INTER-CLASS FEATURE COMPENSATOR
Xin Zhangl. Jiawei Du Ping Liu Joey Tianyi Zhou
Centre for Frontier AI Research, Agency Science, Technology and Research, Singapore
instiiute of High Performance Computing. Agency for Science, Technology and Research, Singapore
University of Nevada, Reno
(zhangx7, dujw, Joey.2hozJ@cfar: astar. edu. sg pingigunr.edi
ABSTRACT
Dataset distillation has emerged as a technique aiming to condense informative features from large, natural datasets into compact and synthetic form. While recent advancements have refined this technique, its performance is bottlenecked by the prevailing class- specifc synthesis paradigm. Under this paradigm, syn- thetic data is optimized exclusively for pre-assigned one-hot label, creating an implicit class barrier in feature condensation. This leads to inefficient utilization of the distillation budget and oversight of inter-class feature distributions, which ultimately limits the effectiveness and efficiency, as demonstrated our analysis. To overcome these constraints, this paper presents the Inter-class Feature Com- pensator (INFER), an innovative distillation approach that transcends the class- specific data-label framework widely utilized in current dataset distillation meth- ods. Specifically, INFER leverages Universal Feature Compensator (UFC) to enhance feature integration across classes, enabling the generation of multiple ad- ditional synthetic instances from single UFC input. This significantly improves the efficiency of the distillation budget. Moreover, INFER enriches inter-class interactions during the distillation, thereby enhancing the effectiveness and gen- eralizability of the distilled data. By allowing for the linear interpolation of labels similar to those in the original dataset, INFER meticulously optimizes the syn- thetic data and dramatically reduces the size of soft labels the synthetic dataset to almost zero, establishing new benchmark for efficiency and effectiveness in dataset distillation. In practice, INFER demonstrates state-of- of-the-art performance across benchmark datasets. For instance, in the ipc 50 setting ImageNet- lk with the same compression level, it outperforms SRe21. by 34.5% using ResNet 18. Codes available at https: / thub. com z zhangx n- xd/UFCCCC
INTRODUCTION
The remarkable success of Deep Neural Networks (DNNs) (Yang et al., 2024; Cai et al., 2024; Zheng al., 2024; Dosovitskiy et al., 2021) in recent years can largely be attributed their ability to extract complex and representative features from vast real-world data (Gao et al., 2020; Benenson al.. 2019). However, the extensive data requirements for training DNNs pose significant chal- lenges. These challenges include not only the time-consuming training process (Liu et al., 2021; Touvron al., 2021; Tolstikhin et al., 2021), but also the substantial costs associated with data storage and computational resources (Kaplan al., 2020; Hoffmann al., 2022).
In response to the rapid growth of computational and storage demands in training DNNs, dataset distillation (DD) (Sachdeva & McAuley, 2023; Lei & Tao, 2023; Yu et al., 2023; Liu & Du, 2025) has emerged as an effective solution. Dataset distillation condenses essential features from extensive datasets into compact, synthetic form, allowing models to maintain comparable performance levels with fewer resources (Wang et al., 2018). Recent advancements in dataset
represents the corresponding author.

---

Published as a conference paper at ICLR 2025
distillation encompass techniques such gradi- ent matching (Zhao et al., 2021: Zhao & Bilen, 2021: Lee et al., 2022b; Shin et al., 2023), tra-
70
G
G-VESM
INFER-Dyn (ours)
INFER (ours)
Top-1 Acc, (9)
jectory matching (Cazenavette et al., 2022; Cui
NTT
et al., 2023; Du al., 2023a;b), data factoriza- tion (Liu et al., 2022; Kim et al., 2022; Wei et al., 2023; Shin et al., 2024), and kernel ridge re- gression (Nguyen et al., 2020; 2021; Loo et al., 2022). These approaches have significantly en- hanced dataset distillation by compressing dense knowledge into single data instances. Despite the diversity of these methods, most of them ad- here to a uniform paradigm: each synthetic data instance is class-specific and optimized exclu- sively for a pre-assigned one-hot label.
40
30
20
Tiny ImageNett ImageNet-1K
Random
10
220
60 Compression Ratio
20
10
Performance vs. compression ra- tio of SOTA dataset distillation methods (G- VBSM (Shao et al., 2024), SRe2L (Yin al., 2024), MTT (Cazenavette al., 2022)) on three benchmarks. Performance is measured as the Top-I accuracy of ResNet-18 (ConvNet128 for MTT) on the respective validation sets, trained from scratch using synthetic datasets. The compression ratio, including the additional soft labels, the proportion of the distilled dataset size to the original dataset size. The star indicates optimal performance.
Figure 1:
While this "one label per instance" paradigm aligns with the traditional data-label pair struc- ture of original datasets, it presupposes that the most effective encapsulation of a dataset's ss knowledge can be achieved through individual instances representing discrete class identities. When the amount of synthetic data is limited, this class-specific paradigm benefits distillation by encouraging synthetic data to condense the most dis- tinctive features of each class. However, as more synthetic instances are assigned to the same class, they tend capture significant but similar features rather than diversifying to include unique, rarer features. This phenomenon, known as "feature duplication" (Jiang et al., 2022; Kim et al., 2022; Cazenavette et al., 2022), leads to Inefficient Utilization of the Distillation Budget, thereby limiting the creation of more diverse and comprehensive synthetic representation.
Another critical downside of the class-specific synthesis paradigm is the "Oversight of Inter-Class Features". By focusing on distinctive class-specific characteristics under the "one label per instance" approach, implicit "Class Barriers" are created between classes. These class barriers prevent the synthetic data instances from capturing inter-class features that bridge different classes in the orig- inal dataset. This oversight, inhibits the formation of thin and clear decision boundaries among classes, which are essential for models to generalize well across complex scenarios. We demon- strate the visualization of frormed decision boundaries in Figure 2.
Recognizing the aforementioned limitations, introduce novel paradigm for dataset distilla- tion, termed the Inter-class Feature compEnsatoR (INFER). Unlike traditional methods that follow the "one instance for one class" paradigm and generate separate synthetic instances for each class, INFER pioneers a "one instance for ALL classes" paradigm by introducing a Universal Feature Compensator (UFC). The UFC, designed to reflect the general representativeness across all classes of the original dataset, depreciates the importance of pre-assigned labels. This feature enables UFC to compensate for inter-class features while INFER randomly incorporates a few natural data in- stances to enhance intra-class features. Notably, INFER integrates one UFC with multiple natural data instances from different classes through a simple additive process without auxiliary generator networks (Liu et al., 2022), allowing the generation of multiple synthetic instances from a single input. This "one instance for ALL classes" paradigm significantly enhances the efficiency of the distillation budget.
Furthermore, we have meticulously designed the optimization of UFCs to encompass inter-class fea- tures. This optimization makes synthetic instances generated from UFCs compatible with MixUp data augmentation, which promotes inter-class interactions and aids in forming thin, clear decision boundaries among classes. Prior works applying MixUp to synthetic data (Yin et al.. 2024) involve dynamic generating and storing an extensive amount of soft labels, which can increase storage re- quirements up to 30-fold. INFER, however, dramatically eliminates the need for such extensive soft label storage by elegantly adopting the linear interpolation of labels used with natural datasets, decreasing the storage requirement by 99.3%. This enhancement not only preserves the distilla- tion budget but also streamlines the entire training process, underscoring INFER's efficiency and effectiveness. Notably, INFER achieves 53.3% accuracy on ImageNet- 1k with a ResNet-50 model, training solely on the synthetic dataset, which is only 4.04% the size of the original ImageNet- 1k.
2

---

Published as a conference paper at ICLR 2025
Univeraal Festure
veusiization decision boundaries daasees)
en image medol
Class
St
Barriers
Breeking Class Banilers
Fefredaay,,
Existing Approach
Existing Approach
One instance for one class
One instance for All classes
Figure 2: Left: Overview of dataset distillation paradigms. The first illustrates the traditional "one instance for one class" approach, where each instance optimized exclusively for its pre-assigned label, creating implicit class barriers. The second illustrates our INFER method, designed for "one instance for ALL classes" distillation. Right: t-SNE visualization of the decision boundaries be- tween the traditional approaches (i.e., SRe2L (Yin et al., 2024)) and our INFER approach. We randomly select seven classes from CIFAR- 100 dataset for the visualization. INFER forms thin and clear decision boundaries among classes, in contrast to the chaotic decision boundaries of the traditional approach.
This performance, which does not require dynamically generated soft labels, outperforms existing approaches in both accuracy and compression ratio.
Our contribution can be summarized follows:
We rethink the prevailing "one label per instance" paradigm that exclusively optimizes each synthetic data instance for a specific class. Through empirically analysis, we identify and address its two main limitations: inefficient utilization of the distillation budget and oversight of inter-class features.
To overcome these issues, we introduce new paradigm INFER, for "one instance for all classes" dataset distillation. Our INFER incorporates a novel Universal Feature Compen- sator (UFC) to efficiently condense and integrate features across multiple classes. Exten- sive experiments across CIFAR, tiny-ImageNet and ImageNet-II Ik datasets demonstrate the state-of- faate-oftth--eee performance of INFER.
2 PRELIMINARIES AND RELATED WORKS
The precursor to dataset distillation in condensing of datasets coresct selection (Bachem et al., 2017: Chen et al., 2010; Har-Peled & Kushal, 2005; Sener & Savarese, 2018; Xin al., 2024). This method involves selecting a coreset of the original dataset that ideally contains the entire repre- sentativeness of the population. However, this approach encounters significant performance drop when compression ratio is small. A plausible explanation for this could be the low density of rep- resentativeness within natural data instances. Therefore, dataset distillation seeks synthesize data instances with densely packed features. We begin with brief formulation of dataset distillation.
Problem Formulation. Assume
given natural and large dataset T {(x,, y.) y.))
(=) where
each element x, E dd is drawn i.idd. from natural distribution D, and the class label y; E y (0. 1 1,..., C 1} with C' representing the number of classes. Dataset distillation aims to synthesize small dataset S {(s; ! where s, Rd and y: V, to serve as approximate solution to the following optimization problem:
S arg min E [E (fog: x. p.y)
SCR'xy (m.a)~
where Al represents the converged weights trained with S, and is the loss function. The class label y; is typically pre-assigned one-hot label (Zhao & Bilen, 2023; Zhao al., 2021; Zhao & Bilen, 2021; Jiang et al., 2022; Kim et al., 2022; Cazenavette et al., 2022) to encourage s; to exhibit more distinct, class-specific features.
Compression Ratio (CR)= synthetic dataset size / original dataset size (Cui et al., 2022). A lower ratio indicates condensed dataset.
3

---

Published as a conference paper at ICLR 2025
Pioneering approach, Wang et al. (Wang al., 2018) is the first to propose DD, method that optimizes S directly after substituting D with T in Equation approach often leads to suboptimal performance, prompting the development of muddient-matching methods (Zhao et al., 2021; Zhao & Bilen, 2021; Lee et al., 2022b; Shin et al., 2023). These methods improve supervision by aligning the models' gradients. The success of gradient-matching has in- spired further research into matching the trajectory of gradients (Cazenavette et al., 2022; Cui et al., 2023; Du et al., 2023a:b), yielding even better performance. Despite these advancements, most of current dataset distillation methods still primarily focus on generating class-specific synthetic in- stances. This ongoing adherence to the class-specific paradigm not only constrains the efficiency of the distillation budget but also results in the neglect of critical inter-class features. These limita- tions drive our investigation into new paradigm, aimed at developing a more efficient and effective dataset distillation solution.
Due to the limited guidance, this
Distillation Budget Consistency. As our INFER depreciates the class-specific paradigm, it be- comes necessary to establish clear criteria for maintaining consistency in the distillation budget. Traditional methods adopt Images Per Class (IPC) as described by (Wang et al., 2018; Zhao et al., 2021; Cazenavette et al., 2022), such that |8] ipc C. This approach provides uniform crite- rion across various datasets. Therefore, we continue to employ IPC as the criterion to measure the distillation budget.
However, IPC does not account for the auxiliary generator (Liu et al., 2022; Lee et al., 2022a) additional soft labels (Yin et al., 2024) that are used enhance the performance of syn- thetic dataset, resulting in an asymmetric advantage compared to methods that solely utilize the data-label pair. Therefore, we also employ the compression ratio, as described by (Cui et al., 2022), to measure the distillation budget. The total bit count of any auxiliary modules and soft labels will be considered part of the synthetic dataset. To compute the compression ra- tio, we divide the total bit count of the synthetic dataset, by that of the original dataset, i.e., CR synthetie dataset
dataset size.
3 METHODOLOGY
In this section, we introduce our novel distillation paradigm. INFER. We begin by detailing the limitations associated with the class-specific distillation paradigm, highlighting inefficiencies and oversight of inter-class feature distributions. Following this, we describe the Universal Feature Compensator (UFC), the cornerstone of our methodology, designed to integrate inter-class features. Lastly, we discuss approach to augmenting synthetic datasets, which aims to facilitate the for- mation thin and clear decision boundaries among classes.
3.1 LIMITATIONS IN
DISTILLATION
Wang et al. (Wang et al., 2018) established the general approach to solve S as outlined in Equation I: Each synthetic data instance is assigned a one-hot label yi and optimized to capture intra-class features by minimizing E (. (Jes x, y). Unlike Equation 1, D replaced by T. given that D is inaccessible and T ~ D. Initially, this class-specific design achieved progress in the carly stages. However, dataset distillation research has evolved, two major limitations become prominent, compelling rethinking of this design.
Inefficient Utilization of Distillation Budget. Recent advancements in dataset distillation (Yin et al., 2024: Loo et al., 2022; Cui et al., 2023) have enabled individual synthetic data instances to capture more features specific to a class, particularly notable in highly compressed scenarios where ipc (Cui et al., 2023: Loo et al., 2022). However, as ipc increases, additional synthetic data instances tend capture distinctive yet duplicated intra-class features, leading redundancy within the synthetic dataset. This redundancy explains the marginal performance gains observed in less compressed scenarios (ipc 50). SeqMatch (Du et al., 2023b) and DATM (Guo al., 2023) addressed this redundancy by dividing the synthetic data into several subsets optimized diversely. We validate this hypothesis through experiments shown in Figure 4 (a). Ideally, newly optimized synthetic data instances should capture rare and diversifying features that complement the distinctive class-specific features.
4

---

Published as a conference paper at ICLR 2025
S = (P".21
To be stored
Oversight of Inter-class Features. The preva- lent focus on optimizing synthetic instances for the most distinctive intra-class features often leads to the oversight of inter-class features. This oversight significantly limits the synthetic data's ability to represent the feature distribu- tions that span across different classes, which is crucial for complex classification tasks. Conse- quently, the potential for these synthetic datasets support the training of models that can gen- eralize across varied scenarios may be signif- icantly hampered. As depicted in Figure 2, this issue is evidenced by the chaotic decision boundaries formed by the "one label per in- stance" synthetic dataset, which neglects the inter-class features necessary for forming thin and clear decision boundaries.
Ui
$11
Sc1
u
S1M
Scm
[$p
.5c }
8 ((5, 1 = u
Integrated while training
Illustration of the integration pro- cess between Universal Feature Compensators (UFCs) and natural data instances as described in Equation 2. The integration is performed through a simple addition process. Consequently, only the sets S (P* 2²) need to be stored as the synthetic dataset. The synthetic dataset Sk is generated on-the-fly during training.
Figure 3:
These limitations drive our investigation into new paradigm, aimed at developing a more effi cient and effective dataset distillation solution.
3.2 UNIVERSAL FEATURE COMPENSATOR: BREAKING CLASS BARRIERS
The Universal Feature Compensators (UFCs), denoted as U = [u)] forms the core of our novel INFER paradigm, designed specifically to address the inefficiencies and oversight inherent in the class-specific distillation approach.
The primary objective of designing the UFC is to enable the generation of multiple synthetic instances from a single compensator. To achieve this, our INFER divides the base synthetic dataset into K subsets, such that S S U U .... US Each base subset S consists of pair (P* ),,, where 4 represents the set of UFCs, and pk T contains natural instances to be integrated with UFCs and |P4| = C. For actual model training on the synthetic dataset, we first integrates UFCs with natural data instances follows:
Design and Functionality.
Sk ((5,, 9i) 8; 0; + Ujj for each æ; pk and each uy E 21"
where Sh represents the UFCs integrated synthetic dataset. This process is illustrated in Figure 3. In practice, multiple architectures participate in UFCs' generation. Assuming the number of archi- tectures is M, Le., 1 = M, each S can be multiplied approximately M times by INFER We repeat the intergration process described above K times, once for each subset of S. This structured approach allows INFER to utilize the distillation budget approximately M times more efficiently compared the class-specific paradigm.
Optimization. Under the INFER paradigm, each Pk contains exactly one instance for each class, randomly selected from the original dataset T. Consequently, p effectively captures the intra-class features with minimal duplication. In contrast, the UFCs are designed to optimize the capture of inter-class features. Therefore, the elements in pk uniformly cover each class, allowing the element Uk to focus on capturing inter-class features by solving the follow minimization problem:
arg min ERd
Σ (E where
[ + U.j yi) + +LBN (fers 21 + +u;)
(3)
LBN (for H; + +u,) E, (1m((5)) - (4
E, - of
(4)
Uj for each x; Pk3 which is the generated synthetic dataset by integrating uj with the corresponding Pt_ CAN is the BN loss inspired by SRe2L (Yin al., 2024) regularize the values of generated 8; fall within the same normalization distribution
we define 8y. (6,, 9.)
5

---

Published as a conference paper at ICLR 2025
those in T. Intuitively, u serves the feature compensator for the natural instances in S, carrying universal inter-class features that are beneficial for classification.
Algorithm Distillation on synthetic dataset via Inter-class Feature Compensator (INFER)
Require: Target dataset T; Number of subsets K; Number of classes C; M networks with different architectures: {f ,,....,,, fMM
1: Initialize S -1
2: for k K do
3: Initialize subset pk = (. the UFCs set Uk = (). and the static labels set )kk 0
4: Randomly select instances, one for each class, to form Pk such that:
5: P = {Zz, y:) (æ,, y:) T and each yı is unique in P&}
6: Initialize 4tth with zeros, where each u, has the same dimensions as æ:
7:
ut = {u, Uj diim(r)), for j M.
8:
for each u, in 24 do
9:
> Construct integrated synthetic instance 8, Let S = { 51) 8; = x + uj for each E pky
10:
11:
repeat
12:
Optimize uj to minimize loss defined in Equation 3
13:
until Converge
14:
Generate static soft labels y, by Equation 6, y* = y* U ( end for
15:
16:
S [Pb Uh }*
17:
= SU (S*}
end for Ensure: Synthetic dataset S
18:
3.3 ENHANCING SYNTHETIC DATA WITH INTER-CLAS CLASS AUGMENTATION
Although UFCs encouraged encapsulate more inter-class features, their limited size, especially compared to the size of the original dataset T, restricts the breadth of features crucial for forming the decision boundaries of neural networks. Therefore, we also leverage MixUP (Zhang al., 2018) as technique to enhance inter-class augmentation.
Data augmentation (Shorten & Khoshgoftaar, 2019) has been well-developed and proven effective in training neural networks with natural datasets. However, the advancements in data augmentation do not generalize well to synthetic datasets generated through dataset distillation. DSA (Zhao Bilen, 2021) designed an adapted augmentation method specialized for dataset distillation, but it is not as effective as standard data augmentation methods. SRe2L (Yin et al., 2024) applies MixUp (Zhang et al., 2018) to the synthetic dataset, achieving superior performance across many datasets. Un- fortunately, the cost of applying MixUp in SRe2L is expensive, due to the massive volume of soft labels. The soft labels are dynamically generated for each augmented instance in each validation epoch, resulting enormous storage requirements or extra training efforts. For example, the synthetic dataset generated by SRe21. for ImageNet-Il 1 requires 0.7GB to store synthetic images, but requires additional 25. 25.9GB to store the soft labels.
Motivated by this, we aim apply MixUp to INFER model in the same way it is used in natural datasets, which we refer to as "static" soft labels. The linear interpolation of labels can be represented mathematically
for [b5; (1 X)8,} Nle (5.) + (8))) V8, E
(5)
where fe) represents the logits output, À ~ Beta(3, 9), and O. To achieve this, we propose three improvements in INFER: (a) We use pk subset of natural datasets, for integration with UFCs because natural instances inherently follow the linear interpolation of labels. (b) We make pk to span across all the classes, rather than limiting it to instances within the same class. As such, the optimized UFCs, 24, which are integrated with p for optimization, also embody the characteristic of linear label interpolation. (c) We employ M neural networks with various architectures (M ((*)))

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with SOTAs CIFAR-10/100 and Image-m Except for SRe2L (Yin et al., 2024), G-VBSM (Shao et al., 2024), and our INFER, all other methods use ConvNet128 for distillation. The distilled synthetic datasets are then evaluated on ConvNet128 and ResNet18. "IN- FER+Dyn" denotes the application of INFER using dynamically generated soft labels, as described in SRe2L (Yin al., 2024). The best performers in each setting are highlighted in red.
CIFAR-10
CIFAR-100
Tiny-ImageNet
1pc
10
50
10
50
100
10
50
Random
31.0
50.6
14.6
33.4
42.8
5.0
15.0
DC (Zhao al., 2021)
44.9
53.9
25.2
DSA (Zhao & Bilen, 2021)
52.1
60.6
32.3
42.8
20.5
1:0.5
2003
2:0.4
KIP (Nguyen et al., 2021)
62.7
68.6
28.3
20.3
4:0
=:0:1
ConvNNt122
RFAD (Loo et al., 2022)
66.3
71.1
33.0
20.5
2:0
2:03
MTT (Cazenavette et al., 2022)
65.4
71.6
39.7
47.7
49.2
23.2
28.0
10.7
1-.2
200.4
1:0.2
30.4
20022
40.3
SeqMatch (Du et al., 2023b)
66.2
74.4
41.9
51.2
23.8
10.6
20
10.5
10.3
G-VBSM (Shao al., 2024)
46.5
54.3
38.7
45.7
INFER
34.0
57.0
41.0
53.8
57.0
22.8
32.3
+0.22
INFER+Dyn
30.1
52.4
37.2
50.7
53.4
24.9
33.9
20.5
1.0-7
220-3
10.3
1..3
1000
Random
29.6
36.7
15.8
32.0
47.5
12.1
17.7
SRe2L (Yin et al., 2024)
27.2
47.5
31.6
49.5
41.1
ResNet18
+0.66
+0.8
G-VBSM (Shao al., 2024)
53.5
59.2
59.5
65.0
47.6
#0.0
+0.44
+00.4
#0.66
+0.3
INFER
32.0
60.4
45.2
62.8
66.3
32.0
52.9
20.5
1:1.6
2:0 05
2:03
10.1
20.1
20.1
INFER+Dyn
30.7
60.7
53.4
68.9 9
73.3
41.0
54.6
20.3
200.0
2.006
200.1
200.2
10.4
20.4
relabel the generated synthetic data instance 8; through averaging, i.e.
ú. -. M Σ for (5:).
(6)
By doing so, INFER does not require the dynamic soft labels for any combinations of [A + (1 A) but only the static soft labels of 8,, 8,, which reduces the size soft labels by up to 99.9%.
Additionally, training synthetic datasets can follow the same paradigm as training natural
datasets. More details of synthesizing and training S can be found Algorithm and Algorithm 2.
EXPERIMENTS
To evaluate the effectiveness of our proposed INFER distillation paradigm, conducted series of experiments across multiple benchmark datasets and compared our results with several state- of-the-art approaches. In this section, we provide details on the experimental setup, the datasets used, and the results obtained. We summarize our main results in Table I and Table 2. Following this, we perform ablation studies to assess the impact of individual components of our method. All experiments were conducted using two Nvidia 3090 GPUs and one Tesla A-100 GPU.
4.1 EXPERIMENTAL SETUP
Baselines and Datasets. We conduct the comparison with several representative distillation meth- ods, including Random, DC (Zhao et al.. 2021), DSA (Zhao & Bilen, 2021). KIP (Nguyen et al., 2021), RFAD (Loo et al., 2022), MTT (Cazenavette et al., 2022), SeqMatch (Du et al., 2023b), VBSM (Shao et al., 2024) and SRe2L (Yin et al., 2024). This evaluation is performed on four popular classification benchmarks, including LLFAR-10/11 (Krizhevsky et al., 2009), Tiny-Imagell (Le & Yang. 2015), and ImageNet- 1k (Deng al., 2009).
Implementation Details. Our INFER uses 4, meaning employs four different architectures for optimizing UFCs: ResNet18 (He al., 2016), MobileNetv2 (Sandler al., 2018), Efficient-

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with SOTAs on ImageNet-Ik. SRe2L (Yin et al., 2024), G-VBSM (Shao al., 2024), and our INFER use 8 for distillation. The distilled synthetic datasets then evaluated on ResNet18, 50, and 101. "INFER+Dyn" denotes the application of INFER using dynamically generated soft labels, as described in SRe2L (Yin al., 2024). We also evaluate SRe2L and G-VBSM under the same compression ratio using our static labeling strategy, denoted as {}+ The best performers in each setting are highlighted in red.
Compression Ratio
ResNet18
ResNet-50
ResNet- 101
ipc
10
50
10
50
10
50
10
50
Random
0.78%
3.90%
10.5
31.4
9.3
31.5
10.0
33.1
#0.4
#00.8
+0.3
40.2
10.4
#0.1
SRe2L" (Yin et al., 2024)
0.81% %
4.04%
9.8
17.3
8.7
17.2
8.8
15.8
20.2
G-VBSM" (Shao al., 2024)
0.81%
4.04%
11.9
32.9
14.5
38.1
13.9
38.9
1:0.2
2:0:1
2:0.2
30.1
10.4
INFER
0.81%
4.04%
28.7
51.8
26.9
53.3 3
26.5
52.2
10.22
20.2
10.8
20
20.1
20.3
SRe2L. (Yin et al., 2024)
4.53%
22.67%
21.3
46.8
28,4
55.6
30.9
60.8
1:0.6
2.0.2
10011
10.3
10.1 1
20.5
G-VBSM (Shao et al., 2024)
4.53%
22.67%
31.4
51.8
35.4
58.7
38.2
61.0
1.0.5
2.0.4
1.0.0
10.3
10.4
20.4
DyF
4.53%
22.67%
36 3
55.6
38.3 3
63 4
38.9
60.7
+0.2
+0.1
NetBO (Tan & Le, 2019), and ShuffleNetv2 (Ma et al., 2018). When distilling ImageNet-IK, only the first three architectures (M = 3) are involved. For reproducibility, the hyperparameter settings Tiny-ImageNet, and ImageNet- Ik, are provided in Appendix A.3. These settings generally follow SRe2L (Yin et al., 2024), with the sole modification being a proportional reduction in the validation epoch number for the dynamic version to ensure fair comparison. All other critical hyperparameters remain unchanged.
for the experimental datasets
Consistant Distillation Budget. As we stated in Section 2, the baselines SRe2L. (Yin et al., 2024) and G-VBSM (Shao et al., 2024) use the dynamic generated soft labels from teacher model in each validation epoch for enhanced performance. However, these additional dynamic soft labels are not considered in the "Images Per Class (IPC)" distillation budget. Therefore, we also adopt the compression ratio (CR = synthetic dataset size/ original dataset size (Cui et al., 2022)) for consistant distillation budget. the size of the soft labels is proportional to the number of validation epochs, we only report the CR in ImageNet- 1 dataset, as shown in Table 2.
Our INFER also employs the soft labels as shown in Equation 6. However, INFER only stores one soft label per instance, which equals one epoch dynamic soft labels. We term it as static soft labels in contrast to the dynamic soft labels generated across every validation epochs. Therefore, INFER reduces the size of soft labels by up 99.3% in ImageNet- 1k dataset (from 300 epoch to 1 epoch). We also implement the dynamic soft labels under INFER, denoted as "INFER+Dyn" As increasing improve the performance, but incurs larger size of soft labels. Another
the validation epoch point to note is, to ensure fair comparison, we reduce the validation epoch to 17. as INFER generates M-fold synthetic instances by integrating UFC (M is for ImageNet-1k). Lastly, we also average the number of UFCs into pp and adjust K = [ipc X
M for fair comparison.
4.2 MAIN RESULTS
Performance results on CIFAR-10/100 and Tiny-ImageNet are summarized in Table 1. While most previous methods rely on ConvNet129 28 due to resource constraints, SRe2L (Yin et al., 2024), G- VBSM (Shao al., 2024), and our INFER method use ResNet18 for synthesis. INFER significantly outperforms competitors. With ipc=50, ResNet18 trained on the distilled CIFAR- 100 achieves 68.9% and 62.8% accuracy, surpassing SRe2L by 19.4% and 13.3%, respectively. INFER also outperforms G-VBSM by 5.3% on Tiny-ImageNee. Despite reducing 99.3% of dynamic soft labels, INFER still outperforms methods using dynamic labels. Static labels suffice for simpler networks and datasets, but dynamic labels provide stronger supervision for complex architectures.
Table provides a detailed performance comparison between SRe2L (Yin et al., 2024), the pi- oneering method for scaling dataset distillation to large datasets like ImageNet-lk, its extension G-VBSM (Shao et al., 2024), and our proposed INFER approach. These methods use ResNet18 for dataset distillation, and their performance is evaluated three architectures: ResNet18, ResNet50, and 101. The table highlights the advantages our INFER method, which consistently outperforms SRe2I. across all evaluation settings. Notably, when distilling datasets with i pc==0,,
8

---

Published as a conference paper at ICLR 2025
[a] Duplication vs ipc
etudy
Feature Duplication
0.85
75.0
</o UPC
+26.3
116.8
(Lower @eelllay
0.75
Top-1 Acc. (%)
60.0
+13.8
45.0
0.65
0.00
gha INFE/R SRe
0.55
0.50
00
1pc
00
0.0
INFER
INFER+Dyn
INFER
INFER+Dyn
(b)
The change in feature duplication with the increase of ipc. To measure the level of feature duplication,
Figure 4: Left:
employ the averaged cosine similarities between each pair of synthetic data instances within the same class. Therefore, a greater value represents higher feature duplication, as SRe2L (Yin et al., 2024) shows. contrast, our INFER obtains lower feature duplication, which is closer to the level observed in natural datasets. Right: The ablation study of UFC. The first two groups are under the pp 10 setting. while the other two are under ipc 50. The purple annotations indicate the performance gains contributed by our UFC.
Table Ensemble of architectures for UFC generation. "R", "M", "E", and "S" represent ResNet18 (He et al., 2016), MobileNetv2 (Sandler et al., 2018), EfficientNeiBO (Tan & Le. 2019), and ShuffleetV2 (Ma al., 2018), respectively. V indicates the network architectures participating in UFC generation. 1 denotes the performance gain contributed by the current ensembles compared with the baseline (only ResNet18). 18). These experiments are conducted on CIFAR 100 dataset.
ipc 10
ipc = 50
R
M E
INFER
f
DyF
INFER
f
INFER+Dyn
1
40.9
+ 0.0
38.1
0.0
59.7
0.0
65.3
0.0
43.2
+ 2.3
46.2
+ 8.1
61.2
+ 1.5
67.3
+.0
2012
1:0.04
44.8
+.9
50.6
12.5
61.7
+ 2.0
68.3
+30
10.
2.0.1
45.2
+ 4.3
53.4
16.3
62.8
+ +.1
68.9
+366
04
10.6
2004
20.1
INFER achieves substantial 5.0% performance improvement on ResNet18 while maintaining an extremely compact synthetic dataset-ont only 4.04% of the original size. When matched to the same compression ratio, the performance gap further widens to 34.5%. This result showcases the superior efficiency and effectiveness of INFER large-scale dataset distillation, particularly in compressing datasets while maintaining performance.
4.3 ABLATION STUDY
Compensator Generation. We examine the effectiveness of the proposed UFC. As shown in Fig- ure 4 (b), regardless of the ipc setting and the labeling strategy, our UFC significantly enhances the quality of the distilled dataset. For example, the Top-1 classification accuracy the model trained with static labels is improved by 25.6% with ipc 10. We also study the influence of network architectures participating compensator generation. According to the results provided Table 3, performance consistently improves with the addition of more ensembled networks. This trend is particularly pronounced with dynamic labeling. For instance, the ensemble of four architectures enhances performance by 4.3% with static labeling and by 16.3% with dynamic labeling. To align with the multi-model-aided compensator generation, we also employ multiple networks for soft la- bel generation. Table Appendix A.5 presents the model performance trained with soft labels generated by various architecture ensembles.
Cross-Architecture Generalization. Table 4 presents the performance evaluation of synthetic CIFAR-100 dataset across different architectures, trained from scratch. When ipc=10, our IN- FER+Dyn and INFER methods outperform SRe2L. across all architectures. For instance, IN- FER+Dyn achieves accuracy of 52.3% on ResNet50, significantly higher than the 22.4% achieved by SRe2L. For ipc=50, the performance advantage of INFER+Dyn and INFER remains evident. INFER+Dyn reaches accuracy of 70.0% ResNet50, far surpassing SRe2L. Our INFER shows a well generalization abilities across different architectures. The cross-architecture results on Ima- geNet, shown in Table further confirm the effectiveness of our methods. Specifically, INFER+Dyn and INFER demonstrate superior performance compared to SRe2L and G-VBSM across various ResNet architectures ImageNet-1k.

---

Published as a conference paper at ICLR 2025
Table 4: Eross-architecture performance of distilled dataset. Here, the synthetic CIFAR- 100 datasets are evaluated by training ResNet-50, ResNet- 101 (He et al., 2016), MobileNetV2 (Sandler et al., 2018), EfficientNeBBO (Tan & Le, 2019), and ShuffleNetV2 (Ma et al., 2018) from scratch. The best performers in each setting are highlighted in red.
pc 10
ipc INFER
50
Networks
SRe2L
INFER
INFER+Dyn
SRc2L
1yF
ResNet50
22.4
52.8
62.4
MobileNetV2
16.1
46.7
43.3
43.2
64.1
67.
10.5
20.1
-20-7
-0.2
20:2
20.2
EfficientNetB0
11.1
32.5
34.9
24.9
55.0
62.7
ShuffleNetV2
11.8
38.7
30.9
27.5
61.5
62.1
3007
20.2
2.0.5
201.
2002
20.2
CrossEntrop/ Loss
Decision Boundary
CrossEntropp Loss
22
Boundary
CrossEntrop Loss
Decision Boundary
~
Centroid of class
Pixel Space
Pixel Space
Pixel Space
Original
SRe2L
INFER (ours)
Figure Visualizations of loss landscapes in pixel space on CIFAR-100 dataset. The optimal decision boundary is supposed have rapid change in cross-entropy loss at the edge, indicating a clear and distinctive decision boundary. Left: distinctive decision boundary trained on the original dataset T. Middle: A less distinctive decision boundary trained the synthetic dataset of outstanding class-specific approach SRe2L. Right: An improved decision boundary trained on the synthetic dataset of INFER.
4.4 MORE DISCUSSIONS WITH SOTA METHOD SRE2L
Feature Duplication Study. We verify our hypothesis that synthetic data instances tend to cap- ture distinctive yet duplicated intra-class features under the traditional class-specifie distillation paradigm. We measure feature duplication by averaging the cosine similarities between each pair of synthetic data instances within the same class. Our experimental results, shown in Figure 4(a), support our hypothesis: the class-specific approach SRe2L. exhibits higher feature duplication as INFER, achieves improved feature uniqueness, more
the number of pc increases. Conversely, closely resembling natural datasets. This reduction in intra-class feature duplication significantly enhances the diversity of the distilled dataset, which, in turn, improves the training performance.
Visualization on Decision Boundaries. To verify our hypothesis regarding the "oversight of inter- class features" in the traditional cpass-specific distillation paradigm, we visualize the decision bound- aries of ResNet-18 models trained with synthetic datasets generated by SRe2L and our INFER, respectively. We randomly select seven classes from the CIFAR-100 dataset and use the t-SNE ap- proach for visualization. As illustrated in Figure 2, INFER forms thin and clear decision boundaries between classes, contrast to the chaotic decision boundaries produced by the traditional approach. Additionally, visualized the 3D loss landscape in pixel spaces of the decision boundaries in Figure 5, which further supports our hypothesis from a different perspective.
For further analysis, refer Appendix A.6, where we provide additional insights into the perfor- mance improvements of INFER.
5 CONCLUSION
In this work, we rethink the current "one class per instance" paradigm in dataset distillation and identified its limitations, including inefficient utilization of the distillation budget and oversight of inter-class features. These issues arise distillation techniques advance, leading to synthetic data that often captures duplicated class-specific features. To address these limitations, we introduce a novel paradigm INFER that employs Universal Feature Compensator (UFC) for "one instance for all classes" distillation. Our experimental results demonstrate that INFER improves the efficiency and effectiveness of dataset distillation, achieving sfate-of-the-art results in several datasets, reduc- ing resource requirements while maintaining high performance. Future work will focus on scaling INFER for extremely large dataset and exploring its application in various real-world scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is supported by Jiawei Du's A*STAR Career Development Fund (CDF) C233312004.
REFERENCES
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint 1733:0/703306766. 2017.
Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR, 2019.
Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation. NewrlPS, 2024.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kemne herding. In UAI, 2010.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. De-bench: Dataset condensation benchmark. arXiv preprint arXiv:2207.09633 2022.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation imagenet- Ik with constant memory. In /CML, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang. Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. CVPR, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Jiawei Du, Yidi Jiang, Vincent Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In CVPR, 2023a.
Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. In NeurlPS, 2023b.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprini arXiv:2101.00 00027, 2020.
Ziyao Guo, Kai Wang, George Cazenavette, HUI LI, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In ICLR, 2023.
Sariel Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. In Proceedings of the twenty-first annual symposium on Computational geometry, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recog- nition. In CVPR, 2016.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.003/1 2022.

---

Published as a conference paper at ICLR 2025
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Jang- Hyuu Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song. Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICMI,, 2022.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-I0 and CIFAR-100 datasets. URI: https://www. cs. toronto. edu/kriz/cffar. himl, 6(1):1, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowl- edge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022a.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset conden- sation with contrastive signals. In ICML, 2022b.
Shiye Lei and Dacheng Tao. comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Ping Liu and Jiawei Du The evolution of dataset distillation: Toward scalable and generalizable solutions. arXiv preprint arXiv:2502.05667, 2025.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In NeurlPS, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In NeurlPS, 2022.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018.
Zhiheng Ma, Anjia Cao, Funing Yang, and Xing Wei. Curriculum dataset distillation. arXiv preprint arXiv:2405.099 00500, 2024.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge- regression. arXiv preprint arXiv:2011. 1.220011..0000 1.00050, 2020.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, 2021.
Noveen Sachdeva and Julian McAuley. Data distillation: survey. Transactions on Machine Learning Research, 2023.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstanti- nos N. Plataniotis. DataDAM: Efficient dataset distillation with attention matching. In ICCV, 2023.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large- scale data condensation via various backbone and statistical matching. CVPR, 2024.
Donghyeok Shin, Scungjae Shin, and II-Chul Moon. Frequency domain-basee dataset distillation. In NewrlPS, 2024.
12

---

Published as a conference paper at ICLR 2025
Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Loss-curvature matching for dataset selection and condensation. In AISTAS, 2023.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: efficient dataset distillation paradigm. In CVPR, 2024.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. In NeurlPS, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Kai Wang. Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang. and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint arXiv:1811.10900 10959, 2018.
Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma. Sparse parameterization for epitomic dataset distillation. In NeurIPS, 2023.
Zhang Xin, Du Jiawei, Li Yunsong, Xie Weiying, and Jocy Tianyi Zhou. Spanning training progress:
Temporal dual-depth scoring (tdds) for enhanced dataset pruning. In CVPR, 2024.
Eric Xue, Yijiang Li, Haoyang Liu, Yifan Shen, and Haohan Wang. Towards adversarially robust dataset distillation by curvature regularization. arXiv preprint arXiv:2403.10045 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. 2023.
Zeyuan Yin, Erie Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at perspective. In NeurlPS, 2024.
imagenet scale from
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lapez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
Guangze Zheng, Shijie Lin, Haobo Zuo, Changhong Fu, and Jia Pan. Nettrack: Tracking highly dynamic objects with a net. CVPR, 2024.
Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: A good compression is all you need. arXiv preprint 07Xiv:240.007976, 2024.
13


Reviews:
[{"review_id": "W0414RZYHC", "reviewer": "Reviewer_RT9a", "summary": "This paper first identifies two limitations of the current dataset distillation methods: inefficient utilization of the distillation budget and oversight of inter-class features. Based on this, this paper proposes to leverage a Universal Feature Compensator to enhance feature integration across classes.\nExperiments demonstrate that the proposed INFER improves the storage efficiency and effectiveness of dataset distillation.", "questions": "1. The authors claim that the existing dataset distillation methods focus on optimizing synthetic data exclusively for a pre-assigned one-hot label. However, RDED [1] introduces a non-optimization method that changes this paradigm. It is recommended that the authors analyze and compare their method with RDED.\n\n2. In Table 1, lines 416-417, the authors claim that static labels perform better on small networks, while dynamic labels excel on large networks. However, in Table 2, static labels sometimes outperform dynamic labels on large networks and datasets, particularly when \nIPC=50 using ResNet-101. Why is this the case?\n\n3. Additionally, in Table 4, the performance comparison between static and dynamic labels is inconsistent. Sometimes static labels are superior, other times dynamic labels are. What factors determine the choice between static and dynamic labels in real-world applications?\n\n4. Analyzing computational cost is crucial in dataset distillation. It would be beneficial to compare both computation cost and peak memory usage against existing methods. \n\n[1] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It analyses the limitations of the current optimization-based dataset distillation methods.\n2. The proposed method can significantly reduce the storage.\n3. The paper is well-written.", "weaknesses": "1. The authors claim that the existing dataset distillation methods focus on optimizing synthetic data exclusively for a pre-assigned one-hot label. However, RDED [1] introduces a non-optimization method that changes this paradigm. It is recommended that the authors analyze and compare their method with RDED.\n\n2. In Table 1, lines 416-417, the authors claim that static labels perform better on small networks, while dynamic labels excel on large networks. However, in Table 2, static labels sometimes outperform dynamic labels on large networks and datasets, particularly when \nIPC=50 using ResNet-101. Why is this the case?\n\n3. Additionally, in Table 4, the performance comparison between static and dynamic labels is inconsistent. Sometimes static labels are superior, other times dynamic labels are. What factors determine the choice between static and dynamic labels in real-world applications?\n\n4. Analyzing computational cost is crucial in dataset distillation. It would be beneficial to compare both computation cost and peak memory usage against existing methods. \n\n[1] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JiajTwGnVw", "reviewer": "Reviewer_hymx", "summary": "In general, dataset distillation methods employ a class-specific synthesis paradigm, each synthetic instance has one label. This work suggests two limitations that these \"one label per instance\" approaches do not utilize the budget efficiently and fails to adequately reflect inter-class features. To overcome these limitations, this work introduces a new approach, Inter-class Feature Compensator (INFER), for “one instance for all classes” framework by utilizing Universal Feature Compensator (UFC). Experimental results show the effectiveness of the proposed framework.", "questions": "1. How to calculate the utilized budget of INFER? According to Algorithm 1, it seems that $P^k$, $U^k$, and $Y^k$ are stored in budget after training. $P^k$ consists of instances equal to the number of classes $C$ (Line 263), and $U^k$ consists of $M$ instances (Line 269). Since there are a total of $K=\\text{IPC}$ (Line 261) sets of $P^k$ and $U^k$, the total number of instances in $P$ and $U$ amounts to $\\text{IPC} \\times (C + M)$, which already exceeds the budget of $\\text{IPC}×C$ used by conventional methods. If my understanding is incorrect, please clarify the budget utilized by INFER.\n\n2. Is there a reason for learning $U^k$ independently for each $k$? Learning $U^k$ independently for each $k$ aims to capture the inter-class features of $P^k$. This approach seems to capture only local inter-class features, which may be suboptimal. Introducing a single $U$ could potentially be beneficial for obtaining global inter-class features. Are there experimental results using a single $U$?\n\n3. In Algorithm 1, all $u_j \\in U^k$ are initialized with zero vector and independently optimized with same ingredient. This suggests a risk that all instances of $u_j$ may converge to the same optimal value. I am curious about how different values of the optimized $u_j$ actually are. Additionally, I am interested in the rationale for initializing with a zero vector rather than a random noise vector, as well as the resulting performance when using the latter.\n\n4. In general, data factorization methods generate more instance than IPC under same budget. Given this characteristic, feature duplication may present a greater limitation in data factorization methods. Does feature duplication occur in data factorization as well? If it does, is there an improvement in performance when combining INFER with data factorization methodologies?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This manuscript is well written and easy to follow along with their reasoning.\n2. The idea of UFC is simple and easy to adapt.\n3. The motivations of this work, inefficient budget utilization and oversight of inter-class features, are straightforward and novel. The proposed method aligns with these motivations.", "weaknesses": "1. The calculation of utilized budget of INFER is unclear. Some components in INFER seem to be consist of several vectors which have same size with original natural instance. However, in the manuscript, there is explicit description of utilized budget of INFER. Therefore, it is essential to clearly specify how the budget used by INFER is calculated and how it is set to ensure a fair comparison with prior studies.\n\n2. There is insufficient baseline used for performance comparison. Although many dataset distillation methods are recently proposed to enhance the performance and budget efficiency, this work only compares INFER to some (possibly outdated) studies. Additionally, comparing INFER’s performance with recent methods raised doubts about its effectiveness. For example, DATM [1], which also address the feature redundancy as the manuscript said (Line 238-240), achieves higher performance than INFER in many settings: 47.2% (IPC=10), 57.5% (IPC=100) for CIFAR-100 and 31.1% (IPC=10). 39.7% (IPC=50) for Tiny-ImageNet.\n\n3. One of the component in INFER, $P^k$, is consist of real instances. Storing real instances can be a drawback in tasks where privacy issues are crucial (e.g., continual learning). Continual learning is one of the primary applications of dataset distillation.\n\n[1] Towards lossless dataset distillation via difficulty-aligned trajectory matching", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1k7Yc1siY", "reviewer": "Reviewer_enyK", "summary": "This paper presents a novel dataset distillation (DD) method, INFER (Inter-Class Feature Compensator), which explores the potential of leveraging inter-class features to significantly enhance distillation performance. In addition, it addresses a critical challenge faced by decoupled DD methods (e.g., SRe2L), where soft labels require nearly 30 times more storage than images. To overcome this limitation, the proposed method introduces a static labelling strategy that not only retains competitive performance but also dramatically reduces storage requirements, making it a more efficient and scalable solution.", "questions": "e. I have a question regarding the IPC used in the experiments, which refers to the number of images per class. If the authors apply the augmentation approach described in Equation 2, the final number of distilled images per class becomes IPC \\times \\left(1 + \\frac{M}{{class number}}\\right) . This seems to differ slightly from the IPC used in other comparison methods. While this difference might have minimal impact on large datasets with many classes, it could be more influential for smaller datasets. Have the authors accounted for this adjustment to ensure a fair comparison?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a. The authors present an innovative analysis of existing class-specific dataset distillation methods. This work distinguishes itself from prior research by proposing the Universal Feature Compensator (UFC), which enhances inter-class feature integration, resulting in a more efficient dataset distillation framework. Relevant related works are appropriately cited. b. The submission is technically sound, with thorough evaluations and comparisons. c. The paper is clearly written and well-organized, making the content easy to follow and understand. d. This work addresses two key limitations of class-specific dataset distillation methods and achieves SOTA performance. Moreover, the proposed static labelling strategy reduces storage requirements by 99.3%.", "weaknesses": "a. The compensator is denoted by $\\mu$ in Fig. 2, but by $u$ in Fig. 3. Maintaining consistent notation across the figures would improve clarity and help avoid potential confusion.\nb. Some results fall short of SOTA benchmarks, and it would be helpful if the authors provided a discussion of potential reasons for this discrepancy.\nc. The current evaluation only focuses on same-architecture distillation performance for ConvNet and ResNet18 on CIFAR and tiny-ImageNet. A broader cross-architecture evaluation would offer better insights into generalization. \nd. While the static labelling approach effectively reduces storage requirements and, in some cases, even outperforms the dynamic method, the paper does not explore whether these benefits extend to other decoupled DD methods, such as SRe2L.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tu8O2WxVrB", "reviewer": "Reviewer_aHjp", "summary": "The authors highlight that the “one label per instance” paradigm reduces intra-class diversity and amplifies inter-class differences within the distilled dataset. These limitations hinder the efficient use of the distillation budget and lead to poorly defined decision boundaries. To overcome these challenges, the authors propose a novel “one instance for all classes” approach, incorporating a Universal Feature Compensator (UFC). This method enhances class overlaps and ambiguities, improving representation quality and achieving state-of-the-art performance across a variety of datasets and architectures.", "questions": "please see the weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The analysis of the two limitations in class-specific dataset distillation is insightful.\n- The proposed Inter-class Feature Compensator is novel and improves the utilization of the distillation budget by leveraging inter-class features.\n- To assess the size of the final distilled dataset, the authors introduce a compression ratio, in addition to the commonly used IPC metric, providing a fairer comparison with methods that utilize soft labels.\n- The paper is well-written and well-structured.\n- It offers extensive validation experiments across multiple datasets and architectures, reinforcing the reliability of the proposed approach.", "weaknesses": "- [1] explores inter-class and inter-class relations in dataset distillation, introducing a class centralization constraint to tighten within-class clustering and improve class discrimination. While both methods report significant performance improvements, the conclusions appear to contrast with those of this paper. Could the authors elaborate on this discrepancy?\n- [2] also employs multiple backbone architectures for dataset distillation. It would be helpful if the authors discuss how their approach aligns with or differs from this.\n- Since the optimizable parameter is not the entire distillation budget but only the M compensators, will this speed up the distillation or synthesis phase?\n- How did the authors accurately calculate the compression ratio in Table 2? Additionally, how did the author estimate the size of soft labels?\n- Does the compensator consume the IPC budget in the experiments? If yes, according to the design of compensator, the IPC values should not be integers.\n- Including comparisons with recent state-of-the-art dataset distillation methods extended from SRe2L [3, 4] would provide a more comprehensive evaluation.\n- There are some typos; for example, in line 401, “Our INFER also employ the soft labels” should be corrected to “Our INFER also employs the soft labels.”\n[1] Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation. CVPR 2024.\n[2] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching. CVPR 2024\n[3] Dataset Distillation in Large Data Era\n[4] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)