Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective
Authors: Zeyuan Yin
Eric Xing
Zhiqiang Shen
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Dataset Condensation and Distillation
ImageNet Scale

Abstract: We present a new dataset condensation framework termed Squeeze, Recover and Relabel (SRe$^2$L) that decouples the bilevel optimization of model and synthetic data during training, to handle varying scales of datasets, model architectures and image resolutions for efficient dataset condensation. The proposed method demonstrates flexibility across diverse dataset scales and exhibits multiple advantages in terms of arbitrary resolutions of synthesized images, low training cost and memory consumption with high-resolution synthesis, and the ability to scale up to arbitrary evaluation network architectures. Extensive experiments are conducted on Tiny-ImageNet and full ImageNet-1K datasets. Under 50 IPC, our approach achieves the highest 42.5\% and 60.8\% validation accuracy on Tiny-ImageNet and ImageNet-1K, outperforming all previous state-of-the-art methods by margins of 14.5\% and 32.9\%, respectively. Our approach also surpasses MTT in terms of speed by approximately 52$\times$ (ConvNet-4) and 16$\times$ (ResNet-18) faster with less memory consumption of 11.6$\times$ and 6.4$\times$ during data synthesis. Our code and condensed datasets of 50, 200 IPC with 4K recovery budget are available at https://github.com/VILA-Lab/SRe2L.

Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective
Zeyuan Yin"!, Eric Xing!:? Zhiqiang Shen"!
Mohamed bin Zayed University of Al CCrnneiee Mellon University {zeyuan. yin, eric. .xing,iinngi............ zhi qiag. shen) Cabzuai. ac.ae
Abstract
We present new dataset condensation framework termed Squeeze Recover and Relabel (X) (SRe²L) that decouples the bilevel optimization model and synthetic data during training, to handle varying scales of datasets, model architectures and image resolutions for efficient dataset condensation. The proposed method demonstrates flexibility across diverse dataset scales and exhibits multiple advantages in terms of arbitrary resolutions of synthesized images, low training cost and memory consumption with high-resolution synthesis, and the ability scale up to arbitrary evaluation network architectures. Extensive experiments are conducted Tiny-ImageNNet and full ImageNet-IK datasets' Under 50 IPC, our approach achieves the highest 42.5% and 60.8% validation accuracy on Tiny-ImageNet and ImageNet- IK, outperforming all previous state-of oa-te-f-thee-. the-ar methods by margins 14.5% and 32.9%, respectively. Our approach also surpasses MTT [1] in terms of speed by approximately 52x (ConvNet-4) and 16x (ResNet- 18) faster with less memory consumption of 11 11.6x and 6.4x during data synthesis. Our code and condensed datasets of 50, 200 IPC with 4K recovery budget are available at link.
Introduction
real synthetic
synthetic
Top-1 Accuracy (%)
Data synthesie time
R18
R50
R101
12.89
SitelL
TESLA
Eplevel
sqquezze (II) decoupled
R18
C4
joint
Figure 1: Left is data synthesis time vs. accuracy on ImageNet-1K with 10 IPC (Images Per Class). Models include ConvNet-4, ResNet- (18, 50, 101), t indicates VIT with 10M parameters [2]. Right the comparison of widely-used bilevel optimization and our proposed decoupled training scheme.
Over the past few years, the task of data condensation or distillation has garnered significant interest within the domain of computer vision [3, 4, 5, 6, 7, 1]. By distilling large datasets into representative, compact subsets, data condensation methods enable rapid training and streamlined storage, while preserving essential information from the original datasets. The significance of data condensation both research and applications cannot be understated, as plays a crucial role in the efficient handling
* "qqual contribution. Project page: https: https://zzeuaaaa //weyy/yyyyinnn.// gi thub tto/projecte /eets// 'We discard small datasets like MNIST and CIFAR to highlight the scalability and capability of the proposed method on large-scale datasets for real-world applications.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Lice
update
Ryeg
Original data
Model w/ BN
Csyn Vsoit> (3) relabel
(1) squeeze
(2) recover
Figure 2: Overview of our framework. It consists of three stages: in the first stage, model is trained from scratch accommodate most of the crucial information from the original dataset. In the second stage, recovery process is performed to synthesize the target data from the Gaussian noise. In the third stage, we relabel the synthetic data in a crop-level scheme to reflect the true label of the data.
and processing of vast. amounts of data across numerous fields. Through the implementation of so- phisticated algorithms, such as Meta-Model Matching [3, 5]. Gradient Matching [8, 9, 7], Distribution Matching [10, 6], and Trajectory Matching [1, 11], data condensation has made remarkable strides. However, prior solutions predominantly excel in distilling small datasets such as MNIST, CIFAR, Tmny-ImageNet [12], downscaled ImageNet [13] featuring low resolution, or subset of ImageNet [7]. This limitation arises due to the prohibitive computational expense incurred from executing a massive number of unrolled iterations during the bilevel optimization process (comprising inner loop for model updates and an outer loop for synthetic data updates). In our study, employing a meticulously designed decoupling strategy for model training and synthetic data updating (as illustrated in Fig. 1 left), the proposed method is capable of distilling the entire large-scale ImageNet dataset at the conventional resolution of 224x224 while maintaining state-of-the-art performance. Remarkably, our training/synthesis computation outstrips the efficiency of prior approaches, even those utilizing reduced resolution subsets of ImageNet. A comparison of efficiency is provided in Table 1.
To address the huge computational and memory footprints associated with training, we propose a tripartite learning paradigm comprised of Squeeze, Recover, and Relabel stages. This paradigm allows for the decoupling of the condensed data synthesis stage from the real data input, as well as the segregation of inner-loop and outer-loop optimization. Consequently, it is not restricted by the scale of datasets, input resolution, or the size of network architectures. Specifically, in the initial phase, rather than simultaneous sampling of real and synthetic data and its subsequent network processing to update the target synthetic data, we bifurcate this process into Squeeze and Recover stages to distinguish the relationship and reduce similarity between real and synthetic data.
In the subsequent phase, we exclusively align the batch normalization (BN) [14] statistics derived from the model trained in the first phase to synthesize condensed data. In contrast to feature matching solutions that perform the matching process solely on individual batches within each iteration, the trained BN statistics original data span the entire dataset, thereby providing more comprehensive and accurate alignment between the original dataset and synthetic data. Given that real data is not utilized in this phase, the decoupled training can considerably diminish computational costs compared to the preceding bilevel training strategy. In the final phase, we employ the models trained in the first phase to relabel the synthetic data, serving data-label alignment process. An in-depth overview of this process is illustrated in Fig. 2.
Ad Advantages. Our approach exhibits the following merits: (1) It can process large resolution conden- sation during training effortlessly with a justified computational cost. (2) Unlike other counterparts, we can tackle the relatively large datasets of ImageNe and ImageNet-1 K to make our method (3) Our approach can directly utilize many off-the-shel pre-trained large models that contain BN layers, which enables to further save the training overhead.
more practical for real-world applications.
and ImageNet- IK datasets. On ImageNet-IK with 224 x 224 resolution and IPC 50, the proposed approach obtains remarkable accuracy of 60.8%, outperforming all previous methods by a large margin. We anticipate that our research will contribute to the community's 's confidence in the practical feasibility of large-scale dataset condensation using decoupled synthesis strategy from the real data, while maintaining reasonable computational costs.
Extensive experiments are performed on

---

Methods
Condensed Arch
Time Cost (ms)
Peak GPU Memory Usage (GB)
DM [10]
ConvNet-4
18.11
10.7
MTT [1]
ConvNet-4
12.64
48.9
ConvNet-4
0.24
4.2
S.eºt. (Ours)
ResNet-18
0.75
7.6
ResNet-50
2.75
33.8
Table 1: Synthesis Time and Memory Consumption Tiny-ImageNet (64x64 resolution) using single RTX-4090 GPU for all methods. Time Cost represents the consumption (ms) when generating one image with one iteration update on synthetic data. Peak value of GPU memory usage measured or converted with batch size of 200 (1 IPC as the dataset has 200 classes).
Contributions.
We propose new framework for large-scale dataset condensation, which involves three-stage learning procedure of squeezing, recovery, and relabeling. This approach has demonstrated excep- tional efficacy, efficiency and robustness in both the data synthesis and model training phases.
We conduct a thorough ablation study and analysis, encompassing the impacts of diverse data augmentations for original data compression, various regularization terms for data recovery, and diverse teacher alternatives for relabeling on the condensed dataset. The comprehensive specifications of the learning process offer valuable insights for subsequent investigations in the domain.
To the best of our knowledge, this is the first work that enables to condense the full ImageNet-IK dataset with an inaugural implementation at standard resolution of 224x224 utilizing widely accessible NVIDIA GPUs such as the 3090, 4090, or A100 series. Furthermore, our method attains the highest accuracy of 60.8% full ImageNet- 1K within an IPC constraint of 50 using justified training time and memory cost, outperforming all previous methods by a significant margin.
Approach
Data Condensation/Distlllaton The objective of dataset condensation acquire a small synthetic dataset that retains substantial amount of information present in the original data. Suppose we have a large labeled dataset T { {<11 3/) ..... (I M(T)) }. we aim learn small condensed dataset e (CC) < ITD that preserves the crucial information in original T. The learning objective on condensed synthetic data is:
Cayn { ....... (á
Ceyn = arg minCe
where Cc(0)= (((a))=(appp) ()) (()00 oay ( y) ÿ is the soft label coresponding to the synthetic data ă.
The ultimate goal of data condensation task is synthesize data for achieving certain/minimal supremum performance gap on original evaluation set when models are trained on the synthetic data and the original full dataset, respectively. Following the definition of coresets [15] and approximate [16], the objective of data condensation task can be formulated as achieving:
sup {1e (0o, (x). y) -(
(No0 (x).
Se
(I.N))~T
where is the performance gap for models trained on the synthetic data and the original full dataset. Thus, aim optimize the synthetic data Csya through:
sup {|e (0o- (x), ( (docom (z))
arg min Cayn.ICl
Then, we can learn <data, label> E Csyn with the associated number of condensed data each class.
Decoupling the condensed data optimization and the neural network training: Conventional solutions, such as FRePo [5], CAFE [6], DC [8], typically choose for the simultaneous optimization of the backbone network and synthetic data within singular training framework, albeit in an iterative fashion. The primary drawback associated with these joint methods is their computational burden due to the unrolling of the inner-loop during each outer-loop update, coupled with bias transference
3

---

from real to synthetic data as result of truncated unrolling. The objective of this study is to devise an efficient learning framework capable of individually decoupling model training and synthetic data optimization. This approach circumvents information bias stemming from real data, concurrently enhancing efficiency in handling diverse scales of datasets, model architectures, and image resolutions, thereby bolstering effective dataset condensation. Our framework is predicated on the assumption that pivotal information within a dataset can be adequately trained and preserved within a deep neural network. The training procedure of our approach is elaborated in the following section.
2.1 Decoupling Outer-loop and Inner-loop Training
Inspired by recent advances in DeepDream [17], Inverting Image [18, 19] and data-free knowledge transfer [20], we propose decoupling approach to disassociate the traditional bilevel optimization inherent to dataset condensation. This is accomplished via a tripartite process to reformulate into a unilevel learning procedure.
Stage-1 Squeeze (e): During this stage, our objective is to extract pertinent information from the original dataset and encapsulate it within the deep neural networks, evaluating the impact of various data augmentation techniques, training strategies, etc. Deep neural networks typically comprise multiple parametric functions, which transform high-dimensional original data (e.g., pixel space of images) into their corresponding liw-dimensional latent spaces. We can exploit this attribute to abstract the original data the pretrained model and then reconstruct them in more focused manner, akin to DeepDream [17] and Inverting Images [18, 19J. It's noteworthy that the purpose of this stage to extract and encapsulate critical information from the original dataset. Hence, excessive data augmentation resulting in enhanced performance does not necessarily lead to the desired models. This approach diverges from previous solutions that sample two data batches from the original large-scale dataset T and the learnable small synthetic dataset C. The learning procedure can be simply cast as regular model training process on the original dataset with a suitable training recipe:
T = arg (i)
where (r) typically uses cross-entropy loss as (r) = (...)) log (p (x))]-
Enabling BN Layers in VIT for Recovering Process: In contrast to distribution matching [10]
that aligns the feature distributions of the original and synthetic training data in sampled embedding spaces, thus allowing for the use of randomly initialized network, our matching mechanism is solely performed on the Batch Normalization (BN) layers using their statistical properties, akin to data-free knowledge transfer [20]. Unlike the feature matching solution which executes the matching process on individual batches within each iteration, the referential BN statistics are calculated over the entirety of the dataset, providing a more comprehensive and representative alignment with the original dataset. Our experiments empirically substantiate that BN-matching can significantly outperform the ceature-matchin method. BN layer is commonly used in ConvNet but is absent in VIT. To utilize both ConvNet and ViT for our proposed data condensation approach, we engineer BN-ViT which replaces all LayerNorm by BN layers and adds additional BN layers in-between two linear layers of feed-fore feed-forward network, as also utilized in [21]. This marks the first instance of broadening the applicability of the data condensation architecture from ConvNets to encompass ViTs as well.
): This phase involves reconstructing the retained information back into the image space utilizing class labels, regularization terms, and BN trajectory alignment. Unlike conforming to batch feature distributions or comprehensive parameter distributions, we solely track the distribution of BN statistics derived from the original dataset. The pairing of BN and predictive probability distribution restricts the optimization process to a singular level, thereby significantly enhancing scalability. By aligning the final classification and intermediary BN statisties (mean and variance), the synthesized images are compelled to encapsulate a portion of the original image distribution. The learning objective for this phase can be formulated as follows:
Stage-2 Recover
arg min (ōo, (Z y) + -Reeg
(5)
Cayn./C
where Reg is the regularization term. Φe, is the model pre-trained in the first stage and it will be frozen in this stage, we solely optimize the Tyyn as single-level training process. Following [18, 20], we discuss three regularizers that can be used and the ablation for them is provided in our experiments. The first two regularizers are image prior regularisers of le and total variation (TV) proposed in [18]:
Retior (msyn) RTV (āsya) + at2 Re, (Tsyn)
(6)

---

where Rτ,- = || 22,, this regularizer encourages image to stay within a target interval instead of diverging. RTV - Eig (wij+1-
(zi+1. Big where 3 is used balance the sharpness of the image with the removal of "spikes" and smooth the synthetic data. While, this is not necessary for dataset condensation since we care more about information recovery.
Learning Condensed Data with BN Consistency: DeepInversion [20] utilizes the feature distribu- tion regularization term to improve the quality of the generated images. Here, we also leverage this property as our recovering loss term. It can be formulated as:
RaN (z) = > [ - E E (T)l2 - > oof - E (o² 17)
~ Σ (m) - BN 112 + +Σ -BN
where 1 is the index of BN layer, (4( and (z) are mean and variance. BN and BN, are running mean and running variance in the pre-trained model at 1-th layer, which are globally counted.
ieroomeesizeeCrror a frequently utilized technique in neural network training, serves as preventative measure against overfitting. Inspired by this, we propose the strategy of multi-crop optimization during the process of image synthesis with the aim of enhancing the informational content of the synthesized images. In practice, on the entire image and subsequently resizing the cropped region to the target dimension of 224 x 224. Under these circumstances, only the cropped region is updated during each iteration. This approach aids in refining the recovered data when viewed from the perspective of cropped regions.
Multi-crop Optimization:
implement by randomly cropping
Stage-3 Relabel (X):
To match our multi- multi-crop optimization strategy, also to reflect the true soft label for the recovered data. We leverage the pre-generated soft label approach as FKD [22].
y 0 (ãRR)
(8)
where कR. is the i-th crop in the synthetic image and ÿ; is the corresponding soft label. Finally, we can train the model @oec.y on the synthetic data using the following objective:
Layn = Σ ys log @0cayn (IR.)
We found that this stage is crucial to make synthetic data and labels more aligned and also significantly improve the performance of the trained models.
How does the proposed approach reduce compute and memory consumption? Existing solutions predominantly employ bilevel optimization [3, 23, 4] or long-range parameter matching strategies [1, 111, which necessitate the feeding of real data into the network to generate guiding variables (e.g., features, gradients, etc.) for target data updates, network training, through an iterative process. These approaches incur considerable computational and memory overhead due the concurrent presence of real and synthetic data on computational hardware such as GPUs, thereby rendering this training strategy challenging to scale up for larger datasets and models. To this end, natural idea is to decouple real and synthetic data during the training phase, thereby necessitating only minimal memory during each training session. This is achieved by bifurcating the bilevel training into a two-stage process: squeezing and recovering. Moreover, we can conveniently utilize off-the-shelff pre-trained models for the first squeezing stage.
Discussion:
well for the backbone
3 Experiments
In this section, evaluate the performance of our proposed SRe"L over various datasets, models and tasks. First, we conduct extensive ablation experiments to investigate the effect of each component in three stages. Next, we demonstrate the superior results of SRe"L in large-scale datasets, cross- architecture generalization, and continual learning application. Finally, we provide the comparison of visualizations distilled data with other Mfae--of-the-ar art methods.
Experiment Setting. We evaluate our method SRe"L on two large-scale datasets Tiny-ImageNet [12] and full ImageNet- K [24]. Detailed comparison among variants of ImageNet-1K is in appendix. For backbone networks, we employ ResNet- 18, 50, 101) [25], VIT-Tiny [26], and our new con- structed BN- SN--VITTiny (Sec. 2.1) the target model training. For distilling ImageNet-IK. 1K, we
5

---

Squeezing Budget (iteration)
Data Augmentation
Recovery model
50
100
200
400
Mixup
CutMix
ResNet-18 ResNet-50
37.88
37.49
32.18
23.88
35,59
25.90
39.19
33.63
30.65
26.43
36.95
29.61
Table 2: Ablation of squeezing budget and data augmentation on Imageee.
recover the data from PyTorch off-the-shelf pre-trained ResNet- 18, 50) with the Top-1 accuracy of [69.76%, 76.13%} for saving re- re-training computational overhead. For distilling Tiny- Iny-lmmaeNet, ResNet- 18, 50} are used as base models, while the first 7 Conv layer is replaced by 3 xx3 Conv layer and the maxpool layer is discarded, following MoCo (CIFAR) [27]. After that, they are trained from seratch on Tiny-ImageNet. More implementation details are provided in appendix.
Evaluation and baselines. Following previous works [3, 1, 8], we evaluate the quality of condensed datasets by training models from scratch on them and report the test accuracy on real val datasets.
3.1 Squeezing Analysis
Numerous training methods exist to enhance the accuracy of models [28], including extended training cycles/budgets and data augmentation strategies such as Mixup [29] and CutMix [30]. We further examine the performance of synthetic data regenerated from models demonstrating varied levels of accuracy. This investigation is aimed at addressing compelling question: Does a model with superior squeezing ability yield more robust recovered data? Here, "a model with superior squeezing ability" is defined model that exhibits strengthened accuracy the validation set.
Squeezing Budget. In Table we observe decrement in the performance of the recovered model as the squeezing budget escalates. This suggests an increasing challenge in data recovery from model that has been trained over more iterations. Consequently, we adopt a squeezing budget of 50 iterations as the default configuration for our experiments on Imny-
Data Augmentation. Table 2 decrease the final accuracy of the recovered data. In summary, results on Tiny--mageNet indicate that extending the training duration and employing data augmentation during the squeeze phase exacerbates the complexity of data recovery from the squeezed model.
also shows that data augmentation methods in the squeeze stage
3.2 Recovering Analysis
The condensed data are crafted and subsequently relabeled utilizing pre-trained ResNet- 18 with temperature of 20. Then, we report the val performance of training a ResNet-18 18 from scratch.
Image Prior Regularization. RTV and R2, are extensively applied in image synthesis meth- ods [18, 31]. However, in the pursuit of extracting knowledge from the pre-trained model, our focus is predominantly Analyzing from the standpoint of evaluation performance as shown in appendix, Re, and Rry barely contribute to the recovery of image semantic information, and may recovery. Consequently, these image prior regularizations are omitted during our recovery phase.
the recuperation of semantic information opposed to visual information.
impediments to data
operation applied to the training data during the subsequent model training phase, we incorporate a corresponding handomResizedCrz augmentation on synthetic data during recovery. This implics that only minor cropped region in the synthetic data undergoes an update in each iteration. Our experimentation reveals that the multi-crop optimization strategy facilitates notable improvement in validation accuracy, in Appendix Table 8.
Multi-crop Optimization. To offset the RandomResizedden
comparative visualization and comparison with other non-crop settings is shown Fig. 3. In the last column (SRe?L), multiple miniature regions enriched with categorical features spread across the entire image. Examples include multiple volcanic heads, shark bodies, bee fuzz, and mountain ridges. These multiple small feature regions populate the entire image, enhancing its expressiveness in terms of visualization. Therefore, the cropped regions on our synthetic images are not only more closely associated with the target categories but also more beneficial for soft label model training.

---

Recovering Budget
Dataset
Recovery model
Time (ms)
0.5k
2k
4h:
ResNet- 18 ResNet-50
34.82
37.88
41.35
41.11
0.75
Imageem
35.92
39.19
39.59
39.29
2.75
ResNet- 18 ResNet-50
38.94
43.69
46.71
46.73
0.83
ImageNet-1K IK
21.38
28.69
33.20
35.41
2.62
Table 3:
Top-1 validation accuracy of ablation using 1e8 50) as recovery model with different updating iterations, and ResNet- 18 student model. "Time" represents the consuming time (ms) when training image per iteration on one single NVIDIA 4090 GPU.
+Rrv + Rig
+Ry Rig
-RV + Rig
Rry Rsz
RTV Rsg + +rop
Figure 3: Visualization of distilled examples on ImageNet- 1K under various regularization terms and crop augmentation settings. Selected classes are (Volcano, Hammerhead Shark, Bee, Valley).
Recover Budget. We conduct various ablation studies to evaluate the impact of varying recovery budgets on the quality of synthetic data. The recovery budgets are designated as [0.5k, 1 2k, 4k). As in Table 3, it indicates that employing longer recovery budget the same model results in superior classification accuracy on the recovered data. In the case of recovery from diverse models, the results demonstrate that the data recovery under the same iterative setting and budget from ResNet-50 is inferior to that from ResNet-18 on both datasets. This suggests that the process data recovery from larger pre-trained models is more challenging large-scale datasets, necessitating ensure that the recovered data achieves comparable performance on downstream classification tasks. To strike balance between performance and time cost, impose recovery budget of 1 iterations on Imageee and 24 iterations on ImageNet-IK for the ablations, and 4k: for the best in Table 4.
iterations to
3.3 Relabeling Analysis
We further study the influence of varying relabeling models and distinct softmax temperatures on different architectural models being optimized. In Fig, 4, present three subfigures that represent

---

Top-1 Accuracy (%)
60
ImageeNet 1K
.00
50
60
10
30
30-
30
90
20
15
20
5
10
20
5
10
15
20
Top-1 Accuracy (%)
40
40
40
Tiny Imageeeeeee
30-
20-
20
15
20
20
10
20
Temperature
Temperature
Temperature
Figure 4: Top-1 val accuracy of models trained on various labels and temperature settings under IPC 50. T and S represent the reference model for relabeling and the target model be trained, separately. R18, R50, and R101 are the abbreviation of ResNet- 18, ResNet-50, and ResNet-101.
the training accuracy of the models on labels generated by three pre-trained models: ResNet- | 18, 50. 101). The training data utilized for these experiments is the same, which is recovered from pre-trained ResNet-18 model.
Relabeling Model. In each row of Fig. 4 (i.e., on the same dataset), the results three subfigures indicate that smaller-scale teacher which is close or identical the recovery model architecture always achieves better accuracy across ResNet-18, ResNet-50 and 101.. Thus, it can be inferred that the labels of the recovered data are most accurate when the relabeling model aligns with the recovery model. Moreover, Top-1 errors tend to escalate with increasing disparity between the relabeling model and the recovery model. Consequently, our three-stage methodology. we opt to employ an identical model for both recovery and relabeling processes.
Temperature Soft Label. We conduct experiments encompassing five different temperature selections [1, 5, 10, 15, 20] specifically for label softmax operations under distillation configura- tion as [22]. The result indicates that the Top-1 accuracy experiences rapid surge initially and subsequently plateaus when the temperature setting exceeds 10. The maximum Top-1 accuracy is recorded as 60.81% when the temperature is fixed 20, employing ResNet- 18 the teacher model and 1esNet-101 as the student model. This observation underscores the beneficial effect of a higher temperature setting for label softmax operations in the training of the student model. Consequently, we elect to utilize temperature value of 20 in our subsequent evaluation experiments.
Model Training. Contrary to previous data condensation efforts, where different architectures could not be effectively trained on the condensed data due to the mismatch of recovery models or overfitting on the limited synthetic data scale, condensed data demonstrate the informativeness and training scalability as the inherent properties in real data. In each subfigure within Fig. 4, we observe a progressive increase in accuracy when training ResNet- 18, ResNet-50, and ResNet-101 the student networks. This indicates that our condensed data does not suffer from overfitting recovered from the squeezing model, and during inference, the Top-1 accuracy is consistently increasing using network with enhanced capabilities.
3.4 Condensed Dataset Evaluation
Tiny-ImageNer The results derived from the group of Table 4. Upon evaluation, it can be observed that MTT achieves 28.0% under the IPC 50. In contrast, our results on ResNet-(15 (18, 50, 101) architectures are 41.1%. 42.2%, and 42.5%, respectively, which significantly surpass the performance of MTT. A noteworthy observation that our stronger backbones not only accomplish superior accuracy but are also robust for different recovery architectures. light of the results under IPC 50 and 100 settings on the relatively small ImageNet dataset, it is apparent that the larger backbone did not yield proportional enhancement in performance from ResNet- 18 to 1es by our method in Table and Fig. While, this is different from the observation on full ImageNet- 1K that we discuss elaborately below.
dataset are presented in the first
8

---

Dataset
IPC
MTT(I]
[ES]
TESLA (VI)
TESLA [11] (RI8)
SRe"L (R18)
SRc (R50)
SRcAL (R101)
50
28.010.3
41.120,4
42.21:0.5
42.510.2
Tiay-I IN
100
49.7:10.3
51.2::0.4
51,5-10.3
10
6..01133
178.13
11.0102
7.710.1
21.3:10.6
28.4100
30.9110.1
IN-1K
50
27.9-1.2
55.64-0.3
100
52.8.10.3
61.011044
62.8.10.2
64.6+-0.3
65.94-0.
Table 4: Comparison with baseline models. indicates the ImageNette dataset, which contains only 10 classes. TESLA [11] uses the downsampled ImageNet- IK dataset. Our results are derived from the full ImageNet- LK, which is more challenging on computation and memory, meanwhile, presenting greater applicability potential in real-world scenarios. The recovery model used in the table is R R18.
Evaluation Model
Squeezed Model
DeiT-Tiny
ResNet-18
ResNet-50
ResNet-101
DeiT-Tiny-BN
25.36
24.69
31.15
33.16
ResNet-18
15.41
46.71
55.29
60.81
Table ImageNet-1K Top-1 on Cross-Archtecture Generalization. Two recovery/squeezed models used: Deii-Tiny-BN and ResNet-18. Four evaluation models: DeiT-Tiny, ResNet-[18, 50, 101).
ImageNet-1K. As shown in the second group of Table employing the same model architecture of ResNet- 18 under IPC 10, our approach improves the performance of TESLA from a baseline of 7.7% significance of 21.3%. Contrary to TESLA, where the performance deteriorates with larger model architectures, our proposed approach capitalizes on larger architectures, displaying an appreciable proportional performance enhancement. This indicates significant promise in the contemporary era of large-scale models. On IPC 50, 100 and 200, our method obtains consistent boosts on accuracy.
3.5
Cross-Architecture Generalization
It is important to verify the generalization prop- erty of our condensed datasets, ensuring abil- ity to effectively generalize to new architectures that has not encountered during the synthe- sis phase. Fig. 4 and Table demonstrate that our condensed dataset exhibits proficient cross- model generalization across ResNet-( 18, 50, 101 and ViT-T. The results reveal that our con- densed datasets maintain robustness across dis- parate and larger architectures. However, we observed suboptimal performance of ViT the condensed dataset, potentially due to the model's 's inherent need for substantial training data as introduced in [26].
Snall
Centipeee
Deason
Triumphal Arch
Keypad
3.6 Synthetic Image Visualization
Fig. 5 provides a visual comparison of selected synthetic images from our condensed dataset and the corresponding images from the MTT condensed dataset. The synthetic images gener- ated by our method manifest higher degree of clarity on semantics, effectively encapsulating the attributes and contours of the target class. In contrast, synthetic images from MTT appear considerably blurred, predominantly capturing color information while only encapsulating minimal details about the target class. Consequently. SRe"L produces superior-quality images that not only embed copious semantic information to augment validation accuracy but also demonstrate superior visual performance.
King Penguin
Cucumber
Garbage Truck
Figure Visualization of MTT [1] and our S The upper two rows are synthetic Tiay-ImageNet and the lower two rows are synthetic ImageNet- (the first row is MTT and second is ours).

---

3.7
Application: Continual Learning
Top-1 Accuracy (%)
Top-1 Accurazy (%)
45-
45
40
J0
35
SRe'L Random
35
SRe"L
Random
40
80
120
1680
2001
20
401
030
80
100
120
100
3011
180
200
Number classes
Number of classes
Figure 6: 5-step and 10-step class-incremental learning on Tiny-ImageN
Many prior studies [10, 5, 32, 7] have employed condensed datasets for continual learning to assess the quality of the synthetic data. We adhere to the method outlined in DM [10] ilass-nncremental learning implementation, which is based on GDumb [33]. This method sequentially stores prior training data in memory and utilizes both new training data and stored data to Iearn model from scratch. To demonstrate the superiority of our method in handling large-scale data, we conduct class incremental learning on Tiny- ImageNet, incorporating an escalating memory budget of 100 images per class, and training with ResNet-1e 18. Fig. 6 shows both 5-step and 10-step class- incremenaat learning strategies, which partition 200 classes into either or learning steps, accommodating 40 and 20 classes per step respectively. Our results are clearly better than the baselines.
Related Work
Data condensation or distillation aims to create a compact synthetic dataset that preserves the essential information in the large-scale original dataset, making it casier to work with and reducing training time while achieving the comparable performance the original dataset. Previous solutions are mainly divided into four categories: Meta-Model Matching optimizes for the transferability of models trained on the condensed data and uses an outer-loop to update the synthetic data when generalized to the original dataset with an inner-loop to train the network, methods include DD [3], KIP [23]. RFAD [4], FRePo [5] and LinBa [34]: Gradient Matching performs a one-step distance matching process on the network trained on original dataset and the same network trained on the synthetic data, methods include DC [8], DSA [32], DCC [9] and IDC [7]: Distribution Matching directly matches the distribution of original data and synthetic data with a single-level optimization, methods
include DM [10], CAFE [6], HaBa [35], IT-GAN [36], KFS [9]; Trajectory Matching matches the training trajectories of models trained original and synthetic data in multiple steps, methods include MTT [1] and TESLA [11]. Our proposed decoupling method presents a new perspective for tackling this task, while our BN-matching recovering procedure can also be considered as special format of Distribution Matching scheme on the synthetic data and global BN statistics distributions.
5
Conclusion
We have presented a novel three-step process approach for the dataset condensation task, providing a more efficient and effective way to harness the power of large-scale datasets. By employing the sequential steps of squeezing, recovering, and relabeling, this work condenses the large-scabe scale ImageNet- 1K while retaining its essential information and performance capabilities. The proposed method outperforms existing state-of-the-art condensation approaches by a significant margin, and has a wide range applications, from accelerating the generating and training process to enabling the method that can be used in resource-constrainee environments. Moreover, the study demonstrates the importance of rethinking conventional methods of data condensation and model training. as new solutions can lead to improvements in both computational efficiency and model performance. As the field of data condensation continues to evolve, the exploration of targeting approaches, such as the one presented in this work, will be crucial for the development of future condensation approaches that are more efficient, robust, and capable of handling vast amounts of data a sustainable manner.
At present, performance disparity persists between the condensed dataset and the original full dataset, indicating that complete substitution of the full data with condensed data is yet to be feasible. Another limitation is the extra storage for soft labels. Moving forward, our research endeavors will concentrate on larger datasets such as the condensation of ImageNet-21K. as well other data modalities encompassing language and speech.
Limitation and Future Work:
10

---

References
[1] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVE Confer-
ence on Computer Vision and Pattern Recognition, 2022. 1, 2, 3, 5, 6, 9, 10, 14, 17, 20
[2] Justin Cui, Ruochen Wang. Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation bench- mark. arXiv preprint arXiv:2207.09639 2022.
[3] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexci A. Efros. Dataset distillation,
2020. 1, 5, 6, 10
[4] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXiv:2210. (tt.:22200//00,,7, 2022. 1,5, 10
[5] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3, 10, 14, 17
[6] Kai Wang. Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2, 10
[7] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteri- zation. Proceedings of the 39th International Conference on Machine Learning. 2022.
10, 14
[8] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05922 2020. 2, 3, 6,
(9] Saehyung Lee, Sanghyuk Chun, Sangwon Jung. Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pages
12352-12364 PMLR, 2022. 2, 10
[10] Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, Jamuary 2-7, 2023, 2023. 2, 3, 4, 10, 14, 17
[11] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- IK with constant memory. In International Conference Machine Learming, pages 6565-6599
PMLR, 2023. 2, 5, 9, 10, 14, 17
[12] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 2,
5, 14
[13] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv preprint / 1707.088/9 2017. 2, 14
[14] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015. 2
[15] Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning, arXiv preprint 1eee1173307766 2017.
[16] Noveen Sachdeva and Julian McAuley.
Data distillation: A survey. arXiv preprint
arXiv:2301.04272, 2023. 3
[17] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Go- ing deeper into neural networks. https://researe ch. googleblog cww//015/oe www.ssarcchggggglllleooo inceptionism going- deeper- seaee-into--uuue html, 2015. 4
[18] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by invert- ing them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5188-5196, 2015. 4,
[19] Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4837, 2016. 4
11

---

[20] Hongxu Yin. Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Dree-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715-8724, 2020. 4,5
[21] Zhuliang Yao, Yue Cao, Yutong Lin, Ze Liu, Zheng Zhang, and Han Hu. Leveraging batch nor- malization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 413-422, 2021. 4
[22] Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part XXIV, pages 673-690. Springer, 2022. 5, 8
[23] Timothy Nguyen. Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems,
34:5186-5199 2021. 5, 10
[24] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009. 5, 14
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition,
pages 770-778, 2016. 5
[26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11999, 2020. 5,9 9
[27] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 1rXiv:191 1,00272, 2019. 6
[28] Tong He, Zhi Zhang, Hang Zhang. Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 558-567, 2018. 6
[29] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 6
[30] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon
Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022-6031 1072-6031,22 2019.
[31] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015. 6
[32] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674 12685. PMLR, 2021. 10
[33] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 524-540. Springer, 2020. 10
[34] Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv: 02916, 2022. 10
[35] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization, Advances in Neural Information Processing Systems, 35:1 100-1113, 2022. 10
[36] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint
SKKv:2204.0051:: 2022. 10
[37] Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from imagenet, March 2019. 14
[38] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020- 16th Furopean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part
XI, 2020. 14
12

---

[39] TorchVision maintainers and contributors.
Torchvision image classification refer- ence training scripts. pettps//githin cotm//ppoorch/iissooo/rrre http://///mmmmmeeeeeooocss////// classification, 2016. 15
[40] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 11); 2008. 17
[41] Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani loannou, and Graham W Taylor. Bounding generalization error with input compression: An empirical study with infinite- width networks. Transactions Machine Learning Research, 2022. 18
[42] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds mutual information. International Conference on Machine Learning, pages
5171-5180. PMLR, 2019. 18
[43] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. IEEE, 2015, 18
[44] Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and generalization deep neural networks, 2019. 18
13

---

Appendix
In the appendix, we provide details omitted in the main text, including:
Section A: Implementation Details.
Section B: Low-Resolution Data.
Section C: Feature Emhedding Distribution.
Section D: Theoretical Analysis.
Section E: More Visualization of Synthetic Data.
A Implementation Details
A.1 Dataset Statisties
Table 6 enumerates various permutations of ImageNet-1K training set, delineated according to their individual configurations. Tiny-ImageNNt [12] incorporates 200 classes derived from ImageNet-1K,, with each class comprising 500 images possessing resolution of 64 Ima- geNette/ImageWoo) [37] (alternatively referred to as subsets of ImageNet) include 10 classes from analogous subcategories, with each image having resolution of 112x 12. The MTT [1] framework introduces additional 10-class subsets of ImageNet, encompassing ImageFruit, ImageSquawk, Image- Meow, ImageBlub, and Image Yellow. ImageNet- 10/100000000111 [38] samples 10/100 classes from ImageNet while maintaining an image resolution of 224 224. Downsampled ImageNet- IK rescales the entirety of ImageNet data to resolution of 64 x 64. our experiments, we choose two standard datasets of relatively large scale: Tiny-ImageNe and the full ImageNet-IK.
Training Dataset
#Class
#Img per class
Resolution
Method
Tiny- MetgeNet [12]
64x64
MIT [1], FRePo [5], DM [10), SRe (Ours)
ImageNette/Imageeeen [37]
~1,000
112x112
MIT FRePo [5]
ImageNet- 10/100 [38]
10/100
~1,20
224x224
IDC [7]
Downsampled ImageNet-18 [13]
1,000
~1,200
64x64
TESLA [11], DM [10)
Full ImageNet- IK [24]
1,000
~,,200
224x224
SRe"L (Ours)
Table 6: Variants of ImageNet-1K training set with different configurations.
A.2 Squeezing Details
Data Augmentation. Table 2 in the main paper illustrates that the utilization of data augmentation approaches during the squeezing phase contributes to a decrease in the final accuracy of the recovered data. To summarize, the results on ImageNet indicate that lengthening the training budget and the application of more data augmentations in the squeezing phase intensify the intricacy involved in data recovery from the compressed model, which is not desired.
Parallel conclusions are inferred from the compressed models for the ImageNet- IK dataset. For our experimental setup, we aimed to extract data from pre-trained ResNet-50 model with available V and V2 weights in the PyTorch model zoo. The results propose that the task of data extraction poses greater challenge from the ResNet-50 model equipped with V2 weights compared the model incorporating V/ weights. This can be attributed to the fact that models utilizing V/ weights are trained employing rudimentary recipe, whereas models with V2 weights encompass numerous training enhancements, such as more training budget and data augmentations, to achieve cutting-edes performance. It is observed that these additional complexities impede the data recovery process even the pre-trained models are stronger. Therefore, the pre-trained models we employ for the recovery of ImageNet- 1K are those integrating V/ weights from the PyTorch model zoo.
Hyper-parameter Setting. We provide
settings for the two datasets in detail.
Tiny- ImageNet: We train modified ResNet- ( (18, 50) models on Imag-Im data with the parameter setting in Table 7a. The well-trained ResNet- (18, 50) models achieve Top-1 accuracy of (59.47%, 61.17%) under the 50 epoch training budget.
14

---

config
value
config
value
optimizer
SGD
optimizer
AdamW
base learning rate
0.2
base learning rate
0.001
weight decay
le-4
weight decay
0.01
optimizer momentum
0.9
optimizer momentum
31. 82 = 0.9, 0.999 1,024
batch size
256
batch size learning rate schedule
learning rate schedule
cosine decay
cosine decay 300
training epoch
100
training epoch augmentation
augmentation
RandomResizedCrop
RandomResizedCrop
Tiny-ImageNe squeezing setting.
(b) ImageNet- IK validation setting.
config
value
config
value
OBN
1.0
OBN
0.01
optimizer
Adam
optimizer
Adam
base learning rate
0.1
base learning rate
0.25
optimizer momentum batch size
81. B2 0.5, 0.9 100
optimizer momentum
31. |2 0.5, 0.9 100
batch size
learning rate schedule
cosine decay 1,000
learning rate schedule
cosine decay 2,000
recovering iteration
recovering iteration
augmentation
RandomResizedCrrp
augmentation
RandomResizedCrop
(c) Imay-ImageNet recovering setting.
(d) ImageNet- recovering setting.
Table 7: Hyper-parameter settings in three stages.
ImageNet-IK: We use PyTorch off-the-shellf ResNet-(18 [8, 50} with V/ weights and Top-1 accuracy of (69.76% 13%) } as squeezed/condensed models. In the original training script [39], ResNet models are trained for 90 epochs with a SGD optimizer, learning rate of 0.1, momentum of 0.9 and weight decay of 14-4
BN-VIT Model Structure. To give a clear illustration of incorporating BN layers into VIT, we provide more details in this subsection. The definitions of vanilla VIT and BN-ViT presented below to show the structure modification. The vanilla ViT can be formulated as:
2/2 MHSA (LN (ze-1)) + ze-1
Z FEN (LN (z))) + z'z
where z is the intermediate representation before Feed-forward Network (FFN), and Z₁ is that after FFN and residual connection. FFN contains two linear layers with a GELU non-linearity in between them, i.e.
FFN(2)) (GELU (z,W3 + +b))) w + 6
The newly constructed BN-VIT is:
z MHSA (BN (zr- 1)) + 26-1
Z3 FFN (BN (zf)) z
where we add one additional BN layer in-between two linear layers of FFN, i.e.,
FFN ( = (GELU (BN (z:W2 + bt))) W + 13
We follow the DeiT official training recipe to train a DeiT-Tiny-BN model for 300 epochs with an AdamW optimizer, cosine decayed learning rate of 10-4, weight decay of 0.05 and 5 warmup epochs.
15

---

Ablation
Top-1 acc. (%)
RTV
Ria
Random Crop
Tiny-lmageNet
ImageNet- IK
/
X
29.87
22.92
X
29.92
23.15
X
/
X
30.11
40,81
X
X
X
30.30
40.37
X
X
37.88
46.71
Table Top-1 validation accuracy under regularization ablation settings. ResNet-18 18 is used in all three stages with the relabeling temperature = 20.
A.3 Recovering Details
Regularization Terms. We conduct large number of ablation experiments under varying regular- ization term conditions, as illustrated in Table 8. The two image prior regularizers, l2 regularization and total variation (TV), are not anticipated enhance validation accuracy our primary focus is on information recovery rather than image smoothness. Consequently, we exclude these two regularization terms from our experiments.
Memory Consumption and Computational Cost. Regarding memory utilization, the memory accommodates pre-trained model, reconstructed data, and the corresponding computational graph during the data recovery phase. Unlike the MTT approach, which necessitates model states across all epochs during training to align with the trajectory, requires the statistical data from each BN layer, stored within the condensed model, for synthetic image optimization. In terms of total computational overhead, it is directly proportional the number of recovery iterations. To establish a trade-off between performance and computational time, we enforce recovery budget of Ik iterations for Tiny-ImageNet and 2k iterations for ImageNet- I in ablation experiments. Our best accuracy, achieved on condensed data from 4k recovery iterations, is presented in Table in the main paper.
proposed methodology, SRe²L, merely
Hyper-parameter Setting. We calculate the total recovery loss Littll = arg min E (ōe, (Z .))
Cayn.IC
RBN and update synthetic data with the parameter setting in Table 7c and Table 7d for Tiny- ImageNet and ImageNet- IK, respectively.
A.4 Relabeling & Validation Details
In this experiment, we utilize architecture identical to that of recovery model to provide soft labels as teacher for synthesized images. We implement a fast knowledge distillation process to isolate the utilization of teacher models in post validation training with a training budget of 300 epochs and temperature setting of 20.
Hyper-parameter Setting. Regarding Tiny-ImageNer, we leverage the condensed data and the retargeted labels train the validation model over span of 100 epochs, with all other training parameters adhering to the condensing configurations outlined in Table 7a. In the case of ImageNet- IK, we apply CutMix augmentation with mix probability p 1.0 and Beta distribution train the validation model in accordance with the parameter configurations presented in Table 7b.
1.0 to
B Low-Resolution Data
To demonstrate our method's effectiveness on now-resolution datasets, we first conduct additional ablation experiments ImageNet- IK dataset, including down-sampled resolutions of 112x 112 and 64x64. In Table 9, the validation accuracy increases as the image resolution size grows, demonstrating that our method is more suitable for handling high-resolution datasets with better effectiveness.
then conduct experiments on the CIFAR datasets demonstrate the efficacy of our method when applied to small-scale datasets with fewer classes and a lower image resolution of 32x The adapted ResNet-15 is used as a backbone model throughout our SRe"L's three phases, and the hyper-parameter settings presented in Table 10. As shown in Table 11, results the relatively
16

---

Dataset (IPC=50)
ResNet-18
ResNet--0
ResNet-101
IN-IK -
35.27
42.26
44.37
IN- 1- 112 112
34.15
42.76
45.25
46.75
55.62
60.81
Table 9: Top-1 validation accuracy on distilled and down-sampled ImageNet-IK datasets.
large CIFAR-100 dataset are on par with those of leading-edge methods, such as DM, FrePo, MTT, and TESLA, among others. However, the smaller CIFAR-10 dataset, clear gap exists. Overall, the additional CIFAR experiments suggest that our approach might also offer significant benefits for the lower-resolution dataset, and we highlight that our approach continues to demonstrate superior computational efficiency and enhanced processing speed when applied to these datasets.
config
value
config
value
optimizer
SGD
OBN
0.01
base learning rate
0.1
optimizer
Adam
momentum
0.9
base learning
0.25
weight decay
5c-4
optimizer momentum
B1. 82 = 0.5, 0.9 100
batch size
128
batch size
learning rate schedule
cosine decay
temperature
30
training epoch
200 (squeeze) 400 (val)
learning rate schedule
cosine decay 1,000
augmentation
RandomCrop
recovery iteration
(a) Squcozing/validation setting.
(b) Recovery setting.
Table 10: Hyper-parameter settings CIFAR-100.
IPC
DM [10]
FrePo [5]
MTT [1]
TESLA [11]
SRe
29.70 0.30
41.30 + 0.20
40.10 0.40
41.70 0.30
23.48
0.80
50
43.60 - 0.40
44.30 + 0.20
47.70 = 0.20
47.90 0.30
51.35
0.79
100
57.06 - 1.23
200
59.64 1.24
Table 11: Top-1 validation accuracy on distilled CIFAR- 100 dataset.
C Feature Embedding Distribution
We feed the image data through a pretrained ResNet-18 model, subsequently extracting the feature embedding prior to the classification layer for the purpose of executing t-SNE [40] dimensionality reduction and visualization. Fig. 7a exhibits Tiny- ImngelmeeNNet data, sourced from 3 classes in MTT's ' and SRe2L's condensed datasets, respectively. Relative to the distribution present in MTT. SRe synthetic data from differing classes displays more dispersed pattern, whilst data from identical classes demonstrates higher degree of clustering. This suggests that the data synthesized by SRe boasts superior discriminability with respect to feature embedding distribution and can therefore be utilized to train models to attain superior performance. Fig. 7b illustrates feature embedding distributions of SRePL's synthetic ImageNet- data derived from classes. Our synthetic ImageNet- IK data also exemplifies exceptional clustering and discriminability attributes.
distinct feature embedding distributions of synthetic
D Theoretical Analysis
provide theoretical analysis of the generalization ability on the condensed dataset. Dataset con- densation task generally aims to train on the condensed data meanwhile achieving good performance on the original val data. Given the significance estimating the generalization error (GE) of deep neural networks method for evaluating their ability to generalize, we adopt this approach for
17

---

SRe
MTT
SReAL
Academic Gown
Refrigerator
Meat Loaf
Tiee Ce:
Caffe Hepe
Tebacce Stop
Nt BuB Turter
Tiny-ImageNe
(b) ImageNet- IK
Figure 7: Feature embedding distribution on synthetic data of Tiny-ImageNet and ImageNet-IK IK. ResNet-18 is used as the feature embedding extractor.
assessing the generalization capability of our condensed dataset for analyzing the generalization/erro bounds between models trained original data and condensed data.
Specifically, we employ the Mutual Information (MI) between original/condenssed input and final layer representations to carry out this analysis, using the same network architecture limit to bound MI, in line with the methodology outlined [41]. To elaborate further:
The MI between two variables X and is:
p(a,
= Ep(a,d) log p(d x))
D); = Ep(e, d) log
p(x)p(d)
p(d)
x.d
where X is the input sample, D is the input representation, i.e., model's output. The *leave one out upper bound (UB) [42] can be utilized to conservatively bound MI:
p (d: xx4)
T(x; D) S
log
JUB
N
Σ j (d: xz))
N-I
Following the information theory fundamentals, by applying the conventional Probably Approxi- mately Correct (PAC) of GE bound, can obtain the bound GE
log()H)) log(1/6)
GE
2Ner
where |7 is the hypothesis-class cardinality and Nee is the number of training examples. For the synthetic data, Nun |Caynl, while for the full data, Num Nori- The confidence parameter, denoted by ó and ranging between 0 and 1, specifies the likelihood that the bound remains consistent with respect to the chosen Num training samples.
According to the property of deep neural networks [41], the cardinality the hypothesis space reduces to [H| = 2/71 where [H] is the number of class Hlass-homgeenoou clusters that the backbone network distinguishes. An estimate for the number of clusters then be obtained by [T] sz 2*((X (x (((:2),
The ability of Input Compression Bound (ICB) [43, 44] predict changes in GE under different dataset interventions, then can be formulated as:
21(::D) + log(1/8)
GE
2.
Thus, we can have the generalization error bound for the condensed data as:
21(X.D) -loog1/// (1/8g))
Glayn
2|C
where the generalization enror bound for full dataset is GEffll
(I(X +log(1/óm))
2
18

---

E More Visualization of Synthetic Data
We provide more visualization comparisons on synthetic Tiny-ImageNet between MTT and SRe in Fig. 8. Additionally, we visualize synthetic samples pertaining to ImageNet- IK in Fig. 9 and Fig. 10 for more comprehensive understanding. It can be observed that our synthetic data has the stronger semantic information than MTT with more object textures, shapes and details, which demonstrates superior quality of our synthesized data.
19

---

MTT
S
Abacus
Academic Gown
Acorn
African Elephant
Albatross
MTT
SRe
Alp
Altar
Alligator
American Lobster
Apron
MTT
S
Camel
Baboon
Backpack
Banana
Bannister
MTT
SReeL
Barbershop
Bam
Barrel
Basketball
Bathtub
Figure 8: Synthetic data visualization on Tiny-Imagen from MTT [1] and St
20

---

Figure 9: Synthetic data visualization on ImageNet- 1K from SL
21

---

10
201
Figure 10: Synthetic data visualization on ImageNet- 1K from St.
22


Reviews:
[{"review_id": "zT0ybNBIun", "reviewer": "Reviewer_4huB", "summary": "The paper introduces a new dataset condensation framework, that is Squeeze, Recover, and Relabel (SRe2L).\nSRe2L decouples the optimization of model and synthetic data during training, enabling effective condensation across varying dataset scales, model architectures, and image resolutions. \n\nThe authors mention in the paper about the advantages such as arbitrary resolution synthesis, low training cost and memory consumption, and scalability to different evaluation network architectures. \n\nExtensive experiments on Tiny-ImageNet and full ImageNet-1K datasets demonstrate  its improved performance compared to state-of-the-art methods. \nSRe2L also outperforms the MTT approach in terms of speed and memory consumption during data synthesis. \nOverall, SRe2L presents a powerful solution for dataset condensation with improved performance and efficiency.", "questions": "see weakness", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Clear Paper Organization: The paper exhibits a well-structured organization that aids in comprehending the presented concepts, methodologies, and experimental results. The logical flow of information allows readers to follow the paper's contributions easily.\n\n2. Novel Framework: The authors propose a new dataset condensation framework, Squeeze, Recover, and Relabel (SRe2L), which offers a fresh perspective on addressing the data condensation problem. While some technical details may not be entirely novel, the paper presents an alternative solution to data condensation, introducing new ideas and approaches.\n\n3. Surprising Performance: The scalability of data condensation to large datasets and deep networks poses significant challenges. The paper's performance on ImageNet-level datasets is particularly impressive, demonstrating the effectiveness and robustness of the proposed framework in handling complex datasets and achieving high validation accuracy.", "weaknesses": "1. Limited Theoretical Analysis: While the paper presents impressive empirical results and demonstrates the effectiveness of the proposed framework, it lacks a comprehensive theoretical analysis. More theoretical analysis, such as error bounds or performance upper bounds, would provide a deeper understanding of the underlying principles and limitations of the proposed approach. Incorporating theoretical analysis could further strengthen the paper's contributions and provide insights into the algorithm's behavior and performance guarantees.\n\nOverall, the paper is well-structured, introduces a novel framework, and achieves remarkable performance on challenging datasets. However, enhancing the theoretical analysis would add a valuable dimension to the paper and provide a more comprehensive evaluation of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you find our work novel with surprising performance on scalability. We would like to address the comments and questions below.\n\n>W1. Limited Theoretical Analysis: While the paper presents impressive empirical results and demonstrates the effectiveness of the proposed framework, it lacks a comprehensive theoretical analysis. More theoretical analysis, such as error bounds or performance upper bounds, would provide a deeper understanding of the underlying principles and limitations of the proposed approach. Incorporating theoretical analysis could further strengthen the paper's contributions and provide insights into the algorithm's behavior and performance guarantees.\n\t\t\t\t\nThanks for the insightful suggestion of incorporating a more theoretical perspective into our current analysis. Given the significance of estimating the generalization error (GE) of deep neural networks as a method for evaluating their ability to generalize, since the data condensation task aims to train on the condensed data meanwhile achieving good performance on the original val data. We've adopted this approach for assessing the generalization capability of our condensed dataset for analyzing the generalization/error bounds between models trained on original data and condensed data.\n\nMore specifically, we employ the Mutual Information (MI) between the original/condensed input and the final layer representations to carry out this analysis, using the same network architecture limit to bound MI, in line with the methodology outlined in [1]. To elaborate further:\n  \nThe MI between two variables $X$ and $D$ is:\n$$I(X ; D) \\equiv \\sum_{x, d} p(x, d) \\log \\frac{p(x, d)}{p(x) p(d)}=\\mathbb E_{p(x, d)}\\left[\\log \\frac{p(d \\mid x)}{p(d)}\\right] \\tag{1}$$\n\nwhere $X$ is the input sample, $D$ is the input representation, i.e., model's output. The *leave one out* upper bound (UB) [2] can be utilized to conservatively bound MI:\n\n$$\nI(X ; D) \\leq \\mathbb{E}\\left[\\frac{1}{N} \\sum_{i=1}^{N} \\log \\frac{p\\left(d_{i} \\mid x_{i}\\right)}{\\frac{1}{N-1} \\sum_{j \\neq i} p\\left(d_{i} \\mid x_{j}\\right)}\\right]=I_{\\mathrm{UB}} \\tag{2}\n$$\n\nFollowing the information theory fundamentals, by applying the conventional Probably Approximately Correct (PAC) of GE bound, we can obtain the bound on GE as:\n\n$$\n\\mathrm{GE}<\\sqrt{\\frac{\\log (|\\mathcal{H}|)+\\log (1 / \\delta)}{2 N_{\\mathrm{trn}}}} \\tag{3}\n$$\n\nwhere $|\\mathcal H|$ is the hypothesis-class cardinality and $N_\\mathrm{trn}$ is the number of training examples. For the synthetic data, $N_\\mathrm{trn}=|\\mathcal C_\\mathrm{syn}|$, while for the full data, $N_\\mathrm{trn}=N_\\mathrm{full}$. The confidence parameter, denoted by $\\delta$ and ranging between 0 and 1, specifies the likelihood that the bound remains consistent with respect to the chosen $N_{\\mathrm{trn}}$ training samples.\n\nAccording to the property of deep neural networks [1], the cardinality of the hypothesis space reduces to $|\\mathcal{H}| \\approx 2^{|\\mathcal T|}$ where $|\\mathcal{H}|$ is the number of class-homogeneous clusters that the backbone network distinguishes.  An estimate for the number of clusters can then be obtained by $|\\mathcal T| \\approx 2^{H(X)} / 2^{H(X \\mid Z)}=2^{I(X ; Z)}$.\n\nThe ability of Input Compression Bound (ICB) [3, 4] is to predict changes in GE under different dataset interventions, it then can be formulated as:\n\n$$\n\\mathrm{GE}_{\\mathrm{ICB}} < \\sqrt{\\frac{2^{I(X ; D)} + \\log (1 / \\delta)}{2 N\\_{\\operatorname{trn}}}} \\tag{4}\n$$\n\nThus, we can have the generalization error bound for the condensed data as:\n\n$$\n\\mathrm{GE}^{\\mathrm{syn}}\\_{\\mathrm{ICB}}<\\sqrt{\\frac{2^{I(X ; D)}+\\log (1 / \\delta_\\text{syn})}{2 | \\mathcal C\\_{\\operatorname{syn}}|}} \\tag{5}\n$$\n\nwhere the generalization error bound for full dataset is $\\mathrm{GE}^{\\mathrm{full}}\\_{\\mathrm{ICB}}<\\sqrt{\\frac{2^{I(X ; D)}+\\log (1 / \\delta_\\text{full})}{2 N\\_{\\operatorname{full}}}}$.\n\nWe will include a more detailed and well-organized theoretical analysis into the revised version of our paper.\n  \n### References\n\n[1] Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani Ioannou, and Graham W. Taylor. \"Bounding generalization error with input compression: An empirical study with infinite-width networks.\" Transactions on Machine Learning Research (TMLR) 2022.\n\n[2] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On Variational Bounds of Mutual Information. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5171–5180. 2019.\n\n[3] Naftali Tishby. Information Theory of Deep Learning, 2017.\n\n[4] Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation Compression and Generalization in Deep\nNeural Networks. OpenReview, 2019."}, {"review_id": "x24zbXkJv7", "reviewer": "Reviewer_QGHe", "summary": "This paper proposes a dataset distillation or dataset condensation method that can support ImageNet-scale compression. The main idea is inspired by some data-free knowledge distillation techniques to optimize the cross-entropy error, BN statistic distance, and some other prior terms for the distilled data. The complexity is simple compared with recent mainstream methods of dataset distillation. And the proposed method achieves impressive accuracy for models trained with distilled data on large-scale datasets.", "questions": "My major questions focus on the detailed ablation studies. Please refer to the 2nd point of the above weaknesses part for details.", "limitations": "The authors should clarify more limitations of the specific solution they proposed instead of those general issues of dataset distillation, e.g., larger datasets and performance gap with models trained on full data.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method is simple yet effective, which enjoys satisfactory scalability.\n2. It is good to know for the community that dataset distillation could achieve promising results on large-scale datasets like ImageNet1k-224 resolution.\n3. The writing is coherent and it's easy for readers to follow the proposals.", "weaknesses": "1. I am a little bit worried about the technical novelty. Since the main idea is largely inspired by data-free knowledge distillation techniques [a], I list this as a weakness for NeurIPS, the top-tier machine learning conference.\n2. Some detailed ablation studies are expected:\n    * I notice that the settings of this paper is different from previous works. For example, previous works typically use Convent for dataset distillation while this work mainly considers ResNet. I am not arguing that the setting must be the same. However, I do think that an ablation study is necessary to show the improvement coming from the architectures.\n    * A sensitivity analysis with respect to $\\alpha_{BN}$ is necessary.\n    * I do understand that the proposed method is mainly for large-scale datasets. But it is also interesting to dynamically increase the size of datasets and compare the performance with the existing methods, to help readers better understand when the proposed method yields advantages.\n\n[a] Dreaming to distill: Data-free knowledge transfer via deepinversion. Yin et al., CVPR 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you find our work simple yet effective,  enjoying satisfactory scalability, and helping the community to know dataset distillation could achieve promising results on large-scale datasets. We would like to address the comments and questions below. \n\n> W1. The technical novelty.\n\nThanks for your kind comments. We highlight our novel contributions in this work beyond data-free knowledge distillation and other related works such as DeepDream, Inverting Image, etc., in the following aspects:\n\n1. In the squeeze phase, we have crafted a new BN-based ViT architecture specifically for the task of data condensation. Interestingly, we've also observed that a higher-performing model in the squeeze phase does not necessarily translate to superior knowledge for the subsequent processes of recover and relabel. This is discovered by our conducted extensive experiments with various data augmentation techniques, which help to identify the most useful strategies that enhance the squeeze procedure.\n2. In the recover phase, we propose the new perspective that not all regularizers in image synthetic are useful for dataset condensation. We performed systematic ablation studies on various regularizers to understand their impact on relabeling and final training. Additionally, we introduced a simple yet highly effective multi-crop optimization technique, which significantly elevated the performance levels.\n3. In the relabel phase, we go beyond the straightforward application of FKD. We enhance it by developing a novel soft label storage mechanism. This innovative solution ensures FKD's compatibility with mixture-based data augmentation techniques like Mixup and CutMix, which is not supported in FKD's vanilla design.\n\n> W2 (1). Previous works typically use Convent for dataset distillation while this work mainly considers ResNet. I am not arguing that the setting must be the same. However, I do think that an ablation study is necessary to show the improvement coming from the architectures.\n\nThanks for the valuable comments. As suggested, we've incorporated additional ablation experiments to reconstruct images using ConvNet, as shown in the table below. To integrate BN-matching within $\\texttt{SRe2L}$, we've implemented BatchNorm operations post each convolutional layer. Subsequently, after training the ConvNet-BN model for 90 epochs, we achieved a well-optimized model boasting a Top-1 accuracy rate of 58.62% as its squeezed model. The results under IPC=10 across various post-training architectures still outperform TESLA by large margins,  highlighting $\\texttt{SRe2L}$'s proficiency even with the non-residual model. \n\n| Squeezed Model  (IPC=10)  | ResNet-18 | ResNet-50 | ResNet-101 |\n| ------- |:---------:|:---------:|:----------:|\n| TESLA  | 7.7  |  -  |  -  |\n| ResNet-18 (paper)  |  21.3 | 28.4 | 30.9 |\n| Our ConvNet  (relabel w/ itself) | 17.0 | 20.5 | 21.2 |\n| Our ConvNet (relabel w/ ResNet-18) | 12.8 | 15.7| 17.1 |\n\n> W2 (2). A sensitivity analysis with respect to $\\alpha_{BN}$ is necessary.\n\nThe experimental ablation results of $\\alpha_{BN}$ are presented in the table below. $\\alpha_{BN}=0.01$ achieves the highest accuracy on ResNet-101, which is adopted in our paper. While, $\\alpha_{BN}=0.1$ obtains slightly better performance on ResNet-18 and ResNet-50 architectures. We will include these ablation results in our revised paper.\n\n| $\\alpha_{BN}$ | ResNet-18 | ResNet-50 | ResNet-101 |\n| ------------- |:---------:|:---------:|:----------:|\n| 0.001         |   45.87   |   54.92   |   56.95    |\n| 0.01          |  46.75   |   55.62   | **60.81**  |\n| 0.1           | **47.83** | **56.19** |   58.36    |\n| 1.0           |   46.87   |   55.24   |   57.70    |\n\n> W2 (3). I do understand that the proposed method is mainly for large-scale datasets. But it is also interesting to dynamically increase the size of datasets and compare the performance with the existing methods.\n\nThank you for the insightful suggestion. As recommended, we undertook further experiments from two dimensions: increasing resolution and increasing the number of classes. The details are presented in the table below.\n\nFor the distillation of the *IN-1K-64x64* dataset, we adhered to the ResNet architecture previously employed for Tiny-ImageNet, adjusting the output dimension to accommodate 1K classes. When distilling the *IN-1K-112x112* dataset, we utilized the aforementioned adapted model and activated the max-pool layer to suit the 112 $\\times$ 112 resolution.\n\nFor the *IN-100-224x224* and *IN-10-224x224* datasets, we curated corresponding sub-datasets and aligned the standard ResNet's output dimension to match the respective class numbers. A trend emerged from our findings. As the number of classes decreased, accuracy exhibited an upward trajectory. This trend aligns seamlessly with established learning principles. The results from our experiments demonstrate that our approach is adaptable, catering not just to large datasets but also to datasets of diverse scales.\n\n| Dataset (IPC=50)| ResNet-18 | ResNet-50 | ResNet-101 |\n| ------- |:---------:|:---------:|:---------:|\n| IN-1K-224x224 (paper) | 46.75 |  55.62 | 60.81 |\n| IN-1K-112x112 | 34.15 | 42.76   | 45.25 |\n| IN-1K-64x64| 35.27 | 42.26 |   44.37    |\n|  |  |  |  |\n| IN-1K-224x224 (paper) | 46.75 |  - | - |\n| IN-100-224x224  | 52.70 | -  |  - |\n| IN-10-224x224  |  73.00 |  -  | -  |\n\n>L1. The authors should clarify more limitations of the specific solution they proposed instead of those general issues of dataset distillation.\n\nThanks for the suggestion. Beyond the usual constraints associated with dataset distillation, we've noticed an additional limitation within our approach that the proposed framework requires storage of extra soft labels for the synthetic dataset, leading to increased disk storage consumption. In other aspects, our approach shows significant advantages over other methods."}, {"review_id": "fjDZWoKA7l", "reviewer": "Reviewer_RJ8s", "summary": "This paper proposes a new dataset condensation termed Squeeze, Recover, and Relabel that decouples the bilevel optimization of model and synthetic data during training. Extensive experiments show the effectiveness and efficiency of the proposed method in several IPC settings.", "questions": "1, One naive question: why the condensed images could be used for training? I am not familiar with this topic.\n\n2, The stage-3 Relabel seems like the process of knowledge distillation. Could the author try another setting, like using R50 to teach R18?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, The paper is well-written and easy to understand.\n\n2, The proposed method becomes efficient due to the decoupled stages.\n\n3, The extensive experiments show the effectiveness of the method.", "weaknesses": "1, Albeit the computation and memory efficiency in the proposed method, the whole training time might be comparable to other methods like DM and MTT. \n\n2, Unclear illustration about BN layers in VIT. Is it means that the proposed method uses the BN layer to replace the LN?\n\n3, The lack of comparison between the IPC and directly sampling the same scale images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you find our work well-written and easy to understand, the proposed method is efficient with extensive experiments. We would like to address the comments and questions below. \n\n>W1. Albeit the computation and memory efficiency, the whole training time might be comparable to DM and MTT.\n\nThanks for pointing out this. We've broken down the whole training time consumption at each stage of our training process in the table below. The timing is evaluated on Tiny-ImageNet using the *ConvNet/ResNet-18* architectures with one RTX 4090 GPU.\n\n|  | Squeeze/Pretrain (h)| Recover/Synthesis (h)| Relabel/Distilling (h)| Total (h)|\n|:----- |:-------:|:------:|:-----:|:-----:|\n| DM| - | 23.78/99.88 |-| 23.78/99.88 |\n| MTT| 0.15/0.63  $\\times$ 100 |16.60/69.72| - | 31.60/132.72 |\n| TESLA | 0.15/0.63  $\\times$ 100 |16.93/71.11| 0.02/0.08| 31.95/134.19 |\n| Ours|0.37/1.54|1.98/8.33 |0.02/0.08|2.37/9.95|\n\nFrom this, it becomes evident that, even when considering the time spent on the squeeze/pre-training stage, our proposed framework still significantly surpasses the efficiency of other methods, such as DM and MTT.\n\nIn the squeeze phase, multiple squeezed models are necessary to be pretrained for sampling and matching multiple trajectories in MTT. However, only one squeezed model is required to match Batch Norm statistics in our method. Consequently, the squeezing phase in our method is considerably more time-efficient compared to that of MTT.\n\nIn the synthesis phase, we have included a time consumption comparison in Table 1 of our paper. This comparison illustrates the time taken to generate one Tiny-ImageNet image with a single iteration update. When considering the iteration settings for synthesis, both DM and MTT models train over 10K iterations. In contrast, our model requires only 4K iterations. Therefore, our model's reduced per-iteration synthesis times, resulting in a shorter total synthesis time.\n\n>W2. Unclear illustration about BN layers in VIT.\n\nThe vanilla ViT can be formulated as:\n\n$$\\mathbf z_{\\ell}^{\\prime}=\\operatorname{MHSA}\\left(\\mathrm{LN}\\left(\\mathbf z_{\\ell-1}\\right)\\right)+\\mathbf z_{\\ell-1}$$\n\n$$\\mathbf z_{\\ell}=\\operatorname{FFN}\\left(\\mathrm{LN}\\left(\\mathbf z_{\\ell}^{\\prime}\\right)\\right)+\\mathbf z_{\\ell}^{\\prime}$$\n\nwhere $\\mathbf z_{\\ell}^{\\prime}$ is the intermediate representation before Feed-forward Network ($\\operatorname{FFN}$), and $\\mathbf z_{\\ell}$ is that after $\\operatorname{FFN}$ and residual connection. $\\operatorname{FFN}$ contains two linear layers with a GELU non-linearity in between them, i.e.,\n\n$$\\operatorname{FFN}(\\mathbf z_{\\ell}^{\\prime})=\\left(\\operatorname{GELU}\\left(\\mathbf z_{\\ell}^{\\prime} W^1_\\ell+b^1_\\ell\\right)\\right) W_\\ell^2+b_\\ell^2$$\n\nThe newly constructed BN-ViT is:\n\n$$\\mathbf z_{\\ell}^{\\prime}=\\operatorname{MHSA}\\left(\\mathrm{BN}\\left(\\mathbf z_{\\ell-1}\\right)\\right)+\\mathbf z_{\\ell-1}$$\n\n$$ \\mathbf z_{\\ell}=\\operatorname{FFN_{BN}}\\left(\\mathrm{BN}\\left(\\mathbf z_{\\ell}^{\\prime}\\right)\\right)+\\mathbf z_{\\ell}^{\\prime}$$\n\nwhere we add one additional BN layer in-between two linear layers of $\\operatorname{FFN}$, i.e.,\n\n$$\n\\operatorname{FFN_{BN}}(\\mathbf z_{\\ell}^{\\prime})=\\left(\\operatorname{GELU}\\left(\\operatorname{BN}\\left(\\mathbf z_{\\ell}^{\\prime} W^1_\\ell+b^1_\\ell\\right)\\right)\\right) W_\\ell^2+b_\\ell^2\n$$\n\nWe will include these details in our revision.\n\t\t\t\t\t\n>W3. Comparison between the IPC and directly sampling the same scale images.\n\nThe results of directly sampling the same scale images are shown in the following table. As suggested by the reviewer, we randomly sampled 50 images per class as the pruned dataset. It can be observed that our dataset condensation results outperform the dataset pruning by large margins across various architectures.\n\n|  IPC=50  | ResNet-18 | ResNet-50 | ResNet-101 |\n|:--------|:--------:|:--------:|:---------:|\n| Random Dataset Pruning|27.47| 26.05 | 26.96 |\n| Dataset Condensation (Ours)|46.75| 55.62| 60.81|\n\n>Q1. One naive question: why the condensed images could be used for training? I am not familiar with this topic.\n\nWe understand this concern stems from the distinctive visual appearance of synthetic data. Despite the condensed images appearing quite different from the natural ones, they are engineered to encapsulate the essence or core characteristics of the original large dataset, particularly:\n\n1. Information Retention: These distilled images retain the important features and patterns required for effective training. In other words, despite their smaller size, distilled images retain crucial information from the original dataset. This information is what the model learns to recognize and apply when it sees new, similar data. \n\n2. Noise Reduction: Dataset condensation/distillation can also serve as a form of noise reduction, filtering out unnecessary or irrelevant information in the synthetic data and allowing the model to focus on the most salient features.\n\nFurthermore, the visualization of distilled examples in Figure 1 of our supplementary material, offers a straightforward interpretation. It reveals that numerous small areas, saturated with categorical features, are scattered throughout the image. This distribution significantly augments the image's expressiveness, enriching its visual representation during model training.\n\n>Q2. The stage-3 Relabel seems like the process of knowledge distillation. Could the author try another setting, like using R50 to teach R18? \n\nThanks for the suggestion. Indeed, we have accommodated a variety of results with diverse relabeling settings including the suggested *using R50 to teach R1* in Figure 3 of the paper. We have also included detailed configurations within the legends of each subfigure. For example, the notation $\\mathrm{T_{R50}S_{R18}}$ is used to indicate the use of a ResNet-50 model for relabeling synthetic data, which subsequently teaches the learning process of a ResNet-18 model."}, {"review_id": "fcNN0lmbn9", "reviewer": "Reviewer_MBn1", "summary": "This paper proposes a 3-step dataset condensation approach. Instead of applying bilevel optimization based approach in the previous work, the proposed method break down 3 decoupled steps: squeeze, recover and relabel. The key idea is decoupling the modeling training on real data and the generation of the synthetic data. At the first squeeze step, the model is trained on the original full dataset. At the second recovery step, synthetic data is generated using pretrained model and class prior with additional regularization (TV loss and BN consistency). At the 3rd stage, soft labels are generated using the pretrained model. And finally, a model on synthetic data is trained using the images from step 2 and labels from step 3.", "questions": "I am not familiar with the field, but I am not convinced about the overall problem being solved.\nIf we would like to condense the original dataset for efficient training, we should at least achieve comparable accuracy on retraining. This is not the case from the IN-1K result.\nIf we would like to obtain a way of retraining a new model without requiring original labeled dataset, semi-supervised learning on a unlabeled data would be a more promising direction instead of image synthesis.", "limitations": "Please see my concerns in the Questions section", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Compared with previous method on Tiny-IN & IN-1K, the proposed approach achieves superior performance. \nThe proposed method generates more visually appealing images from the example images in Fig 4.\nThe first work condenses the full IN-1K, based on the claim from the paper.\nEfficient way of using decoupled steps during the generation.", "weaknesses": "1. Lack of strong technical novelty. The proposed method combines a few prior work on image synthesis, such as deepDream & Inverting Image, and applies it directly into a new problem.\n2. Lack of comprehensive study with baseline approaches, and demonstration of the improvement of the key novelty. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments. We are appreciated that you find our work achieving superior performance with more visually appealing images. We would like to address the comments and questions below. \n\n>W1. Strong technical novelty. The proposed method combines a few prior work on image synthesis, such as deepDream & Inverting Image, and applies it directly into a new problem.\n\nThanks for the valuable feedback. We highlight our novelty in each stage of *squeeze*, *recover* and *relabel* comparing to prior methods such as DeepDream, Inverting Image, etc.\n\n1. In the squeeze phase, we have crafted a new BN-based ViT architecture specifically for the task of data condensation. Interestingly, we've also observed that a higher-performing model in the squeeze phase does not necessarily translate to superior knowledge for the subsequent processes of recover and relabel. This is discovered by our conducted extensive experiments with various data augmentation techniques, which help to identify the most useful strategies that enhance the squeeze procedure.\n2. In the recover phase, we performed systematic ablation studies on various regularizers to understand their impact on relabeling and final training. Additionally, we introduced a simple yet highly effective multi-crop optimization technique, which significantly elevated the performance levels.\n3. In the relabel phase, we go beyond the straightforward application of FKD. We enhance it by developing a novel soft label storage mechanism. This innovative solution ensures FKD's compatibility with mixture-based data augmentation techniques like Mixup and CutMix, which is not supported in FKD's vanilla design.\n\n>W2. Comprehensive study with baseline approaches, and demonstration of the improvement of the key novelty.\n\nThanks for the suggestion. In response, we have expanded our analysis to include additional comparisons with baseline approaches as detailed below:\n\nWe recover images from ConvNet in $\\texttt{SRe}^2\\texttt{L}$, the IPC equals 10 in post-training for the comparison with baseline method TESLA. We conducted two configurations: relabel using the same squeezed model and relabel using the large pretrained ResNet-18. The results are shown in the following table and our approach demonstrates superior performance over the baseline. The results also reflect the observation and novelty in our paper that a correct relabeling model is necessary for the final accuracy. We will include these comparisons in our revised paper.\n\n| Squeezed Model      | ResNet-18 | ResNet-50 | ResNet-101 |\n| ----------------------------------- |:---------:|:---------:|:----------:|\n| TESLA   (IPC=10)                    |    7.7    |    --     |     --     |\n| Our ConvNet (relabel w/ itself, IPC=10)   |   17.0\t|20.5\t| 21.2 |\n| Our ConvNet (relabel w/ ResNet-18, IPC=10) |   12.8 |\t15.7 | 17.1 |\n| Our ResNet-18 (IPC=10)                   |   21.3   |   28.4   |    30.9   |\n\t\n>Q1. I am not familiar with the field, but I am not convinced about the overall problem being solved. If we would like to condense the original dataset for efficient training, we should at least achieve comparable accuracy on retraining. This is not the case from the IN-1K result. If we would like to obtain a way of retraining a new model without requiring original labeled dataset, semi-supervised learning on a unlabeled data would be a more promising direction instead of image synthesis. \n\nThank you for your insights. While our proposed method may not completely resolve the dataset condensation problem, it significantly advances the ability to address this challenge within this domain:\n\n1. It is worth noting that ours is the only approach capable of distilling the entire ImageNet-1K under 224 $\\times$ 224 resolution, while still achieving commendable performance (60.8%) on the original validation set. It is an absolute 32.9% higher than the previous SOTA method TESLA (ICML 2023) using the same IPC training samples.\n\n2. Moreover, our compression rate is 25$\\times$. This implies that we are training our model on just 1/25 of the samples compared to conventional model training. Given this drastic reduction in training samples, it's reasonable that the performance may not fully reach that of standard model training.\n\nWe also clarify that our objective is not to pursue a method that eliminates the need of the original labeled dataset. Instead, our focus is on data condensation/distillation where the goal of this task is to achieve a similar level of accuracy as the original full dataset but with **much fewer samples**. This target makes data distillation work apart from semi-supervised learning, which has a different set of objectives.\n\nWe appreciate the reviewer's valuable feedback. We will persist in our efforts to enhance and optimize the performance of data distillation within this area.\n"}, {"review_id": "AAsi5GfKOc", "reviewer": "Reviewer_7q3V", "summary": "The paper addresses the dataset condensation task and proposes a new framework termed Squeeze, Recover and Relabel. In this three step approach, the authors first train a model from scratch to accommodate most of the crucial information from the original dataset. In the second stage, target data is synthesized from Gaussian noise. And in the final stage, the generated synthetic data is relabelled using a crop-level scheme to align with the true label of the data. Extensive and controlled experimentation showed significant performance improvement compared to previous state-of-the-art methods.", "questions": "The paper in its current form needs clarification on Cross-Architecture Generalization. Please refer to the “Weakness” section for details.\n\nThe authors acknowledge the performance disparity between condensed dataset in the limitation section. I feel that this might limit the adaptability of the proposed approach.\n\nMinor typo:\n1) Line 277 refers to Figure 3, it should refer to Table 5.\n2) “Deit” -> “DeiT”", "limitations": "Yes, the authors discuss the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This publication has several strengths including:\n\n1) The writing is clear and easy to understand.\n\n2) Generalizability of the framework to scale of datasets, input resolution, and the size of network architectures.\n\n3) Good experimental methodology with carefully designed ablations that justifies architectural design decisions especially impacts of  squeezing budget, diverse data augmentations for original data compression, recovery budget and regularization terms for data recovery and insights on model choice and training for relabeling process.\n\n4) Very exhaustive in-depth empirical comparison with state-of-the-art methods demonstrating strong performance as well as incur reduced compute and memory consumption.", "weaknesses": "I am confused by the claims in the section “Cross-Architecture Generalization”. The authors attribute the suboptimal performance of DeiT-Tiny on the condensed datasets due to the model’s inherent need for substantial training data. \n\nHowever, from Table 5 there appears to be a “cross-architecture gap”: ResNet based evaluation models perform poorly on condensed data based on ViT based squeezed model compared to when evaluated on condensed data based on ResNet based squeezed model. Similar observation for DeiT-Tiny evaluated on condensed data based on DeiT-Tiny-BN and ResNet-18 squeezed model. This shows that the condensed does not generalize well to different network architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments. We are encouraged that you find our work clear and easy to understand, and provide an exhaustive in-depth empirical comparison with state-of-the-art methods. We would like to address the comments and questions below.\n\n>Q1. Clarification on Cross-Architecture Generalization. \n\nThanks for the insightful suggestion. It is a well-known observation in prior dataset condensation/distillation works that performance degradation often results from a mismatch between the synthesis and final training architectures. Yet, as indicated in the following results, the gap between ViT and ResNet-18 in our case is significantly narrower compared to the previous method TESLA. This suggests that our approach will be better at managing this mismatch with stronger *Cross-Architecture Generalization* ability.\n\n|       | ViT  | ResNet-18 |        Gap        |\n|:----- |:----:|:---------:|:-------------|\n| TESLA | 11.0 |    7.7    | $\\downarrow$ 30.0%  |\n| Ours  | 25.4 |   24.7    | $\\downarrow$ 2.8% |\n\nMoreover, Figure 3 clearly exhibits the proficiency of cross-model generalization across ResNet-{18, 50, 101}, and Table 5 presents cross-architecture generalization between ViT and ResNet. We will make this clearer in our revision.\n\n>Q2. The authors acknowledge the performance disparity between condensed dataset in the limitation section. I feel that this might limit the adaptability of the proposed approach.\n\nThanks for your comments, we clarify the *adaptability* ability of our method manifests in at least two aspects: \n\n1. The proposed method is the only approach capable of distilling the entire ImageNet-1K under 224 $\\times$ 224 resolution, while still achieving commendable performance (60.8%) on the original ImageNet-1K validation set, which is 32.9% higher than the previous SOTA method TESLA (ICML 2023) using the same IPC of training samples. This has been shown the strongest *adaptability* ability in the task.\n\n2. Moreover, our compression rate is 25$\\times$. This implies that we are training our model on just 1/25 of the samples compared to conventional model training. This will be a good fit/adaptability for various resource-constrained training scenarios. Given this drastic reduction in training samples, the performance gap in comparison to full-data model training is surprisingly narrow.\n\t\t\t\t\n>Minor typo: 1. Line 277 refers to Figure 3, it should refer to Table 5.  2. “Deit” -> “DeiT” \n\t\t\t\t\nThanks for pointing them out. Line 277 refers to both Figure 3 and Table 5 in our paper. We have corrected the typos in our revision and will polish the whole paper thoroughly.\t\n\t"}]

Author rebuttal:
Dear Reviewers,

We would like to express our gratitude for your insightful feedback and comments, which have been helpful in updating and enhancing our submission. We kindly invite you to review our author rebuttal so that we may address any further questions you may have or clarify any points that remain unclear. In summary, our rebuttal mainly includes the following:

* We carry out theoretical analysis of generalization error bound through Input Compression Bound (ICB) and Mutual Information (MI) between the original/condensed input and the final layer representations. (Reviewer 4huB)
* We present results on ConvNet for dataset distillation, a sensitivity analysis with respect to $\alpha_{BN}$, and additional experiments on increasing resolution and increasing the number of classes. (Reviewer QGHe)
* We break down the whole training time consumption at each stage to demonstrate the superior efficiency of our proposed approach. (Reviewer RJ8s)
* We provide a detailed explanation of BN-ViT architecture. (Reviewer RJ8s)
* We provide the comparison between our dataset condensation and directly sampling the same scale of images. (Reviewer RJ8s)
* We expand our analysis to include additional comparisons with the baseline approach. (Reviewer MBn1)
* We present the comparison and discussion of the cross-architecture generalization ability. (Reviewer 7q3V)

We hope our responses can adequately address your concerns. We will integrate all the comments presented in the rebuttal into the revised paper, and we sincerely appreciate your valuable feedback.

Best,  
Authors

Meta-review: The paper was received well by the reviewers. Particularly, the reviewers commended 
- clear presentation,
- novel framework,
- impressive performances,
- and extensive experiments. 

Some of the issues, such as 
- limited theoretical analysis, 
- ablation studies, 
- and cross-architecture generalisation
are effectively addressed in the rebuttal. 

Our final decision is an acceptance. We kindly ask the authors to include the additional contributions made throughout the rebuttal process in the final version.
Decision: Accept (spotlight)