Heavy Labels Out! Dataset Distillation with Label Space Lightening

## Abstract

Abstract Dataset distillation or condensation aims to condense a large-scale training dataset into a much smaller synthetic one such that the training performance of distilled and original sets on neural networks are similar. Although the number of training samples can be reduced substantially, current state-of-the-art methods heavily rely on enormous soft labels to achieve satisfactory performance. As a result, the required storage can be comparable even to original datasets, especially for large-scale ones.
To solve this problem, instead of storing these heavy labels, we propose a novel label-lightening framework termed HeLlO aiming at effective image-to-label projectors, with which synthetic labels can be directly generated online from synthetic images.
Specifically, to construct such projectors, we leverage prior knowledge in open-source foundation models, e.g. , CLIP, and introduce a LoRA-like fine-tuning strategy to mitigate the gap between pre-trained and target distributions, so that original models for soft-label generation can be distilled into a group of low-rank matrices.
Moreover, an effective image optimization method is proposed to further mitigate the potential error between the original and distilled label generators.
Extensive experiments demonstrate that with only about 0.003% of the original storage required for a complete set of soft labels, we achieve comparable performance to current state-of-the-art dataset distillation methods on large-scale datasets.
Our code will be available.

## 1 Introduction

Dataset distillation Wang et al. () is proposed to deal with the issues caused by large-scale datasets, e.g., high computational overhead for training and heavy burden for storage and transmission.
It aims to condense a large dataset into a much smaller synthetic one, which preserves the original training performance, so that it can serve as an effective and efficient surrogate to train downstream neural networks.
For instance, it has been demonstrated that a network trained with merely 1 synthetic image per class (IPC) can perform well on CIFAR-10 Krizhevsky et al. ().
However, with such a high compression ratio, it is challenging for the distilled sets to encapsulate the whole knowledge of the original dataset used for training in a very limited space.
Thus, classic methods in this field like Wang et al. (); Zhao et al. (); Zhao & Bilen (; ) still have a significant performance gap between the original set and the synthetic one, especially when handling large-scale datasets Yu et al. ().

To compensate for such dramatic information loss, recent state-of-the-art dataset distillation methods Shao et al. (); Sun et al. (); Yin et al. () turn to data augmentation, to make the best use of the limited synthetic data.
Specifically, strategies such as Mixup Zhang et al. () and Cutmix Yun et al. () are applied in downstream network training, which effectively enhance the performance of distilled datasets and scale dataset distillation up to larger and more complex datasets like ImageNet Deng et al. ().

Nevertheless, these recent works heavily rely on soft labels generated by a pre-trained teacher model on the original dataset.
According to RDED Sun et al. (), networks trained with 10 IPC on ImageNet-1k achieve only 15.2% accuracy with categorical hard labels, compared to 42.1% with soft labels.
Since each augmented sample corresponds to a distinct soft label, as shown in Fig. 1 (left), there are a number of generated soft labels that far exceeds the basic synthetic samples.
Consequently, storage costs for these soft labels are non-negligible, especially for large-scale datasets with numerous categories.
For example, on ImageNet-1K with 1 IPC, the required storage for distilled images is $\sim 15$ MB, whereas the storage for soft labels exceeds 572 MB—more than 38 times greater.
Furthermore, with 200 IPC, the storage required for soft labels reaches 110 GB, making it even comparable to the original dataset size.

To address the issue of such heavy labels, we propose a novel label-lightening framework termed Heavy Labels Out, or HeLlO in short.
Fig. 1 (right) illustrates the overall framework of the proposed HeLlO.
By creating an effective and lightweight projector from the image to the label space, it reduces the required storage significantly.
Specifically, we build the projector upon recent foundation models like CLIP Radford et al. () that has been pre-trained on massive data and can readily adapt to various target datasets.
To achieve this, we propose an effective LoRA-like Hu et al. () knowledge transfer method that efficiently transforms the original feature space of CLIP into that of the target data.
As an efficient alternative to the teacher model trained on the target dataset for soft-label generation, the derived low-rank matrices can be seen as a transferable and lightweight representation for the original label space.

Interestingly, by leveraging the vision-language alignment capability in CLIP Zhang et al. (), we propose initializing the projector with the textual representation of label categories, providing a strong starting point that improves training and convergence.
Moreover, we propose an effective image optimization method to further reduce the potential error between the original and distilled label generators.
Our extensive experiments show that with only 0.003% of the original storage for soft labels, we achieve performance comparable to, or even better than, state-of-the-art large-scale dataset distillation methods.

Figure: Figure 1: The soft label generation part of the current state-of-the-art large-scale dataset distillation (left), and our proposed online lightening image-to-label projector framework (right). For the current state-of-the-art large-scale dataset distillation, for each downstream training epoch, soft labels are generated for each augmented image and stored all the soft labels. For our proposed method, we adopt the open-source foundation models as the base models, which are fixed during the whole training process, and introduce a LoRA-like knowledge transfer method to narrow the gap between the original label space and the target one. We only need to store the low-rank matrices, which significantly reduces the storage costs.
Refer to caption: x1.png

In summary, our contributions are as follows:

- •
We are the first to focus on the issue of heavy labels in dataset distillation to our best knowledge and propose an effective label-lightening framework termed HeLlO to address the problem.
- •
By leveraging pre-trained CLIP, the proposed HeLlO method compresses the storage of massive soft labels into a set of lightweight low-rank matrices and tailors an initialization method based on CLIP’s textual representation to enhance optimization.
- •
We introduce an image-level optimization technique that further minimizes the gap between the original and distilled label generators.
- •
Extensive experiments validate the comparable or even superior performance to state of the arts using just 0.003% of the storage required for synthetic labels.

## 2 Related Works

Dataset distillation or condensation Wang et al. (); Zhao et al. (); Yu et al. () aims to solve the issues of massive storage, transmission burden, and computational costs for downstream tasks caused by large-scale datasets. Specifically, it condenses the whole knowledge of the original large-scale datasets into a much smaller space and preserves the performance. Based on the optimization objectives, the mainstream dataset distillation methods can be roughly divided into three categories: performance matching Wang et al. (); Deng & Russakovsky (); Loo et al. (); Zhou et al. (); Nguyen et al. (; ), parameter matching Zhao et al. (); Zhao & Bilen (); Cazenavette et al. (); Cui et al. (); Du et al. (); Guo et al. (); Liu et al. () and distribution matching Zhao & Bilen (); Wang et al. (); Zhao et al. (); Sajedi et al. ().

Traditional dataset distillation methods suffer scaling-up problems due to the bi-level optimization problems, such that the gradients should backpropagate through an unrolled computation graph Yu et al. (). Recent work SRe^2L Yin et al. () proposes a variant distribution matching paradigm to decouple the bi-level optimization and scales up to the full-size ImageNet-1K dataset. It matches the distribution in feature space of the synthetic dataset and the statistical information of the original dataset stored in the batch normalization layers of the pre-trained model. Further, G_VBSM Shao et al. () utilize multiple pre-trained teachers to provide more statistical information and improve the transferability across different architectures. RDED Sun et al. () is the current state-of-the-art large-scale dataset distillation method, which is based on selection instead of synthesizing. It selects and concatenates the most representative patches evaluated by the pre-trained teacher model.

However, due to the significant reduction in the size of the datasets, an apparent performance gap still exists between the original dataset and the distilled one.
For small-scale dataset distillation, a series of works Bohdal et al. (); Sucholutsky & Schonlau (); Cui et al. (); Deng & Russakovsky (); Nguyen et al. (); Loo et al. (); Zhou et al. () expand the label space by transforming the one-hot labels to soft labels, which apparently improve the performance for downstream tasks and also provides a new perspective to condense dataset comprehensively.
However, simply transforming the one-hot label to a soft label for each synthetic image is not effective for large-scale dataset distillation, as the plain soft labels do not provide sufficient extra knowledge for downstream tasks.
In order to solve this issue and compensate for the huge reduction in the number of data, current large-scale dataset distillation methods Shao et al. (); Sun et al. (); Yin et al. () adopt the extensive data augmentation strategies, e.g., Mixup Zhang et al. () and Cutmix Yun et al. (), and generate soft labels for each augmented image.
It will increase the diversity of the distilled data for downstream training, and significantly improve the performance for downstream tasks. However, generating such labels requires restoring huge amount of soft labels, and for large-scale datasets, it will cause non-negligible storage costs. Focusing on this issue, our proposed method only requires 0.003% storage space while obtaining comparable performance with the state-of-the-art large-scale dataset distillation methods.

## 3 Methods

### 3.1 Preliminary

For the large-scale dataset $\mathcal{T}=(X_{t},Y_{t})$, where $X_{t}\in\mathbb{R}^{N_{t}\times D}$ and $Y_{t}\in\mathbb{R}^{N_{t}\times C}$, dataset distillation aims to learn a much smaller dataset $\mathcal{S}=(X_{s},Y_{s})$, where where $X_{s}\in\mathbb{R}^{N_{s}\times D}$ and $Y_{s}\in\mathbb{R}^{N_{s}\times C}$, such that the models train on both two datasets can obtain similar performance. Here, $N_{t}$ and $N_{s}$ refer to the number of samples in $\mathcal{T}$ and $\mathcal{S}$, $N_{t}\gg N_{s}$, and $D$ and $C$ are the dimension of the images and labels, respectively. Current state-of-the-art large-scale dataset distillation methods Shao et al. (); Sun et al. (); Yin et al. () all follow the real-time teacher(s)-guided soft label generation strategy. It generates a soft label for each augmented image, and the label space is expanded to $\mathbb{R}^{K\times N_{s}\times C}$, where $K$ is the number of training iterations for downstream tasks.

Here, the current large-scale dataset distillation methods can be formulated as follows:

$$ $\begin{split}X_{s}^{*}&=\mathop{\arg\min}\limits_{X_{s}}\mathcal{L}(\mathcal{S },\mathcal{T}),\\ Y_{s}^{*}&=\frac{1}{|\Theta_{\mathcal{T}}|}\sum f_{\theta\sim\Theta_{\mathcal{ T}}}(\mathcal{A}(X_{s}^{*})),\end{split}$ (1) $$

where $\mathcal{L}$ is the optimization objectives to update the distilled images, $\Theta_{\mathcal{T}}$ refers to teacher model(s) ($|\Theta_{\mathcal{T}}|\geq 1$), and $\mathcal{A}$ is the augmentation methods. Each augmented distilled image requires generating the corresponding soft labels, which will cause a huge storage burden.

### 3.2 Efficient Initialization of Surrogate Projection

To effectively and efficiently transfer the label space in a lightweight way and easily adapt it to different datasets, we adopt the open-source and pre-trained foundation model, CLIP, as our base model. It does not require extra storage space and can be accessed on demand. Specifically, we adopt the paradigm of linear probe CLIP by utilizing the image encoder part of CLIP and following with a linear transformation. The image encoder of CLIP is pre-trained on numerous paired data and can provide accurate and knowledge-rich features, which makes the linear probe CLIP a powerful classifier.
Here, the parameters required to store is only the linear transformation part.

However, the storage cost of the linear transformation part depends on the number of classes of the original dataset, which will be non-negligible for large-scale datasets with a large number of classes. Also, there still exists a gap between the original label space and the lightening one, which may make transferring to downstream tasks difficult. Here, to reduce the storage cost for the linear transformation part and improve the ability to transfer, we propose a novel storage-efficient initialization strategy.
Here, given a pre-trained multi-modal foundation model, e.g., CLIP, we denote the image encoder part as $\mathcal{E}_{I}$ and the text part as $\mathcal{E}_{T}$. For any dataset $\mathcal{D}=(X,Y)$, we can simply obtain the text descriptions $R=\{r^{(i)}\}_{i=0}^{C-1}$ for the whole dataset by utilizing the vanilla prompt engineering technique Radford et al. () with fixed templates.
We adopt the fixed normalized text embedding of the descriptions for all classes as the initialization of the linear transformation, which significantly saves storage space as we do not need to store the initial parameters. Also, it can improve the basic performance of our label projector as the proposed initialization is equivalent to the pre-trained zero-shot classification. Following we will provide the theoretical analysis.

###### Proposition 1 .

Text embedding initialized linear transformation is equivalent to the pre-trained zero-shot classification.

###### Proof.

For basic zero-shot CLIP prediction, we have:

$$ $\begin{split}&c^{*}=\mathop{\arg\max}\limits_{i\in\{0,\dots,C-1\}}\mathcal{S} im(x,r^{(i)}),\\ &\mathcal{S}im(x,r^{(i)})=v_{I}\cdot(v_{T}^{(i)})^{T},where\\ &v_{I}=\frac{\mathcal{E}_{I}(x)}{||\mathcal{E}_{I}(x)||},v_{T}^{(i)}=\frac{ \mathcal{E}_{T}(r^{(i)})}{||\mathcal{E}_{T}(r^{(i)})||},\end{split}$ (2) $$

where $x$ refers to the input image(s), $r_{i}$ is the text description for class $i$, and $v_{I}\in\mathbb{R}^{B\times d_{f}}$ and $v_{T}\in\mathbb{R}^{C\times d_{f}}$ refer to the normalized embedding for the input image and the text description for $i^{th}$ class. $B$ is the batch size of the input image(s), and $d_{f}$ is the dimension of the output embedding.
As for linear probe one, denote the parameters of the linear transformation is $W\in\mathbb{R}^{d_{f}\times C}$, $W=\{w^{(i)}\}_{i=0}^{C-1}$, and here, numerically, $W=(v_{T})^{T}$ and $w^{(i)}=(v_{T}^{(i)})^{T}$. The classification can be formally written as:

$$ $c^{*}=\mathop{\arg\max}\limits_{i\in\{0,\dots,C-1\}}v_{I}\cdot w^{(i)}+b,where \ w^{(i)}=(v_{T}^{(i)})^{T}.$ (3) $$

Here, we set the bias $b$ zero, and these two operations are equivalent.
∎

Figure: Algorithm 1 HeLlO Framework

###### Proposition 1 .

###### Proof.

### 3.3 LoRA-Like Low-Rank Knowledge Transfer

As mentioned before, we adopt the fixed initialization for the linear transformation part, which will not introduce any extra storage costs and can improve the basic classification ability of the linear probe CLIP. However, there still exists a significant gap between the original label space and the lightening one, which may cause difficulties transferring to downstream tasks.
Here, one typical way to solve the above issues is fine-tuning the whole projector to the target label space, but it requires huge extra computational costs to train the complex foundation model and non-negligible storage space to save the tuned parameters.

In order to reduce the computational costs and the storage costs, while narrowing the gap and further improving the transferability of the projector to the downstream tasks, we propose a novel parameter-efficient knowledge transfer method. First of all, to minimize the cost of fine-tuning, we follow the idea of LoRA Hu et al. (), which decomposes the weight matrix of the foundation models into low-rank matrices. It will preserve the pre-trained knowledge and enhance efficiency by reducing the number of updated parameters. Formally, for specific fine-tuning target $\mathcal{L}$, we have:

$$ $\begin{split}&\theta^{*}=\mathop{\arg\min}\limits_{\theta}\mathcal{L}(\mathcal {D};\theta),where\\ &\theta^{*}=\theta_{0}+\Delta\theta,\ \Delta\theta=A\cdot B.\end{split}$ (4) $$

Here, $\mathcal{D}$ refers to the target dataset, $\theta_{0}\in\mathbb{R}^{d\times k}$ is the initial pre-trained parameters of the model, and $\Delta\theta$ is the incremented weight, which is updated during the fine-tuning procedure. $A\in\mathbb{R}^{d\times r}$ and $B\in\mathbb{R}^{r\times k}$ are the decomposed low-rank matrices of $\Delta\theta$, where $r\ll d$ and $r\ll k$, largely relieving the computational and storage burden. Specifically, we apply LoRA to both the image encoder and the linear transformation parts (while with different ranks), avoiding fine-tuning the whole model and saving storage space.
Moreover, to further improve the transferability to the downstream tasks, we combine the original LoRA target optimization objective with the multi-teacher knowledge transfer metric as follows:

$$ $\begin{split}\mathcal{L}(\mathcal{D};\theta)=MSE(f_{\theta}(X),Y^{{}^{\prime}} )+\lambda CE(f_{\theta}(x),Y).\end{split}$ (5) $$

Here, $\theta$ is the parameters of the projector, $f_{\theta}(X)=\mathcal{E}_{I}(X)W$, and $Y^{{}^{\prime}}$ refers to the soft labels generated by the weak teachers $\Theta_{\mathcal{T}}^{{}^{\prime}}$, such that $Y^{{}^{\prime}}=\frac{1}{|\Theta_{\mathcal{T}}^{{}^{\prime}}|}\sum f_{\theta
\sim\Theta_{\mathcal{T}}^{{}^{\prime}}}(X)$. Practically, we adopt the original dataset $\mathcal{T}$ as the target dataset, and weak teachers are from the single training trajectory for easy to obtain and transfer.

### 3.4 Synthetic Dataset Initialization and Update

Here, we follow RDED Sun et al. (), to initialize the distilled dataset $\mathcal{S}$. Specifically, as the image patches can effectively represent object features, they select patches based on their difficulty and concatenate the patches to form an image. Specifically, they adopt the teacher model $\theta_{t}$ as the observer to evaluate the difficulty of the patches, and the most representative patches will be selected. The selection metric is as follows:

$$ $p^{*}=\mathop{\arg\min}\limits_{p\sim\mathcal{P}}CE(f_{\theta_{t}}(p),y_{p}),$ (6) $$

where $\mathcal{P}$ is a bunch of patches random cropped from the images of the original dataset $\mathcal{T}$, and $y_{p}$ is the corresponding labels of the original image. However, to reduce storage costs, we propose a surrogate parameter-efficient model to replace the original teacher model. This substitution introduced a performance gap, as the observer model is not the projector model for the downstream tasks. To narrow this gap, we further update the synthetic dataset to minimize the information loss of patches on the surrogate projector. Here, we follow LIC Anonymous () to do the image update, and the adapted optimization metric is as follows:

$$ $\mathcal{G}(\mathcal{E}_{I},p)=MSE(\mathcal{E}_{I}(p),\mathcal{E}_{I}(\hat{p})),$ (7) $$

where $\hat{p}$ is the transformed one with first down-sampled and then up-sampled to the original size. It will further reduce the information loss on the projector, and narrow the performance gap between the observer and the projector.

### 3.5 Algorithm Summary

In summary, we propose a novel label-lightening framework, HeLlO, building an effective and efficient image-to-label projection with lower storage requirements. The framework of HeLlO is shown in Algorithm 1.
Here, we first initialize the synthetic dataset $\mathcal{S}$ with the metric Eq. 6, which selects the most representative patches of the dataset. Then, we initialize the linear transform part using the normalized text embedding, generated by the fixed text descriptions and the pre-trained text encoder without any extra storage space. Following we adopt the LoRA-like knowledge transfer method Eq. 5 to efficiently fine-tune the projector with weak teachers’ guidance, and this step will only cause very low storage costs. As we use the projector to replace the observer model to relabel the images for downstream training, there exists a performance gap. To further narrow this gap and reduce the information loss on the projector model, we adopt Eq. 7 to update the synthetic data. Lastly, for the downstream training, the synthetic labels can be directly generated online from synthetic images through the projector.

## 4 Experiments

In this section, we conduct extensive experiments to show the effectiveness of our proposed method. Firstly, we compare the performance of our proposed method with the current state-of-the-art large-scale dataset distillation methods. Then we evaluate the cross-architecture generalization ability of our method with various architectures. We also conduct comprehensive ablation studies to show the efficacy of each step of our method and explore the impact of the key factors. Lastly, we also evaluate the performance of our distilled dataset applying to the continual learning task.

### 4.1 Experiment Setting

#### 4.1.1 Datasets and Networks

Our proposed method HeLlO aims to solve the heavy-label issue in the large-scale dataset distillation methods. Here, we adopt the ImageNet-100 and ImageNet-1K as the validation datasets to show the efficacy of our proposed method. Both of these two datasets are 224 $\times$ 224 in size.

As for networks, we adopt CLIP (ResNet-50) from the official Open-AI as the base model, followed by a linear transformation. For baseline comparison, we follow the prior works Yin et al. (); Shao et al. (); Sun et al. (), adopting ResNet-18 He et al. () as the evaluation model. Also, to show the generalization ability across various architectures of our proposed method, we select ShuffleNet-V2 Ma et al. (), MobileNet-V2 Sandler et al. (), EfficientNet-B0 Tan & Le (), Swin-V2-Tiny Liu et al. (), and VGG-11 Simonyan () as the evaluation architectures.

**Table 1: Comparison with baseline methods. ^∗ indicates the evaluation results reproduced by us, bold refers to the best results and underline refers to the second best results. Here, all methods adopt ResNet-18 as the evaluation model. Here, the #Params refers to the number of parameters of the teacher model(s) adopted during the downstream training, the size of labels refer to the soft labels required to store, for IPC 1, 10, and 50.**
| Datasets |  | SRe^2L Yin et al. () | G_VBSM Shao et al. () | RDED Sun et al. () | Ours |
| --- | --- | --- | --- | --- | --- |
| ImageNet-100 | 1 | 3.0 $\pm$ 0.3 | - | 8.1 $\pm$ 0.3 | 12.5 $\pm$ 0.2 (+ 4.4) |
| 10 | 9.5 $\pm$ 0.4 | - | 36.0 $\pm$ 0.3 | 48.9 $\pm$ 0.1 (+ 12.9) |  |
| 50 | 27.0 $\pm$ 0.4 | - | 61.6 $\pm$ 0.1 | 69.4 $\pm$ 0.1 (+ 7.8) |  |
| #Params | 10.7M | - | 10.7M | 0.6M (0.06$\times$) |  |
| Size of Labels | (5.7M, 57.2M, 286.1M) | (1e-1$\times$, 1e-2$\times$, 2e-3$\times$) |  |  |  |
| ImageNet-1K | 1 | 0.1 $\pm$ 0.1 | 1.7 $\pm$ 0.1^∗ | 6.6 $\pm$ 0.2 | 12.9 $\pm$ 0.3 (+ 6.3) |
| 10 | 21.3 $\pm$ 0.6 | 31.4 $\pm$ 0.5 | 42.0 $\pm$ 0.1 | 43.7 $\pm$ 0.1 (+ 1.7) |  |
| 50 | 46.8 $\pm$ 0.2 | 51.8 $\pm$ 0.4 | 56.5 $\pm$ 0.1 | 52.2 $\pm$ 0.1 (-) |  |
| #Params | 11.1M | 20.8M | 11.1M | 0.8M (0.07$\times$) |  |
| Size of Labels | (572.2M, 5722.0M, 28610.2M) | (1e-4$\times$, 1e-5$\times$, 3e-6$\times$) |  |  |  |

**Table 2: Evaluation results of cross-architecture generalization under the ImageNet-100 and ImageNet-1K with IPC 10 setting. ^∗ indicates the evaluation results reproduced by us.**
| Datasets |  | ShuffleNet-V2 | MobileNet-V2 | EfficientNet-B0 | Swin-V2-Tiny | VGG-11 |
| --- | --- | --- | --- | --- | --- | --- |
| ImageNet-100 | RDED | 27.7 $\pm$ 0.6^∗ | 35.7 $\pm$ 0.3^∗ | 37.9 $\pm$ 0.1^∗ | 18.0 $\pm$ 0.1^∗ | 21.2 $\pm$ 0.4^∗ |
| Ours | 32.7 $\pm$ 0.8 (+ 5.0) | 40.6 $\pm$ 0.8 (+ 4.9) | 47.0 $\pm$ 0.1 (+ 9.1) | 24.2 $\pm$ 0.3 (+ 6.2) | 27.6 $\pm$ 0.1 (+ 6.4) |  |
| ImageNet-1K | RDED | 23.3 $\pm$ 0.1^∗ | 34.4 $\pm$ 0.2 | 42.8 $\pm$ 0.5 | 17.8 $\pm$ 0.1 | 22.7 $\pm$ 0.1 |
| Ours | 26.5 $\pm$ 0.2 (+ 3.2) | 38.1 $\pm$ 0.5 (+ 3.7) | 44.4 $\pm$ 0.2 (+ 1.6) | 29.5 $\pm$ 0.1 (+ 11.7) | 24.2 $\pm$ 0.3 (+ 1.5) |  |

#### 4.1.2 Implementation Details

For surrogate projector training, we first initialize the linear transformation part with text embedding. We use the official prompt engineering templates provided by the CLIP code base to generate the text description and use the text encoder (from official CLIP with ResNet-50) to generate the text embedding. During the training process, we propose a LoRA-like knowledge transfer method to further improve the transferability of our method to the downstream tasks. Here, we efficiently fine-tune the convolution layer in the image encoder part and the linear transformation part. Specifically, for ImageNet-100, we use rank 8 for the image encoder part, and 64 for the linear transformation part, and for ImageNet-1K, we use rank 8 for the image encoder part, and 128 for the linear transformation part.
We also utilize multi-weak teachers as guidance to generate the soft labels for projector learning. In practice, we train a ResNet-18 model from scratch using the PyTorch official code base and select some checkpoints along the training trajectory. The teachers are in different stages for different IPCs, and we use 9 teachers for projector training. For more implementation details, please refer to the supplementary materials.

### 4.2 Results on Baselines

Our method aims to solve the heavy-label issues in the large-scale dataset distillation methods. Here, we compare our proposed method with prior state-of-the-art large-scale dataset distillation methods, SRe^2L Yin et al. (), G_VBSM Shao et al. (), and RDED Sun et al. (). Following the experiment setting with previous works and fair comparison, we use the distilled dataset to train several random initialized ResNet-18 from scratch, and the mean and standard deviation of the accuracy on the corresponding real test dataset are reported in Table 1. From the results, our proposed method only requires very low storage space costs for label generation that can get comparable performance.
Particularly for the smaller distilled dataset generation (smaller IPCs or classes), our proposed method demonstrates superior performance, achieving state-of-the-art results that exceed those of previous methods by a remarkable margin of up to 12.9% under the setting of ImageNet-100 with IPC 10. Moreover, it accomplishes this while simultaneously siginificantly reducing associated storage costs.

### 4.3 Results on Cross-Architecture Generalization

The ability to generalize to different architectures is an important standard to measure the performance of the distilled dataset, which shows the practicality to the downstream tasks. Here, we evaluate the cross-architecture performance of the previous state-of-the-art method RDED and our proposed method on the ImageNet-100 and the ImageNet with IPC 10. We adopt five different architectures ShuffleNet-V2, MobileNet-V2, EfficientNet-B0, Swin-V2-Tiny, and VGG-11. The results are shown in Table 2. From the results, our proposed method demonstrates state-of-the-art performance across various architectures. For residual-like architectures (ShuffleNet-V2, MobileNet-V2, and EfficientNet-B0), and convolutional networks (VGG-11), our proposed method shows a superior transferability than the previous state-of-the-art method RDED.
Surprisingly, our method demonstrates exceptional transferability on transformer architectures, surpassing the previous state-of-the-art by an impressive margin of 6.2% (ImageNet-100) and 11.7% (ImageNet-1K) on Swin-V2-Tiny, while only requires very low storage costs for the label generation.

**Table 3: The results of the ablation studies for the effectiveness of each step of our proposed method. From left to right, each step is incremented based on the former one.**
|  | Probe Linear CLIP | + Multi-Weak-Teacher Guided | + LoRA-Like Knowledge Transfer | + Text-Embedding-Based Init. | + Image Update |
| --- | --- | --- | --- | --- | --- |
| Acc. | 28.2 $\pm$ 0.2 | 30.1 $\pm$ 0.1 (+ 1.9) | 43.5 $\pm$ 0.1 (+ 13.4) | 43.6 $\pm$ 0.1 (+ 0.1) | 43.7 $\pm$ 0.1 (+ 0.1) |
| #Params | 1.0M | 1.0M (-) | 1.5M ($\uparrow$ 0.5) | 0.8M ($\downarrow$ 0.7) | 0.8M (-) |

**Table 4: The results of the ablation studies for the impact of the different learnable parameters in LoRA-like transfer learning (left), and the different stages of teachers (right). The experiments are conducted under the setting of IPC 10 for ImageNet-100 and ImageNet-1K.**
|  | 0.3M / 0.4M | 0.6M / 0.8M | 0.9M / 1.2M | 1.2M / 1.6M | 1-41 | 11-51 | 21-61 | 31-71 | 41-81 | 51-91 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ImageNet-100 | 45.4 $\pm$ 0.2 | 48.9 $\pm$ 0.1 | 49.7 $\pm$ 0.1 | 49.8 $\pm$ 0.2 | 48.9 $\pm$ 0.1 | 47.5 $\pm$ 0.1 | 45.5 $\pm$ 0.1 | 44.2 $\pm$ 0.1 | 43.9 $\pm$ 0.7 | 41.5 $\pm$ 0.4 |
| ImageNet-1K | 38.8 $\pm$ 0.1 | 43.7 $\pm$ 0.1 | 44.2 $\pm$ 0.1 | 44.4 $\pm$ 0.1 | 42.3 $\pm$ 0.1 | 43.7 $\pm$ 0.1 | 42.1 $\pm$ 0.1 | 41.2 $\pm$ 0.1 | 40.6 $\pm$ 0.1 | 39.7 $\pm$ 0.3 |

### 4.4 Ablation Study

In this section, we will conduct comprehensive ablation studies to thoroughly evaluate the improvements in performance achieved by our method. Also, we provide a detailed analysis of the impacts of the key factors.

#### 4.4.1 The Impact of Key Factors

To validate the effectiveness of our proposed method, we designed a series of ablation experiments to evaluate each component of our method. The results are shown in Table 3. Here, we start from the plain linear probe CLIP. We directly use the original dataset to train the linear probe CLIP and use it to online generate the labels during the downstream tasks training. As the results shown in Table 3, it only obtains 28.2% accuracy, while requiring 1.0M parameters to store. Based on that, we adopt multi-weak teachers to guide the linear probe CLIP training, which gains 1.9% improvement and maintains the storage costs.
Then, we introduce the LoRA-like knowledge transfer method, which significantly improves the performance of downstream training by 13.4% but causes an increase in storage. Following we propose the text-embedding-based initialization strategy, such that we do not need to store the whole linear transformation part but the low-rank matrices. It helps largely reduce the storage costs by 0.8M while maintaining the performance. Lastly, we narrow the gap of the original distribution and the target one by updating the images, which improves the performance of the distilled dataset.

#### 4.4.2 The Impact of Different Rank

We also explore the impact of ranks of the low-rank matrices in the LoRA-like knowledge transfer part. It also reflects the relation between the number of learnable parameters and the performance. The results are shown in Table 4 (left). From the results, we find that the ranks of the low-rank matrices or the number of learnable parameters can significantly influence the performance of the downstream tasks. However, this effect is pronounced only when the number of learnable parameters is insufficient; once a sufficient level is reached, further increases in learnable parameters do not lead to notable improvements in performance. The inflection point in the results occurs at 0.6M/0.8M for ImageNet-100 and ImageNet-1K. This also indicates that our method is robust to the selection of the ranks; as long as ranks reach a sufficient level, the results remain stable without significant fluctuations.

#### 4.4.3 The Impact of Different Stages of Teachers

In our proposed method, we adopt multi-weak teachers to guide the projector training. Here, we explore the impact of the stage of the teachers on the performance of the downstream tasks. Here, the experiments are under the setting of IPC 10 for both ImageNet-100 and ImageNet-1K. The results are shown in Table 4 (right). The results indicate that the stage of the teachers has a particularly significant impact on the performance of the downstream tasks. For smaller IPCs, earlier-stage teachers are more beneficial for transferring to downstream tasks. In contrast, later-stage teachers tend to contain more complex knowledge that is difficult to decouple and learn effectively.

Figure: Figure 2: The results on the continual learning for 5-step (left) and 10-step (right). All experiments are conducted under the setting of IPC 10 for ImageNet-100.
Refer to caption: x2.png

### 4.5 Results on Continual Learning

Continual learning De Lange et al. (); Wang et al. (); Rebuffi et al. () is an important application for dataset distillation Yu et al. (). Here, for fair comparison, we follow the previous works Zhao & Bilen (); Yin et al. (), adopting the GDumb Prabhu et al. () framework to evaluate the performance on continual learning. The experiments are conducted under the setting of ImageNet-100 with IPC 10, and we evaluate both the 5-step and the 10-step settings. The results are shown in Fig. 2. From the results, our proposed method is significantly superior to the previous state-of-the-art method RDED.

## 5 Conclusion

In this paper, we propose a novel label-lightening framework termed HeLlO, aiming to solve the heavy-label issue in large-scale dataset distillation. Our method involves an effective image-to-label projector, with which the synthetic labels can be directly generated online from synthetic images during training downstream networks. Specifically, we leverage the prior knowledge in open-source foundation models and introduce a parameter-efficient LoRA-like fine-tuning method to narrow the gap between the label distribution of the pre-trained and target ones, which improves the transferability of the projector to the downstream tasks as well. Moreover, we propose a text-guided initialization strategy for the projector that enhances training. To further address the gap between the original label generator and the projector, we also develop a strategy to optimize synthetic images within the projector. Extensive experiments demonstrate that the proposed HeLlO achieves performance comparable or even superior to current state-of-the-art dataset distillation techniques while using just about 0.003% of the original label storage space.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Why there is no 'Size of Labels' for L Yin et al. (2024) and Sun et al. (2024)
soundness: 3
strengths: 1. The motivation of this paper is clear and easy to understand.
2. The proposed method is simple and effective.
summary: This paper addresses the high storage cost of soft labels in dataset distillation that generates synthetic labels directly from images. It leverages open-source models like CLIP and a LoRA-like fine-tuning strategy to extract the labels, bridging the gap between pre-trained and target distributions.
weaknesses: I am convinced of the motivation of this paper, but there are significant concerns on using CLIP to generate labels from images:

1. Why not use stronger supervised models or self-supervised models? If we can use another stronger model like CLIP to generate labels from images, why not use it directly for the target task, or replace CLIP (weakly-supervised) with other models like supervised ViT or self-supervised DINOv2? Authors finetune CLIP vision encoder with LoRA, and text embeddings to adapt the specific dataset, which can be seen as a kind of supervised trained model. If we change it to supervised ResNet, ViT, or self-supervised DINOv2 (versus the performance reported with R18 from scratch), the performance may be better. Similarly, why not finetune the foundation models (CLIP or DINOv2) with LoRA on the distilled datasets if we have already used a foundation model?

2. The effectiveness of the proposed method. The storage of classifier weights is not important compared to labels, only 0.7MB in Table 3. In addition, the improvements of Text-Embedding-Based Init and Image Update are quite marginal (0.1 in accuracy). Most improvements come from LoRA-Like Knowledge Transfer, that is to say, adapting the supervised information of the dataset to CLIP vision encoder. 'M' is a bit confusing, please change 'M' to 'MB' in the paper.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: - Ambiguous notations
    1. In line 198, $r_i$ is described as the text description for class $i$. What is $r^{(i)}$ in Equation (2)?
    2. In line 199, $v_T$ is used for the text description of the $i$-th class. What is the meaning of $v_T^{(i)}$ in Equation (3)?
    3. In line 201, $W$ is defined as a matrix of size $d_f \times C$, yet the authors also write $W = \textbraceleft w^{(i)} \textbraceright_{i=0}^{C-1}$, which suggests $W$ is a set. Could the authors clarify this discrepancy?
- What is the definition of weak teacher in line 237? Does is differ from the teacher defined in line 162?
- The reviewer is curious about the potential outcomes of using a model pre-trained on the original real dataset as the label projector. Could the authors provide more insights or results related to this scenario?
soundness: 1
strengths: The work effectively tackles the critical issue of high storage requirements for soft labels.
summary: This work introduces HeLlO (Heavy Labels Out), a framework designed to address the challenge of high storage requirements for soft labels in large-scale dataset distillation. Instead of storing the entire soft label, an off-the-shelf pre-trained feature extractor is employed to generate labels in an online fashion during downstream network training. This approach significantly reduces the storage burden, especially when dealing with extensive datasets that necessitate numerous soft labels. Moreover, to mitigate the distribution gap between the target label space and the label space inherent to the pre-trained model, a LoRA-like fine-tuning strategy is applied.
weaknesses: - The comparison presented in Table 1 appears unfair. Unlike the baseline methods, the proposed method requires storing a teacher model (or its low rank version), which complicates a direct comparison. It is unclear why the storage size of the teacher model is relevant here. Please provide a more comprehensive comparison that includes the storage requirements for both the teacher model and the labels across all methods.
- Table 1 contains numerous missing values, which hinders a comprehensive comparison. The reviewer encourages the authors to provide complete results. At a minimum, please include the IPC 50 results for SRe2L and G-VBSM on the Places365-Standard dataset, as the proposed method performs lower than the baseline in this setting.
- Poor writing, there are many typos and ambiguous notations in the proof of Proposition 1, e.g. both $r_i$ and $r^{(i)}$ represent the text description for class $i$. Detailed feedback on these issues is provided in the questions part.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Some questions I think that could be helpful here:

1. Can we apply this to more complex tasks (like segmentation of vision language models)? This would widely increase the scope of the paper, and understandably storage requirements on these tasks would be significantly more prominent (i.e. segmentation would have softlabels for each pixel so the computational costs would be far larger).

2. How does the method compare with naive compressions such as quantization etc. 

I think the biggest things for the authors to focus on during the rebuttal/discussion phase should be comprehensive comparisons against existing storage compression techniques (particularly simple ones, such as quantization or even interpolation). Additionally, I would find this work's significance far more compelling as it has been applied to a task where the storage costs of soft labels are a clear bottleneck, like semantic segmentation.
soundness: 3
strengths: I do think this paper approaches a problem that isn't really being avidly researched in the field of data distillation. In particular, most works aim to reduce the costs associated with the number of images, or the learning paradigms for distillation. However, this work looks at an add on to existing distillation techniques in an effort to reduce the storage requirements of the labels. From a novelty perspective, I would agree that this paper incorporates new or innovative techniques into solving a unique problem. I believe there is value in the idea of an image-label-projector, particularly in distillation, and it is interesting to see how CLIP and LoRA have influenced the author's designs, showing a blend of recent developments in adjacent domains. The results are quite convincing, and the ablations studies appear throughout, including downstream applications, such as continual learning. I believe this contribution has strengths primarily in novelty and target scope. However, the latter is also a weakness I elaborate on below.
summary: This paper proposes an interesting alternative to label storage in the context of data distillation. By developing image-to-label projectors based on existing works in CLIP and LoRA, their proposed method, HeLlo can dramatically reduce the original storage requirements of the soft label set by generating them in an online fashion. Understandably with large datasets (i.e. large number of classes), the size of softlabels will scale linearly and can create storage bottlenecks for further downstream usage of the distilled data. Experimentally, we see that the proposed method can far surpass the performance of SoTA work RDED on the distillation benchmarks -- including a variety of different downstream architectures (both convolutional and transformer).
weaknesses: Despite the interesting approach taken in this paper, I find there to be a few crucial weaknesses that may overshadow the benefit of the approach. Data Distillation computational costs are often computed regarding the size/number of images as these often take up more storage space than the labels. Understandably at lower image/class ratios, this distribution may deviate (as soft labels would remain scaled at the number of classes, say 1K on ImageNet) -- however I am not convinced that the storage cost of soft labels is indeed that significant. In particular, the asses data distillation methods are on Image Classification, hence each image is associated with one vector of soft labels. Thus at a reduced number of images, 1 or even 50 images per class, the incurred cost of soft labels is actually quite small, (appears to be a couple GB's on ImageNet1K at 10 ipc). Finally, I think alternate methods of logit compression should be compared, such as simple quantization of the logits to a lower precision, as they may not indeed cause significant performance degradation, but will greatly reduce the storage costs. I think the primary weakness here is whether the contribution in this paper actually solves a problem that is being faced in data distillation. As currently I think the scope is not actually a commonly faced problem in the field.

Additionally, I think the professionalism/clarity of the figures can be improved, particularly Figure 1.0, where certain words are covered by symbols, etc.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Does the method demonstrate cross-architecture generalization capability on ViT models?
2. Can HeLlO maintain strong performance at higher IPCs, such as 100 or 200?
3. How does the proposed framework perform on other downstream tasks like object detection or segmentation?
soundness: 2
strengths: [S1] This paper tackles one of the key challenges in efficient training and dataset distillation (DD).

[S2] The writing is clear and easy to follow.

[S3] The results show comparable performance to some of the state-of-the-art methods on large-scale datasets, but the experiments could be more extensive, especially in terms of comparisons and generalizability.
summary: Dataset distillation aims to condense large training datasets into smaller synthetic ones, but current methods still demand substantial storage for soft labels. The proposed method, HeLlO, tackles this issue by generating synthetic labels directly from images using foundation models like CLIP and fine-tuning with low-rank matrices, significantly cutting down storage requirements. This approach delivers performance on par with state-of-the-art methods while using just 0.003% of the original storage.
weaknesses: [W1] There are concerns about the method's generalizability, especially regarding its applicability to high-resolution datasets and various model architectures. I suggest testing DRUPI on 256x256 ImageNet-1k, as well as practical datasets like clinical and aerial images, to better assess its versatility. Additionally, evaluating its performance on ViT models would help determine how well the method generalizes across different architectures.

[W2] An analysis of distillation costs is missing from the paper, which is crucial for assessing the efficiency of data-efficient training algorithms. A detailed comparison of the training costs for both the proposed method and state-of-the-art (SOTA) approaches, especially in terms of memory usage and training time, is necessary.

[W3] The paper lacks comparisons with key state-of-the-art methods such as DREAM, DataDAM, and SeqMatch in Table 2. Adding these comparisons would offer a more complete evaluation of DRUPI's effectiveness. If these methods were excluded for specific reasons, it would be helpful for the authors to explain their omission and clarify how DRUPI compares to these recent approaches.

**References**:
- [a] Liu, Yanqing, et al. "DREAM: Efficient dataset distillation by representative matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
- [b] Sajedi, Ahmad, et al. "DataDAM: Efficient dataset distillation with attention matching." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
- [c] Yin, Zeyuan, et al. "Squeeze, recover and relabel: Dataset condensation at ImageNet scale from a new perspective." Advances in Neural Information Processing Systems 36 (2024).

Meta-review: The paper introduces an innovative method to tackle the high storage costs related to soft labels in dataset distillation by employing open-source models and a fine-tuning strategy to derive labels directly from images. However, the primary concerns revolve around the reliance on CLIP for label generation and doubts about the comparative effectiveness of the proposed method when pitted against more robust supervised or self-supervised models. In addition, the paper's contribution is considered marginal, with minimal gains in accuracy and an emphasis on classifier weights storage that may not be as significant as portrayed.

Decision: Reject