SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Discussion. Our method effectively takes advantage of the low intrinsic dimension of natural images, and has the following advantages: (i) The proposed low-rank DC plugin significantly improves condensing efficiency (i.e., fewer training parameters) and reduces the cost of synthetic dataset storage. This is particularly beneficial when the image height  $(H)$  and width  $(W)$  are very large. Due to the low intrinsic dimension of the image, the value of  $r$  can be very small, such as 2. (ii) Under the same memory consumption or the number of learnable parameters as traditional DC methods, our plugin can reconstruct a larger number of synthetic images, preserving sufficient information for the large training dataset  $\mathcal{T}$ . (iii) Our plugin is orthogonal to existing dataset distillation work and can be further combined with them.
3.3 Incorporating Low-rank DC Plugin to SOTA Methods
Our proposed low-rank manifolds DC plugin can be easily incorporated into existing DC solutions [56, 66, 8, 64, 65]. Without loss of generality, we integrate it into gradient matching-based DC [66] and distribution matching-based DM [65], and define the two new methods as Low-rank DC (LoDC) and Low-rank DM (LoDM), respectively. We verified the effectiveness of the low-rank plugin in Sec. 4.1. Additionally, in the appendix, we combine our low-rank DC plugin with more advanced dataset distillation methods, including MTT [8], IDC [23], HaBa [33], and RemenberThePast [13].
Low-rank DC (LoDC). To achieve the goal stated in Eq. 3, we match the gradients of the large dataset  $\mathcal{T}$  and the small dataset  $\Omega (\mathcal{A},\mathcal{B})$  within a low-rank manifold. Specifically, we use the synthetic dataset  $\Omega (\mathcal{A},\mathcal{B})$  to train a deep neural network  $\phi_{\theta}$  (the initial parameter is  $\theta_0$ ). After optimizing the network, we aim to minimize the distance between the gradient  $\nabla_{\theta}\mathcal{L}^{\mathcal{T}}(\theta_t)$  of the loss  $\mathcal{L}^{\mathcal{T}}$  w.r.t  $\theta_t$  on
Algorithm 2: LoDM: Low-rank Dataset Condensation with Distribution Matching [65]
Input: A large training dataset  $\mathcal{T}$    
Required: Randomly initialized  $\mathcal{A},\mathcal{B}$  of rank  $r$  for  $|Y|$  classes, the probability distribution of randomly initialized weights  $P_{\theta_0}$  , neural network  $\psi_{\theta}$  , number of iterations  $T_{ou}$  , learning rate  $\eta_{dc}$    
for iteration  $k = 1,\dots ,T_{ou}$  do Initialize  $\theta_0\sim P_{\theta_0}$  Initialize  $\mathcal{L}_s^{DM} = 0$  for class  $c = 1,\dots ,|Y|$  do Sample a minibatch real pair  $\mathbf{B}_c^{\mathcal{T}}\sim \mathcal{T}$  and a minibatch synthetic pair  $\mathbf{B}_c^{\Omega (\mathcal{A},\mathcal{B})}\sim \Omega (\mathcal{A},\mathcal{B})$  Compute  $\mathcal{L}_c^{DM}(\mathbf{B}_c^{\mathcal{T}},\mathbf{B}_c^{\Omega (\mathcal{A},\mathcal{B})}) =$  d  $\left(\frac{1}{N\tau}\sum_{(\mathbf{x}_i,y)\in \mathbf{B}_c^{\mathcal{T}}}\psi_{\theta_0}(\mathbf{x}_i),\frac{1}{N\mathcal{A}\mathcal{B}}\sum_{(\mathcal{A}_i\mathcal{B}_i,y)\in \mathbf{B}_c^{\Omega (\mathcal{A},\mathcal{B})}}\psi_{\theta_0}(\mathcal{A}_i\mathcal{B}_i)\right)$  Compute  $\mathcal{L}_s^{DM} = \mathcal{L}_s^{DM} + \mathcal{L}_c^{DM}(\mathbf{B}_c^{\mathcal{T}},\mathbf{B}_c^{\Omega (\mathcal{A},\mathcal{B})})$  Update  $\mathcal{A}\gets \mathcal{A} - \eta_{dc}\nabla_{\mathcal{A}}\mathcal{L}_{s}^{DM}$  and  $\mathcal{B}\gets \mathcal{B} - \eta_{dc}\nabla_{\mathcal{B}}\mathcal{L}_{s}^{DM}$  Output: A small dataset  $\Omega (\mathcal{A},\mathcal{B})$
the large dataset  $\mathcal{T}$  and the gradient  $\nabla_{\theta}\mathcal{L}^{\Omega (\mathcal{A},\mathcal{B})}(\theta_t)$  of the loss  $\mathcal{L}^{\Omega (\mathcal{A},\mathcal{B})}$  w.r.t  $\theta_t$  on the small dataset  $\Omega (\mathcal{A},\bar{\mathcal{B}})$  at step  $t$  as DC [66]. That is, we solve the following optimization problem:
$$
\min  _ {\mathcal {A}, \mathcal {B}} \mathrm {E} _ {\theta_ {0} \sim P _ {\theta_ {0}}} \left[ \sum_ {t = 1} ^ {T _ {i n}} d \left(\nabla_ {\theta} \mathcal {L} ^ {\mathcal {T}} \left(\theta_ {t} \mid \mathcal {T}\right), \nabla_ {\theta} \mathcal {L} ^ {\Omega (\mathcal {A}, \mathcal {B})} \left(\theta_ {t} \mid \Omega (\mathcal {A}, \mathcal {B})\right)\right) \right], \tag {4}
$$
where  $d(\cdot, \cdot)$  is a distance function,  $T_{in}$  is the number of training iterations,  $\mathcal{A}$  and  $\mathcal{B}$  are updated using gradients obtained by backpropagation in Eq. 4, and  $\theta$  is trained using the whole small dataset  $\Omega(\mathcal{A},\mathcal{B})$  by gradient descent optimization. We provide the pseudocode of our LoDC in Algorithm 1. Additionally, LoDC can be easily extended to DSA [64] methods that incorporate differentiable data augmentations into gradient matching [66].
Low-rank DM (LoDM). To achieve the goal of Eq. 3, our LoDM requires the distribution of the small dataset  $\Omega (\mathcal{A},\mathcal{B})$  to accurately approximate the distribution of the real training large dataset  $\mathcal{T}$  as DM [65]. Specifically, we first transform each input image  $\mathcal{A}_i\mathcal{B}_i\in \mathbb{R}^{(C\times H\times W)}$  into a different space through a family of parametric functions  $\psi_{\theta}\colon \mathbb{R}^{(C\times H\times W)}\to \mathbb{R}^{d'}$ . We then use the maximum mean difference (MMD) [16] to estimate the distance between the real and compact data distributions:  $\sup_{\| \psi_{\theta}\|_{\mathcal{H}}\leq 1}(\mathbb{E}\left[\psi_{\theta}(\mathcal{T})\right] - \mathbb{E}\left[\psi_{\theta}(\Omega (\mathcal{A},\mathcal{B}))\right])$ , where  $\mathcal{H}$  is reproducing kernel Hilbert space. As the real data distribution is unavailable, we utilize the empirical estimate of MMD as DM [65], that is, LoDM to solve the following optimization problem:
$$
\min  _ {\mathcal {A}, \mathcal {B}} \mathbb {E} _ {\theta_ {0} \sim P _ {\theta_ {0}}} \left[ d \left(\frac {1}{N _ {\mathcal {T}}} \sum_ {i = 1} ^ {N _ {\mathcal {T}}} \psi_ {\theta_ {0}} \left(\boldsymbol {x} _ {i}\right), \frac {1}{N _ {\mathcal {A B}}} \sum_ {i = 1} ^ {N _ {\mathcal {A B}}} \psi_ {\theta_ {0}} \left(\mathcal {A} _ {i} \mathcal {B} _ {i}\right)\right) \right], \tag {5}
$$
where  $N_{\mathcal{T}}$  and  $N_{\mathcal{AB}}$  represent the sample sizes of  $\mathcal{T}$  and  $\Omega (\mathcal{A},\mathcal{B})$  respectively. The low-rank matrices  $\mathcal{A}$  and  $\mathcal{B}$  are updated by performing a gradient descent on Eq. 5. DM [65] indicates that network  $\psi_{\theta_0}$  can perform well when a family of random initializations is adopted. Therefore,  $\theta_0$  does not need to be updated but is randomly sampled in each training iteration. We have adopted this default setting in this paper. Algorithm 2 provides the optimization process of our LoDM.
3.4 Application to Continual Learning
The goal of class continual learning (CL) is to use a model to learn from a continuously arriving sequence of new classes while retaining knowledge from previous classes [68, 58]. Mainstream memory-based CL methods typically involve carefully selecting a small number of old samples from a large training dataset and storing them in a limited memory buffer  $\mathcal{M}$  to mitigate catastrophic forgetting [59, 7, 41, 57]. The degree of forgetting of old classes is directly affected by the amount of data information in the samples stored in the limited memory buffer  $\mathcal{M}$ . Our dataset condensation plugin can compress a raw large dataset into a smaller dataset  $\Omega(\mathcal{A},\mathcal{B})$  by Alg. 1 or Alg. 2 to story in  $\mathcal{M}$ , thereby to preserve more information about older classes under strict memory constraints. In this work, similar to DM [65], we establish a baseline using the simple yet effective GDumb [41] approach
Table 1: Comparison with coreset selection methods and dataset condensation methods.
DataSet	Img/Cls	Ratio%	Coreset Selection Methods	Dataset Condensation Methods
Random	Herding	Forgetting	DD	LD	DC	DSA	DM	LoDM(Ours)
MNIST	1	0.017	64.9±3.5	89.2±1.6	35.5±5.6	-	60.9±3.2	91.7±0.5	88.7±0.6	89.7±0.6	91.2±0.4
10	0.17	95.1±0.9	93.7±0.3	68.1±3.3	79.5±8.1	87.3±0.7	97.4±0.2	97.1±0.1	96.5±0.2	97.7±0.1
50	0.83	97.9±0.2	94.8±0.2	88.2±1.2	-	93.3±0.3	98.8±0.2	99.2±0.1	97.5±0.5	98.2±0.1
CIFAR10	1	0.02	14.4±2.0	21.5±1.2	13.5±1.2	-	25.7±0.7	28.3±0.5	28.8±0.7	26.0±0.8	43.8±0.8
10	0.2	26.0±1.2	31.6±0.7	23.3±1.0	36.8±1.2	38.3±0.4	44.9±0.5	51.1±0.5	48.9±0.6	59.8±0.4
50	1	43.4±1.0	40.4±0.6	23.3±1.1	-	42.5±0.4	53.9±0.5	60.6±0.5	63.0±0.4	64.6±0.1
CIFAR100	1	0.2	4.2±0.3	8.4±0.3	4.5±0.2	-	11.5±0.4	12.8±0.3	13.9±0.3	11.4±0.3	25.6±0.5
10	2	14.6±0.5	17.3±0.3	15.1±0.3	-	-	25.2±0.3	32.3±0.3	29.7±0.3	37.5±0.8
TinyImageNet	1	0.2	1.4±0.1	2.8±0.2	1.6±0.1	-	-	4.61±0.2	4.79±0.2	3.9±0.2	10.3±0.2
10	2	5.0±0.2	6.3±0.2	5.1±0.2	-	-	11.6±0.3	14.7±0.2	12.9±0.4	18.3±0.3
Table 2: Comparison with dataset distillation methods on the same image (SI) or same memory (SM).
DataSet	Img/Cls	DC	LoDC(SI)	LoDC(SM)	DSA	LoDSA(SI)	LoDSA(SM)	DM	LoDM(SI)	LoDM(SM)
MNIST	1	91.7±0.5	-	93.0±0.3	88.7±0.6	-	90.6±0.6	89.7±0.6	87.0±0.7	91.2±0.4
10	97.4±0.2	96.0±0.2	97.6±0.3	97.1±0.1	95.3±0.2	97.7±0.1	96.5±0.2	92.0±0.6	97.7±0.1
CIFAR10	1	28.3±0.5	28.2±0.5	35.2±0.5	28.8±0.7	28.3±0.6	41.0±0.2	26.0±0.8	24.8±0.3	43.8±0.8
10	44.9±0.5	42.8±0.4	50.8±0.3	51.1±0.5	47.6±0.4	56.5±0.3	48.9±0.6	46.0±0.7	59.8±0.4
CIFAR100	1	12.8±0.3	12.3±0.2	18.6±0.1	13.9±0.3	13.5±0.2	22.9±0.2	11.4±0.3	9.6±0.4	25.6±0.5
10	25.2±0.3	24.4±0.2	27.1±0.6	32.3±0.3	29.8±0.1	33.7±0.7	29.7±0.3	27.7±0.2	37.5±0.8
TinyImageNet	1	4.61±0.2	4.21±0.2	6.80±0.1	4.79±0.2	4.70±0.3	9.90±0.4	3.9±0.2	3.6±0.1	10.3±0.2
in CL. GDumb first stores training samples in memory in a greedy manner while ensuring balanced samples per class. During testing, the model is trained from scratch exclusively using samples from memory  $\mathcal{M}$ . More specifically, we compare the effects of storing randomly selected sample sets [41], Herding sample sets [59, 44, 7], traditional DC condensed dataset (i.e., DC [66]/DSA [64], DM [65]), and our low-rank manifold condensed dataset (i.e., LoDC or LoDM in Sec. 3.3) in memory  $\mathcal{M}$  for CL in Sec. 4.2.
4 Experiment
In this section, we conduct experiments to verify the effectiveness of the proposed low-rank DC plugin. Due to space constraints, some experiment results are included in the Appendix.
4.1 Data Condensation for Deep Learning
Datasets. We evaluate our low-rank DC plugin on four benchmark datasets as DM [65], including MNIST [29], CIFAR10 [26], CIFAR100 [26], and TinyImageNet [28]. MNIST contains 60,000 grayscale images of size  $28 \times 28$ , totaling 10 classes. CIFAR10 and CIFAR100 contain 50,000 RGB images of size  $32 \times 32$ . The former has 10 classes, and the latter has 100 classes. TinyImageNet contains 100,000 images resized to  $64 \times 64$ , with a total of 200 classes.
Baselines. We compare two kinds of methods to reduce the amount of data: coreset selection (i.e., Random, Herding [59, 4], Forgetting [53]) and synthetic dataset distillation (i.e., DD [56], LD [5], DC [66], DSA [64], DM [65]). Specifically, Random sampling selects images randomly, Herding heuristically selects the sample closest to the center of the class, and Forgetting selects the sample that is most likely to be forgotten during model training.  $DD$  is one of the earliest works on dataset distillation, which updates the synthetic dataset by solving a bi-level optimization problem.  $LD$  performs label distillation rather than image distillation.  $DC$  performs gradient matching between large training dataset and condensed dataset, and DSA further considers differentiable siamese augmentation strategies based on DC.  $DM$  condenses datasets through distribution matching. We further combine the proposed low-rank plugin into DC, DSA and DM to obtain LoDC, LoDSA and LoDM, respectively. Comparisons with other dataset distillation methods (e.g., MTT [8], IDC [23], HaBa [33], and RememberThePast [13]) are provided in the appendix.
Experimental Details. In each experiment, we first choose the coreset (for Random, Herding, Forgetting) or learn a synthetic dataset (for DD, LD, DC, DSA, DM, LoDC, LoDSA, LoDM) and then employ it to train 20 randomly initialized networks (default ConvNet [46] architecture). By default, we set the rank  $r$  of our low-rank plugin to 2 for MNIST, CIFAR10, CIFAR100, and 4 for TinyImageNet. In a few cases, the rank  $r$  will be searched in  $\{1, 2, 4, 8\}$ . Other hyperparameters are the same as baselines [66, 64, 65]. We repeat the experiment multiple times for each method and
Table 3: Cross-architecture testing performance on CIFAR10 (10 images per class). LearnOn means condensing the dataset on this architecture, and TestOn means using the condensed dataset to train a new architecture.
Method	LearnOn \ TestOn	ConvNet	LeNet	AlexNet	VGG11	ResNet18
DSA	AlexNet	30.4±0.7	24.2±0.4	28.3±0.4	27.2±1.0	27.8±1.1
ConvNet	31.4±1.1	21.7±1.6	25.9±0.8	27.6±0.8	27.6±1.4
DM	AlexNet	41.4±0.8	31.4±0.2	37.5±0.9	36.8±0.5	34.9±1.1
ConvNet	42.2±0.5	33.4±0.6	38.8±1.3	36.2±1.0	34.6±0.5
LoDM(ours)	AlexNet	56.2±0.3	32.9±0.9	49.9±0.5	51.0±0.6	50.9±0.5
ConvNet	56.4±0.3	45.5±0.6	53.4±0.6	50.7±0.6	50.6±0.7
(b)
Figure 1: Distribution of real (all images) and synthetic images (50 images per class) on MNIST dataset: (a) Real, (b) DC, (c) Our LoDC  $(r = 8)$ , (d) DM, (e) Our LoDM  $(r = 2)$ , and (f) Our LoDM  $(r = 8)$ .
report the average test accuracy. Additionally, in Sec. 4.1.2, we also tested the cross-architecture effectiveness of the synthetic dataset on five standard deep network architectures: ConvNet [46], LeNet [29], AlexNet [27], VGG11 [52], and ResNet18 [19].
4.1.1 Compared to SOTA DC Baselines
We compare the performance of our LoDC, LoDSA, LoDM with other baseline methods under different sample sizes (1/10/50 image(s) per class) in Tab. 1 and Tab. 2. We have the following observations from Tab. 1: (i) When the sample size is small (e.g., 1 image per class), both Random and heuristic Forgetting sample selection perform poorly, significantly lower than Herding's method, because the latter samples best represent the class centers. As the sample size of each class increases, Herding's advantage becomes less obvious. (ii) The dataset condensation methods are significantly better than the coreset methods. For example, in CIFAR10 (10 images per class), Random and Herding only achieve  $26.0\%$  and  $31.6\%$  accuracy, while DD achieves  $36.8\%$  accuracy, DC, DM, and DSA achieve  $44.9\%$ ,  $48.9\%$ , and  $51.1\%$ , respectively. (iii) Our method condenses dataset in a low-rank manifold, effectively reducing memory cost (or the number of parameters to be optimized per image). Therefore, by utilizing the same memory, our low-rank LoDM can represent a more significant number of images, which is significantly better than other SOTA dataset compression methods, especially when the sample size of each class is small. For example, when using one image per class, our LoDM achieved accuracies of  $43.8\%$ ,  $25.6\%$ , and  $10.3\%$  on CIFAR10, CIFAR100, and TinyImageNet, respectively, while the best baseline achieved only  $28.8\%$ ,  $13.9\%$ , and  $3.9\%$ .
In Tab. 2, we compare the performance of our methods (LoDC, LoDSA, LoDM) in the low-rank manifold with traditional dataset distillation methods (DC, DSA, DM) using the same number of images (SI) or the same memory consumption (SM). It can be observed that: (i) Our method (rank  $r = 2$ ) reduces storage consumption by  $7 \times$ ,  $8 \times$ , and  $8 \times$  under MNIST, CIFAR10, CIFAR100 datasets, respectively, when the number of images is the same. However, the performance is still relatively close to the traditional DC method. We further explain the correspondence between rank size and memory cost in the appendix. (ii) Under the same memory consumption, our method can condense the large dataset into a smaller dataset with more samples, thereby preserving more information from the original large datasets. Therefore, compared to traditional condensing methods, our method exhibits a significant improvement. For example, on the CIFAR10 (10 Img/Cls) dataset, LoDSA and LoDM have shown an improvement of  $8.9\%$  and  $13.8\%$  compared to DSA and DM, respectively. Furthermore, as shown in Fig. 2(a), our LoDM method can achieve a higher final performance than DM after only 2,000 iterations, compared to 20,000 iterations.
4.1.2 Ablation Study
Cross-architecture Transferability Analysis. We verify that the condensed dataset learned in the low-rank manifold still has good cross-architecture transfer ability. We learn condensed dataset on a particular architecture (i.e., AlexNet/ConvNet), and then use the learned small dataset to train five different architectures (use Batch Normalization [21] as DM) from scratch, and finally verify on the test set of CIFAR10 dataset. As shown in Tab. 3, we observe that in each method (DSA, DM or
(a) iterations
(b) rank size
(c) learning rate
Figure 2: Ablation study on CIFAR10 (10 images per class): (a) Accuracy changes during DM, LoDM(SI) and LoDM(SM) iterations, (b) rank size of synthetic datasets, and (c) learning rate of synthetic datasets  $(r = 2)$ .
(a) real
Figure 3: Visualization of (a) real and (b) DM synthetic, and (c) LoDM(Ours) synthetic images on CIFAR10 dataset (10 images per class) and (d) LoDM(Ours) on MNIST dataset (10 images per class).
(b) DM
(c) LoDM
(d) LoDM
LoDM), the condensed data on AlexNet and ConvNet perform very similarly when training different architectures, so they have cross-architecture versatility. Furthermore, our LoDM method achieves the best performance in almost all architectures, benefiting from its ability to hold more images in low-rank space at the same memory cost.
Data Distribution Analysis. We verify that the condensed data learned in the low-rank manifold can also capture the characteristics of the original data distribution. We first train a ConvNet model using all the original data as a feature extractor. Then, we input the original images, DC, DM, LoDC, and LoDM learned data (50 images per class) into the network to extract features and perform dimensionality reduction through T-SNE [54] to visualize. As shown in Fig. 1, we observe that both DC and LoDC cannot capture the distribution of raw data well, since they aim to perform dataset distillation with the goal of gradient matching. DM considers distribution matching as the distillation goal, which captures the data distribution effectively. LoDM inherits the properties of DM, particularly when the rank increases, e.g., in Fig. 1(d) and Fig. 1(e), the rank is 2 and 8, respectively.
Hyper-parameter Analysis. We analyze the effect of two hyperparameters, rank size  $r \in \{1, 2, 4, 8\}$  and learning rate  $\eta_{dc} \in \{0.1, 0.5, 1, 2, 5\}$ , on CIFAR10 based on the LoDM method. Fig. 2(b) shows that when using the same number of images as DM, LoDM(SI) gradually achieves similar accuracy to DM in the high-dimensional pixel space, as the rank size increases. When the rank is 8, the performance is nearly identical to that of DM. We found that LoDM(SM) consistently outperforms DM when using the same memory, and the rank equal to 2 is a good choice based on empirical evidence. We also provide image visualizations generated by different ranks in the appendix. Fig. 2(c) shows that updating the generated data with a relatively large learning rate, such as 1 or 5, yields better results.
Visualization of Images. We visualize in Fig. 3 the condensed images generated by our LoDM method (rank  $r = 2$ ) for the CIFAR10 and MNIST datasets, ten images per class. We observe that compared to the images condensed by DM in high-dimensional pixel space, the images we recover under the low-dimensional manifold will be sharper, but still capture the main features of the class, especially on the MNIST dataset. This further verifies that images are locally connected and have lower intrinsic dimensions, so dataset condensation in traditional pixel space may be inefficient.
4.2 Data Condensation for Continual Learning
In this section, we apply the low-rank dataset distillation plugin to continual learning tasks. We perform class-incremental learning with strict memory constraints, specifically using 10 images per class for the CIFAR10 dataset and 20 images per class for the CIFAR100 dataset. Based on Sec. 3.4,
(a) 5-tasks on CIFAR10
(b) 5-tasks on CIFAR100
(c) 10-tasks on CIFAR100
Figure 4: Test accuracy on the class-incremental learning task.
we combine the coreset method (Random, Herding) and various dataset distillation methods (DSA, DM, and our LoDM) into a simple and effective CL method GDumb [41]. We conduct experiments on two benchmark datasets, CIFAR10 and CIFAR100, where the CIFAR10 dataset is divided into 5 tasks, and the CIFAR100 dataset is divided into 5 tasks and 10 tasks, respectively. Based on Fig. 4, we observe that in the three subfigures (a-c), GDumb+LoDM achieves the best results. For example, on CIFAR100 with 5 tasks, the final accuracies of GDumb+Random, GDumb+Herding, GDumb+DSA, GDumb+DM, and GDumb+LoDM are  $27.9\%$ ,  $27.0\%$ ,  $30.0\%$ ,  $33.81\%$ , and  $37.92\%$ , respectively. This suggests that our condensed data in a low-rank manifold is also meaningful for continual learning with limited memory.
5 Conclusion and Future Works
In this work, inspired by natural images that are locally connected and have low intrinsic dimensions, we propose a simple yet effective plugin that condenses a large dataset into a smaller dataset in a low-dimensional manifold. We apply this plugin to the existing dataset condensation methods and observe significant performance improvements while maintaining the same memory cost. Additionally, the analysis revealed that the dataset condensed in the low-dimensional manifold exhibits similar characteristics to the traditional high-dimensional pixel space dataset condensing method, such as matching the distribution of the large dataset and cross-architecture transferability. Furthermore, the plugin effectively addresses the issue of catastrophic forgetting in continual learning tasks. Our work has two directions for further improvement in the future: (i) Rank is a manually tuned hyperparameter in this paper. Therefore, how to adaptively assign the best rank to each dataset in the future is a feasible direction. (ii) We plan to apply the proposed dataset distillation plugin to more downstream tasks in the future, such as network architecture search, federated learning, and meta-learning.
Li Shen was supported by the STI 2030—Major Projects (No. 2021ZD0201405). Enneng Yang and Guibing Guo were supported by the National Natural Science Foundation of China under Grants No. 62032013 and No. 61972078, the Science and technology projects in Liaoning Province (No. 2023JH3/10200005), and the Fundamental Research Funds for the Central Universities under Grants No. N2217004 and No. N2317002. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
[1] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
[2] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pages 11849-11860. Curran Associates, Inc., 2019.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.