Title: An Efficient Dataset Condensation Plugin and Its Application to Continual Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Data Condensation

Continual Learning

Few-shot Learning
Abstract: Dataset condensation (DC) distills a large real-world dataset into a small synthetic dataset, with the goal of training a network from scratch on the latter that performs similarly to the former. State-of-the-art (SOTA) DC methods have achieved satisfactory results through techniques such as accuracy, gradient, training trajectory, or distribution matching. However, these works all perform matching in the high-dimension pixel spaces, ignoring that natural images are usually locally connected and have lower intrinsic dimensions, resulting in low condensation efficiency.  In this work, we propose a simple-yet-efficient dataset condensation plugin that matches the raw and synthetic datasets in a low-dimensional manifold. Specifically, our plugin condenses raw images into two low-rank matrices instead of parameterized image matrices. Our plugin can be easily incorporated into existing DC methods, thereby containing richer raw dataset information at limited storage costs to improve the downstream applications' performance.  We verify on multiple public datasets that when the proposed plugin is combined with SOTA DC methods, the performance of the network trained on synthetic data is significantly improved compared to traditional DC methods. Moreover, when applying the DC methods as a plugin to continual learning tasks, we observed that our approach effectively mitigates catastrophic forgetting of old tasks under limited memory buffer constraints and avoids the problem of raw data privacy leakage.

An Efficient Dataset Condensation Plugin and Its Application to Continual Learning

Abstract

Dataset condensation (DC) distills a large real-world dataset into a small synthetic dataset, with the goal of training a network from scratch on the latter that performs similarly to the former. State-of-the-art (SOTA) DC methods have achieved satisfactory results through techniques such as accuracy, gradient, training trajectory, or distribution matching. However, these works all perform matching in the high-dimension pixel space, ignoring that natural images are usually locally connected and have lower intrinsic dimensions, resulting in low condensation efficiency. In this work, we propose a simple-yet-efficient dataset condensation plugin that matches the raw and synthetic datasets in a low-dimensional manifold. Specifically, our plugin condenses raw images into two low-rank matrices instead of parameterized image matrices. Our plugin can be easily incorporated into existing DC methods, thereby containing richer raw dataset information at limited storage costs to improve the downstream applications' performance. We verify on multiple public datasets that when the proposed plugin is combined with SOTA DC methods, the performance of the network trained on synthetic data is significantly improved compared to traditional DC methods. Moreover, when applying the DC methods as a plugin to continual learning tasks, we observed that our approach effectively mitigates catastrophic forgetting of old tasks under limited memory buffer constraints and avoids the problem of raw data privacy leakage.

1 Introduction

Modern machine learning techniques utilize large-scale real-world datasets and advanced deep neural networks (DNNs) to achieve amazing success in various fields, such as models like SAM [24] and GPT [6, 38] in the fields of computer vision and natural language processing, both of which have surpassed classical models [51, 42, 43] trained on small datasets. However, training a wellperforming model in the machine learning community requires repeated tuning of various aspects of the model [14], such as the number of layers, learning rate, and other important hyper-parameters. When the dataset is large, the cost of data management and repeated model training becomes unacceptable. As stated in [65], NAS-Bench-101 [63] spent 100 TPU-years of computing time on the CIFAR10 dataset [26] for an exhaustive neural architecture search. In contrast, it only takes dozens of TPU minutes to train the best model from scratch using the optimal configuration discovered [65]. Additionally, in continual learning [41, 2, 60, 61], to prevent forgetting old tasks while learning new tasks, a certain amount of old task data is typically stored for replay [58, 57, 2, 41]. Nevertheless, storing large old task datasets is unrealistic due to strict memory and privacy constraints. Therefore, reducing the data size becomes a valuable and emerging research direction.

A direct way to reduce data size is to select a representative subset from the original dataset [17]. This paradigm calculates the importance score for each sample based on specific importance criteria and selects a subset to replace the entire training dataset [62]. The calculation criteria include the distance between the sample and the class center [10], the gradient norm of sample [39], and the diversity among samples [47, 57], etc. However, the selection-based method is not always effective, particularly when the task-condition data information is uniformly distributed in the original data [30]. In such cases, the heuristic coreset selection method can only capture a limited amount of information. Recent research [56] has shown that generating a small dataset that performs similarly to the original dataset, instead of selecting a subset, is a promising direction. Therefore, dataset condensation (DC) or dataset distillation (DD) [56, 66, 64, 65, 37, 11, 15] has recently received increasing attention. This novel paradigm distills information from a large real-world dataset into a compact synthetic dataset that produces comparable results to the original dataset. The earliest DD [56] method uses accuracy value as the direct distillation objective and describes the distillation process as a bi-level optimization problem. However, matching accuracy directly through bi-level optimization involves high computational costs and memory overhead [15]. Therefore, state-of-the-art (SOTA) DC methods perform condensation through surrogate objectives, such as gradient [66, 64, 22], training trajectory [8], feature [55], or distribution [65] matching, and achieve more satisfactory results.

However, existing DC methods all optimize parameterized condensed images in the original highdimensional pixel space, overlooking the fact that natural images typically exhibit local connectivity and have low intrinsic dimensionality [31]. More specifically, in the visual self-supervision task, Masked Autoencoder [18] divides the image into multiple patches and reconstructs the complete image by randomly masking a certain proportion of patches. Experimental results show that masking 75% of the patches can still reconstruct the original image. Also, Maximum Likelihood Estimation [31, 40] of the image's intrinsic dimensions on ImageNet [12] dataset shows that although each image contains 150 , 528 pixels, its intrinsic dimension is only between 26 and 43 . We further verify in the appendix that both the original images and the images generated by traditional dataset distillation (e.g., DSA [64], DM [65]) are low rank. These results imply that compressing data in high-dimensional pixel space is an inefficient approach.

In this work, we propose a simple yet efficient plugin for dataset condensing that compresses a large dataset into a compact synthetic dataset on a low-dimensional manifold. Specifically, unlike existing DC methods that train a compact dataset S of size N × D × H × W , where N , D , H , and W are the number, channels, height, and width of images, respectively, we decompose an image representation under each channel ch ∈ { 1 , . . . , D } into a low-dimensional space, and learn two low-rank matrices A ∈ R N × D × H × r and B ∈ R N × D × r × W , where r << min { H,W } represents the rank size. Obviously, our low-rank method is orthogonal to existing DC methods and can be integrated into SOTA DC methods as a flexible plugin to improve the learning efficiency of DC. Without loss of generality, we conduct extensive experiments on multiple publicly real-world datasets, integrating the proposed plugin into gradient matching-based [66, 64] and distribution matchingbased [65] DC methods. Experimental results show that using our low-rank plugin significantly reduces storage consumption for compact datasets and achieves comparable performance as SOTA DC methods based on high-dimensional pixel spaces, with the same number of images. Furthermore, under the same memory consumption, our plugin can effectively store more information from the large dataset, thus significantly improving the performance of condensed data in downstream tasks. Lastly, when applying low-rank DC methods to continual learning (CL) tasks, we observe that our approach effectively mitigates catastrophic forgetting of old tasks under the constraints of limited memory buffers and avoids data privacy issues by storing condensed rather than raw data.

The main contributions of this paper are as follows:

We propose a simple yet effective dataset distillation plugin that condenses a large dataset into a compact synthetic dataset on a low-dimensional manifold, offering an orthogonal approach to existing DC methods.

Experiments on deep learning tasks demonstrate that the proposed plugin achieves performance comparable to that of existing SOTA DC methods while significantly reducing memory consumption. It achieves significantly better accuracy than existing SOTA DC methods with the same memory consumption.

We also verify that the dataset condensed in the low-dimensional manifold has good crossarchitecture transferability and maintains the excellent characteristics of existing DC methods, such as learning the class distribution information of the large dataset.

When applying low-rank DC as a plugin for CL tasks, our DC plugin approach effectively mitigates the problem of catastrophic forgetting of old tasks while protecting the data privacy.

2 Related Works

In this section, we summarize the most related work to this work as three-fold, including coreset sample selection and dataset distillation; continual learning; and low-rank manifolds.

Coreset Selection and Data Condensation. Coreset selection [17] and dataset condensation [56] are two methods to eliminate data redundancy, which help to improve the model's training efficiency and reduce the cost of data management. Coreset selection has been widely studied in active learning [48] and continual learning [34], which tries to identify the most informative training samples [62, 59, 10, 44, 3]. Unfortunately, these methods fail when the task-condition information is evenly distributed among the original samples [30]. Empirical studies [56, 66] show that the benefit of existing coreset selection methods is marginal compared with random selection across multiple benchmarks. In recent years, dataset condensation [56] has been proposed to distill a large real dataset onto a small synthetic dataset and achieve better performance than coreset selection. DD [56] formulates the dataset distillation as a bi-level optimization. The inner loop utilizes the condensed dataset to train a network, while the outer loop minimizes the prediction error of the trained network on the original dataset. It updates the synthesized dataset pixel by pixel using Back-Propagation Through Time [35] to compute the meta gradient. KRR [36] transforms the distillation problem into a kernel ridge regression problem, simplifying the expensive nested optimization in DD [56] to a first-order optimization. Unlike the aforementioned works that optimize based on accuracy value, recent works believe that the effectiveness of models trained on the larger dataset and the compact dataset can be reflected in their corresponding parameter states or gradient states [15]. Therefore, they choose to optimize more fine-grained surrogate objectives [66, 64, 30, 8, 65]. Notably, DC [66] and DSA [64] minimize the gradient matching loss between the large dataset and the synthetic dataset at each training step, MTT [8] matches the training trajectory of parameters, LCMat [50] matches loss curvature, CAFE [55] aligns layer-wise features, and DM [65] generates synthetic samples that resemble the distribution of real samples in the feature space. There has recently been a small amount of work on parameter-efficient dataset distillation. For example, IDC [23] and IDM [67] take advantage of the local similarity characteristics of images to partition and expand condensed data to generate more images at the same memory cost. HaBa [33] and RememberThePast [13] utilize bases to produce images or share bases among all classes respectively. However, these methods ignore that natural images have low intrinsic dimensionality [40], which leads to inefficient condensation.

Continual Learning. Continual learning (CL) aims to enable a neural network model to learn new tasks continuously without forgetting the old tasks [68, 58]. Inspired by the working mechanism of the human brain, mainstream memory-based CL methods consolidate previously learned knowledge by replaying old data, thereby avoiding catastrophic forgetting. Due to strict memory and privacy constraints, usually only a small portion of old task data can be kept, many above-mentioned coreset selection methods are used to select informative samples for storage. For example, some heuristic method is used to select the most representative samples from each class [59] (e.g., iCaRL [44]) or the sample closest to the decision boundary (e.g., Rwalk [9], MIR [2], Shim [49]). However, the number of new task's samples and stored old samples in memory is often highly unbalanced, leading the model to be biased towards learning new tasks with more data. Recent works [7, 41] addressing this class imbalance in memory-based CL has achieved impressive and satisfactory performance: End-to-End Incremental Learning [7] samples the data after training on the new classes and combines it with the data in memory to create balanced data for fine-tuning. GDumb [41] greedily stores samples in memory and trains the model from scratch on new tasks using only the samples in memory.

Low-rank Manifolds. The low-rank structure is prevalent in machine learning, such as computer vision [40, 18, 32], natural language processing [20, 1], recommendation systems [25, 45], etc. For example, MLE [31, 40] found that natural image datasets containing thousands of pixels can actually be described with fewer variables. LoRA [20] can fine-tune the GPT-3 175B [6] with comparable accuracy to full-parameter fine-tuning while reducing the number of trainable parameters by a factor of 10,000 and GPU memory requirements by a factor of three. In recommender systems, the original highly sparse rating/interaction matrix is usually decomposed into low-rank user embedding matrices and item embedding matrices [25] to capture user interests and item features. However, to the best

of our knowledge, no research has explored the exploitation of low-rank structure to improve the condensation efficiency for the dataset distillation task, and this work is the first to fill this gap.

3 Low-Rank Data Condensation Plugin

In this section, we first define the problem of DC in Sec. 3.1, then introduce our proposed low-rank plugin for DC in Sec. 3.2, and then describe how to integrate our plugin with existing DC methods in Sec. 3.3 and further apply it to continual learning in Sec. 3.4.

3.1 Problem Definition

The goal of dataset condensation (DC) is to distills a large target dataset T = { x i , y i } N T i =1 containing N T training image x i ∈ R d and its label y i ∈ { 1 , 2 , . . . , | Y |} into a small dataset S = { s i , y i } N S i =1 with |S| synthetic image s i ∈ R d , where N S ≪ N T ( 2 -3 orders of magnitude), | Y | represents the number of classes, and R d defines a d -dimensional space. We expect a network ϕ θ S trained on the small dataset S to have similar performance to a network ϕ θ T trained on the large training set T on the unseen test dataset, that is:

where P T represents the real distribution of the test dataset, x i represents the input image, y represents the ground truth, and ℓ ( · ) represents a loss function such as cross-entropy loss.

To achieve the goal stated in Eq. 1, existing DC methods [56, 66, 64, 8, 65] first initialize the dataset S ∈ R N S × D × H × W as a set of learnable parameters in high-dimensional pixel space. Here, N S denotes the number of synthetic images, and C , H , and W represents the number of channels, the image's height, and the image's width, respectively. Then, the dataset S is updated pixel-bypixel based on accuracy value matching [56, 36] or surrogate objective matching [66, 8, 65] of the condensed dataset S and the large dataset T . For example, in the first dataset distillation work DD [56], dataset S is treated as a hyperparameter in a bi-level optimization problem as follows:

where the inner loop trains a randomly initialized network ϕ (parameters denoted as θ ) on the synthetic dataset S until convergence, and the outer loop uses the large target dataset T as a validation set to optimize S . The small dataset S is updated by solving the meta-gradient [35] of the bilevel optimization in Eq. 2, allowing the trained model ϕ on the dataset S to perform well on the real dataset T . Instead of optimizing directly based on the accuracy value of the distilled data S , SOTA DC methods are based on surrogate objectives to make the model trained on S and T approximate each other in the parameter space [66], i.e., θ T ≃ θ S , or in the gradient space [64, 66], i.e., ∇ θ L T ( θ ) ≃∇ θ L S ( θ ) , or in the feature distribution space [65], i.e., ϕ θ ( x i ) ≃ ϕ θ ( s i ) . However, these methods all focus on pixel-by-pixel optimizations of S , ignoring that natural images are locally connected [18] and usually have low intrinsic dimensionality [32, 40]. In the appendix, we perform principal component analysis on real images and images obtained by traditional dataset distillation methods (such as DSA [64] and DM [65]) and show that they have low-rank properties. Hence, optimizing S in a high-dimensional pixel space is inefficient.

3.2 Our Low-Rank Data Condensation Plugin

In this work, we introduce a low-rank dataset condensation plugin that distills knowledge from a large dataset T to a small synthetic dataset S in a low-rank manifold. Specifically, instead of directly initializing S as a variable with shape N S × D × H × W , we conduct a low-rank decomposition of the content x i,ch ∈ R H × W in the channel ch ∈ { 1 , . . . , D } of an image x i and employ two variables, A i,ch ∈ R H × r and B i,ch ∈ R r × W , to reconstruct a high-dimensional image by x i,ch ≈ A i,ch B i,ch ,

where the rank r ≪{ H,W } . Therefore, the goal of data condensation in the low-rank manifold is to optimize A ∈ R N S × D × H × r and B ∈ R N S × D × r × W such that the network ϕ θ Ω( A , B ) , trained on the small reconstructed data Ω( A , B ) , achieves similar performance to the network ϕ θ T trained on the high-dimensional large dataset T . Therefore, the DC in the low-rank manifold is formalized as:

where Ω( A , B ) is an operation that reconstructs x i channel-wise from low-rank matrices A and B , i.e., x i = A i B i = [ A i, 1 B i, 1 | . . . |A i,D B i,D ] ∈ R D × H × W , where [ ·|· ] means channel-wise stacked the image representation.

Discussion . Our method effectively takes advantage of the low intrinsic dimension of natural images, and has the following advantages: (i) The proposed low-rank DC plugin significantly improves condensing efficiency (i.e., fewer training parameters) and reduces the cost of synthetic dataset storage. This is particularly beneficial when the image height ( H ) and width ( W ) are very large. Due to the low intrinsic dimension of the image, the value of r can be very small, such as 2 . (ii) Under the same memory consumption or the number of learnable parameters as traditional DC methods, our plugin can reconstruct a larger number of synthetic images, preserving sufficient information for the large training dataset T . (iii) Our plugin is orthogonal to existing dataset distillation work and can be further combined with them.

3.3 Incorporating Low-rank DC Plugin to SOTA Methods

Our proposed low-rank manifolds DC plugin can be easily incorporated into existing DC solutions [56, 66, 8, 64, 65]. Without loss of generality, we integrate it into gradient matching-based DC [66] and distribution matching-based DM [65], and define the two new methods as Lo w-rank DC (LoDC) and Lo w-rank DM (LoDM), respectively. We verified the effectiveness of the low-rank plugin in Sec. 4.1. Additionally, in the appendix, we combine our low-rank DC plugin with more advanced dataset distillation methods, including MTT [8], IDC [23], HaBa [33], and RemenberThePast [13].

Low-rank DC (LoDC) . To achieve the goal stated in Eq. 3, we match the gradients of the large dataset T and the small dataset Ω( A , B ) within a low-rank manifold. Specifically, we use the synthetic dataset Ω( A , B ) to train a deep neural network ϕ θ (the initial parameter is θ 0 ). After optimizing the network, we aim to minimize the distance between the gradient ∇ θ L T ( θ t ) of the loss L T w.r.t θ t on

the large dataset T and the gradient ∇ θ L Ω( A , B ) ( θ t ) of the loss L Ω( A , B ) w.r.t θ t on the small dataset Ω( A , B ) at step t as DC [66]. That is, we solve the following optimization problem:

where d ( · , · ) is a distance function, T in is the number of training iterations. A and B are updated using gradients obtained by backpropagation in Eq. 4, and θ is trained using the whole small dataset Ω( A , B ) by gradient descent optimization. We provide the pseudocode of our LoDC in Algorithm 1. Additionally, LoDC can be easily extended to DSA [64] methods that incorporate differentiable data augmentations into gradient matching [66].

Low-rank DM (LoDM) . To achieve the goal of Eq. 3, our LoDM requires the distribution of the small dataset Ω( A , B ) to accurately approximate the distribution of the real training large dataset T as DM [65]. Specifically, we first transform each input image A i B i ∈ R ( C × H × W ) into a different space through a family of parametric functions ψ θ : R ( C × H × W ) → R d ′ . We then use the maximum mean difference (MMD) [16] to estimate the distance between the real and compact data distributions: sup ∥ ψ θ ∥ H ≤ 1 ( E [ ψ θ ( T )] -E [ ψ θ (Ω( A , B ))]) , where H is reproducing kernel Hilbert space. As the real data distribution is unavailable, we utilize the empirical estimate of MMD as DM [65], that is, LoDM to solve the following optimization problem:

where N T and N AB represent the sample sizes of T and Ω( A , B ) respectively. The low-rank matrices A and B are updated by performing a gradient descent on Eq. 5. DM [65] indicates that network ψ θ 0 can perform well when a family of random initializations is adopted. Therefore, θ 0 does not need to be updated but is randomly sampled in each training iteration. We have adopted this default setting in this paper. Algorithm 2 provides the optimization process of our LoDM.

3.4 Application to Continual Learning

The goal of class continual learning (CL) is to use a model to learn from a continuously arriving sequence of new classes while retaining knowledge from previous classes [68, 58]. Mainstream memory-based CL methods typically involve carefully selecting a small number of old samples from a large training dataset and storing them in a limited memory buffer M to mitigate catastrophic forgetting [59, 7, 41, 57]. The degree of forgetting of old classes is directly affected by the amount of data information in the samples stored in the limited memory buffer M . Our dataset condensation plugin can compress a raw large dataset into a smaller dataset Ω( A , B ) by Alg. 1 or Alg. 2 to story in M , thereby to preserve more information about older classes under strict memory constraints. In this work, similar to DM [65], we establish a baseline using the simple yet effective GDumb [41] approach

in CL. GDumb first stores training samples in memory in a greedy manner while ensuring balanced samples per class. During testing, the model is trained from scratch exclusively using samples from memory M . More specifically, we compare the effects of storing randomly selected sample sets [41], Herding sample sets [59, 44, 7], traditional DC condensed dataset (i.e., DC [66]/DSA [64], DM [65]), and our low-rank manifold condensed dataset (i.e., LoDC or LoDM in Sec. 3.3) in memory M for CL in Sec. 4.2.

4 Experiment

In this section, we conduct experiments to verify the effectiveness of the proposed low-rank DC plugin. Due to space constraints, some experiment results are included in the Appendix .

4.1 Data Condensation for Deep Learning

Datasets . We evaluate our low-rank DC plugin on four benchmark datasets as DM [65], including MNIST [29], CIFAR10 [26], CIFAR100 [26], and TinyImageNet [28]. MNIST contains 60,000 grayscale images of size 28 × 28 , totaling 10 classes. CIFAR10 and CIFAR100 contain 50,000 RGB images of size 32 × 32 . The former has 10 classes, and the latter has 100 classes. TinyImageNet contains 100,000 images resized to 64 × 64 , with a total of 200 classes.

Baselines . We compare two kinds of methods to reduce the amount of data: coreset selection (i.e., Random, Herding [59, 4], Forgetting [53]) and synthetic dataset distillation (i.e., DD [56], LD [5], DC [66], DSA [64], DM [65]). Specifically, Random sampling selects images randomly, Herding heuristically selects the sample closest to the center of the class, and Forgetting selects the sample that is most likely to be forgotten during model training. DD is one of the earliest works on dataset distillation, which updates the synthetic dataset by solving a bi-level optimization problem. LD performs label distillation rather than image distillation. DC performs gradient matching between large training dataset and condensed dataset, and DSA further considers differentiable siamese augmentation strategies based on DC. DM condenses datasets through distribution matching. We further combine the proposed low-rank plugin into DC, DSA and DM to obtain LoDC , LoDSA and LoDM , respectively. Comparisons with other dataset distillation methods (e.g., MTT [8], IDC [23], HaBa [33], and RemenberThePast [13]) are provided in the appendix.

Experimental Details . In each experiment, we first choose the coreset (for Random, Herding, Forgetting) or learn a synthetic dataset (for DD, LD, DC, DSA, DM, LoDC, LoDSA, LoDM) and then employ it to train 20 randomly initialized networks (default ConvNet [46] architecture). By default, we set the rank r of our low-rank plugin to 2 for MNIST, CIFAR10, CIFAR100, and 4 for TinyImageNet. In a few cases, the rank r will be searched in { 1 , 2 , 4 , 8 } . Other hyperparameters are the same as baselines [66, 64, 65]. We repeat the experiment multiple times for each method and

(a)

(b)

(c)

(d)

(e)

(f)

Figure 1: Distribution of real (all images) and synthetic images (50 images per class) on MNIST dataset: (a) Real, (b) DC, (c) Our LoDC( r =8 ), (d) DM, (e) Our LoDM( r =2 ), and (f) Our LoDM( r =8 ).

report the average test accuracy. Additionally, in Sec. 4.1.2, we also tested the cross-architecture effectiveness of the synthetic dataset on five standard deep network architectures: ConvNet [46], LeNet [29], AlexNet [27], VGG11 [52], and ResNet18 [19].

4.1.1 Compared to SOTA DC Baselines

We compare the performance of our LoDC, LoDSA, LoDM with other baseline methods under different sample sizes (1/10/50 image(s) per class) in Tab. 1 and Tab. 2. We have the following observations from Tab. 1: (i) When the sample size is small (e.g., 1 image per class), both Random and heuristic Forgetting sample selection perform poorly, significantly lower than Herding's method, because the latter samples best represent the class centers. As the sample size of each class increases, Herding's advantage becomes less obvious. (ii) The dataset condensation methods are significantly better than the coreset methods. For example, in CIFAR10 (10 images per class), Random and Herding only achieve 26.0% and 31.6% accuracy, while DD achieves 36.8% accuracy, DC, DM, and DSA achieve 44.9%, 48.9%, and 51.1%, respectively. (iii) Our method condenses dataset in a low-rank manifold, effectively reducing memory cost (or the number of parameters to be optimized per image). Therefore, by utilizing the same memory, our low-rank LoDM can represent a more significant number of images, which is significantly better than other SOTA dataset compression methods, especially when the sample size of each class is small. For example, when using one image per class, our LoDM achieved accuracies of 43.8%, 25.6%, and 10.3% on CIFAR10, CIFAR100, and TinyImageNet, respectively, while the best baseline achieved only 28.8%, 13.9%, and 3.9%.

In Tab. 2, we compare the performance of our methods (LoDC, LoDSA, LoDM) in the low-rank manifold with traditional dataset distillation methods (DC, DSA, DM) using the same number of images (SI) or the same memory consumption (SM). It can be observed that: (i) Our method (rank r = 2 ) reduces storage consumption by 7 × , 8 × , and 8 × under MNIST, CIFAR10, CIFAR100 datasets, respectively, when the number of images is the same. However, the performance is still relatively close to the traditional DC method. We further explain the correspondence between rank size and memory cost in the appendix. (ii) Under the same memory consumption, our method can condense the large dataset into a smaller dataset with more samples, thereby preserving more information from the original large datasets. Therefore, compared to traditional condensing methods, our method exhibits a significant improvement. For example, on the CIFAR10 (10 Img/Cls) dataset, LoDSA and LoDM have shown an improvement of 8.9% and 13.8% compared to DSA and DM, respectively. Furthermore, as shown in Fig. 2(a), our LoDM method can achieve a higher final performance than DM after only 2,000 iterations, compared to 20,000 iterations.

4.1.2 Ablation Study

Cross-architecture Transferability Analysis . We verify that the condensed dataset learned in the low-rank manifold still has good cross-architecture transfer ability. We learn condensed dataset on a particular architecture (i.e., AlexNet/ConvNet), and then use the learned small dataset to train five different architectures (use Batch Normalization [21] as DM) from scratch, and finally verify on the test set of CIFAR10 dataset. As shown in Tab. 3, we observe that in each method (DSA, DM or





























(a) iterations

























(b) rank size



























(c) learning rate

(a) real

(b) DM

(c) LoDM

(d) LoDM

LoDM), the condensed data on AlexNet and ConvNet perform very similarly when training different architectures, so they have cross-architecture versatility. Furthermore, our LoDM method achieves the best performance in almost all architectures, benefiting from its ability to hold more images in low-rank space at the same memory cost.

Data Distribution Analysis . We verify that the condensed data learned in the low-rank manifold can also capture the characteristics of the original data distribution. We first train a ConvNet model using all the original data as a feature extractor. Then, we input the original images, DC, DM, LoDC, and LoDM learned data (50 images per class) into the network to extract features and perform dimensionality reduction through T-SNE [54] to visualize. As shown in Fig. 1, we observe that both DC and LoDC cannot capture the distribution of raw data well, since they aim to perform dataset distillation with the goal of gradient matching. DM considers distribution matching as the distillation goal, which captures the data distribution effectively. LoDM inherits the properties of DM, particularly when the rank increases, e.g., in Fig. 1(d) and Fig. 1(e), the rank is 2 and 8, respectively.

Hyper-parameter Analysis . We analyze the effect of two hyperparameters, rank size r ∈ { 1 , 2 , 4 , 8 } and learning rate η dc ∈ { 0 . 1 , 0 . 5 , 1 , 2 , 5 } , on CIFAR10 based on the LoDM method. Fig. 2(b) shows that when using the same number of images as DM, LoDM(SI) gradually achieves similar accuracy to DM in the high-dimensional pixel space, as the rank size increases. When the rank is 8, the performance is nearly identical to that of DM. We found that LoDM(SM) consistently outperforms DM when using the same memory, and the rank equal to 2 is a good choice based on empirical evidence. We also provide image visualizations generated by different ranks in the appendix. Fig. 2(c) shows that updating the generated data with a relatively large learning rate, such as 1 or 5, yields better results.

Visualization of Images . We visualize in Fig. 3 the condensed images generated by our LoDM method (rank r = 2 ) for the CIFAR10 and MNIST datasets, ten images per class. We observe that compared to the images condensed by DM in high-dimensional pixel space, the images we recover under the low-dimensional manifold will be sharper, but still capture the main features of the class, especially on the MNIST dataset. This further verifies that images are locally connected and have lower intrinsic dimensions, so dataset condensation in traditional pixel space may be inefficient.

4.2 Data Condensation for Continual Learning

In this section, we apply the low-rank dataset distillation plugin to continual learning tasks. We perform class-incremental learning with strict memory constraints, specifically using 10 images per class for the CIFAR10 dataset and 20 images per class for the CIFAR100 dataset. Based on Sec. 3.4,































(a) 5-tasks on CIFAR10



































(b) 5-tasks on CIFAR100

        



























(c) 10-tasks on CIFAR100

we combine the coreset method (Random, Herding) and various dataset distillation methods (DSA, DM, and our LoDM) into a simple and effective CL method GDumb [41]. We conduct experiments on two benchmark datasets, CIFAR10 and CIFAR100, where the CIFAR10 dataset is divided into 5 tasks, and the CIFAR100 dataset is divided into 5 tasks and 10 tasks, respectively. Based on Fig. 4, we observe that in the three subfigures (a-c), GDumb+LoDM achieves the best results. For example, on CIFAR100 with 5 tasks, the final accuracies of GDumb+Random, GDumb+Herding, GDumb+DSA, GDumb+DM, and GDumb+LoDM are 27 . 9% , 27 . 0% , 30 . 0% , 33 . 81% , and 37 . 92% , respectively. This suggests that our condensed data in a low-rank manifold is also meaningful for continual learning with limited memory.

5 Conclusion and Future Works

In this work, inspired by natural images that are locally connected and have low intrinsic dimensions, we propose a simple yet effective plugin that condenses a large dataset into a smaller dataset in a lowdimensional manifold. We apply this plugin to the existing dataset condensation methods and observe significant performance improvements while maintaining the same memory cost. Additionally, the analysis revealed that the dataset condensed in the low-dimensional manifold exhibits similar characteristics to the traditional high-dimensional pixel space dataset condensing method, such as matching the distribution of the large dataset and cross-architecture transferability. Furthermore, the plugin effectively addresses the issue of catastrophic forgetting in continual learning tasks. Our work has two directions for further improvement in the future: (i) Rank is a manually tuned hyperparameter in this paper. Therefore, how to adaptively assign the best rank to each dataset in the future is a feasible direction. (ii) We plan to apply the proposed dataset distillation plugin to more downstream tasks in the future, such as network architecture search, federated learning, and meta-learning.

References

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 , 2020.

Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems , pages 11849-11860. Curran Associates, Inc., 2019.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems , 32, 2019.

Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 1266-1275, 2020.

Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. Neural Information Processing Systems Workshop , 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV) , pages 233-248, 2018.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV , pages 532-547, 2018.

Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence , pages 109-116, 2010.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Dc-bench: Dataset condensation benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. Advances in Neural Information Processing Systems , 35:3439134404, 2022.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research , 20(1):1997-2017, 2019.

Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. arXiv preprint arXiv:2305.01975 , 2023.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing , pages 291-300, 2004.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 , pages 770-778. IEEE Computer Society, 2016.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z. Pan. Delving into effective gradient matching for dataset condensation. In IEEE International Conference on Omni-layer Intelligent Systems, COINS 2023 , pages 1-6. IEEE, 2023.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICML , pages 11102-11118. PMLR, 2022.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4015-4026, 2023.

Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer , 42(8):30-37, 2009.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning , pages 12352-12364. PMLR, 2022.

Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems , 17, 2004.

Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank approximation via alternating minimization. In International Conference on Machine Learning , pages 2358-2367. PMLR, 2016.

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. Advances in Neural Information Processing Systems , 35:1100-1113, 2022.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

Berndt Müller, Joachim Reinhardt, Michael T Strickland, Berndt Müller, Joachim Reinhardt, and Michael T Strickland. Btt: Back-propagation through time. Neural Networks: An Introduction , pages 296-302, 1995.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations , 2021.

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34:5186-5198, 2021.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34:20596-20607, 2021.

Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations , 2021.

Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In The European Conference on Computer Vision (ECCV) , pages 524-540. Springer, 2020.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Steffen Rendle. Factorization machines. In 2010 IEEE International conference on data mining , pages 995-1000. IEEE, 2010.

Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6148-6157, 2017.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations , 2018.

Burr Settles. Active learning literature survey. 2009.

Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021 , volume 35, pages 9630-9638, 2021.

Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Losscurvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics , pages 8606-8628. PMLR, 2023.

Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost: Joint appearance, shape and context modeling for mulit-class object recognition and segmentation. In European conference on computer vision (ECCV) , 2006.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations , 2015.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations , 2019.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12196-12205, 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Tiehang Duan, and Mingchen Gao. Improving task-free continual learning by distributionally robust memory evolution. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 22985-22998. PMLR, 2022.

Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09218 , 2023.

Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1121-1128, 2009.

Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3014-3023, 2021.

Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, and Xingwei Wang. Data augmented flatness-aware gradient projection for continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5630-5639, 2023.

Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. The Eleventh International Conference on Learning Representations , 2023.

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning , pages 7105-7114. PMLR, 2019.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , pages 12674-12685. PMLR, 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Ninth International Conference on Learning Representations 2021 , 2021.

Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7856-7865, 2023.

Zhu Fei, Zhang Xu-Yao, and Liu Cheng-Li. Class incremental learning: A review and performance evaluation. Acta Automatica Sinica , 49:1-26, 2023.

A Implementation Details

Experiment Details . We train a model using condensed synthetic images and measure the Top-1 accuracy on raw testing images. For a fair comparison, our main experimental settings follow the settings of DC, DSA and DM. Experiments are conducted on a 3-layer convolutional network [46] (ConvNet-3) with 128 filters, without any specific emphasis. The batch size of the raw images is set to 256 during the matching process. For the DC [66], DSA [64], LoDC, and LoDM methods, the outer loop is set to 1 and the inner loop is set to 1 when using 1 image per class in all experimental datasets. When using 10 images per class, the outer loop is set to 10 and the inner loop is set to 50. In the MNIST and CIFAR10 datasets, when using 50 images per class, the outer loop is set to 50 and the inner loop is set to 10. For the DM [65] and LoDM methods, the loop is set to 20,000 in all experiments. The optimizers all use SGD. When optimizing the condensed data, the learning rate for DC/DSA/LoDC/LoDSA is set to 0.1 by default, and the learning rate for DM/LoDM is set to 1.0 by default. When using condensed data to train the network, the update times of the network are set to 1,000, and the learning rate is set to 0.01.

Relationship between Memory and Rank . Suppose the shape of each channel of a real image is ( H,W ) , where H and W are the height and width of the image, respectively. The memory consumed by traditional dataset distillation can be recorded as N × D × H × W , where N and D represent the number of images and the number of channels. The memory consumed by our low-rank dataset distillation method is N × D × ( H × r + r × W ) , where r is the rank size. For example, in the CIFAR10 dataset, both H and W are 32 , and our r is generally set to 2 , which means that it can save 8 times the storage compared to traditional dataset distillation methods (i.e., DC, DSA and DM).

B Additional Experimental Results

B.1 Experimental Analysis

Compare with Other Adavanced Methods . In our main paper, the low-rank plugin is mainly used in the traditional DC [66]/DSA [64] based on gradient matching and DM [65] based on distribution matching. Recently, a research work (i.e., MTT [8]) shows that better dataset condensation performance can be obtained based on finer-grained gradient trajectory matching. This is strictly orthogonal to our work. As shown in the Tab. 4, when our low-rank plugin is further applied to the MTT, our work achieved 58.7% accuracy on CIFAR10 (Img/Cls=1) and 31.0% on CIFAR100 (Img/Cls=1). This is a significant improvement compared to MTT, which is 46.3% and 24.3%.

In addition, related to our work are recent works on efficient dataset distillation [23, 67, 33, 13]. Among them, (i) IDC-I/IDC [23] and IDM [67] consider the local similarity of image pixels, and they partition and expand the synthetic image to obtain more images. This is orthogonal to our work, i.e., we can perform partitioning and expansion operations on images reconstructed by our low-rank plugin to further increase the number of images under the same memory constraints. For example, on the CIFAR10 dataset, compared with IDC-I/IDC, our low-rank versions (i.e., LoIDC-I/LoIDC) bring performance improvements of 12.5% and 6.6%, respectively. (ii) HaBa [33] inputs the bases into the Hallucinator networks to reconstruct condensed images, while RememberThePast [13] shares bases among all classes and constructs condensed images by addressing matrices. We note that their basis is also equal to the original image size. Therefore, under the same memory cost, we can use our low-rank plugin on their bases to obtain more images. For example, on the CIFAR10 dataset, our LoHaBa and LoRememberThePast improved by 17.8% and 4.0%, respectively, compared to Haba and RememberThePast in Tab. 4.

Principal Component Analysis of Images . In this section, we verify that both original images and images condensed by traditional dataset distillation methods (e.g., DSA [64] and DM [65]) have low-rank properties. Therefore, it is inefficient to perform dataset condensation directly in high-dimensional pixel space. Specifically, we randomly sample some images from the CIFAR10 dataset and further perform singular value decomposition (SVD) on each channel of these images. As shown in Fig. 5, the diagonal matrix represents the distribution of the average eigenvalues on all channels of all selected images. We can observe that both the original images (Fig. 5(a)) and the images synthesized by the traditional data distillation methods (e.g., DSA [64] in Fig. 5(b) and DM [65] in Fig. 5(c)) can be approximated by low rank, that is, the first few eigenvalues on the diagonal dominate. Therefore, it is reasonable to perform low-rank dataset distillation in this paper. Fig. 5(d) shows the distribution of eigenvalues of images produced by the proposed LoDM.

(a) Real

(b) DSA

(c) DM

(d) LoDM

0

5

10

15

20

25

30

0

5

10

15

20

25

30

0

5

5

10-

30

10

20

15 -

20

15 -

10

10

20-

20-

-0

25

25

0

30

0

5

10

15

20

25

30

0

5

10

15

20

25

30

0

0

5

5

10 -

30

10

60

20

40

15

15 -

10

20

20

20-

0

0

25-

25-

30

30

B.2 Image Visualization

Training Process of Synthetic Data . Fig. 6 and Fig. 7 illustrate the intermediate state images of different ranks ( r = { 2 , 8 } ) of our LoDM during the iterative process of synthesizing images on the CIFAR10 dataset. In addition, Fig. 10 and Fig. 11, we visualize the intermediate state images of different ranks ( r = { 2 , 4 } ) of our LoDM in the iterative process of synthesizing images on the MNIST dataset. Visualization of Real and Synthetic Images . We provide more visualizations of the synthetic images from the different datasets: MNIST in Fig. 8, and CIFAR10 in Fig. 9. As the rank size increases, the synthesized image captures increasingly detailed features.

(a) 0-th It

(b) 1000-th It

(c) 5000-th It

(d) 20000-th It

(a) 0-th It

(b) 1000-th It

(c) 5000-th It

(d) 20000-th It

(a) DC

(b) DSA

(c) DM

(d) Our LoDC(r=2)

(e) Our LoDSA(r=2)

(f) Our LoDM(r=2)

(g) Our LoDM(r=4)

(h) LoDM(r=8)

(i) Real

?

0

1

2

2

2

2

2

2

2

以

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

M

y

Q

41

4

4

4

4

力

4

4

5

5

S

S

S

5

S

5

5

5

S

5

S

5

6

b

9

6

y

2

7.

2

>

2

7

2

7

7

7

7

&

8

8

B

8

8

?

8

8

P

B

8

8

8

8

8

8

8

8

8

9

9

9

9

9

9

3

6

g

9

9

9

9

9

38

6

TOT

D

R

2:2

2

2

2

2

2

人

2

1

2

3

3

3

3

33

3

3

3

3

3

3

3

3

3

3

3

3

3

3

W

3

4

D

十

y

乡

4

4

4

y

4

4

4

y

4

力

4

y

6

5

S

.5

5

3

S

3

5

S

S

5

3

6

6

6

6

子

1

7

7

7

7

7

7

子

7

7

7

1

7

8

g

8

8

8

8

8

8

8

8

8

8

8

8

8

&

8

8

8

8

9

q

9

a

9

b

9

b

a

b

b

9

b

9

b

b

(a) Real

(b) DM

(c) Our LoDM(r=1)

(d) Our LoDM(r=2)

(e) Our LoDM(r=4)

(f) Our LoDM(r=8)

8

(a) 0-th It

(b) 1000-th It

(c) 5000-th It

(d) 20000-th It

3

L

名

8

(a) 0-th It

(b) 1000-th It

(c) 5000-th It

(d) 20000-th It

。

2

2

95G7

3

毕

4

3

3

乡

q:5:78ar

3

3

y

一分57

E

95G18

6

3

3

+

3

y

3

乡

5

5

567

S6

5

5

厂

1

7

1

L

6

1

2

8

81

护

8

P

8

g

8

8

8

子

8

9

99

9

9.

9

9

6

9

？

9

9

9

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present paper studies the task of dataset condensation (DC) with a specific focus on its computational and condensation efficiency. The main idea is to condense the dataset in a low-rank representation space instead of in the high-dimensional pixel space. To do so, the author decomposes an image representation into two low-rank matrices, where the proposed method is orthogonal to the existing DC methods and can be readily integrated with existing DC methods. The author considers the gradient matching-based and distribution matching-based DC methods as examples to show how the proposed method can significantly improve them. Extensive experiments are conducted on common classification benchmarks, and the proposed method outperforms the state-of-the-art DC methods with large margins.\n\nSTRENGTHS:\n1. The idea of reducing the input distribution's dimensionality to increase the effective memory size and facilitate the DC is novel. Though leveraging the data's low-rank manifolds/inductive bias is not novel in general, as stated in lines 133-144, it is still interesting to see this idea can effectively work on DC, which is a surging interest in the community.\n\n2. The paper is well-written, with clear motivation and excellent clarity. The technical details are easy to follow.\n\n3. The proposed method can significantly improve the existing SOTA DC methods by a large margin, and it can also sufficiently benefit continual learning, which strongly demonstrates the significance and generality of the proposed method.\n\nWEAKNESSES:\n1. The main weakness is on the experimental protocol. Although the reviewer understands that the author wants to align with the existing methods in DC and choose the same input size for experiment, the input size of the MNIST (28 x 28), CIFAR100 (32 x 32), and TinyImageNet (64 x 64) are too small to be realistic. Given that the proposed method aims to reduce the input space dimensions, it is more favorable to see how effective the proposed method will be when the input size is large enough, e.g., at least 512 x 512. Only by doing so can we clearly see the benefits of the proposed method.\n\nOn the other hand, using realistic input size also provides us with a comprehensive understanding of the proposed method, especially its potential limitations. The reason is that we may intuitively expect that reducing a realistic size of the image to a low dimension (e.g., with rank size 2) can not retain the sufficient information and fine-grained visual context of the image. This is a clear limitation of the proposed method, and the author should definitively provide more analysis of it to inspire future study.\n\n2. The author should at least provide the results of DC and DSA in Table 1 for the TinyImageNet, given that they may achieve SOTA in datasets like MNIST and CIFAR10. It would be helpful to see whether the proposed method significantly outperforms all the comparison methods in a more realistic dataset like TinyImageNet. \n \n3. The result of the proposed method on MNIST is weird when Img/Cls is 10, as it is even worse than the original DM method. Can the author provide any plausible reason to explain it?\n\n4. For the same memory setting in Table 2, how many images have been used by the proposed method? What if the comparison methods in Table 2 also leverage the same number of images as the one used in the same memory setup of the proposed method? Will the proposed method still outperform those compared methods?\n\n5.  In Figure 1, we can observe that only when rank $r=8$ can the proposed method approximate the DM method's latent space structure. Then, how much memory has the proposed method reduced when rank $r=8$? Is this memory reduction still significant?\n\n6.  In lines 284-286, the author states that the proposed method (rank $r=2$) reduces storage consumption by 7x, 8x, 8x, and 8x under different datasets. Please provide the exact number of storage between the proposed method and the comparison method.\n\n7. In lines 322-326, the author states that \"We observe that compared to the images condensed by DM in high-dimensional pixel space, the images we recover under the low-dimensional manifold will be sharper, but still capture the main features of the class, especially on the MNIST dataset.\" However, from Table 1, it is evident that the proposed method is much inferior to the compared methods on the MNIST dataset, which contradicts the claim above-mentioned. How to interpret this phenomenon? \n\nMinor:\n\n1. For subfigure (a) of Figure 1, how does the author obtain this T-SNE plot? Does the author directly plot the T-SNE over the pixel space or the feature embedding of the input images?\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nFinal Acknowledgement on Sep 2nd:\n\nThank you so much for your detailed response. After reading all the reviewer's comments and the corresponding author's response, my concerns have been appropriately addressed.\n\nQUESTIONS:\nPlease refer to the Weakness section for more details.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an efficient dataset condensation (DC) plugin that utilizes a low-dimensional manifold, namely two low-rank matrices, to match the large raw and smaller synthetic condensed dataset. This aims to reduce inefficient condensation when matching in image pixel space. They show that their method is modular, and easy to incorporate into existing DC methods, and is effective, improving model performance when trained on the smaller synthetic dataset. Their method reduces catastrophic forgetting when the entire raw dataset is not available for replay, and reduces privacy leakage from the initial raw dataset.\n\nAfter reading the authors rebuttal and discussing with them, as well as seeing the discussion with other reviewers, I will decrease my score to 7.\n\nSTRENGTHS:\nThis work makes nice use of the structure implicit in natural images, namely local connectivity and similarity of nearby pixels and the low intrinsic dimensionality of natural images, to improve dataset condensation. They propose a simple but effective plugin to any modern DC method that performs DC in a low-dimensional manifold as opposed to in pixel space, significantly improving learning efficiency and memory consumption while maintaining downstream performance of models trained on the condensed datasets. The authors also demonstrate the benefit of their low-rank approach for continual learning, an intuitive application as memory is often constrained in continual learning frameworks. The paper is clearly written, the method well-described, and the analysis is thorough. I particularly like the modularity of the method, and that it is applicable and useful across DC methods.\n\nWEAKNESSES:\nIt’s interesting that the synthetic digits in Figure 3d no longer always resemble the real categories. What are potential risks associated with this? Is it possible that selecting a condensation dataset via model selection on a fixed set of data could have downstream implications if this model is used on real-world data or data under a slightly different distribution? How would this method fare when datasets are imbalanced or biased, as is often the case in real-world settings? The datasets considered all are uniform and IID, unless I’m mistaken?\n\nNits:\n\nThe images in figure 1 and figure 2 should be labeled with (a) (b) etc for easier legibility\n\nQUESTIONS:\nWhy do the authors think their method underperforms on MNIST in table 1, which has perhaps the lowest-dimensionality of any of the considered datasets?\n\nLIMITATIONS:\nI would like to see a discussion of failure modes of limitations of the method, where there any cases that surprised the authors where the model fails? How might this perform in practice, where categories of interest are often highly imbalanced and data is often biased or, in the case of continual learning, where distributions shift over time?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel dataset distillation technique, referred to as LoDM.  Instead of storing data in pixel space, LoDM propose to store each condensed image in a low dimensional space which is represented by multiplication of two matrixes. By decreasing the memory budget allocated for each image, LoDM significantly reduces memory cost. LoDM can be easily combined with SOTA data condensation loss. The evaluation results on four datasets show that LoDM outperforms the presented baselines.\n\nSTRENGTHS:\n1.\tThe paper proposes a novel data parameterization method to store the condensed information, which improves the storage efficiency for data condensation.\n2.\tThe evaluation results show that LoDM outperforms other baselines which store condensed information in images.\n3.\tThe writing is clear and easy to follow.\n\nWEAKNESSES:\nMany thanks to paper, I really enjoy reading the paper and the idea of LoDM, but I observed a notable absence in the discussion on a highly relevant area - data parameterization, within the realm of data condensation…\n\nIn fact, LoDM could be viewed as a variant of the data parameterization method. Storing condensed information in other formats rather than single image is not new for data condensation, which is known as data parametrization. Several data parameterization methods[1][2][3] have been proposed, yet this paper fails to discuss or compare them. Moreover, these data parameterization methods deliver better performance than LoDM.\n\nWithout a thorough discussion and comparison with these related works, it becomes challenging to truly evaluate the novelty and effectiveness of LoDM.\n\n[1] Kim, Jang-Hyun, et al. \"Dataset condensation via efficient synthetic-data parameterization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Liu, Songhua, et al. \"Dataset distillation via factorization.\" Advances in Neural Information Processing Systems 35 (2022): 1100-1113.\n\n[3] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404.\n\nQUESTIONS:\nCompared to SOTA data parameterization methods, what is the advantage of LoDM?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposed a low-rank technique to reduce the storage of the compressed dataset. It successfully applied the LoRA technique in DC and DD problems, and also apply it to the CL problem.

After applying the LoRA technique, the method can still achieve similar performance compared to the original DM.

STRENGTHS:
++ It is valuable and meaningful to reduce the storage of the synthetic data for dataset distillation.
++ The LoRA trick is successfully applied to various conditions.

WEAKNESSES:
-- The paper looks like directly applies the LoRA trick to the DC and DM methods.  Could the author give more explanation on how they adjust the LoRA for the data distillation task? What are the challenges they face and how do they make changes to overcome them?

-- The method names are not consistent, LoDM and LoraDM are used in the main paper.

-- It will be better if more analysis can be added to discuss how the proposed method makes use of the insight of "natural images that are locally connected and have low intrinsic dimensions". 

--It will be better to discuss the relationship between the saved computations/ storage and the different ranks.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
The author has discussed the limitation and social impact in the supplementary.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
This paper proposes a simple yet effective plugin for data condensation/distillation methods. Specifically, utilizing the low-dimensional nature of image manifolds, it proposes to decompose each synthetic image into two low-rank matrices and optimize them instead. Experimental results demonstrate the effectiveness of the proposed method.

STRENGTHS:
- The paper is well-written and easy to follow.

- The idea proposed is novel and interesting.

- The results are promising.

WEAKNESSES:
- There seems to be a gap between the motivation and the method. The motivation is based on the low-dimensionality of natural images while the method is applied to synthetic images. It would be better to include a discussion and experiment on synthetic images to show that they are also intrinsically low-dimensional.

- Missing references and discussions. Although different, the idea proposed is of the same category as the multi-formation in [1] and the Partitioning and Expansion Augmentation scheme in [2]. It would be better to include relevant discussions (preferably with experimental results) in the paper to better position the proposed method in the literature.

[1] Kim, J.H., Kim, J., Oh, S.J., Yun, S., Song, H., Jeong, J., Ha, J.W. and Song, H.O., 2022, June. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning (pp. 11102-11118). PMLR.

[2] Zhao, G., Li, G., Qin, Y. and Yu, Y., 2023. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7856-7865).

QUESTIONS:
Please address the weaknesses above.

LIMITATIONS:
The limitations are discussed in the supplementary materials.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14