An Efficient Dataset Condensation Plugin and Its Application to Continual Learning
Authors: Enneng Yang
Li Shen
Zhenyi Wang
Tongliang Liu
Guibing Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: Data Condensation
Continual Learning
Few-shot Learning
TL;DR: In this work, we propose a simple-yet-efficient dataset condensation plugin that matches the raw and synthetic datasets in a low-dimensional manifold.

Abstract: Dataset condensation (DC) distills a large real-world dataset into a small synthetic dataset, with the goal of training a network from scratch on the latter that performs similarly to the former. State-of-the-art (SOTA) DC methods have achieved satisfactory results through techniques such as accuracy, gradient, training trajectory, or distribution matching. However, these works all perform matching in the high-dimension pixel spaces, ignoring that natural images are usually locally connected and have lower intrinsic dimensions, resulting in low condensation efficiency.  In this work, we propose a simple-yet-efficient dataset condensation plugin that matches the raw and synthetic datasets in a low-dimensional manifold. Specifically, our plugin condenses raw images into two low-rank matrices instead of parameterized image matrices. Our plugin can be easily incorporated into existing DC methods, thereby containing richer raw dataset information at limited storage costs to improve the downstream applications' performance.  We verify on multiple public datasets that when the proposed plugin is combined with SOTA DC methods, the performance of the network trained on synthetic data is significantly improved compared to traditional DC methods. Moreover, when applying the DC methods as a plugin to continual learning tasks, we observed that our approach effectively mitigates catastrophic forgetting of old tasks under limited memory buffer constraints and avoids the problem of raw data privacy leakage.

An Efficient Dataset Condensation Plugin and Its Application to Continual Learning
Enneng Yang', Li Shen?*, Zhenyi Wang3 Tongliang Liu', Guibing Guo
Northeesteern University, China RJD Explore Academy, China
3 University of Maryland, College Park, USA 'The University of Sydney, Australia ennengyang @ stumail.neu..duu mathshenli @ gmail.com; 2 @dm tongliang. liu sydney.edu.au; guogb @ww.:ess.ed.....mm
Abstract
Dataset condensation (DC) distills a large real-world dataset into a small synthetic dataset, with the goal of training network from scratch the latter that per- forms similarly to the former. State-of-the-art (SOTA) DC methods have achieved satisfactory results through techniques such as accuracy, gradient, training tra- jectory. or distribution matching. However, these works all perform matching in the high-dimension pixel space, igmoring that natural images are usually locally connected and have lower intrinsic dimensions, resulting in low condensation dataset condensation
efficiency. In this work, we propose simple-yet-efiie plugin that matches the raw and synthetic datasets in a low-dimensional manifold. Specifically, our plugin condenses raw images into two low-rank matrices instead of parameterized image matrices. Our plugin can be easily incorporated into ex- isting DC methods, thereby containing richer raw dataset information limited storage costs improve the downstream applications' performance. We verify on multiple public datasets that when the proposed plugin is combined with SOTA DC methods, the performance of the network trained on synthetic data is significantly improved compared to traditional DC methods. Moreover, when applying the DC methods plugin to continual learning tasks, we observed that approach effectively mitigates catastrophic forgetting of old tasks under limited memory buffer constraints and avoids the problem of raw data privacy leakage.
Introduction
Modern machine learning techniques utilize large- seaee-ccaleine real-world datasets and advanced deep neural networks (DNNs) to achieve amazing success in various fields, such models like SAM [24] and GPT [6. 38] in the fields of computer vision and natural language processing, both of which have surpassed classical models [51, 42, 43] trained on small datasets. However, training well- performing model in the machine learning community requires repeated tuning of various aspects of the model [14], such the number of layers, learning rate, and other important syper-parameters. When the dataset is large, the cost of data management and repeated model training becomes unacceptable. As stated in [65], NAS-Bench-100 [63] spent 100 TPU-years of computing time the CIFAR10 dataset [26] for an exhaustive neural architecture search. In contrast, it only takes dozens of TPU minutes to train the best model from scratch using the optimal configuration discovered [65]. Additionally, in continual learning [41, 2, 60, 61], to prevent forgetting old tasks while learning new tasks, certain amount of old task data is typically stored for replay [58, 57, 2, 41J. Nevertheless, storing large old task datasets is unrealistic due to strict memory and privacy constraints. Therefore, reducing the data size becomes a valuable and emerging research direction.
"Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A direct way to reduce data size is to select representative subset from the original dataset [17]. This paradigm calculates the importance score for each sample based on specific importance criteria and selects subset to replace the entire training dataset [62]. The calculation criteria include the distance between the sample and the class center [10], the gradient norm of sample [39], and the diversity among samples [47, 57], etc. However, the selection- raaiio method is not always effective, particularly when the task-condition data information is uniformly distributed in the original data [30]. In such cases, the heuristic coreset selection method can only capture limited amount of information. Recent research [56] has shown that generating a small dataset that performs similarly to the original dataset, instead of selecting subset, is a promising direction. Therefore, dataset condensation (DC) or dataset distillation (DD) [56, 66, 64, 65, 37, 11, 15] has recently received increasing attention. This novel paradigm distills information from large real-world dataset into a compact synthetic dataset that produces comparable results to the original dataset. The earliest DD [56] method uses accuracy value as the direct distillation objective and describes the distillation process bi-level optimization problem. However, matching accuracy directly through bi-level optimization involves high computational costs and memory overhead [15]. Therefore, state-of-the-an (SOTA) DC methods perform condensation through surrogate objectives, such as gradient [66, 64, 22], training trajectory [8], feature [55], or distribution [65] matching, and achieve more satisfactory results.
However, existing DC methods all optimize parameterized condensed images in the original high- dimensional pixel space, overlooking the fact that natural images typically exhibit local connectivity and have low intrinsic dimensionality [31]. More specifically, in the visual self-supervision task, Masked Autoencoder [18] divides the image into multiple patches and reconstructs the complete image by randomly masking a certain proportion of patches. Experimental results show that masking 75% of the patches can still reconstruct the original image. Also, Maximum Likelihood Estimation [31, 40] of the image's s intrinsic dimensions on ImageNet [12] dataset shows that although each image contains 150, 528 pixels, its intrinsic dimension is only between 26 and 43. We further verify in the appendix that both the original images and the images generated by traditional dataset distillation (e.g..
DSA [64], DM [65]) are low rank. These results imply that compressing data in high-dimensional pixel space is an inefficient approach.
In this work, we propose simple yet efficient plugin for dataset condensing that compresses large dataset into a compact synthetic dataset on a -iw-eimensional manifold. Specifically, unlike existing DC methods that train compact dataset S of size N x D x H x W, where N. D, H, and W are the number, channels, height, and width of images, respectively, we decompose an image representation under each channel ch E {1..... D} into low-dimensional space, and learn two low-rank matrices A E IR N xDxxx xr and B E RR-xXxxXM where r << min H, W} represents the rank size. Obviously, our low-rank method is orthogonal to existing DC methods and can be integrated into SOTA DC methods as a flexible plugin to improve the learning efficiency of DC. Without loss of generality, we conduct extensive experiments on multiple publicly real-world datasets, integrating the proposed plugin into gradient betcching-bssee [66, 64] and distribution matching- based [65] DC methods. Experimental results show that using our low-rank plugin significantly reduces storage consumption for compact datasets and achieves comparable performance as SOTA DC methods based on digeedimmnnional pixel spaces, with the same number of images. Furthermore, under the same memory consumption, our plugin can effectively store more information from the large dataset, thus significantly improving the performance of condensed data in downstream tasks. Lastly, when applying low-rank DC methods to continual learning (CL) tasks, we observe that our approach effectively mitigates catastrophic forgetting of old tasks under the constraints of limited memory buffers and avoids data privacy issues by storing condensed rather than raw data.
The main contributions of this paper are as follows:
We propose a simple yet effective dataset distillation plugin that condenses a large dataset into compact synthetic dataset on diw-dimensional manifold, offering orthogonal approach to existing DC methods.
Experiments deep learning tasks demonstrate that the proposed plugin achieves per- formance comparable to that of existing SOTA DC methods while significantly reducing memory consumption. It achieves significantly better accuracy than existing SOTA DC methods with the same memory consumption.
We also verify that the dataset condensed in the low-dimensional manifold has good cross- architecture transferability and maintains the excellent characteristics of existing DC meth- ods, such as learning the class distribution information of the large dataset.
2

---

When applying low-rank DC as plugin for CL tasks, our DC plugin approach effectively mitigates the problem of catastrophic forgetting of old tasks while protecting the data privacy.
2
Related Works
In this section, we summarize the most related work to this work as three-fold, including coreset sample selection and dataset distillation; continual learning; and low-rank manifolds.
Coreset Selection and Data Condensation. Coreset selection [17] and dataset condensation [56] are two methods to eliminate data redundancy, which help to improve the model's training efficiency and reduce the cost of data management. Coreset selection has been widely studied in active learning [48] and continual learning [34], which tries to identify the most informative training samples [62, 59, 10, 44, 3]. Unfortunately, these methods fail when the task-condition information is evenly distributed among the original samples [30]. Empirical studies [56, 66] show that the benefit of existing coreset selection methods is marginal compared with random selection across multiple benchmarks. In recent years, dataset condensation [56] has been proposed to distill a large real dataset onto a small synthetic dataset and achieve better performance than coreset selection. DD [56] formulates the dataset distillation as a bi-level optimization. The inner loop utilizes the condensed dataset to train a network, while the outer loop minimizes the prediction error of the trained network on the original dataset. It updates the synthesized dataset pixel by pixel using Back- Propagation Through Time [35] to compute the meta gradient. KRR [36] transforms the distillation problem into kernel ridge regression problem, simplifying the expensive nested optimization in DD [56] to first-order optimization. Unlike the aforementioned works that optimize based on accuracy value, recent works believe that the effectiveness of models trained on the larger dataset and the compact dataset can be reflected in their corresponding parameter states or gradient states [15]. Therefore, they choose optimize more fine-grained surrogate objectives [66, 64, 30, 8, 65J. Notably, DC [66] and DSA [64] minimize the gradient matching loss between the large dataset and the synthetic dataset at each training step, MTT [8] matches the training trajectory of parameters, LCMat [50] matches loss curvature. CAFE [55] aligns layer-wise features, and DM [65] generates synthetic samples that resemble the distribution of real samples in the feature space. There has recently been a small amount of work on parameter-efficiee dataset distillation. For example, IDC [23] and IDM [67] take advantage of the local similarity characteristics of images to partition and expand condensed data to generate more images at the same memory cost. HaBa [33] and SemmemerrThhPare [13] utilize bases to produce images or share bases among all classes respectively. However, these methods ignore that natural images have low intrinsic dimensionality [40], which leads to inefficient condensation.
Continual Learning. Continual learning (CL) aims to enable a neural network model to learn new tasks continuously without forgetting the old tasks [68, 58]. Inspired by the working mechanism of the human brain, mainstream memory- based CL methods consolidate previously learned knowledge by replaying old data, thereby avoiding catastrophic forgetting. Due to strict memory and privacy constraints, usually only a small portion of old task data can be kept, many obove-mentioned coreset selection methods are used to select informative samples for storage. For example, some heuristic method is used to select the most representative samples from each class [59] (e.g.. iCaRL [44)) or the sample closest to the decision boundary (e.g.. Rwalk [9], MIR [2], Shim [49]). However, the number of new task's 's samples and stored old samples in memory is often highly unbalanced, leading
the model to be biased towards learning new tasks with more data. Recent works [7, 41] addressing
this class imbalance in memory- -based CL has achieved impressive and satisfactory performance: End-to-End Incremental Learning [7] samples the data after training on the new classes and combines it with the data in memory to create balanced data for fine-tuning. GDumb [41] greedily stores samples in memory and trains the model from scratch on new tasks using only the samples in memory.
Low-rank Manifolds. The low-rank structure is prevalent in machine learning, such as computer vision [40, 18, 32], natural language processing [20, 1], recommendation systems [25, 45], etc. For example, MLE [31, 40] found that natural image datasets containing thousands of pixels can actually be described with fewer variables. LoRA [20] can fine-tune the GPT-3 175B [6] with comparable accuracy to full-parameter fine-tuning while reducing the number of trainable parameters by factor of 10,000 and GPU memory requirements by factor of three. In recommender systems, the original highly sparse rating/interaction matrix is usually decomposed into low-rank user embedding matrices and item embedding matrices [25] to capture user interests and item features. However, to the best
3

---

of our knowledge, no research has explored the exploitation of low-rank structure to improve the condensation efficiency for the dataset distillation task, and this work is the first to fill this gap.
Low-Rank Data Condensation Plugin
In this section, we first define the problem of DC in Sec. 3.1, then introduce our proposed low-rank plugin for DC in Sec. 3.2, and then describe how to integrate our plugin with existing DC methods in Sec. 3.3 and further apply continual learning in Sec. 3.4.
3.1 Problem Definition
The goal of dataset condensation (DC) is to distills large target dataset T (x;, yi | NT containing Ny training image E Rd and its label y; E f1.2..... [Y]} into a small dataset S = {s:, Vil Ns with S] synthetic image S; E Rd where Ns < NT (2 3 orders of magnitude). Y represents the number of classes, and R defines a d-dimensional space. We expect a network Φys trained on the small dataset S to have similar performance to a network Ogr trained on the large training set T on the unseen test dataset, that is:
Ex [E (ØeT (x:), v)] = Ex (x:),
s.t. AT arg min L (0T) = arg min 8T
Σ
e (ØOT (2)),
lT
NT (x..))TT
0 arg min (0())) arg min
Σ
l ( (x:).
0s
Ns
(X..))S
where Pr represents the real distribution of the test dataset, represents the input image, y represents the ground truth, and E(-) represents loss function such as cross-entropy loss.
To achieve the goal stated in Eq. 1, existing DC methods [56, 66, 64, 8, 65] first initialize the dataset S E xx as set of learnable parameters in high-dimensional pixel space. Here, Ng denotes the number of synthetic images, and C, H, and W represents the number of channels, the image's height, and the image's width, respectively. Then, the dataset S is updated pixel-by- pixel based on accuracy value matching [56, 36] or surrogate objective matching [66, 8, 65] of the condensed dataset S and the large dataset T. For example, in the first dataset distillation work DD [56], dataset S is treated as hyperparameter in a bi-level optimization problem as follows:
S* = arg min L (dys), subject to g = arg min L (de).
(2)
where the inner loop trains a randomly initialized network & (parameters denoted as 0) on the synthetic dataset S until convergence, and the outer loop uses the large target dataset T validation set to optimize S. The small dataset S is updated by solving the meta- sata-grrddnttt [35] of the bi- level optimization in Eq. 2, allowing the trained model & on the dataset S to perform well on the real dataset T. Instead of optimizing directly based on the accuracy value of the distilled data S, SOTA DC methods are based on surrogate objectives to make the model trained on S and T approximate each other in the parameter space [66], i.e., =065 or in the gradient space [64, 66], i.e. V
L (f), or the feature distribution space [65], i.e., (x(x;) (s. However, these methods all focus pixel-by-pixel optimizations of S, ignoring that natural images are locally connected [18] and usually have low intrinsic dimensionality [32, 40]. In the appendix, we perform principal component analysis on real images and images obtained by traditional dataset distillation methods (such as DSA [64] and DM [65]) and show that they have low-rank properties. Hence, optimizing S in nigh-dimensional pixel space is inefficient.
3.2 Our Low-Rank Data Condensation Plugin
In this work, we introduce a low-rank dataset condensation plugin that distills knowledge from a large dataset T to small synthetic dataset S in low-rank manifold. Specifically, instead of directly initializing S as a variable with shape Ns x D x H x W, we conduct low-rank decomposition of the content X4.ch R XW in the channel ch {1...., .., D} of image x; and employ two variables, Aich I2 Hxr and Bichh R W to reconstruct high-dimensional image by Xi & Ar,

---

Algorithm 1: LoDC: Low-rank Dataset Condensation with Gradient Matching [66]
Input: A large training dataset T
Required: Randomly initialized A, of rank for |Y] classes, the probability distribution of randomly initialized weights f%,, neural network de. number outer- iier-iteeaaioos Tow and number of inner-iterations Tun. learning rates and Tje, minibatch size Ny, NAB
for outer-iteration l- = I, You do Initialize 0, Pa.
for inner-iteratonn t 1, 1,... T.. for class e 1, 1,... | do
Sample minibatch real pair BT ~ T and minibatch synthetie pair B (.A.B). Compute 22? (0. (0,IB.)
((A,
NT Σ Exx.e)B E(po, (x1),
Compute C. (0, (0.
NAW (A (. ((0o(A.1)),))
Compute L. ?, (14.5)) = d Vooc% (0, B VoC.IA.8) (0. (0 (B
Update A + A Tide V ALe DC (B. B. (B((A and B +- B nue V VBL. DC (B? B (.AA))))
10
Update 1c+1 + 8. - 130 V 2 (B(A,))
11
Output: small dataset ((AA B))
where the rank r & W). Therefore, the goal of data condensation in the low-rank manifold is to optimize A x and B E R Dxrxxxxxww>>. such that the network @gac4.e), trained on the small reconstructed data ((A,, B3), achieves similar performance to the network Øor trained on the high-dimensional large dataset T. Therefore, the DC in the low-rank manifold is formalized as:
Ex [e (ØOr (x+), v) Ex, [e (chanca.s) (x:),
s.t. OT= arg min L (00))= = arg min 0T
Σ
e (dho (x.),v),
NT
0T
(X..V)ET
(3)
1
(a(A.)) arg min (.)B))) (0 arg min gulA.B)
Σ
( (A.B.); y),
g((A.))
Ns
(A.B.)))
where ()A, B) is an operation that reconstructs x, channel-wise from low-rank matrices A and B, i.c., = A,B; = [A,,, DA, E RDxHXW where [-]-] means channel-wise stacked the image representation.
Discussion. Our method effectively takes advantage of the low intrinsic dimension of natural images, and has the following advantages: (i) The proposed low-rank DC plugin significantly improves condensing efficiency (i.e., fewer training parameters) and reduces the cost of synthetic dataset storage. This is particularly beneficial when the image height (H) and width (W) are very large. Due to the low intrinsic dimension of the image, the value of can be very small, such as 2. (ii) Under the same memory consumption the number of learnable parameters traditional DC methods, our plugin can reconstruct larger number of synthetic images, preserving sufficient information for the large training dataset T. (iii) Our plugin is orthogonal to existing dataset distillation work and can be further combined with them.
3.3 Incorporating Low-rank DC Plugin to SOTA Methods
Our proposed low-rank manifolds DC plugin can be casily incorporated into existing DC solutions [56. 66, 8, 64, 65]. Without loss of generality, we integrate it into gradient matching-bases DC [66] and distribution matching-baid based DM [65], and define the two new methods as Low-rank DC (LoDC) and Low-rank DM (LoDM), respectively. We verified the effectiveness of the low-rank plugin in Sec. 4. Additionally, in the appendix, we combine our low-rank DC plugin with more advanced dataset distillation methods, including MTT [8], IDC [23], HaBa [33], and RemenberThhPast [13].
Low-rank DC (LoDC). To achieve the goal stated Eq. 3, we match the gradients of the large dataset T and the small dataset Q(A, B3) within low-rank manifold. Specifically, we use the synthetic dataset (1A, B) to train a deep neural network do (the initial parameter 0g). After optimizing the network, we aim to minimize the distance between the gradient V VoL (0,) of the loss L w.r.t 0,
5

---

Algorithm 2: LoDM: Low-rank Dataset Condensation with Distribution Matching [65]
Input: A large training dataset T
Required: Randomly initialized A, rank r for |Y] classes, the probability distribution of randomly initialized weights Pap neural network Wh, number of iterations Tow, learning rale T]dc-
for iteration k: = 1,... Tou do
Initialize 0p ~ P%
Initialize L. DM =
for class 1,... Y do
Sample minibatch real pair B ~ T and minibatch synthetic pair B ((4.8)) ~ S2(AA Compute L. (B² B )
d K (xx:))ENT Vio (m:). NAB Σ WR ((4.8) Po (A.Bi) Compute C. L.M + Lo DM (B. B B.(.AB)))
Update A - A nde V.C. and t B - nde VBC. DM
Output: A small dataset (1AA, 13)
the large dataset T and the gradient V (OA.BB))) of the loss (S(A.B) w.r.t 0g on the small dataset (2A,, 13) at step t as DC [66]. That is, we solve the following optimization problem:
Σ (VocT (O.1T),VoL VoL (0
min
Ego- Peo A.B
(4)
where .(:,-) is a distance function, Tin is the number of training iterations. A and B are updated using gradients obtained by backpropagation in Eq. 4, and is trained using the whole small dataset ((A, B) by gradient descent optimization. We provide the pseudocode of our LoDC in Algorithm 1. Additionally, LoDC can be easily extended DSA [64] methods that incorporate differentiable data augmentations into gradient matching [66].
Low-rank DM (LoDM). To achieve the goal of Eq. 3, our LoDM requires the distribution of the small dataset Q(AA, B3 to accurately approximate the distribution of the real training large dataset T as DM [65]. Specifically, we first transform each input image A,B, E RCC*HxW) into a different space through a family of parametric functions whg (C xW) - TR'" We then use the maximum mean difference (MMD) [16] estimate the distance between the real and compact data distributions: sup]ive n ş
(E [tho (T) - E [ubo (S2(A, B) where H is reproducing kernel Hilbert space. As the real data distribution is unavailable, we utilize the empirical estimate of MMD as DM [65], that is,
LoDM to solve the following optimization problem:
NAB
Σ tho (A,B.)) ]
min A.8
d
Moo (aT.)
(5)
N
NAB
where Ny and NAS represent the sample sizes T and (2A,, B)) respectively. The low-rank matrices A and B are updated by performing gradient descent on Eq. 5. DM [65] indicates that network tho, perform well when family of random initializations is adopted. Therefore, Ag does not need to be updated but is randomly sampled in each training iteration. We have adopted this default setting in this paper. Algorithm provides the optimization process of our LoDM.
3.4 Application to Continual Learning
The goal of class continual learning (CL) is to use model to learn from continuously arriving sequence of new classes while retaining knowledge from previous classes [68, 58]. Mainstream eeeorry-bssed CL methods typically involve carefully selecting small number of old samples from a large training dataset and storing them in limited memory buffer M mitigate catastrophic forgetting [59, 7, 41, 57]. The degree of forgetting of old classes is directly affected by the amount of data information in the samples stored the limited memory buffer M. Our dataset condensation plugin can compress a raw large dataset into a smaller dataset Q(A, B3) by Alg. I or Alg. 2 to story in M. thereby to preserve more information about older classes under strict memory constraints. In this work, similar to DM [65], we establish baseline using the simple yet effective GDumb [41] approach

---

Table Comparison with coreset selection methods and dataset condensation methods.
Coreset Selection Methods
Dataset Condensation Methods
Img/Cls
Ration
Random
Herding
Forgetring
DD
LD
DC
DSA
DM
LaDM(Ours)
0.017
64,913355
89.2::166
35.5:15.6
60.9::322
91.71055
88.720.6
89,710066
91,210.4
MNIST
10
0.17
95.11099
93.7.10.3
68.1133.3
8..
87.3.10.7
97.4.10.2
97.1
96.510.2
97.7.10.1
50
0.83
97.91022
94.8.02.
88.2.11.2
93.3.10.3
98.8.10.2
99.2.1..1
97.510.5
98.2.10.1
0.02
14.4120
21.511:2
13.511.2
25.7.10077
28.3.00.5
28..
26.0.00888
43...0.8
CIFARIO
10
26.0.1:22
31.61077
2..3:1..0
1..81122
38.1:004
44.9....5
51.1.0.5
48.9:10.6
59.8:10.4
1..41:.
40.4::06
2.334
42.54:04
53.940.5
60.6:1.55
63.0104
64.6:0.11
0.2
4.24:033
8.4:0.33
4,52002
11.5404
12.8:0033
13
11.4:003
25.6:10.5
CIFAR100
10
2
14.61.0.5
17.310033
15.1:10.3
25.2.1..3
32.3
29.7..0.3
37.5..0.8
Tiny ImageeeeN 1
0.2
14+0.1
2.8:0.2
1.600.1
4.61+0.2
4.79=0.2
3.9:002
10.3:00.2
10
2
5.0+0..2
6.3
0.1
11.6+03
14.7=0.2
12,9+04
18.3:0033
Table 2: Comparison with dataset distillation methods on the same image (SI) or same memory (SM).
DataSet
Img/Cls
DC
LoDCISL)
LoDCISM)
DSA
LoDSA(SD)
LoDSA(SM)
DM
LoDMISD)
LoDMISMD
MNIST
91. 0.33
93.022033
88.720.6
90.6:2066
8977406
87.020.7
91.2-0.4
10
4...2
96.0=022
97.6:103
97.1
95.340022
97.711.11
9654002
92.020.6
97.710011
CIFARIO
1
28. 2..3:00.
28.2=0.3
35.2:0.5
28.8:0.7
28.3:0.6
41.0:0,2
260408
24.8=033
438-08
10
44.9::0.5
4.
0.3
51.100.5
47.640.
56.540.3
48.940.66
46.0=0.7
5,8:044
CIFARIO0
1
12.8.0.33
12.3.002
18.6
139.10.3
13.510.2
22.910.2
114103
9.1.0.4
25.6....5
2.3
24.4=02
2..1-00.
32.3+00.3
29.8+0..1
33.710.7
29.74:0.3
27.7:0.2
3.5-008
ImylmageNet
4.11102
4.21.022
022
6.000.
4.79.10.2
4.70.0..3
9.9010.44
39102
3.1.0.1
10.3
in CL. GDumb first stores training samples in memory in greedy manner while ensuring balanced samples per class. During testing, the model is trained from scratch exclusively using samples from memory M. More specifically, we compare the effects of storing randomly selected sample sets [41], Herding sample sets [59, 44, 7]. traditional DC condensed dataset (i.e., DC [66]/DSA [64], DM [65]), and our low-rank manifold condensed dataset (i.e., LoDC LoDM in Sec. 3.3) in memory M for CL in Sec. 4.2.
4 Experiment
In this section, we conduct experiments verify the effectiveness of the proposed low-rank DC plugin. Due to space constraints, some experiment results are included in the Appendix.
4.1 Data Condensation for Deep Learning
Datasets. We evaluate our low-rank DC plugin on four benchmark datasets as DM [65]. including MNIST [29], CIFARIO [26], CIFARIO0 [26), and TinyImageNet [28]. MNIST contains 60,000 grayscale images of size 28 x 28, totaling 10 classes. CIFARIO and CIFARIO0 contain 50,000 RGB images of size 32 x 32. The former has classes, and the latter has 100 classes TinylmageNet contains 100,000 images resized to 64 x 64, with total of 200 classes.
Baselines. We compare two kinds of methods reduce the amount of data: coreset selection (i.e.. Random, Herding [59, 4], Forgetting [53]) and synthetic dataset distillation (i.e., DD [56], LD [5], DC [66], DSA [64], DM [65]). Specifically, Random sampling selects images randomly, Herding heuristically selects the sample closest to the center of the class, and Forgetting selects the sample that is most likely to be forgotten during model training. DD is one of the earliest works on dataset distillation, which updates the synthetic dataset by solving bi-level optimization problem. LD performs label distillation rather than image distillation. DC performs gradient matching between large training dataset and condensed dataset, and DSA further considers differentiable siamese augmentation strategies based on DC. DM condenses datasets through distribution matching. We further combine the proposed low-rank plugin into DC, DSA and DM to obtain LoDC, LoDSA and LoDM, respectively. Comparisons with other dataset distillation methods (c.g., MTT [8], IDC [23], HaBa [33], and RemenberTheeast [13]) are provided in the appendix.
Experimental Details. In each experiment, we first choose the coreset (for Random, Herding, Forgetting) or learn synthetic dataset (for DD, LD, DC, DSA, DM, LoDC, LoDSA, LoDM) and then employ to train 20 randomly initialized networks (default ConvNet [46] architecture). By default, we set the rank of our low-rank plugin to 2 for MNIST, CIFAR1O, CIFARI00, and 4 for TinyImageNet In a few cases, the rank r will be searched {1,2, 4, 8). Other hyperparameters are the same baselines [66, 64, 65]. We repeat the experiment multiple times for each method and
7

---

Table 3: Cross architecture testing performance CIFARIO (10 images per class). LearnOn means condensing
the dataset on this architecture, and TestOn means using the condensed dataset to train architecture.
Method
LearnOn \ TestOn
ConvNet
LeNet
AlexNet
VGG11
ResNet18
AlexNet
30.4+0.7
24.2+00.4
28.3+0.4
27.2=1.0
27.8+1.1
DSA
ConvNet
31.411.11
21.7111.
25.91108
27.6.1.8
27.6.11.4
AlexNet
41.4+0.8
31.4+0.2
37.5+0099
36.8=0.5
34,9
DM
ConvNet
42.2+0.55
33,440.6
38.8+1.3
36.2+1.0
34.6+0.55
AlexNet
56.2.10.3
32.910,9
49.910.5
$1.010.6
50.910.5
LoDM(ours)
ConvNet
56.4+0..
45.5+0.6
53,420.6
50.7=0.6
50.6
(a)
(b)
(c)
(d)
(e)
Figure 1: Distribution real (all images) and synthetic images (50 images per class) on MNIST dataset: (a)
Real, (b) DC, (c) Our LoDC(r=8), 8), (d) DM, (c) Our LoDM(r= 2), and (f) Our LoDM(r=8)
report the average test accuracy. Additionally, effectiveness of the synthetic dataset on five standard deep network architectures: ConvNet [46],
Sec. 4.1.2, we also tested the arcsttrcciteecture
LeNet [29], AlexNet [27], VGG11 [52], and ResNet18 [19].
4.1.1 Compared to SOTA DC Baselines
We compare the performance of our LoDC, LoDSA, LoDM with other baseline methods under different sample sizes (1/10/50 image(s) per class) in Tab. 1 and Tab. 2. We have the following observations from Tab. 1: (i) When the sample size is small (e.g.. image per class), both Random and heuristic Forgetting sample selection perform poorly, significantly lower than Herding's ss method, because the latter samples best represent the class centers. As the sample size of each class increases, Herding's advantage becomes less obvious. (ii) The dataset condensation methods are significantly better than the coreset methods. For example, in CIFAR1O (10 images per class), Random and Herding only achieve 26.0% and 31.6% accuracy, while DD achieves 36.8% accuracy, DC, DM, and DSA achieve 44.9%, 48.9%, and 51. 1%, respectively. (iii) Our method condenses dataset in low-rank manifold, effectively reducing memory cost (or the number of parameters to be optimized per image). Therefore, by utilizing the same memory, our low-rank LoDM can represent a more significant number of images, which is significantly better than other SOTA dataset compression methods, especially when the sample size of each class is small. For example, when using one image per class, our LoDM achieved accuracies of 43.8%, 25.6%, and 10.3% on CIFARIO, CIFARI00, and TinylmageNet, respectively, while the best baseline achieved only 28.8%, 13.9%, and 3.9%.
In Tab. 2, we compare the performance of our methods (LoDC, LODSA, LoDM) the low-rank manifold with traditional dataset distillation methods (DC, DSA, DM) using the same number of images (SI) or the same memory consumption (SM). It can be observed that: (i) Our method (rank 2) reduces storage consumption by 7x, 8x, and 8x under MNIST, CIFAR10 10, CIFAR 100 datasets, respectively, when the number of images is the same. However, the performance still relatively close to the traditional DC method. We further explain the correspondence between rank size and memory cost in the appendix. (ii) Under the same memory consumption, our method can condense the large dataset into smaller dataset with more samples, thereby preserving more information from the original large datasets. Therefore, compared to traditional condensing methods, method exhibits significant improvement. For example, the CIFAR1O (10 Img/Cls) dataset, LoDSA and LoDM have shown an improvement of 8.9% and 13.8% compared to DSA and DM, respectively. Furthermore, shown in Fig. 2(a), our LoDM method can achieve higher final performance than DM after only 2,000 iterations, compared to 20,000 iterations.
4.1.2 Ablation Study
Cross-architecture Transferability Analysis. We verify that the condensed dataset learned in the low-rank manifold still has good cross-architecture transfer ability. We learn condensed dataset on a particular architecture (i.e., AlexNet/ConyNet), and then use the learned small dataset to train five different architectures (use Batch Normalization [21] as DM) from scratch, and finally verify on the test set of CIFAR10 10 dataset. As shown in Tab. 3, we observe that in each method (DSA, DM or
8

---

65
DM LOOKIERI
DM
LC0M550
LeOXA554
LODN
ACC (%)
LOONI (SR))
ACC (%)
65
40
ACC
20
45
Iteration
1=1
2
F=4
8=8
0.1
0.5
2
5
(a) iterations
(b) rank size
(c) learning rate
Figure 2: Ablation study on CIFARIO (10 images per class): (a) Accuracy changes during DM, LoDM(SI) and
LoDM(SM) iterations, (b) rank size of synthetic datasets, and (c) learning rate of synthetic datasets (r 2).
J.O
J
3
:
5
ESTO
0
8
real
(b) DM
(c) LoDM
(d) LoDM
Figure 3: Visualization of (a) real and (b) DM synthetic, and (c) LoDM(Ours) synthetic images on CIFARIO
dataset (10 images per class) and (d) LoDM(Ours) on MNIST dataset (10 images per class).
LoDM), the condensed data AlexNet and ConvNet perform very similarly when training different architectures, so they have cross-architecture versatility. Furthermore, our LoDM method achieves the best performance in almost all architectures, benefiting from its ability hold more images in low-rank space at the same memory cost.
Data Distribution Analysis. We verify that the condensed data learned in the low-rank manifold can also capture the characteristics the original data distribution. We first train ConvNet model using all the original data feature extractor. Then, imput the original images, DC, DM, LoDC, and LoDM learned data (50 images per class) into the network to extract features and perform dimensionality reduction through T-SNE [54] to visualize. As shown in Fig. I, observe that both DC and LoDC cannot capture the distribution of raw data well, since they aim to perform dataset distillation with the goal of gradient matching. DM considers distribution matching the distillation goal, which captures the data distribution effectively. LoDM inherits the properties of DM, particularly when the rank increases, e.g., in Fig. (dd) and Fig. (e)), the rank is 2 and 8, respectively.
Hyper-paree parpm--er Analysis. We analyze the effect of two hyperparameters, rank size r 21, 4, 4,8} and learning rate Tie E (0.1, 0.5, 1, 2, 5). on CIFAR 10 based on the LoDM method. Fig. 2(b) shows that when using the same number of images as DM, LoDM(SI) gradually achieves similar accuracy to DM in the high-dimensional pixel space, the rank size increases. When the rank is 8, the performance is nearly identical to that of DM. We found that LoDM(SM) consistently outperforms memory, and the rank equal 2 is good choice based empirical
DM when using the evidence. We also provide image visualizations generated by different ranks in the appendix. Fig. 2(c) shows that updating the generated data with a relatively large learning rate, such as 1 or 5, yields better results.
Visualization of Images. We visualize in Fig. 3 the condensed images generated by our LoDM method (rank = 2) for the CIFARIO and MNIST datasets, ten images per class. We observe that compared to the images condensed by DM in high-dimensional pixel space, the images recover under the fow-dimensional manifold will be sharper, but still capture the main features of the class, especially on the MNIST dataset. This further verifies that images are locally connected and have lower intrinsic dimensions, so dataset condensation traditional pixel space may be inefficient.
4.2 Data Condensation for Continual Learning
In this section, we apply the low-rank dataset distillation plugin to continual learning tasks. We perform dass-incremennal learning with strict memory constraints, specifically using 10 images per class for the CIFARIO 10 dataset and 20 images per class for the CIFAR100 dataset. Based on Sec. 3.4,

---

Testing Accuraco
100
Randon
DM
Testing Accuracy(
80
Randon Hedding
Testing Accuracy
Randen
CM
05A
Harding
60
DSA
8 & s
DSA
60
en
60
2
10
20
40
60
100
10 20 30 40 50 60 70 80 90 100
Number of Classes
Number of Classes
Number of Classes
(a) 5-tasks on CIFARIO
(b) 5-tasks on CIFARIO0
(c) 10-tasks on CIFAR100
Figure 4: Test accuracy the class-incremente learning task.
we combine the coreset method (Random, Herding) and various dataset distillation methods (DSA, DM, and our LoDM) into simple and effective CL method GDumb [41]. We conduct experiments on two benchmark datasets, CIFARIO and CIFAR100, where the CIFAR10 dataset is divided into 5 tasks, and the CIFARIO0 dataset divided into tasks and 10 tasks, respectively. Based on Fig. 4, we observe that in the three subfigures (a-c), GDumb+LoDMe achieves the best results. For example, on CIFAR 00 with 5 tasks, the final accuracies of GDndom, GDumb+Herding, GDumb+DSA, GDumb+DM, and GDumb+LoDM are 27.9% 27 27.0%. 30.0%, 33.81 33.81% %. and 37 .22%, respectively. This suggests that our condensed data in low-rank manifold is also meaningful for continual learning with limited memory.
Conclusion and Future Works
In this work, inspired by natural images that are locally connected and have low intrinsic dimensions, we propose simple yet effective plugin that condenses a large dataset into smaller dataset in a low- dimensional manifold. We apply this plugin the existing dataset condensation methods and observe significant performance improvements while maintaining the same memory cost. Additionally, the analysis revealed that the dataset condensed in the low-dimensional manifold exhibits similar characteristics to the traditional high-dimensional pixel space dataset condensing method, such as matching the distribution of the large dataset and architerchee transferability. Furthermore, the plugin effectively addresses the issue of catastrophic forgetting in continual learning tasks. Our work has two directions for further improvement the future: (i) Rank a manually tuned hyperparameter in this paper. Therefore, how to adaptively assign the best rank to each dataset in the future is feasible direction. (ii) We plan to apply the proposed dataset distillation plugin to more downstream tasks in the future, such as network architecture search, federated learning, and meta-learning.
Acknowledgements
Li Shen was supported by the STI 2030-Majoo Projects (No. 2021ZD0201405), Enneng Yang and Guibing Guo were supported by the National Natural Science Foundation of China under Grants No. 62032013 and No. 61972078, the Science and technology projects in Liaoning Province and the Fundamental Research Funds for the Central Universities under Grants No. N221 and No. N2317002. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
(No. 00233H3/10200005
References
[1] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint 20X.::20/2./3255: 2020.
[2] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pages 1 11849-1188 1860. Curran Associates, Inc., 2019.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
10

---

[4] Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1275-1275, 2020.
[5] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. Neural Information Processing Systems Workshop, 2020.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[7] Francisco M Castro, Manuel J Marfn-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pages 233-248, 2018.
[8] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pages 4750-4759, 2022.
[9] Arslan Chaudhry, Puneet K Dokamia, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, pages 532-547,20 2018.
[10] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sin Sixth Conference on Uncertainty in Artificial Intelligence pages 109-116, 2010.
[11] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Dc-bench: Dataset condensation bench- mark. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 leee, 2009.
[13] Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. Advances Neural Information Processing Systems, 35:34391- 34404, 2022.
[14] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 20(1):1997-2017, 2019.
[15] Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. arXiv preprint arXiv:2305.0197. 2023.
[16] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[17] Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering, In Proceedings of the thitty-sixth annual ACM symposium on Theory of computing, pages 291-300, 2004.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 770-778. IEEE Computer Society, 2016.
[20] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
11

---

[21] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
[22] Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z. Pan. Delving into effective gradient matching for dataset condensation. In IEEE International Conference on Omni-layer Intelligent Systems, COINS 2023, pages 1-6. IEEE, 2023.
[23] Jang- Hyyn Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteri- zation. In ICMI, pages 11 11102-11111 1118- PMLR, 2022.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 4015-4026, 2023.
[25] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[28] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7,7):3, 2015.
[29] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6:1172:27-2334, 1998.
[30] Sachyung Lee, Sanghyuk Chun, Sangwon Jung. Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pages 22352-12364 PMLR, 2022.
[31] Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances neural information processing systems, 17, 2004.
[32] Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank approximation via alternating minimization. In International Conference on Machine Learning. pages 2358-2367. PMLR, 2016.
[33] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinehao Wang. Dataset distillation via factorization. Advances in Neural Information Processing Systems, 35:1 100-1113, 2022.
[34] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[35] Berdtt Müller, Joachim Reinhardt, Michael T Strickland, Berndt Müller, Joachim Reinhardt, and Michael T Strickland. Btt: Back-propagation through time, Neural Networks: An Introduction. pages 296-302, 1995.
[36] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021.
[37] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5195 2021.
[38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances Neural Information Processing Systems, 05:27730-27744, 2022.
12

---

[39] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training, Advances in Neural Information Processing Systems, 34:20596-20607 2021.
[40] Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations, 2021.
[41] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions progress in continual learning. In The European Conference Computer Vision (ECCV), pages 524-540. Springer, 2020.
[42] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
[44] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In IEEE Conference on Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
[45] Steffen Rendle. Factorization machines. In 2010 IEEE International conference data mining, pages 995-1000. IEEE, 2010.
[46] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6148-6157, 2017.
[47] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
[48] Burr Settles. Active learning literature survey. 2009.
[49] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremmntal continual learning with adversarial shapley value. In Thirty- AAAI Conference on Artificial Intelligence, AAA/ 2021, volume 35, pages 9630-9638, 2021.
[50] Seungjae Shin, Hcesun Bac, Donghyeok Shin, Weonyoung Joo, and I1-Chul Moon. Loss- curvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics, pages 8606-8628, PMLR, 2023.
[51] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost: Joint ap- pearance, shape and context modeling for mulit-class object recognition and segmentation. In European conference on computer vision (ECCV), 2006.
[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference Learning Representations, 2015.
[53] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
[54] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[55] Kai Wang. Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12196-122 12201 2022.
[56) Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint
2018.
13

---

[57] Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Tichang Duan, and Mingchen Gao. Improving task free continual learning by distributionally robust memory evolution. In ICML, volume 162
of Proceedings of Machine Learning Research, pages 22985- PMLR, 2022.
[58] Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv.2307.09218, 2023.
[59] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual Interna- tional Conference Machine Learning, pages 1121-1128, 2009.
[60] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2021.
[61] Enneng Yang, Shen, Zhenyi Wang. Shiwei Liu, Guibing Guo, and Xingwei Wang. Data augmented flatness-aware gradient projection for continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5630-5639, 2023.
[62] Shuo Yang, Zeke Xie, Hanyu Peng. Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. The Eleventh International Conference on Learning Representations, 2023.
[63] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105-7114, PMLR, 2019.
[64] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning. pages 12674 12685. PMLR, 2021.
[65] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 6514-6523, 2023.
[66] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Ninth International Conference on Learning Representations 2021, 2021.
[67] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7856-7865, 2023.
[68] Zhu Fei, Zhang Xu- Yao, and Liu Cheng-Li. Class incremental learning: A review and perfor- mance evaluation. Acta Automatica Sinica, 49:1-26, 2023.
14

---

A Implementation Details
Experiment Details. We train model using condensed synthetic images and measure the Top-1 accuracy on raw testing images. For a fair comparison, our main experimental settings follow the settings of DC, DSA and DM. Experiments are conducted on 3-layer convolutional network [46] (ConyNet-3) with 128 filters, without any specific emphasis. The batch size of the raw images is set to 256 during the matching process. For the DC [66]. DSA [64], LoDC, and LoDM methods, the outer loop is set to I and the inner loop is set to when using image per class in all experimental datasets. When using 10 images per class, the outer loop is set 10 and the inner loop is set to 50. In the MNIST and CIFARIO 10 datasets, when using 50 images per class, the outer loop is set to 50 and the inner loop is set to 10. For the DM [65] and LoDM methods, the loop set to 20,000 in all experiments. The optimizers all use SGD. When optimizing the condensed data, the learning rate for OC/DSA/LoDC//Loll LoDSA is set to 0. by default, and the learning rate for DM/LoDM is set to 1.0 by default. When using condensed data to train the network, the update times of the network are set to 1,000, and the learning rate set to 0.01.
Relationship between Memory and Rank. Suppose the shape of each channel of a real image is (H, W). where H/ and W are the height and width of the image, respectively. The memory consumed by traditional dataset distillation can be recorded as N x D H x W, where N and D represent the number of images and the number of channels. The memory consumed by our low-rank dataset distillation method is N x D (H xr + rx),, where r is the rank size. For example, in the CIFARIO 10 dataset, both H and W are 32, and our is generally set 2, which means that can save times the storage compared to traditional dataset distillation methods (i.e., DC, DSA and DM).
B Additional Experimental Results
B.1 Experimental Analysis
Compare with Other Adavanced Methods. In our main paper, the low-rank plugin is mainly used in the traditional DC [66]/DSA [64] based on gradient matching and DM [65] based on distribution matching. Recently, a research work (i.e., MTT [8]) shows that better dataset condensation performance can be obtained based on finer- mner-ggained gradient trajectory matching. This is strictly orthogonal to our work. As shown in the Tab. 4, when our low-rank plugin is further applied to the MTT. our work achieved 58.7% 7% accuracy on CIFARIO (Img/Cls=1) and 3100% on CIFAR100 (Img/Cls=1). This is significant improvement compared to MTT, which is 46.3% and 24.3%.
In addition, related to work recent works on efficient dataset distillation [23, 67, 33, 13). Among them, (i) IDC-I/IDC [23] and IDM [67] consider the local similarity of image pixels, and they partition and expand the synthetic image obtain more images. This is orthogonal our work, i.e, we can perform partitioning and expansion operations on images reconstructed by our low-rank plugin further increase the number of images under the same memory constraints. For example, on the CIFARIO 10 dataset, compared with IDC-I/IDC, our low-rank versions (i.e., LoIDC-ILoIDC) bring performance improvements of 12.5% and 6.6%, respectively. (ii) HaBa [33] inputs the bases into the Hallucinator networks to reconstruct condensed images, while DememberThePPai [13] shares bases among all classes and constructs condensed images by addressing matrices. We note that their basis is also equal to the original image size. Therefore, under the same memory cost, we can use our low-rank plugin on their bases to obtain more images. For example, on the CIFARIO dataset, our improved by 17.8% and 4.0%, respectively, compared to Haba
LoHaBa and LoRememberTThel and RememberThePan in Tab. 4.
Table 4: Compare with other advanced dataset condensation methtods.
MTT
IDC-I
IDC
HaBa
RememberThePast
46.3%
36.7%
50.6%
48.3%
66.4%
CIFARIO (Img/Cls=1)
LoMTT
LoIDC-I
LoIDC
LoHaBa
TnRememberThePaan
58.7%
49.2%
57.2%
66.1%
68.4%
MTT
IDC-I
IDC
HaBa
RememberTheeast
24.3%
16.6%
24.9%
33.4%
CIFAR100 (Img/Cls=1)
LoMTT
LoIDC-I
LoIDC
LoHaBa
MnkememberThePann
31.0%
26.9%
33.1%
36.1%
15

---

Principal Component Analysis of Images. In this section, we verify that both original images and images condensed by traditional dataset distillation methods (e.g. DSA [64] and DM [65]) have low-rank properties. Therefore, it is inefficient to perform dataset condensation directly in high-dimensional pixel space. Specifically, we randomly sample some images from the CIFARIO dataset and further perform singular value decomposition (SVD) each channel of these images. As shown in Fig. 5, the diagonal matrix represents the distribution of the average eigenvalues on all channels of all selected images. We can observe that both the original images (Fig. 5(a)) and the images synthesized by the traditional data distillation methods (e.g., DSA [64] in Fig. 5(b) and DM [65] in Fig. 5(c)) be approximated by low rank, that is, the first few eigenvalues diagonal dominate. Therefore, is reasonable to perform low-rank dataset distillation in this paper. Fig. 5(d) shows the distribution of cigenvalues of images produced by the proposed LoDM.
the
10
15
20
25
10
10
10
15
20 15
10
25-
50
30-
(a) Real
(b) DSA
10
15
30
0-
10-
10-
15-
15-
10
70
30-
32
(c) DM
(d) LoDM
Figure 5: Principal component analysis of images on CIFARIO dataset: (a) Real images: (b) Synthetic images
of DSA; (c) Synthetic images of DM; and (d) Synthetic images of LoDM(r 2).
B.2 Image Visualization
Training Process of Synthetic Data. Fig. 6 and Fig. 7 illustrate the intermediate state images of different ranks (r (2, 8)) of our LoDM during the iterative process of synthesizing images on the CIFAR10 dataset. In addition, Fig. 10 and Fig. 11, we visualize the intermediate state images of different ranks (r (2, 4}) of our LoDM in the iterative process of synthesizing images on the
MNIST dataset. Visualization of Real and Synthetic Images. We provide more visualizations of the synthetic images from the different datasets: MNIST in Fig. 8, and CIFARIO in Fig. 9, As the rank size increases, the synthesized image captures increasingly detailed features.
(a) 0-th It Figure
(b) 1000-th It
(c) 5000-th It
(d) 20000-th It
Visualization of the synthetic image process of LoDM(r=2) on the CIFARIO dataset.
16

---

(a) O-th
(b) 1000-th
(c)
5000-th
(d) 20000-th
Figure
Visualization of the synthetic image process of LoDM(r=8) on the CIFARIO datasct.
0
G 6
0
0
0
0
0
0
0
O
/
1
7
f
{
f
1
1
2
2
2
3
2
A.
2
6 3
2
2
2
2
2
2
2
2
2 3 4 5
2
2
2
2
2
2
2
2
2
2
3
3
3
5
3
9
8
3 Y 5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
+ 5
4
4 4
4
Y
4
4
4
8
4
4
4 5
4 5
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5 6
5
5
5
5
5
5
5
5
5
5
5 6
6
6
6
6
6
4 6 7
6
6
4
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7 7
7
7
2
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
3
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
4
9
4
9
9
9 9
9
9
9
9
9
9
9
9 9
9
9
9
9
9
(a)
DC
(b) DSA
(c)
DM
3 0
8
10
8
O
O
#
t
f);
$
1
0 3
0
4
2
2
L
8
8
3
3
3
J
9
J
8 Y r
8
3
3
1
4
6 5
Y
#
M
+
H 5
TT
H
3
5
r
3
8
5
5
S
6
9 6 T
S r 6 /1
F 1
&
7 x f
8 9
0
1 T
7
6
L
0
E 6 E 8
1 8
7
7
3
9
9
T
7
T
8
8
8
8
H
8
$
B
O
I
(d) Our LoDC(r=2)
(c) Our LoDSA(r=2)
(f)
Our LoDM(r=2)
5
O
0
0
0
6
6
0
D
0
0
0
0
0
0
0
1
1
[
1
1
1
1
1
/
/
/
/
1
3.
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 2 2
2
2
2
2
>
2
2
2
2
5
1335
3
3
3
3
3
3
5
3
3
3
3
3
3 3
3
3
3
3
3
3
3
3
3 4
4
4
4
4
4
u
H
+
4
4
4
4
4
4
4
4
4 144
4
4
4
4
4
5
5
5
8
5
5
S
5
5
5
5
5
6
5
5
5
5
5
G
6
6
6
6
6 7 8 9
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
T
7
7 8
7 8
77
9
7
7
7
7
7
7
7
7
7
M
8
g
8
8
8
/ 8
8
8
4
8
8
8
8
8 9
8
8
8
8
8
8
8
8
8
8
9
4
9
9
8
4
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(g) Our (oDM(r=4)
(h) LoDM(r=8)
(i) Real
Figure 8:
Visualization on MNIST dataset (10 Img/Cls).
17

---

(a) Real
DM
(c) Our LoDM(rII)
(d) Our LoDM(r=2)
(e) Our LoDM(r=4)
(f) Our LoDM(r=8)
Figure 9: Visualization on the CIFARIO dataset (10 Img/Cls).
1000
2
2
&
8
8
8
8
3
3 3
N
9
F
S
PESTS
5
0
8
4
C
m
8
8
8
(a) 0-th
(b) 1000-th It
(c) 5000-th
(d) 20000-th
Figure 10:
Visualization of the synthetic image process LoDM(r=2) on the MNIST dataset.
J
Ø
J
O
O
D
6 0
Ø
O
O
6
6
6
Ø a
/
A
2
2
2
A
2
2
2
2
2
1
2
2
5
3
3
3
3
3
5
3
3
3
3
1
4
4
+
4
4
4
4 4
9
$
5
5
5
15555
5
5
5
5
5
G
à
6666666
6
6
7 7
7
2
772
f
8
4 9
8
(a) 0-th
(b) 1000-th It
(c) 5000-th
(d) 20000-th
Figure 11: Visualization of synthetic image process LoDM(r=4) on MNIST dataset.
18


Reviews:
[{"review_id": "pVpw7QsftO", "reviewer": "Reviewer_JHfi", "summary": "This paper proposes a simple yet effective plugin for data condensation/distillation methods. Specifically, utilizing the low-dimensional nature of image manifolds, it proposes to decompose each synthetic image into two low-rank matrices and optimize them instead. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "Please address the weaknesses above.", "limitations": "The limitations are discussed in the supplementary materials.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n\n- The idea proposed is novel and interesting.\n\n- The results are promising.", "weaknesses": "- There seems to be a gap between the motivation and the method. The motivation is based on the low-dimensionality of natural images while the method is applied to synthetic images. It would be better to include a discussion and experiment on synthetic images to show that they are also intrinsically low-dimensional.\n\n- Missing references and discussions. Although different, the idea proposed is of the same category as the multi-formation in [1] and the Partitioning and Expansion Augmentation scheme in [2]. It would be better to include relevant discussions (preferably with experimental results) in the paper to better position the proposed method in the literature.\n\n[1] Kim, J.H., Kim, J., Oh, S.J., Yun, S., Song, H., Jeong, J., Ha, J.W. and Song, H.O., 2022, June. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning (pp. 11102-11118). PMLR.\n\n[2] Zhao, G., Li, G., Qin, Y. and Yu, Y., 2023. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7856-7865).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Our motivation and method matching:** As shown in $\\underline{\\text{Fig.1(a,b) in the response}}$, we perform principal component analysis on real images and synthetic images of DM on the CIFAR10 dataset. It can be observed that both are dominated by a small number of top eigenvalues, so both the real image and the synthetic image of the traditional dataset distillation method(e.g.,DM) are of low rank. Based on this, we propose the dataset distillation in low-rank plugin, which directly performs dataset distillation in the low-rank manifold instead of the original high-dimensional pixel space. The proposed plugin can be applied to arbitrary dataset distillation methods to improve their performance. As shown in the CIFAR10(Img/Cls=1) results in the table below, when our plugin is applied to existing dataset distillation methods, the performance improvement is very significant.\n\n| DC | LoDC(Ours)| DSA | LoDSA(Ours)| DM | LoDM(Ours)| MTT | LoMTT(Ours)| \n| :------:| :------: |:------:| :------: |:------:| :------: |:------:| :------: |\n| 28.3% |  35.2% | 28.8% |  41.0% | 26.0% |  43.8% | 46.3% |  58.7% | \n\n\n- **Related literature discussion:** Thank you for your suggestions for related work. Our motivation and approach are fundamentally different from [1,2], where multi-formation images (in IDC[1]) and partition and expand enhancement schemes (in IDM[2]) exploit the local similarity of image pixels. However, our work exploits the global low-rank properties of images from the perspective of principal component analysis.  Therefore, they are orthogonal, that is, the pixel similarity of the image and the low-rank characteristics of the image are considered at the same time when the dataset is distilled. In our revision, we will discuss these correlations and differences according to your suggestion.  In addition, we also applied our dataset distillation plugin on the latest dataset distillation work (MTT based on gradient trajectory matching). The results on CIFAR10 (Img/Cls=1) are shown below, and our method(LoMTT) shows a clear improvement.\n\n| MTT | IDC[1] | IDM[2] | Ours| \n| :------:| :------: | :------: | :------: \n| 46.3% | 50.6%   |  45.6%  | 58.7% | "}, {"review_id": "SPVxXEisT9", "reviewer": "Reviewer_Lynm", "summary": "This paper proposed a low-rank technique to reduce the storage of the compressed dataset. It successfully applied the LoRA technique in DC and DD problems, and also apply it to the CL problem.\n\nAfter applying the LoRA technique, the method can still achieve similar performance compared to the original DM.", "questions": "Please see the weakness.", "limitations": "The author has discussed the limitation and social impact in the supplementary.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "++ It is valuable and meaningful to reduce the storage of the synthetic data for dataset distillation.\n++ The LoRA trick is successfully applied to various conditions.", "weaknesses": "-- The paper looks like directly applies the LoRA trick to the DC and DM methods.  Could the author give more explanation on how they adjust the LoRA for the data distillation task? What are the challenges they face and how do they make changes to overcome them?\n\n-- The method names are not consistent, LoDM and LoraDM are used in the main paper.\n\n-- It will be better if more analysis can be added to discuss how the proposed method makes use of the insight of \"natural images that are locally connected and have low intrinsic dimensions\". \n\n--It will be better to discuss the relationship between the saved computations/ storage and the different ranks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **The significance of our low-rank method:** As described in our related work, low-rank is a long-standing and fundamental research point in the field of machine learning. LoRA is not the first work to study low-rank. The contribution of LoRA is to consider the characteristics of low intrinsic dimension when fine-tuning the parameters of large language models, thus proposing a method for parameter-efficient tunings. Our work is the first to consider the low-rank characteristics of images in the field of dataset distillation. As shown in $\\underline{\\text{Fig.1(a) and Fig.1(b) in the response}}$, we found that both the real image (Fig.1(a)) and the image generated by the traditional dataset distillation method (Fig.1(b)) are of low rank , that is, a small number of eigenvalues dominate. Based on this strong motivation, we propose to perform dataset distillation directly in the low-rank manifold, which exploits low-rank properties to store more raw data information with as little space as possible. Our proposed low-rank dataset distillation plugin is general and can be combined with arbitrary dataset distillation methods to significantly improve their performance, for example in the table below, we compared the baseline method on CIFAR10 (Img/Cls=1). There are very significant improvements. We believe that filling this research gap in the dataset distillation is significant.\n\n| DC | LoDC(Ours)| DSA | LoDSA(Ours)| DM | LoDM(Ours)| MTT | LoMTT(Ours)| \n| :------:| :------: |:------:| :------: |:------:| :------: |:------:| :------: |\n| 28.3% |  35.2% | 28.8% |  41.0% | 26.0% |  43.8% | 46.3% |  58.7% | \n\n- **Consistency of method names:** Thanks, we fixed the issue.\n- **Insights of our approach:** In $\\underline{\\text{Fig.1(a) and Fig.1(b) in the response}}$, we performed principal component analysis on the real and synthetic images and found that a small number of eigenvalues on the diagonal are dominant. This result shows that natural images are low-rank, and images condensed by traditional dataset distillation method(i.e., DM) also low-rank. Based on this motivation, we propose a general low-rank dataset distillation plugin, which directly learns low-rank synthetic images instead of learning synthetic images in the original high-dimensional pixel space.\n- **Relationship between memory and rank:** Suppose the shape of each channel of a real image is $(H, W)$, where $H$ and $W$ are the height and width of the image, respectively. The memory consumed by traditional dataset distillation can be recorded as $N \\times D \\times H  \\times W$, where $N$ and $D$ represent the number of images and the number of channels. The memory consumed by our low-rank dataset distillation method is $N  \\times D \\times (H  \\times r + r  \\times W)$, where $r$ is the size of the rank. For example, in the CIFAR10 dataset, both $H$ and $W$ are $32$, and our $r$ is generally set to $2$, which means that it can save $8$ times the storage compared to traditional dataset distillation methods. It is obvious that our method improves significantly when the original image resolution is higher. In $\\underline{\\text{Fig.2 in the response}}$, we give an example of images of different resolutions $(64 \\times 64, 128 \\times 128, 256 \\times 256, 512 \\times 512)$ on a subset of ImageNet, which effectively verifies this point. We will provide a detailed analysis of the relationship between memory and rank in the revised version."}, {"review_id": "WkSdjJIQOw", "reviewer": "Reviewer_FpMf", "summary": "This paper introduces a novel dataset distillation technique, referred to as LoDM.  Instead of storing data in pixel space, LoDM propose to store each condensed image in a low dimensional space which is represented by multiplication of two matrixes. By decreasing the memory budget allocated for each image, LoDM significantly reduces memory cost. LoDM can be easily combined with SOTA data condensation loss. The evaluation results on four datasets show that LoDM outperforms the presented baselines.", "questions": "Compared to SOTA data parameterization methods, what is the advantage of LoDM?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1.\tThe paper proposes a novel data parameterization method to store the condensed information, which improves the storage efficiency for data condensation.\n2.\tThe evaluation results show that LoDM outperforms other baselines which store condensed information in images.\n3.\tThe writing is clear and easy to follow.", "weaknesses": "Many thanks to paper, I really enjoy reading the paper and the idea of LoDM, but I observed a notable absence in the discussion on a highly relevant area - data parameterization, within the realm of data condensation…\n\nIn fact, LoDM could be viewed as a variant of the data parameterization method. Storing condensed information in other formats rather than single image is not new for data condensation, which is known as data parametrization. Several data parameterization methods[1][2][3] have been proposed, yet this paper fails to discuss or compare them. Moreover, these data parameterization methods deliver better performance than LoDM.\n\nWithout a thorough discussion and comparison with these related works, it becomes challenging to truly evaluate the novelty and effectiveness of LoDM.\n\n[1] Kim, Jang-Hyun, et al. \"Dataset condensation via efficient synthetic-data parameterization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Liu, Songhua, et al. \"Dataset distillation via factorization.\" Advances in Neural Information Processing Systems 35 (2022): 1100-1113.\n\n[3] Deng, Zhiwei, and Olga Russakovsky. \"Remember the past: Distilling datasets into addressable memories for neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 34391-34404.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n- **Relationship to works [1,2,3]:** Thanks for your suggestions for related works. Although our work and [1,2,3] seem to be data parameterization methods, the motivation of our work is fundamentally different from theirs: IDC[1] constructs multi-formation synthetic images by upsampling, using neighborhood similarity of image pixels. Factorization[2] inputs the base into multiple hallucination networks to generate multiple images, and uses contrastive learning to enhance the diversity of images to enhance performance. RememberThePast[3] shares a set of bases among images of all classes, exploiting image-to-image correlations. In summary, these works [1-3] all consider the image characteristics at the pixel level, and do not consider the global low-rank characteristics of the image. As shown in $\\underline{\\text{Fig.1(a) and Fig.1(b) in the response}}$, we found that both real images and images generated by traditional dataset distillation method(e.g., DM) are of low rank, that is, a small number of eigenvalues dominate. Based on this, we propose to perform dataset distillation in low-rank manifolds, so our method is complementary to these works. We proposed low-rank plugin can be applied to any existing dataset distillation methods, including the above three methods. Specifically, when combined with IDC[1], we can separately learn a set of multi-formation images and a set of low-rank images to simultaneously consider the local similarity and low-rank properties of images. When combined with Factorization[2] and RememberThePast[3], we can decompose the parameterized basis into two low-rank vectors, thus storing more basis at the same memory cost. For example, as shown in bellow table, when our low rank is combined with Factorization[2], the accuracy on the CIFAR10 (Img/Cls=1) dataset is improved from 48.3% to 66.1%. We believe that applying our low-rank plugin to IDC[1] and RememberThePast[3] will further improve their performance as well. In our revision, we will discuss the mentioned papers[1,2,3] in detail.\n\n| Factorization[2] | LoFactorization(Ours)|\n| :------:| :------: |\n| 48.3%   | 66.1% |\n\n- **Performance comparison with works [1,2,3]:** In our paper, the low-rank plug-in is mainly used in the traditional DC/DSA based on gradient matching and DM based on distribution matching. However, recent works (e.g. Factorization[2]) are mainly based on better-performing gradient-trajectory-matching-based methods (MTT). As shown in bellow table, when our low-rank plug-in is further applied to the MTT, our work achieved 58.7% accuracy on CIFAR10 (Img/Cls=1). This is a significant improvement compared to MTT (46.3%), DIC[1] (50.6%), Factorization[2] (48.3%), second only to RemeberThePast[3] (66.4%). However, our low-rank plug-in can be further applied to [1,2,3], for example, when combined with Factorization[2], our method is further improved to 66.1%. Due to time constraints, the code of our low-rank method applied to RememberThePast[3] is still running, and we will present it after subsequent executions are complete. Intuitively, our low-rank plugin allows us to store more images/basis than RememberThePast[3] in the same memory, so we think our method(LoRememberThePast) will further improve compared to RememberThePast[3].\n\n\n| MTT | IDC[1] | Factorization[2] | RememberThePast[3] | LoMTT(Ours)| LoFactorization(Ours)|\n| :------:| :------: | :------: | :------: | :------: | :------: |\n| 46.3% | 50.6%   |  48.3%   | 66.4%   | 58.7% | 66.1% |\n\n- **Advantage of our method:** We found that the images generated by the dataset distillation method have a global characteristic of low rank, which has not been considered in previous image parameterization methods, and their core idea is to use the local characteristics of image pixels. Furthermore, our proposed method based on this property is simple-yet- effective, and can be combined with existing methods as a versatile plug-in to save more information of the original dataset at the same memory cost, greatly improve the performance of traditional dataset distillation methods."}, {"review_id": "OsvRSXbhVo", "reviewer": "Reviewer_VZYQ", "summary": "The authors propose an efficient dataset condensation (DC) plugin that utilizes a low-dimensional manifold, namely two low-rank matrices, to match the large raw and smaller synthetic condensed dataset. This aims to reduce inefficient condensation when matching in image pixel space. They show that their method is modular, and easy to incorporate into existing DC methods, and is effective, improving model performance when trained on the smaller synthetic dataset. Their method reduces catastrophic forgetting when the entire raw dataset is not available for replay, and reduces privacy leakage from the initial raw dataset.\n\nAfter reading the authors rebuttal and discussing with them, as well as seeing the discussion with other reviewers, I will decrease my score to 7.", "questions": "Why do the authors think their method underperforms on MNIST in table 1, which has perhaps the lowest-dimensionality of any of the considered datasets?", "limitations": "I would like to see a discussion of failure modes of limitations of the method, where there any cases that surprised the authors where the model fails? How might this perform in practice, where categories of interest are often highly imbalanced and data is often biased or, in the case of continual learning, where distributions shift over time?", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This work makes nice use of the structure implicit in natural images, namely local connectivity and similarity of nearby pixels and the low intrinsic dimensionality of natural images, to improve dataset condensation. They propose a simple but effective plugin to any modern DC method that performs DC in a low-dimensional manifold as opposed to in pixel space, significantly improving learning efficiency and memory consumption while maintaining downstream performance of models trained on the condensed datasets. The authors also demonstrate the benefit of their low-rank approach for continual learning, an intuitive application as memory is often constrained in continual learning frameworks. The paper is clearly written, the method well-described, and the analysis is thorough. I particularly like the modularity of the method, and that it is applicable and useful across DC methods.", "weaknesses": "It’s interesting that the synthetic digits in Figure 3d no longer always resemble the real categories. What are potential risks associated with this? Is it possible that selecting a condensation dataset via model selection on a fixed set of data could have downstream implications if this model is used on real-world data or data under a slightly different distribution? How would this method fare when datasets are imbalanced or biased, as is often the case in real-world settings? The datasets considered all are uniform and IID, unless I’m mistaken?\n\nNits:\n\nThe images in figure 1 and figure 2 should be labeled with (a) (b) etc for easier legibility", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n- **The impact of synthetic data:** Indeed, in $\\underline{\\text{Fig.3(d) in the original manuscript}}$, it seems intuitively that the generated image is no longer similar to the real image, but from the perspective of model learning, the impact of this difference may be weak. On the one hand, as shown in the $\\underline{\\text{Fig.1 (a),(d) and (f) in the original manuscript}}$, traditional dataset distillation and the proposed low-rank dataset distillation can well preserve the distribution close to the real images. On the other hand, when there are downstream tasks, such as in the continual learning task shown in $\\underline{\\text{Fig.4 in the original manuscript}}$, the data distribution of the new task/class is significantly different from the old task/class. In this case, the method of dataset distillation still performs significantly better than the model trained on the real data (i.e., Random and Herding).\n- **Dataset distillation under imbalanced dataset:** Yes, the original experimental datasets are class balanced. In order to verify the effect when the category data distribution is unbalanced, we sample the data of the original 10 classes of CIFAR10, and the 10 classes respectively sample the original data of $1.0, 0.9, 0.8, ..., 0.1$, thus constructing an imbalanced-CIFAR10. We perform dataset distillation on this imbalanced dataset, and as shown in $\\underline{\\text{Fig.3(a) in the response}}$, the accuracy of dataset distillation methods does not decrease significantly on the imbalanced dataset. This is because the data set is performed class by class (refer to the fifth line of $\\underline{\\text{Alg.1 and Alg.2 in the original manuscript}}$ (that is, *for class $c = 1, 2, ..., |Y|$ do*, where $|Y|$ represents the total number of classes), and the data imbalance between classes will not affect the accuracy of distillation.\n- **Subfigure identification:** We will add subfigure identification as you suggested.\n- **Reason for poor performance on MNIST:** As shown in $\\underline{\\text{Tab.2 in the original manuscript}}$, our LoDC and LoDSA have significant performance improvements compared to DC and DSA. The reason why LoDM is not as effective as DM when Img/Cls=10 in $\\underline{\\text{Tab.1 in the original manuscript}}$ is that we strictly follow the hyperparameter settings of DM, which are obtained by searching on DM. As shown in bellow table, when we fine-tune the hyperparameters of LoDM (learning rate or rank), LoDM achieves $97.7\\%$ performance, which is better than $96.5\\%$ of DM. Similarly, after simply performing hyper-parameters search, our LoDM method also achieves $64.6\\%$ accuracy on CIFAR10(Img/Cls=50), which is better than DM's $63.0\\%$.\n\n\n| Dataset | Base method | Our method(Update version) |\n| :------:| :------: | :------: | \n| MNIST(Img/Cls=10)   | DM: 96.5% | LoDM: 97.7% | \n| CIFAR10(Img/Cls=100)   | DM: 63.0% | LoDM: 64.69% | \n| CIFAR100(Img/Cls=10)   | DSA: 32.3% | LoDSA: 33.7% | \n\n- **The reason why the method fails:** As mentioned above, our low-rank version (such as LoDM) strictly follows the hyperparameter setting of the base method LoDM), and we found that after simply adjusting the hyperparameters of the low-rank method, the proposed method is better than the traditional method on all datasets.\n- **Continual Learning with imbalanced data:** Based on the imbalanced dataset defined in the 2-th question for your reply, we performed a continual learning task on this dataset, which is the more challenging \"data imbalance and distribution drift\". The experimental results are shown in $\\underline{\\text{Fig.3(b) in the response}}$, and there is no significant difference in accuracy when we perform continual learning under the imbalanced version of CIFAR10 and the balanced version of CIFAR10. This is because the dataset distillation is performed class by class (refer to the fifth line of $\\underline{\\text{Alg.1 and Alg.2 in the original manuscript}}$), so the data imbalance between classes will not affect the accuracy of the dataset distillation. Further, the data of each class is distilled into a fixed number(Img/Cls=20) of synthetic images, so it does not affect the performance under continual learning.\n\n"}, {"review_id": "odpPKHBYYx", "reviewer": "Reviewer_Lbw7", "summary": "The present paper studies the task of dataset condensation (DC) with a specific focus on its computational and condensation efficiency. The main idea is to condense the dataset in a low-rank representation space instead of in the high-dimensional pixel space. To do so, the author decomposes an image representation into two low-rank matrices, where the proposed method is orthogonal to the existing DC methods and can be readily integrated with existing DC methods. The author considers the gradient matching-based and distribution matching-based DC methods as examples to show how the proposed method can significantly improve them. Extensive experiments are conducted on common classification benchmarks, and the proposed method outperforms the state-of-the-art DC methods with large margins.", "questions": "Please refer to the Weakness section for more details.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of reducing the input distribution's dimensionality to increase the effective memory size and facilitate the DC is novel. Though leveraging the data's low-rank manifolds/inductive bias is not novel in general, as stated in lines 133-144, it is still interesting to see this idea can effectively work on DC, which is a surging interest in the community.\n\n2. The paper is well-written, with clear motivation and excellent clarity. The technical details are easy to follow.\n\n3. The proposed method can significantly improve the existing SOTA DC methods by a large margin, and it can also sufficiently benefit continual learning, which strongly demonstrates the significance and generality of the proposed method.", "weaknesses": "1. The main weakness is on the experimental protocol. Although the reviewer understands that the author wants to align with the existing methods in DC and choose the same input size for experiment, the input size of the MNIST (28 x 28), CIFAR100 (32 x 32), and TinyImageNet (64 x 64) are too small to be realistic. Given that the proposed method aims to reduce the input space dimensions, it is more favorable to see how effective the proposed method will be when the input size is large enough, e.g., at least 512 x 512. Only by doing so can we clearly see the benefits of the proposed method.\n\nOn the other hand, using realistic input size also provides us with a comprehensive understanding of the proposed method, especially its potential limitations. The reason is that we may intuitively expect that reducing a realistic size of the image to a low dimension (e.g., with rank size 2) can not retain the sufficient information and fine-grained visual context of the image. This is a clear limitation of the proposed method, and the author should definitively provide more analysis of it to inspire future study.\n\n2. The author should at least provide the results of DC and DSA in Table 1 for the TinyImageNet, given that they may achieve SOTA in datasets like MNIST and CIFAR10. It would be helpful to see whether the proposed method significantly outperforms all the comparison methods in a more realistic dataset like TinyImageNet. \n \n3. The result of the proposed method on MNIST is weird when Img/Cls is 10, as it is even worse than the original DM method. Can the author provide any plausible reason to explain it?\n\n4. For the same memory setting in Table 2, how many images have been used by the proposed method? What if the comparison methods in Table 2 also leverage the same number of images as the one used in the same memory setup of the proposed method? Will the proposed method still outperform those compared methods?\n\n5.  In Figure 1, we can observe that only when rank $r=8$ can the proposed method approximate the DM method's latent space structure. Then, how much memory has the proposed method reduced when rank $r=8$? Is this memory reduction still significant?\n\n6.  In lines 284-286, the author states that the proposed method (rank $r=2$) reduces storage consumption by 7x, 8x, 8x, and 8x under different datasets. Please provide the exact number of storage between the proposed method and the comparison method.\n\n7. In lines 322-326, the author states that \"We observe that compared to the images condensed by DM in high-dimensional pixel space, the images we recover under the low-dimensional manifold will be sharper, but still capture the main features of the class, especially on the MNIST dataset.\" However, from Table 1, it is evident that the proposed method is much inferior to the compared methods on the MNIST dataset, which contradicts the claim above-mentioned. How to interpret this phenomenon? \n\nMinor:\n\n1. For subfigure (a) of Figure 1, how does the author obtain this T-SNE plot? Does the author directly plot the T-SNE over the pixel space or the feature embedding of the input images?\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nFinal Acknowledgement on Sep 2nd:\n\nThank you so much for your detailed response. After reading all the reviewer's comments and the corresponding author's response, my concerns have been appropriately addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n - **Advantages in high-resolution images:** Thanks for your suggestion, in fact our method works better when the original image resolution is higher. As shown in $\\underline{\\text{Fig.2 in the response}}$, we sampled a subset of ImageNet and tested the performance of DM and our LoDM at different resolutions $(64 \\times 64, 128 \\times 128, 256 \\times 256, 512 \\times 512)$. We can observe that the performance of DM does not change significantly at different resolutions, while LoDM, which considers low-rank characteristics, can save more information than DM at the same memory cost when the resolution is increased, so the effect continues to improve. In addition, the size of the rank is closely related to the number of images, although a low rank may not be able to restore the fine-grained information of a single high-resolution image, but it can save more images. As shown in $\\underline{\\text{Fig.2 in the original manuscript}}$, we empirically find that the performance across ranks is generally good.\n - **TinyImageNet results:** We supplement the results of DC and DSA under TinyImageNet. For example, as shown in below table, when Img/Cls=1, the performance of DC and LoDC are $4.61\\%$ and $6.8\\%$, respectively; the accuracy of DSA and LoDSA are $4.79\\%$ and $9.9\\%$, respectively. Our method significantly improves the DC and DSA algorithms. We will update the results in the revised version.\n \n| DC | DSA | LoDC(Ours) | LoDSA(Ours)|\n| :------:| :------: | :------: | :------: |\n| 4.61%   | 6.8% | 4.79% | 9.9% |\n\n - **Reason for poor performance on MNIST(Img/Cls=10):** Thanks for pointing this out. The reason we do not perform well on MNIST(Img/Cls=10) is that we strictly follow the hyperparameters on DM, which are fine-tuned for DM. When we simply adjust the hyperparameters (learning rate or rank) of our LoDM in MNIST, LoDM has achieved performance improvement. For example, the accuracy of LoDM in MNIST (Img/Cls=10) is $97.7\\%$, which exceeds $96.5\\%$ of DM. Similarly, CIFAR10(Img/Cls=100) has an accuracy of $64.69\\%$ after fine-tuning, which exceeds $63.0\\%$ of DM; for CIFAR100(Img/Cls=10), the accuracy of LoDSA is $33.7\\%$, while that of DSA is $32.3\\%$. \n\n| Dataset | Base method | Our method(Update version) |\n| :------:| :------: | :------: | \n| MNIST(Img/Cls=10)   | DM: 96.5% | LoDM: 97.7% | \n| CIFAR10(Img/Cls=100)   | DM: 63.0% | LoDM: 64.69% | \n| CIFAR100(Img/Cls=10)   | DSA: 32.3% | LoDSA: 33.7% | \n\n - **Number of images in Table 2:** Suppose the shape of each channel of the image is $(H, W)$, where $H$ and $W$ are the height and width of the image, respectively. The memory consumed by traditional dataset distillation (DC/DSA/DM) can be recorded as $N \\times D \\times H \\times W$, where $N$ and $D$ represent the number of images and the number of channels. The memory consumed by our low-rank dataset distillation method is $N \\times D \\times (H \\times r+r \\times W)$, where $r$ is the size of the rank. Therefore, under the same memory, when the traditional method stores an image in MNIST($28 \\times 28$), CIFAR10($32 \\times 32$), CIFAR100($32 \\times 32$), if $r=2$, then our LoDC(SM)/LoDSA( SM)/LoDM(SM) can store $7, 8, 8$ pictures respectively. On the contrary, when using the same image, LoDC(SI)/LoDSA(SI)/LoDM(SI) is slightly worse than DC/DSA/DM, but they also reduce the memory consumption by $7, 8, 8$ respectively.\n - **Memory consumption:** When $rank=8$ in  $\\underline{\\text{Fig.1 in the original manuscript}}$, on the MNIST dataset, our method saves $(28/16=1.75)$ times the memory for the same number of images.\n - **Relationship between memory and rank:** Please refer to the reply to your 4-th question.\n - **MNIST performance issues:** Please refer to the reply to your 3-th question, our method is significantly better than the traditional method on the MNIST dataset after simple fine-tuning the hyper-parameters.\n - **T-SNE visualization:** We first trained a CNN network using the complete MNIST raw data set, then used it as a feature extractor to extract high-dimensional features of the original image or distilled image, and finally reduced the high-dimensional features through T-SNE visualization to obtain the results of $\\underline{\\text{Fig.1 in the original manuscript}}$.\n\n"}]

Author rebuttal:
Dear reviewers, the figures in the responses are provided in this supplementary material, which contains 3 figures in total:
- Fig.1 is the principal component analysis of real and synthetic images of the dataset distillation method, which shows that both real and synthetic images are low-rank and used to answer the questions of $\underline{\text{reviewers JHfi, Lynm and FpMf}}$.
- Fig.2 is about the distillation performance of the ImageNet dataset at high resolution to answer the questions of $\underline{\text{reviewers Lbw7 and Lynm}}$.
- Fig.3 is about the performance of dataset distillation and continual learning under unbalanced datasets to answer $\underline{\text{reviewer VZYQ}}$'s question.


Meta-review: The authors did a good rebuttal, while the AC agrees with Reviewer FpMf that a thorough analysis and comparison with data parameterization methods will make this work stronger.

The AC would like to support this work, conditioned that a careful revision will be done in the camera-ready
Decision: Accept (poster)