MGDD: A Meta Generator for Fast Dataset Distillation
Authors: Songhua Liu
Xinchao Wang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Dataset Distillation
Dataset Condensation
Efficient Learning
Conditional Generation
Meta Learning
TL;DR: We propose a meta learning algorithm to train a meta generator to generate dataset distillation results efficiently so that the distilled datasets can be generated in a one-stop feed-forward inference after adaptation.

Abstract: Existing dataset distillation (DD) techniques typically rely on iterative strategies to synthesize condensed datasets, where datasets before and after distillation are forward and backward through neural networks a massive number of times. Despite the promising results achieved, the time efficiency of prior approaches is still far from satisfactory. Moreover, when different sizes of synthetic datasets are required, they have to repeat the iterative training procedures, which is highly cumbersome and lacks flexibility. In this paper, different from the time-consuming forward-backward passes, we introduce a generative fashion for dataset distillation with significantly improved efficiency. Specifically, synthetic samples are produced by a generator network conditioned on the initialization of DD, while synthetic labels are obtained by solving a least-squares problem in a feature space. Our theoretical analysis reveals that the errors of synthetic datasets solved in the original space and then processed by any conditional generators are upper-bounded. To find a satisfactory generator efficiently, we propose a meta-learning algorithm, where a meta generator is trained on a large dataset so that only a few steps are required to adapt to a target dataset. The meta generator is termed as MGDD in our approach. Once adapted, it can handle arbitrary sizes of synthetic datasets, even for those unseen during adaptation. Experiments demonstrate that the generator adapted with only a limited number of steps performs on par with those state-of-the-art DD methods and yields $22\times$ acceleration.

MGDD: A Meta Generator for Fast Dataset Distillation
Songhua Liu Xinehao Wang"
National University of Singapore songhua, liu@u, nus edu, xinchao@nus edu.
Abstract
Existing dataset distillation (DD) techniques typically rely on iterative strategies to synthesize condensed datasets, where datasets before and after distillation are forward and backward through neural networks a massive number of times. Despite the promising results achieved, the time efficiency of prior approaches still far from satisfactory. Moreover, when different sizes of synthetic datasets are required, they have to repeat the iterative training procedures, which highly cumbersome and lacks flexibility. In this paper, different from the time-consuming forward- backward passes, we introduce generative fashion for dataset distillation with significantly improved efficiency. Specifically, synthetic samples are produced by a generator network conditioned on the initialization of DD, while synthetic labels are obtained by solving least-squares problem in feature space. Our theoretical analysis reveals that the errors of synthetic datasets solved in the original space and then processed by any conditional generators satisfactory generator efficiently. we propose a meta-learning algorithm, where a meta generator is trained on large dataset so that only a few steps are required adapt to a target dataset. The meta generator is termed as MGDD in our approach. Once adapted, it can handle arbitrary sizes of synthetic datasets, even for those unseen during adaptation. Experiments demonstrate that the generator adapted with only limited number steps performs on par with those ofte--ff-the-rr DD methods and yields 22x acceleration.
upper-bounded To find
Introduction
Dataset distillation (DD) introduced by Wang et al. [51] aims to compress an original dataset T into a much smaller synthetic set S, such that the performance of neural network, trained with the condensed dataset S, is similar the network trained with T. The derived synthetic datasets not only save the cost of storage and transmission but also significantly reduce the computational resources and time required by training models using original datasets. As such, DD finds its application across wide spectrum of domains and is receiving increasing attention from the community.
The typical paradigm of DD is optimize in an iterative loop, as shown in Fig. ((a). In each iteration, new network is sampled, leveraged by which a matching metric is calculated for S and T. and the matching loss is then back propppaee through the network to update S. Recently, large number of approaches have been dedicated to exploring advanced matching objectives to improve the training performance of distilled datasets, including matching training performance via meta learning [51, 7]. matching feature regression performance [37, 38, 61, 32, 33], matching training gradients [60, 58, 20], matching training trajectories [2, 8, 4], and matching feature statistics [59, 50].
Although impressive results have been achieved and it has been demonstrated that neural networks trained by synthetic dataset with even only I image per class can yield reasonable performance
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

XN Times
T
Solve
T
Match
Random Select
Analytical
Label
Generator
Backprop
Initialized 8
Final &
(a) Previous DD Paradigm
(b) MGDD Paradigm
Figure I: Illustration of previous back- back-propagation-basee and our generative fashions after adapta- tion for dataset distillation.
real data, the iterative optimization process adopted in existing works results in significant computational overhead. As shown in Fig. 2, for FRePo [61]. the method with the best performance requires above 3 hours to obtain synthetic dataset with image per class for CIFAR 100 [23], let alone RTP [7], which a back-propagation tirriiinnntrouhttiiitm method and requires over 9 days for optimization. Such dramatic latency makes existing approaches hard to be applied in scenarios requiring high efficiency, like handling data streams. Moreover, when the memory budget for a synthetic dataset changes, existing methods have repeat the time-consuming optimization for the different sizes of S, which lacks flexibility.
To alleviate the drawbacks of the conventional iterative hewaardbackward process, in this paper, we propose a generative dataset distillation approach, where the optimization loop is replaced by single feed-forward propagation, shown in Fig. 1(b). Specifically, given initialization of synthetic samples S, we first obtain the corresponding synthetic labels by analytically solving least-squares problem in feature space. Then, generator adopted to transfer the initialized samples to the final ones. Our theoretical analysis indicates that S solved in an original space can be transferred to the final result by any conditional generator with upper-boure -ounded error, which validates the feasibility of this pipeline.
Then, the key problem of our framework lies in finding suitable generator as quickly possible for the feed-forward generation process of S. this end, we propose method called MGDD, where meta generator is learned with meta learning algorithm on large database like ImageNet [6]. Trained in a learning-to-eert of adaptation steps are required for target dataset unseen in the meta learning. Experiments demonstrate that our approach yields 22x acceleration 2 and comparable performance with existing adapted can also handle
fashion, the meta generator is optimized such that only a small number
-fate-of-tee the-ar art methods, unseen sizes of S during adaptation, which improves the flexibility cross-size generalization in existing DD methods significantly. We also validate that MGDD gets competitive performance target datasets with large domain shifts from those seen in the meta learning.
shown in Fig. 2. Beyond that, the generator
Our contributions can be summarized from the following three aspects:
We propose an innovative feed-forward generation fashion for DD without backward propa- gation after adaptation which significantly improves the efficiency of existing methods;
We introduce MGDD which uses meta-learning algorithm learn a meta generator and helps the generator adapt to a target dataset rapidly;
The proposed method achieves significant acceleration and improvement in the flexibility of cross-size generalization for existing DD approaches with comparable performance.
2
Related Works
Unlike conventional efficient learning schemes that mainly focus on lightening models [9, 34, 55, 54, 10, 18, 19, 17], dataset distillation (DD) looks into data compression: given real large dataset DD DD aims at a smaller synthetic dataset S which can match the training performance of T. The seminal work by Wang al. [51] proposes meta learning approach model this objective: in meta training, network is trained with the current for multiple times, while in meta test, the loss for the updated network is evaluated on T. which is then back- proppaaapd through the bi-level optimization
'The acceleration factor estimated here includes adaptation time. If only feed-forwand time is considered, the acceleration can be 1650x.

---

to update S. The following work by Deng et al. [7] improves performances by adopting momentum during meta training.
Considering the concerns on memory and time complexity of unrolling the computational graph in meta learning, variety of works introduce various surrogate matching objectives to update S. Zhao et al. [60] propose to match training gradients of S with those of T, and following researches [20, 27, 16, 57, 45] focus on improving the classical gradient-matching objective. Beyond the single-step gradient, Cazenavette et al. [2] and subsequent works [4, 8] consider regulating multi- step training effects and propose matching training trajectories. Without the necessity of calculating higher order gradients, the distribution matching methods [59, 50] minimize the distance between feature statisties of S and T, and result in satisfactory computational efficiency. Another branch of methods [37, 38, 61, 32, 33] turn to kernel ridge regression (KRR) model to improve the efficiency of the seminal meta learning based solution, since KRR enjoys the analytical form of solution, which gets rid of the meta-training process and yields best trade-off between performance and efficiency.
Different from works focusing on objectives of DD, some other researches explore methods of synthetic data parameterization, where synthetic samples can be stored in some other formats except the raw one to improve the data efficiency, and raw samples are recov- ered via some functions during down-stream training, e.g., data augmentation [58]. up-sampling [20], linear transformation [7], and neural networks [30, 26, 3, 49].
28
Ours-DS
FRePo
26
24
Ours
MTT
22
RTP
20
18
IDC
16
DSA
14-
12
DC
DM
For thorough introduction to dataset distillation, refer readers to the recent surveys for this area [56, 13, 43, 28]. For existing methods, no matter what objec- tives and parameterizations are adopted, they all rely on an intensive loop of borward-backward propagation through a massive number of neural networks. Al- though concurrent work [31] also adopts similar pre- tr-inriinng and adaptation pipeline, it still relies on an iterative loop to solve an initialization of synthetic data. Different from prior works, we innovatively introduce feed-forward fashion for dataset distillation in this paper. fact, our method is orthogonal to different training objectives and data parameterizations, and can be built upon any combination of them. In this paper, we consider the KRR-based objective thanks to its favorable performance and computational efficiency and experiment with both raw and down-sampled parameterizations.
10
10a
104
10s
106
Log-Seale Rumning Time (sec.)
Figure 2: Results of different DD meth- ods on CIFARI00 with image per class. Our MGDD achieves state-of-the-art ef- ficiency and at least comparable perfor- mance. DS denotes down-sampling pa- rameterization.
Methods
In this section, we elaborate on the technical methods of the proposed MGDD pipeline. We first introduce in Sec. 3.1 some preliminary information related to the matching objective. Then, for the main method, according to the overview of the whole pipeline in Fig. 1(b), the final synthetic labels and samples are derived by solving a least-squares problem and conditional generator given initial synthetic data, which would be illustrated in detail in Sec. 3.2 and 3.3 respectively. Finally in Sec. 3.4, we analyze the feasibility of this two-step pipeline theoretically.
3.1 Preliminary
Let us denote datasets before and after distillation as T = (Xe, Yi) and S = (X,, Y.) respectively, where X, E Re Y, E Re X, E Raaxd Y, E R n4 and n, are number of real and synthetic data respectively, d and e are input and output dimensions respectively. Typically, for the RGB image classification task, d is equal to h x w x 3, e is the number of classes, and Y, is organized in the one-hot format. For the objective of DD, in this paper, we mainly consider the KRR-based methods in neural feature spaces [61, 32] due to its overall superior performance in terms of accuracy and efficiency. Specifically, assume there is an optimal neural network so- to projects X, and X, to feature space with p dimensions and n, < Re < p. The prediction error on T for the optimal KRR
3

---

parameter solved by S, denoted as W is adopted as the loss function:
((S;0") f (X.) - Y1 I (o- (X.) (fo- (X.) (X.) Y. (1)
In practice, since the optimal parameter #* is unknown, it is approximated by different random initializations [32] or alternately optimization with S [61].
3.2 Solving Synthetic Labels
Through Eq. we can find that the loss neural space 0 upper-bounded by the distance between parameters solved by S and T:
((S:0) = ((ss0)==1fe(xx)WW33 Il - Yell2 ll fo- (Xx)
=I) - (x 2 ( w (2) -I
w:"
where 1 denotes the pseudo-inverse of a matrix. In our MGDD framework, synthetic samples X, are initialized as some random real samples in X,. Given a fixed X, and a random network fe. the upper bound in Eq. forms a least-squares problem with respect synthetic labels Y,. which can be minimized by an analytically optimal solution:
Y = (x.) = (o(X)))o(X.) (TXX))))TTXXXXXXXXXXX (xx(X)))o(x))))>>>)
(3)
Y obtained with Eq. serves final synthetic labels. In the subsection, we will introduce the generation of synthetic samples conditioned on their initialization.
3.3 Learning a Synthetic Sample Generator
Conditioned initialized synthetic samples X.. generator yu is adopted to predict the final synthetic data X;, where the parameter w can encode useful information of the target dataset T and the optimal neural space parameterized by 0" We expect that the generator can acquire such knowledge through a fast learning process within a limited number of training steps. To this end, we propose Marning-to-lcane algorithm based on MAML [12]. where meta generation network is learned to optimize the performance of the network adapted for few steps from the meta one.
Specifically, to ensure the generality of the meta generator for different target datasets, we perform the training algorithm on ImageNetlk [6]. a large-scale dataset for image classification. In each training iteration, a subset of all classes is randomly sampled from it to mimic different target datasets that the generator may encounter practice. And the meta generator is learned in a bi-level learning framework including a meta-training loop and meta-testing step, and the meta-testing loss is back- propagated through the computational graph of meta-training steps to update the parameter of the meta generator.
In each meta-training and meta-testing step, from the selected classes, we randomly sample a batch of real images as T and initialize synthetic data X, with part of them. With random and fixed neural network as fo, the synthetic labels Y are solved via Eq. 3. Then, the final synthetic samples X are predicted by the current generator in a forward pass, and S = (X; Y,") is evaluated by the loss L(S; 8*) in Eq. In this paper, following Loo al. [32], we approximate the optimal neural parameter 0** via random sampling in different steps from the distribution for initialization. The loss signal is back- paok-propagated to the current generator and the meta generator to update parameters in meta-training and meta-testing respectively. It is worth noting that in different meta-training and meta- testing times, we use different sizes of synthetic data, which enhances the cross-size generalization ability on target datasets since the meta-testing losses on sizes probably unseen during meta- meta-training are optimized. The main algorithm is summarized Alg. 1. Given a trained meta generator, a limited number of adaptation steps are performed for target dataset. The procedure of adaptation is similar to the meta- trtttttriiing loop in Alg. 1.
As for the architecture of the generator, in this paper, we adopt simple encoder-decoder model, where the encoder consists of three convolutional blocks with two average pooling layers while the decoder has symmetric structure. Notably, we observe in practice that it is beneficial for different sizes of synthetic datasets adopt different transfer functions. Taking various sizes into consideration, concatenate additional size-embedding channels to the bottle-necked layer of the

---

Algorithm MGDD: Meta-I Metaing Framework for Training a Synthetic Sample Generator
Input: 2: A Large Dataset; 0: Parameter of a Random Neural Network; T: Number of Meta- Training Steps; oc: Learning Rate in Meta-Training: Learning Rate in Meta-Testing.
Output: w: Parameter of a Meta Generator.
procedure GETTRAININGLOSS(((
2: Initialize X, with some random real images from T;
3: Obtain Y with the analytical solution in Eq. 3;
4: Forward propagation with X < 9e (Xx);;
5: Randomly sample 8* and compute the loss [((X;, Y,);; 0*) in Eq. 1;
6: return [((X;, (²))
7: end procedure
8: Initialize randomly;
9: repeat
10: w' copp(w))
11: Randomly choose subset of classes C from Z;
12: Sample batch of images of C' as T;
13:
for
i < T do
14:
C = GetTrainingLosss()
Meta-Training
15:
Back propagation and update w': w' t w' - aW
16:
end for
17:
C GetTrraiiingoos
Meta-Testing
18:
Back propagation and update
BV
19:
until convergence
generator, inspired by the design of the position embedding in Transformer models [47] and the time-step embedding in diffusion models [15, 39, 41]. Please refer the appendix for detailed architecture configurations.
3.4 Theoretical Analysis
There are two key steps the proposed MGDD framework: solving the optimal synthetic labels Y as introduced in Sec. 3.2 and generating the corresponding synthetic samples X as introduced in Sec. 3.3. Define the error L(S; 0) in Eq. using the fixed X, and the optimal Y. in Eq. 3 with the projection function of fo as The reason we pursue the optimal Y, is that the final X; is transferred from the initial X,, whose error in the optimal neural space parameterized by 0" is upper-bounded by derived in the following theorem.
Theorem 1. Given fe IRd + R², fo- Bd RR,, X1 E Rn.xd Y, E Ro X, E Rn, < p. Y obtained by Eq. 3 with the optimal L((X,, 0) denoted as e, and an arbitrary transfer function Iw parameterized by taking Xa input, the transferred you (X.) yields upper-bounded loss L((g (Xs), (;);)))
Proof. We first rewrite the given condition:
0)): = ( ( - Yell3 =
(4)
Then, we have:
[((g,(X)), = fe (X.) Yoll3
fe- (o (g (X.)) (fe- (fx (x) Yell3
(5)
I fo- (Xx) (o( + -Y11
= [e (X.) Wx1
where the inequality based on the triangle inequality and the last equation is due to Eq. 4.
Theorem I indicates that achieve feed-forward dataset distillation, we do not need to design neural network taking the whole original dataset T as input. We can in fact adopt conditional
5

---

Dataset
CIFARIO
CIFART00
1D
Ratio (%)
0.02
0.2
0.2
Random
Acc. (9)
21.87:0.30
38.86-10.31
57.07:40.54
0.06 0.16
24.50=10.08
Full Dalaset
Acc. (%)
84.8100.1
56.21.0.3
Acc. (%)
28.20:10.71
43.74::0.41
53.43:1..28
12.44.10.18
25.08:10.17
[C]
Time (sec.)
153
3605
20090
1496
33749
(%)
28. 10:40022:22
52.155+0:48
60.58:40.29
13.81
32.49::0.30
DSA [58]
Time (SCC.)
172
3871
21217
1667
35905
(%)
35.34:1::887
08.50:10.39
69.32:100300
17.93:20.15
36,08-10.38
IDC [20]
Time (sec.)
39062
40244
45888
189811
198101
Acc. (%)
45 29-4-0.86
62.77:20.56
0109920.34
24.17=0.57
39.43-00.26
[2]
Time (sec.)
2972
8588
8601
8063
10295
Acc. (%)
27 07.36
48-80-1-0.31
02.94
11.51
29.13:::0.33
DM [59]
Time (sCO.)
1123
1182
1423
10942
11355
Acc. (%)
49.1=:0.6"
62.4=00.4"
70.5:1:.4"
21.3+:0.6"
34.7:+0,.5
RIP 17]
(sec.)
816212
1647559
1660726
1672205
Acc. (%)
43.24:10.32
65.76:10.72
71.03.1.334
27.07:1:.26
39.97.10.32
FRePo [61]
Time (sec.)
8507
11133
29185
11231
37607
Acc. (%)
46 06.267:0
60.76::0.38
69. 09.50==0117
24.6620.15
36,47::0.27
Time (scc.)
120 6802
505
1395 x
505 x05x22
3004x13
Ours
Lahel (sec.)
7
7
7
7
Generator (scc.)
113
498
1388
498
2997
Forward
2
20
Table Comparisons on test accuracy and running time with state of the arts. The acceleration marked by the red subscript is computed against the method with the best accuracy. We also provide detailed analysis for our method on the time cost for each component in adaptation, including solving synthetic labels, updating the generator from the meta generator, and feed- forward generation. IPC: Number of Images Per Class; Ratio: ratio of distilled images to the whole training set. denotes results from the original paper.
generation function that transfers the initial synthetic samples to the desired ones, which has an error upper bound as shown in Theorem 1. Since the upper bound is related to the error in the original space parameterized by 8, is crucial solve the optimal synthetic labels with respect @ in the first step. Also, we notice from Eq. 5 that the optimal generator is dependent on T and 0*. Given that l intractable and can only be approximated by iterative sampling, we build meta learning algorithm in Alg. I for the MGDD framework to enforce an efficient process to model this dependency in only a few steps.
Experiments
4.1 Implementing Details
As discussed in the previous section, there are 3 stages in the proposed MGDD, including training, adaptation, and inference stages. In the training stage, aim at meta generator In and adopt Alg. 1 to train I., on a large dataset. In this paper, to ensure that the meta generator can acquire general knowledge for fast adaptation on target dataset, we use ImageNetlk [6], a large-scale image recognition dataset popular in the computer vision and machine learning communities, as the dataset for meta learning. There images to the 32 x 32 resolution. In each outer loop of Alg. 1, we randomly select 100 classes at most and a batch of data with 2,000 images in maximal from the selected class as current target dataset T, with which we initialize a synthetic dataset S with 1,000 samples at most in each inner step. The training objective is based on Eq. I and the implementation follows the open-source code of FRePo [61]. For computational efficiency, the generator processes each sample independently and the configuration of the architecture can be found in the appendix. The meta generator trained by [21] and the learning rate 3 is set as 10-5, The learning rate in meta- meta-training set as 10-4 and the number of meta-training steps T is The meta generator trained with 200,000 meta-testing iterations. We use a cloud server with a single A40 GPU for meta learning and a workstation with a single 3090 GPU for the subsequent adaptation. The meta learning takes roughly 2 days while the time cost of adaptation is analyzed in Tab. 1.
roughly 1,280,000 images in total of 1,000 classes. We resize all
the Adam optimizer

---

IPC Steps Methods
A
PACS
Path
Blood
PathBlood
CUB200
Baseline 53.54=0.99
33.53=3.19
1.26 39.76:11.74
38.03=00.60 54.23:3..12 68.38:00.55 1..1
4.84:10199
1,000 w/o 5..74
00.29
0.32
0.77 42.73::0.25 1.306:1:.36
58.03:00551
6.73:1:.133
Ours 57.31=1.41
41.18
1.00
43.98=0.55
43.46-:0.89
1.19
00.8
59.66
7.40-10.20
Baseline 55.80=
0.52
38.53:0.800
54, 70 07
68.55
39
60.62 1.14
5.65:20.26
2,000 w/o Meta
57.37 1.38 42.15
62.84
41.47
1.05
43.89
58.33
1. 71.28
1.61 60.72.10.17 7.25 1.009
Ours 60.18=1.99
42.50=0.09
63.96
0.94 04..0099444442..0 44.92=00448 59.06
72
62.22::0.27 0.68
Bascline
68.64:0.52
41.39
59.38
52.85 011.:551114144...0 44.87:00228
64.74:::
1.10
64.771166633 72
02.41::0.20
Full
Ours 66.77=1.881 44.24
64.48
0.68
53.40
48.29
70.25-10.11
72.93
1.70 65.30:10.44
12.21
Baseline
76.33=0.31
1.61
10.92=1002
63.
2:26:::::222
39.54:10.34
76.18:20.87
80.09:10.822
74,4620091 10.33-40.64
1,000 w/o Meta
77.2 54.57=0.75
72.81=0.10
10.1
58.33
75.00:00.74
79. 19. 74.45=0.39 13, 03.09::0.09
79.40=0.29 56.18: 0.34
74.49
0.36
61.41:10.13
06.21:10.10
00.20:10.41 7.69 13.99:10.09
Baseline
77.59=0..41 $1.18.12.31 72.36.
67.68
0.86
59.86.10.21
06.9610..70
81.73:1..37 75.01:10.51
11.25
10
2,000 w/o Meta.
78.24=0.42 55.52=0.63
74.89=0.23
01.09=0.63
60.69:20.35
77.06=0.118
81.71=00377
75.26==0.07 13.83:40.24
Ours
80.78=1.05 56.82=0.51 76.12=0.63 72.11=0.48 62.33:::.15 77.44:t0.11 82.31=:0.45 76.32:+0.14 14.69:10..77
Baseline
85.40=0..24 60.64 1.64 78. 0.54 76.87 1.09 65. 65.38:10.48 77. 17.44111.5 1.53 85.81 76.58:10.255 16.84.10,12
Full
Ours
59 39.47= 0.88
1,20 0.. 0..91 68.76:10.26 78.21 1.42 84, 84.47:40.23 77.07:10.98 15.05:00.37
Table 2: Evaluations on PACS, PathMNIST, BloodMNIST, and CUB200 datasets. The baseline is
FRePo [61]. w/o Meta denotes adapting from scratch instead of meta generator.
In the adaptation stage, we load the meta generator obtained by the training stage and try to adapt it for target dataset. The optimization is similar to that in the meta-training step of Alg. and the difference is that we use the Adam optimizer to update the parameters instead of pure gradient decent. Notably, there are two ways to initialize X, in each adaptation step, namely single-initialization and multi-initializatio
modes. In the single-initallization mode, X. is set the same group of random real samples in the real dataset for all adaptation steps, while in the multi-initialization mode, it can be set differently, with various random seed or/and various sizes of synthetic datasets. We experiment with both ways in this section.
4.2 Comparisons with State of the Arts
In this part, we compare the proposed MGDD with existing state-of-the-art DD methods on standard benchmarks, including CIFARIÓ and CIFAR100 [23] datasets. There are 50,000 training images and 10,000 testing images in the 32 x 32 resolution for both datasets, and the numbers of classes are 10 and 100 respectively. For CIFARIO, we compare our method with state of the arts on settings of 1, 10, and 50 images per class (IPC) in synthetic datasets and for CIFAR 100. IPCs are and 10. The candidates are three gradient-matenn mathinngttttthinggg methods DC [60]. DSA [58], and IDC [20], one trajectory-
method MTT [2]. one distribution- matching method DM [59], one back propaggttion-hhouge titm time (BPTT) method RTP [7], and one
method FRePo [61]. We obtain synthetic datasets using each algorithm on 3-layer ConvNet whose architecture is the same as that [61] and evaluate the accuracy on testing datasets for networks with the same architecture trained on synthetic datasets. For our method, we adapt the generator from the meta model on each comparison setting for 10,000 steps, and for each setting, we report mean and standard deviation over the accuracy of 3 networks initialized from scratch. Moreover, we report the cost of time to derive synthetic datasets.
As shown in Tab. 1, our method achieves comparable performance with DD methods based on heavy iterative loops and yields significant acceleration. For maddinn-mmaching based based solutions, they typically rely on computing second-order derivatives over large amount of neural networks to back-propagate the matching loss to synthetic data, which results high computation overhead. Even worse, for vajectory-matchinn and BPTT methods, higher-order derivatives are necessary. As a result, their time cost for optimizing synthetic datascts is formidable, and our method achieves up to 6800 acceleration compared with BPTT. For DM and FRePo, although they do not rely on high-order gradients, they typically require a massive number of forward-backwer update synthetic datasets, e.g., the FRePo baseline requires 500,000 iterations for optimization, and the efficiency is still unsatisfactory. On the contrary, the results of our method are produced by a generator in one-stop fashion, which is trained by meta-learning algorithm and learns to adapt to a target dataset rapidly. It thus enjoys the superior running efficiency of DD. More analysis of the accuracy performance given the same running time can be found in the appendix.
iterations to
Notably, although the generator is only trained under 32 x 32 RGB images in meta-learning, given that it has calyyccomvllutional structure and maintains the same resolution for input and output, it
7

---

Dataset
CIEARIO
CIFARIOD
Train Arch.
Unseen Arch.
IPC
10
50
10
IPC Method
ConvNet
AlexNet
VGG
ResNet
FRePo
26.8 49.6=0.1
62.0-04
10. 1+0.3
29 02.6
Baseline
49.610.11
44.5.10.7
33.0:0.11
31.81.6
Ours
FRePo
42.6:00.3
58.9=0.4
66.8=02
20.8+:22
02.
Ours
58.920.4
55.11004
35.9.066
32.7
Ours DC
DDC
24.7::004 43.1=0.3
34.3:104 46.2..06
6..0.0.4
5..0=033|
14.110.11
6.620.2
22.31004
19.8::0.3
10
Baseline
Ous
26.8=0.77
42.6:b0.3
23,4+:0.3
39.6::0.8
22.9:b0.6
16.900.1
19.1
15.1
25.74
45.5=0.4 57.4=0.5
9.640.2
21.9::.33
Baseline
62.0::0.4
$9.2:10033
48.7::1.11
48.2
Ours DM
31,4+0.2 48.9=0.2 60.8=0.4
16.6+02 22.9+0.3
Ours
66.8.10.2
62.8.10.2
50.9.10.7 5.241:.2
Table 3: Performance of using different DD training objectives during adaptation.
Table 4: Performance of different network architectures on CIFARIO.
is applicable to datasets with larger resolutions. It can also be adapted for datasets with different numbers of input channels with minor modifications. In the appendix, we provide more experimental results of such cross rrosstccsouution and wross-channel-numbee generalization cases.
4.3 Empirical Studies
In this part, we focus on some interesting properties of the proposed MGDD method, including cross- dataset, cross-objective, cross-architectur,, cross-IPC settings. More studies including cross-resolution, cross-chanel-nnumeer and cross-class- number that cannot fit into the main content are put in the appendix.
paas-synthetiic-nitaaizzat
cross- pross-parameterization and
Cross-Datases Generalization: MGDD proposed in this paper is expected to be generalized to any downstream target datasets, including those unseen and with large domain shifts from datasets used in the meta learning. To evaluate the cross-dataset generalization performance of MGDD, we conduct experiments more datasets including one domain generalization dataset PACS [29], two medical classification datasets PathMNIST and BloodMNIST [53], and one fine-grain image classification dataset CUB200 [48]. PACS contains 9,991 images from 7 classes and 4 domains: Photo (P), Art Painting (A), Cartoon (C), and Sketch (S). The style variations across the 4 domains are drastic. We perform dataset distillation on each domain both independently and jointly, which formulates a 28-class dataset. PathMNIST and BloodMNIST contain 107, images of 9 classes and 17,092 images from 8 classes respectively. We also combine them together to form a 17-class dataset denoted as PathBloodMNIST. CUB200 contains 6,000 images of 200 bird species. We process all the images to the 32 x 32 resolution in the RGB format and compare the performance with the FRePo baseline [61] and a generator from scratch instead of the meta generator, under 1,000 and 2,000 steps as well as full convergence. The quantitative results shown in Tab. 2 validate the robustness of MGDD under various datasets and domain shifts.
Cross-Objective Generalization: By default, both the meta-learning and adaptation objectives used in this paper for MGDD are the KRR objective in Eq. following FRePo [61]. Empirically, we find that it is also feasible to adopt different objectives for adaptation. Here, switch the adaptation objective to DC [60] and DM [59] respectively. The optimization steps for both baselines and our method are set as 2,000. As shown in Tab. 3, our methods yield consistent improvement over different baselines in a limited number of optimization steps.
Poos-AArcittcturg Generalization: Adapted on an original architecture, a satisfactory generator expected to produce results that are also valid to train networks with different structures, namely arosttrrciee generalization. Tab. 4 shows the performance on CIFAR1O, 10, where ConvNet is used in adaptation while AlexNet [24], VGG11 [46], and ResNet18 [14] are used as unseen architectures for evaluation. The results indicate that the accuracy improvement on the original model still holds for unseen structures.
Cross- Mooss-Intiilliationn Generalization: We find that the single- ingtaaiiiitiiiiiniion scheme used in Tab. I may lead to over-fitting of the generator to the adopted single initialization. As shown in Tab. 5, if we change the initialization of synthetic data, the performance would drop dramatically. Fortunately, multi-initialization is an alternative to account for this drawback, wherein each adaptation step, a new initialization of the synthetic dataset is sampled from the original dataset. Tab. 5 indicates that multi-initializatinn typically requires more adaptation steps for convergence and can perform on par with single-initializatin privacy, given that visualization results produced by the KRR objective are somehow realistic, as illustrated in Fig. 6 and [61]. In such cases, replacing the current dataset with the results of another
It is useful when samples in the synthetic dataset are concerned with user
8

---

Default
Multi-Init
Dataset
CIFARIO
CIFARIOO
Setting
Original
Difterent Init Different Init
Acc. (%)
0..76=0.38
43.27:4:0.6
60.24:4:125
Ours
14622640..7 60.76++0.38 24.06=
Time (sec.)
505
1108
Ours DS
48.17:10.77 63.97.10.00 27.76.10.4
Table 5: Using multiple initialized samples during adaptation can enhance the cross-initialee moss-initialization performance. Performance on CIFARIO with 10 IPC is shown here.
Table 6: Our method is orthogonal synthetic dataset parameterization methods and provides improvement gain on small synthetic datasets, DS denotes down-sampling parameterization.
65
Oun from Maa Adgg Ouuc
Tost Accumcy (%)
Test A Acuracy (1)
53.0
Test Accuracy
.55
Besellie
$2.5
50-
55-
50.0
45
50-
Mot w/o Ensbel Randen
47.5
Ao Mite
49-
w. Ane. 1abel
33
30
Ous
25
10
20
30
40
$0
100
200
100
200
400 S00 1000 2000 Adaptation Steps
300
10
IPC
CL Task Index
Figure 3: Performance of vari-
Figure 4: Performance of gen- erators with various adaptation steps on CIFARIO with 10 IPC.
Figure 5: Performance of con- tinual learning on CIFARIO0
ous IPCs on CIFARIO. Only 10 and 50 are seen in adaptation.
initialization would help solve the problem efficiently, without the necessity to re-run the whole optimization loop of existing methods.
Generalization: Beyond different training objectives, the proposed MGDD also orthogonal with different synthetic data parameterization tricks. In Tab. 6, we consider storing 2x down-sampled synthetic images instead of the raw ones. Thus, 4x synthetic samples can be stored given the same storage budget. We find that the simple strategy can lead to additional performance gain for relatively small budgets. The observation is consistent with previous works [20, 30, 7].
Cross-IPC Generalization: One crucial benefit of the MGDD is the cross-IPC generalization. Once the generator is adapted for target dataset, when the storage budget changes, we do not need to perform the optimization again, unlike previous solutions relying on iteratively updating synthetic datasets. To demonstrate the cross-IPC generalization performance, we conduct experiments on CIFAR10 and adapt the meta generator using multi- initialization, with 10 and 50 IPCs for 2,000 steps. The adapted generator is evaluated on unseen IPCs 20, 30, and 40. The results are shown in the red curve of Fig. 3, where the generator produces satisfactory synthetic datasets with unseen IPCs.
To make the generator aware of the sizes of synthetic datasets, we concatenate size embedding channels to features the middle layer of the generator. To understand how the embedding works, we remove these channels and conduct the same evaluation. As shown in the yellow curve of Fig. 3, the performance degrades without size embedding. In Fig. 6, we visualize some samples before and after the adapted generator on CIFAR10 with I and 50 IPCs. We can observe that the results are quite different: for small sizes, the generated results are vague, while for large sizes the results are more realistic, and their styles are also different. Thus, it is reasonable to take different sizes of synthetic datasets into consideration in the inference stage.
We also try training the generator from scratch instead of the meta model on the target dataset. As shown in Fig. 3, the worse performance in the green curve indicates that meta-learning is crucial for finding a satisfactory initial checkpoint for downstream adaptation.
Moreover, we also evaluate the performance on higher IPCs like 100 and 200, and the results are still encouraging compared with random real samples, which indicates that our method can serve as an alternative when the computational resource cannot support optimization for larger IPCs directly.
Various Adaptation Steps: In Fig. 4, we visualize the accuracy under different adaptation steps on CIFAR 10 with 10 IPC as the red curve. Compared with training from scratch and the baseline FRePo [61], as shown the yellow and green curves respectively, our method results in significantly faster adaptation convergence, which would be attributed to the good initial generator found by the meta-learning algorithm. Thus, our method more applicable in scenarios requiring high efficiency, like processing streaming data.

---

IPC before Generator
50 IPC before Gemerater
afCle Generalor
50 aflle Generatir
Figure 6: Visualizations of samples before and after generator on CIFARIO with and 50 IPC.
Furthermore, to demonstrate the effectiveness of analytical labels, we replace them with vanilla one-hot labels in synthetic datasets and the performance is shown in the blue curve. The considerable performance drop indicates the importance of minimizing the error in an original space via analytical labels, which is consistent with the theoretical analysis in Theorem 1.
4.4 Application: Continual Learning
Continual learning (CL) aims to learn from stream of data, where historical and future data are unavailable when learning the current data batch. One important issue is catastrophic forgetting [22]: model tends to forget knowledge acquired in previous data when learning newly-coming data. Focusing on this drawback, many works introduce buffer with limited memory to store core data and knowledge of past experience for future use [40, 1]. Dataset distillation benefits this field by generating informative samples [36, 5, 42, 35, 44] to prevent forgetting as much as possible.
In this paper, we evaluate the CL performance of the proposed MGDD on CIFAR100, following the same protocol of [61, where all 100 classes are divided into 10 tasks randomly with 10 classes for each task. For each task, a buffer with 20 images for each class is allowed for synthetic data. We first try adapting the generator on each new task from the meta model for 2,000 steps and the performance is shown in the yellow curve in Fig. 5. Alternatively, we can choose to adapt the generator from the checkpoint of the previous task, which has already learned some global knowledge of full data and yields better performance, as shown in the red curve. Notably, it is also feasible only adapt the generator on the first task and the remaining tasks directly adopt this generator to output synthetic data. As shown in the blue curve, with the most significant flexibility, the performance is still comparable with the FRePo baseline [61] shown in the green curve, which suggests great practical value for processing data streams.
Conclusion
In this paper, we propose MGDD, a novel feed-forward paradigm for dataset distillation (DD). Specifically, in our pipeline, synthetic labels obtained by solving least-squares problem equipped with an analytical solution, and synthetic samples are transferred from their initial results by a conditional generator instead of taking the whole original training dataset input. Theoretical derivation indicates an error upper bound of the proposed framework. On the one hand, unlike existing DD approaches requiring time-consuming borward-backward iterations through massive number of networks, MGDD generates distilled results with a generator adapted rapidly from a meta generator, which improves the efficiency of DD significantly. On the other hand, existing techniques have to repeat the whole iterative algorithms for different sizes of synthetic datasets, while MGDD can perform inference flexibly on various sizes once adapted to the target dataset. Focusing on the efficiency of adaptation on target datasets, we propose meta- meta-learning algorithm to train meta generator, such that can acquire knowledge of target datasets sufficiently in only few steps. Experiments demonstrate that the proposed MGDD performs on par with existing state-of-the art DD baselines under x acceleration. also exerts strong cross-size generalization ability even on sizes of synthetic datasets unseen during adaptation. Future works may explore advanced feed-forward fashions of DD, focusing on generation pipelines, training algorithms, and network architectures, making improvements the datased cross-size, and cross-ane architeccee generalization.
10

---

Acknowledgment
This work is supported by the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant (project number: A0005947-21-00 project refer- ence: ECT-RP2), and the Singapore Ministry of Education Academic Research Fund Tier (WBS: A0009440-01-00).
References
[1]
Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking experience replay: a bag of tricks for continual learning. In 2020 25th International Conference on Pattern Recognition (ICPR). pages 2180-2187. IEEE, 2021.
[2] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories, arXiv preprint 0rXiv22203.19922 2022.
[3] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexci Efros, and Jun Yan Zhu. Generalizing dataset distillation via deep generative prior. arXiv preprint arXiv:2305.01644 2023.
[4] Justin Cui, Ruochen Wang, Si, Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- with constant memory. arXiv preprint 0rXiv:2211.10558 2022.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3385 2021.
[6] Deng, Dong, Richard Socher, Li-Jia Kai Li, Fei-Fei. Imagenet: Acrge-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255.. leee, 2009.
[7] Zhiwei Deng and Olga Russakovsky. Remember past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206.02916 2022.
[8] Jiawei Du, Yidi Jiang, Vincent TF Tan, Jocy Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. arXiv preprint 1. 2022.
[9] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
[10] Gongfan Fang. Xinyin Ma, and Xinchao Wang. Structural Pruning for Diffusion Models. In Advances in Neural Information Processing Systems, 2023.
[11] Fastai, Fastai/imagenette: A smaller subset of 10 easily classified classes from imagenet, and a little more french.
[12] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agmostic meta-lcarning for fast adaptation of deep networks. In International conference machine learning, pages 1135. PMIRR 2017.
[13] Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. arXiv preprint arXiv:230..01975. 2023.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[16] Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208, 00311, 2022.
[17] Yongeheng Jing. Efficient Representation Learning With Graph Neural Networks. PhD thesis, 2023.
[18] Yongcheng Jing, Yiding Yang, Xinchao Wang. Mingli Song, and Dacheng Tao. Amalgamating knowledge from heterogeneous graph neural networks. In CVPR, 2021.
[19] Yongeheng Jing, Chongbin Yuan, Li Ju, Yiding Yang. Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In CVPR, 2023.

---

[20] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun. Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. arXiv preprint urXiv:2205.14955 2022.
[21] Diederik Kingma and Jimmy Ba. Adam: arXiv: 1412.690, 2014.
method for stochastic optimization. arXiv preprint
[22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11<(13)3551-33526 2017.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[25] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yaun.eccu..om/// cct/e///.omm/ex//nisssss.. 2010.
[26] Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.000 00719, 2022.
[27] Sachyung Lee, Sanghyuk Chun, Sangwon Jung. Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. arXiv preprint arXiv:2202.029100 2022.
[28] Shiye Lei and Dacheng Tao. A comprchensive survey to dataset distillation. 1iXiv:2301.05600 2023.
arXiv preprint
[29] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEFE international conference computer vision, pages 5542-5550, 2017.
[30] Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Proceedings of the Advances in Neural Information Processing Syxtems (NeurIPS), 2022.
[31] Songhua Liu and Xinchao Wang. Few-shot dataset distillation via translative pre-training Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 18654-18664, October 2023.
[32] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using ran- dom feature approximation. In Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), 2022.
[33] Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755, 2023.
[34] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLMM-runer: On the Structural Pruming of Large Language Models. In Advances in Neural Information Processing Systems, 2023.
[35] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual lcarning in image classification: An empirical survey, Neurocomputing, 469:28-51, 2022.
[36] Wojeiech Masarczyk and Ivona Tautkute. Reducing catastrophic forgetting with learning on synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Workshop, 2020.
[37] Timothy Nguyen, Zhourong Chen, and Jachoon Lec. Datasct meta-lcarning from kernel ridge-regression. arXiv preprint arXiv:201 1,00000 2020.
[38] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34, 2021.
[39] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising difffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[40] Sylvestre- Rebufti, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
12

---

[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffussin models, 2021.
[42] Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. arXiv preprint arXiv:2103. 158511,,.1.,,...1. 2021.
[43] Noveen Sachdeva and Julian McAuley. Data distillation: A survey, arXiv preprint arXiv:2301.04272. 2023.
[44] Mattia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu, Sample condensation in online continual learning. In Proceedings of the International Joint Conference on Neural Networks (IJNN)) pages 1-8, 2022.
[45] Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and I1-Chul Moon. Loss-curvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics, pages 8606 8628. PMLR, 2023.
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion, arXiv preprint arXiv: 1...144099555. 2014.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[49] Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling datasct into generative model. arXiv preprint arXiv:2303.04707, 2023.
[50] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang. and Yang You. Cafe: Learning to condense dataset by aligning features. arXiv preprint arXiv:2203.0153// 2022.
[51] Tongzhou Wang. Jun- Yan Zhu, Antonio Torralba, and Alexci A Efros. Datasct distillation. arXiv preprint arXiv: 10959, 2018
[52] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for henchmarking machine learning algorithms. CoRR, abs/1708.07747 2017.
[53] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):1,, 2023.
[54] Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge neural networks. In European Conference on Computer Vision, 2022.
[55] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly, In Advances in Neural Information Processing Systems, 2022.
[56] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. JEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
[57] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Xu Dongkuan. Accelerating dataset distillation via model augmentation. arXiv preprint 222:022212.06155. 2022.
[58] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685 PMLR, 2021.
[59] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 651 651146523, 2023.
[60] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, arXiv preprint arXiv:2006. 05929, 2020.
[61] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00771 2022.
13

---

In this part, we include more details about the technical methods, more experimental results of our MGDD method, and more discussion on limitations and future works, which cannot be accommodated in the main paper due to the page limit. Our method first trains meta generator to generate synthetic samples and then an adaptation stage is executed for a target dataset. We provide algorithmic details of the adaptation stage, summary of hyper-parameters, and configurations of our generator architecture. Then, we conduct more evaluations on the rros-number-of-chae cross-umbbrr-ffcllaee
cross-resolution, cross-ipe, and performance of our method. More discussions of the adaptation performance and more qualitative examples are also included. Finally, we discuss limitations of the proposed method and potential future works.
A More Details
Adaptation Algorithm: Alg. demonstrates the procedure of meta learning obtain meta synthetic sample generator. On a downstream target dataset, the meta network is adapted to specific network with limited number of steps. The adaptation algorithm is similar the meta-training step of the meta learning algorithm. Here, we present the full details in Alg. 2. We can prepare multiple initialization of synthetic samples through randomly sampling from the target dataset. Recall that the main pipeline of our algorithm is to first obtain analytical synthetic labels in a random neural space A: Here, the optimal kernel-ridee rideell-iidg---eressioo parameters of the target dataset W can Y fo),), be computed by W fo(),) (fo(X,) (X., ()) if the number of real samples n, is smaller foxx)))>>o((x) Y..
than the feature dimension Otherwise, W = (fo(X,)
Algorithm 2 Adaptation Algorithm of Synthetic Sample Generator for a Target Dataset
Input: (X,, Vi): Target Dataset; T: Number Adaptation Steps; a: Learning Rate of Generator; 0: Parameter of Random Neural Network; w: Parameter of Meta Generator; I: A Set of Randomly Initialized Synthetic Samples.
Output: w': Parameter of a Tareet-Specific Generator.
1: W = -X) (X. )-;
2: for Each X, in I do
3: Y = = foo,( W:.
b Eq.
4: end for
5: Initialize generator parameters with w;;
6: for 1 1iis 5 T do
7: Sample a batch of real data (X;, Y7) of from (X,, Yt);
8: Sample initialized synthetic data (X,, Y?) from I;
9:
X 9w (X+);
D Forward propagation
Sample neural parameters 0" from random distribution;
11: c= (xl.f- (X,) (X;) (X.-(())aa (-;) Yr1
> Eq. I
Update w/ via w/ t w' V.
D. Back propagation
13:
end for
After the calculation of analytical labels, we fix them and train the synthetic sample generator initialized by parameters of the meta generator for some steps. The optimization objective is similar to those in Zhou et al. [61] and Loo et al. [32]. The difference is that the optimization target is parameters of the generator instead of synthetic samples.
Summary
For clear view, we summarize the hyper-parameter and their values in both meta learning and adaptation stages as shown in Tab. 7. All experiments follow these default settings of hyper-parameter if not specified. Other configurations unmentioned follow the settings of the baseline FRePo [61].
Parameters:
Generator Architecture: We illustrate the detailed configurations of our generator architecture in Fig. 7. It essentially adopts encoder-decoder structure with 3 -ar---Bart-o---BBee blocks and AvgPool layers for down- wwwn-namplnnggaaam for the encoder and a symmetric structure for the decoder. Notably, to make the network aware of different sizes of synthetic datasets, we concatenate the size embedding to bottle-necked features after the encoder. Inspired by the positional embedding in Transformer models [47] and the time-step embedding in diffusion models [15, 39], we encode the size by sinusoidal signals and learnable non-linear transformation function. Embedding features are replicated and expanded along the spatial axes before concatenation with features from the encoder.
14

---

Hyper-Parameter
Notation
Value
Conv
Learning Stuge
ReLU
Sinusoid
Conv 3
Number of Meta Training Steps
Numher Meta Testing Steps
T
T
200,000
Maximal Number of Classes
max(C)
100
Minimal Number of Classes
min(C)
Maximal Number of Synthetic Samples
max(n.)
1,000
Cons 256
ReLU
Minimal Number of Synthetic Samples
min(n.))
Number of Real Samples
2,000
BN
ReLUU
BN
Learning Rate in Meta- Beta-Training
a
le.4
Learning Rate in Meta-Testing
Io-5
KeLU
Conv 128
Parameter of Adam Optimizer
(3,, P2)
(0.9. 0.999)
AugPool
Expand
UySample
Parameter Cosine Learning Rate Schoduler
n
Aduptation Stage
Number of Adaptation Steps
T
1,000
Conv
Batch Size of Real Data
1,024
BN
HN
Learning Rate of Generator
1c-4
Parameter of Adam Optimizer
(8 A2)
(0.9. 0.999) 0.1
KeLU
Concat,
ReLU
Parameter Cosine I Learingg Rate Scheduler
"
Figure 7: Architecture of our generator network.
Table 7: List of hyper-parameters.
Dataset
MNIST
FashionMNIST
50
Ratio (56)
0.017
0.17
0.83
0.017
0.17
0.83
Random
Acc. (%)
64.943.
95.1
97.920022
3..4::3.8
73.8:::.77
82.5
Full
Acc. (%)
99.61:0.0
93.5:1:0.1
Acc. (%)
91.7++0.5
97,4+00,2
98.8=0.2
70.3::0.7
83,4+:0.3
82.9+0.2
DC [60]
Time (sec.)
157
3581
19811
155
3597
19829
(%)
88.7.00.6
98.810.22
99.2.10.1
70.3
84.6.00.1
88.710.1
DSA [58]
(sec.)
172
21259
3854
Acc. (%)
89.1:10.1
97.810..1
98.810.11
70.6:10.4
85.2.104
88.910.11
IDC|20]
Time (sec.)
22062
22798
28389
21929
23160
28499
Acc. (%)
88.7
96.6:20.4
98.1
75.3::0.9
87.2400.3
88.3+0.1
MTT [}]
Time (sec.)
3114
9323
9987
3107
9305
10042
(%)
89.7+00.6
97.5+0.11
98.620.1
71.5::0.5
83.8::0.2
88.2+0..3
DM [59]
Time (sec.)
1115
1457
1456
Acc. (%)
93.0.10.4
98.610011
99.2:10.0
75.4.10.55
85.5.10.2
89.2.10.1
FRoPo [61]
Time (sec.)
6112
9174
21678
6115
8463
21549
Acc. (%)
91.311022
97.810.22
99.0.10.0
73.8.10.88
84.7.10.2
88.31.0.1
Time (sec.)
x5
392x10
1012x21
147 x42
432 x22
1005
Table 8: Comparisons on test accuracy and running time with state of the arts on single-channel datasets. The acceleration marked by the red subscript is computed against the method with the best accuracy. IPC: Number of Images Per Class: Ratio: ratio of distilled images the whole training set. Results demonstrate the cross-channel generalization ability of our meta generator.
B
More Results
Tros-Nummerrof-Chhame Generalization: In the meta learning stage, meta generator is trained taking RGB images as input and output. Here, we demonstrate that it also be feasible for the meta generator to be adapted for target datasets that have different numbers of channels. Specifically, we additionally train convolution layers for channel adaptation to map the number of channels from the original number to 3 and from 3 to original number the beginning and the ending positions of the generator, respectively. The parameters of these adaptors are initialized from a uniform distribution and are optimized jointly with parameters of the generator.
Here, we conduct experiments on MNIST [25] and FaShionMNIST [52] datasets. Both of them contain 10 classes with 60,000 single-channel images. Results are shown in Tab. 8 following the same comparison protocols as Tab. 1, where the generator our method is adapted for 10,000 steps in each setting. Experiments demonstrate that our method can achieve comparable performance with those state-of-ther ones in a significantly shorter period of time. The conclusion the same as that in the main paper.
15

---

# of Classes
20
50
IPC
IPC
Biseline
28.20-10.77
48.26:1: 1,26
Baseline
23.42:1008
49.40:10.53
16.8420.30
0.221
36.51.10.47
Ours
37.9510444
41.90.1..88
Table 9: Comparisons with the baseline FRePo on ImageNette under 128 resolution. Results demonstrate the cross-resolution generalization ability of our meta generator.
Table 10: Comparisons with baseline FRePo on various CIFAR 100 subsets. Results demonstrate the
cross-number-of-cleee generatlization ability of our meta generator.
Dataset
CIFARIO
CIFARI00
IPC
DC
41.8:0.66
25.9::0.
13.3+0.3
6.7-00.2
DSA
41.5::04
27.6:0022
14.920.3
8.1
IDC
51.940.5
302200
13.320.3
11.0:0.11
MIT
55.9:1..3
29.1:0.4
26.72:0.5
13.7:10.3
DM
46.8.10.5
25.3:.0.3
15.7...33
8.0
FRcPo
59.1
38.3+0.9
30.0=0.6
19.9+0.3
Ours
60.8:04
46.11:0.8
30.620.3
25.4:10.5
Table 11: Comparisons with state of the arts on cross- IPC generalization.
Dalusel
IPC
DC [60]]
DSA [58]
IDC [20]
MIT 121
DM [59]
HRaPo [61]
Ours
08.
87.7.10.6
76.11.0.1
73.110.8
87.810,7
64.8.10.9
87.8.10.2
MNIST
10
96.2::022
96.7::0.1
05.1
92.8+002
96.240.1
96.3:t:.11
97.2:10.1
95.7.10.2 50
98.3.10.1
98.410.11
96.6.10.1
98.0100.1
98.5.10.1
98.6.1...1
70.3:10.7
70.3:10.7
64.100
7..5
71.110.3
61.5110.3
71.9:00.4
FashionMNIST
10
79.8:10.2
79.0:10.3
82.91002
80.1
83.020.11
81.2::0.2
83.410,22
50
78.5:10.2
86.9::0.1
87.0+0.1
86.2
86.8:022
85.9::0.1
87.2:10.11
28.2:10.7
28.11:0.7
25:31::00
36.810.55
26.81:088
27.2::0.5
42.6:1.0.3
CIFARIO
10
39.7.10.5
48.7:10.3
49.5..0.3
50.8.0.55
48.810.2
49.4:10..3
5..910044
50
1..1
56.0::044
61.7#0.2
56.5+005
$7.740.3
61.8+00.2
66.8:10.2
12.4:022
13.8:-022
15.4+:.2
13.24:0.
11.900,2
10.14:0.2
20.8:10.2
CIFARI00
10
21.1:10.2
31.3100
28.9100.3
30.21044
30.0:0.44
26.6:1044
32.2:10.3
Table 12: Comparisons with state of the arts various benchmarks under the same number of training steps. IPC: Number of Images Per Class. Results demonstrate the superior efficiency our method.
Cross-Resolution Generalization: Although the meta generator trained under 32 resolution, is possible for it to be adapted for datasets with different resolutions, thanks to the convoluooalll architecture of the generator. We demonstrate the cross-resolution generalization performance on ImageNette []]], which contains 10 classes and 9,469 images. Following the FRePo baseline [61], we conduct experiments on and 10 IPCs under 128 resolution. Results in Tab. 9 demonstrate the feasibility of such cross-teso generalization.
Cross-Number-off offClsss Generalization: Here, we conduct experiments on CIFAR100 subsets with random 20 and 50 classes respectively and compare the performance with the FRePo baseline [61] Results in Tab. demonstrate that the meta generator performs robustly on datasets with various numbers of classes.
Cross-IPC Generalization: For existing methods, when budgets for synthetic datasets change, they have to either repeat the time-consuming training loop of dataset distillation, which is inconvenient infeasible at all, or prune some synthetic data heuristically, which leads inferior performance. For example, shown in Tab. 11, on CIFARIO 100, if the original synthetic IPC is 50 and the new IPC becomes 20 5, random pruning would lead to unsatisfactory performance for existing methods. By contrast, the generator in our MGDD can work for arbitrary sizes of synthetic datasets once adapted, which makes handle such scenarios better. We present another example on CIFAR 100, the original IPC is 10 and the new IPC is 5 or 2.
16

---

90
97.5
98.7
Test Accuracy (%)
Test Accuracy (%)
97.0
Tost Accuracy (%)
98.6
96.5
98.5
70
96.0
98.4
60
w/o Mata
95.5
w/o Meta
wio Meta
w/o Ans. Labd
Ama. Label
98.3
wio Ana. Label
Baseline
95.0
Baseline
Baseline
Ours
Ours
98.2
Ours
100
500 1000 2000
100
300
500
1000 2000
100
500
1000 2000
Adaptation Steps (1 IPC)
Adaptation Steps (10 IPC)
Adaptation Steps (50 IPC)
Figure 8: Performance generators with various adaptation steps on MNIST.
Comparisons under the Same Steps: To better demonstrate the superiority of the proposed method, we compare our method with state of the arts with the number of training/adaptationn steps controlled the same. As shown Tab. 12, under 1000 steps, our method outperforms others significantly especially relatively challenging datasets with more patterns, like CIFARIO and CIFAR100 Furthermore, in Fig. 8, 9, 10, and 11, we visualize the performance of generators in each setting with different adaptation steps MNIST, FashionMNIST. CIFARIO, and CIFARI00 datasets respectively as supplements to Fig. 4. It can be shown that our method can achieve the most satisfactory performance with only limited number of adaptation steps compared with the baseline FRePo and generators from scratch, which indicates that the proposed method is more suitable for scenarios requiring high efficiency, like processing data streams. Note that for IPC, we observe that using analytical labels would often lead to inferior performance compared with vanilla one-hot labels. We speculate that it is because soft labels by the analytical solution are relatively not good leading the generator to synthesize slass-discrininative patterns when the size of synthetic dataset is small. Thus, we do not use analytical labels for 1 IPC by default.
Qualitative Results: In Fig. 12, we supply qualitative visualization of initialized synthetic samples and results by generator under 1 and 10 IPC CIFAR1O and IPC CIFARIO0 as supplements to Fig. 6.
C Limitations and Future Works
Our MGDD method mainly focuses on the efficiency issue in existing methods. Although it can be demonstrated that our method can result in better performance in only limited time, it does not reduce the time and memory complexity of computing the matching metrics since we adopt the same objectives as previous approaches. When adapting for large synthetic datasets, it may still face the issue on GPU memory in existing works. Nevertheless, it possible for our method to adapt on some small IPCs and then generalize to large synthetic datasets, discussed in the main paper, which can serve as a remedy to this limitation. Besides, initialized samples of synthetic datasets come from real data, and results by generator still look somehow realistic, which may potentially make the method vulnerable to privacy attack, especially for data like personal information. Also, in scenarios like storing synthetic samples of human faces, the generator may break the integrity of faces and lead to an infringement of portrait rights being misused.
Future works may focus on more effective training objective, training pipeline, and architecture of the generator in meta learning or/and adaptation stages further improve the cross-dataset, cross-ipe, and cross- crossarcchteecure generalization. It would also be valuable to extend the MGDD to other tasks and modalities beyond image classification and explore advanced input and output parameterizations of the generator.
17

---

72.5
87
70.0
Test Accuracy (%)
Test Accuracy (%)
Test Accuracy (%)
67.5
82
81
85
62.5
80
84
w/o
w/o Meta
60.0
wio Ana. Label
79
wrf Ana. Label
wio Ana. Label
83
57.5
Baseline
Baseline
Baseline
Ours
Our
Ours
82
100
300
500
1000
2000
100
300
500
1000
2000
100
300
500
1000
2000
Adaplation Steps (1 IPC)
Adaptation Steps (10 IPC)
Adaptation Steps (50 IPC)
Figure 9: Performance of generators with various adaptation steps on FashionMNIST.
67.5
60.0
Test Accuricy (%)
Test Acuracy (%)
$7.5
Test Accuracy (%)
65.0
40
62.5
55.0
$2.5
60.0
57.5
50.0
30
47.5
55.0
Label
45.0
Ana. Label
52.5
Baseline
Baseline
50.0
Baseline
42.5
Ours
Ours
Ours
47.5
20
40.0
100
300
500
1000
2000
100
300
500
1000
2000
100
300
S00
1000
2000
Adaptation Steps (1 IPC)
Adaptation Steps (10 IPC)
Adaptation Steps (50 IPC)
Figure 10: Performance of generators with various adaptation steps on CIFARIO.
22
wio Meda
32
Test Accuracy (%)
20
Ana Label
Baseline
Test Accuracy (%)
30
18
28
14-
24
12-
wlo Meta
10
wio Ana. Label
20
Baseline
Ours
300
1000
2000
300
2000
Adaptation Steps (1 IPC)
Adaptation Steps (10 IPC)
Figure 11: Performance of generators with various adaptation steps CIFARI00.
18

---

Before Generator
After Generator
IPC, CIFARIO
Before Generutor
After Generator
10 IPC, CIFARIO
Before Generntor
After Generator
IPC, CIFARI00
Figure 12: More visualizations of samples before and after generator on CIFARIO and CIFAR100.
19


Reviews:
[{"review_id": "rIb7YQN8v3", "reviewer": "Reviewer_jcg2", "summary": "This paper proposes a meta-learning conditional generator method that takes some initialization of synthetic images and quickly learns to render it for Dataset Distillation purposes. The authors extensively verified the performance and show that the proposed method is substantially faster and achieves good performance.", "questions": "See above.", "limitations": "Didn't find the equivalent paragraph for limitation. The authors should add and highlight it?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well motivated and a good contribution to the Dataset Distillation community\n+ The proposed algorithm is interesting. The algorithm is almost like treating the conditional generator as an updater/optimizer that learns to iteratively refine the synthetic set for a customized loss\n+ The experimental results support the claims", "weaknesses": "- From figure 6, it looks like the updated images aren't moving a lot after the refinement by generator. Could the authors explain the potential reasons?\n- The performance of DD are eventually not better than previous results.\n- The method requires an additional neural network component for update/refinement\n\nMisc. line 40 RTP -> RTP[7], missing citation number.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer jcg2 for the constructive comments. We are happy that the reviewer finds our work well-motivated, the method interesting, and the experiments extensive. We would like to address the concerns and questions reflected in the review below.\n\n* > Why do the updated images not move a lot after the refinement by the generator?\n  \n  Thanks for the insightful question. The reason is on the training objective of DD we apply, *i.e.*, the NFR loss proposed in FRePo [a], where the visualizations of final distilled images are somewhat realistic and do not move much from the initialization. Thus, we assume that the image-to-image transformation is not complex and can be learned with a simple auto-encoder structure. Since operations in the generation model, like Conv and BN, are all global-wise, *i.e.*, different spatial locations in a feature map share a common transformation function, it is understandable that basic semantic structures do not change too much by the generator.\n\n* > The performance of DD is eventually not better than previous results.\n\n  Thanks for the comment. On the one hand, the major focus of the work is on the efficiency, or the performance in a limited number of iterations, which has been extensively verified in the main paper and the supplement. On the other hand, if enough time is given, the results can surpass previous ones in some cases. Here are some examples on CIFAR10:\n\n  |        IPC        |       1        |       10       |       20       |\n  | :---------------: | :------------: | :------------: | :------------: |\n  | Acc. of FRePo (%) | 43.24$\\pm$0.32 | 65.76$\\pm$0.72 | 67.60$\\pm$0.49 |\n  | Acc. of Ours (%)  | 48.01$\\pm$0.43 | 66.01$\\pm$0.15 | 68.07$\\pm$0.10 |\n\n* > The method requires an additional neural network component for update/refinement.\n\n  Thanks for pointing this out. Indeed, the proposed method comprises an additional generator, which introduces additional overhead for computing the matching metrics and it does not reduce the complexity. As mentioned in Sec. C of the supplement, we would like to view this as a limitation. Nevertheless, the introduced memory consumption is acceptable in fact, *e.g.*, 3182 MB v.s. 3288 MB for CIFAR10 with 1 IPC. Moreover, as discussed in Lines 90 to 92 of the supplement, it is possible for our method to adapt on some small synthetic datasets and then generalize to larger ones, which can serve as a remedy to this limitation. \n\n* > Misc. line 40 RTP -> RTP[7], missing citation number.\n\n  Thanks for the detailed review. We will definitely fix this in the next version.\n\n* > Didn't find the equivalent paragraph for limitation.\n\n  Thanks for the suggestion. Please refer to Sec. C of the supplement for details and we will highlight it in the final version.\n\nWe would like to thank Reviewer jcg2 again for the valuable feedback. Hope our responses address the reviewer's concerns and we are happy to answer additional questions if there are. \n\n[a] Dataset Distillation Using Neural Feature Regression, Zhou et al., NeurIPS 2022."}, {"review_id": "khXUdvGhEm", "reviewer": "Reviewer_pTHr", "summary": "- This paper proposing a meta-learning version of dataset distillation (DD)\n- The main idea is that dataset distillation is slow, and meta-learning can be used to amortize the dataset distillation procedure\n- The goal a function $g_\\omega (X_{init}) = X_S$, which changes an initialized distilled dataset to the final one\n- The labels are learned to minimize the KRR loss, using the feature space of randomly initialized networks\n- Compared to MAML, in this case $\\omega_{init}$ is functionally equivalent to $\\theta_{init}$ in MAML, and $\\omega$ is updated for a few parameter steps. In this sense the goal is to learn an optimal $\\omega_{init}$", "questions": "- I'm not sure if Theorem 1 is particular relevant to the method. As mentioned in the weaknesses, this theorem takes $\\omega$ to be independent of $X_t$ but in reality it is not because of meta-adaptation. Furthermore, there is not really interpretation of what the upper bound means. What does the right hand side term in eq. 6 mean? In particular why does it make sense to consider the product $f_{\\theta^*}(X_s)W^{\\theta}_s$, considering that one is based on the feature space $\\theta$ and the other based on $\\theta^*$? While the feature spaces are expected to induce similar kernels, multiplying the features together directly does not make sense because they could be randomly permuted versions of one another, for example\n\n- What is the effect of using a larger learning rate for FRePo in figure 4?\n\n- Why do you base $Y_s^*$ on the initialization images $X_{s}$ instead of the transformed images $g_\\omega(X_s)$ in Alg. 1 line 5?", "limitations": "See weaknesses", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- A meta-learning approach to DD is new, and is valuable considering that current dataset distillation algorithms are very slow (and often the cost of doing DD is much much greater than the cost of training networks on the full data, often negating their value)\n\n- The improvements in computation time are very dramatic, and the cost in terms of performance on the reported baselines is low\n\n- I am very impressed with the cross-IPC generalization (line 281). Being able to growing/shrink a distilled dataset in response to changing needs without completely retraining a network is very desirable, and is lacking with current DD algorithms", "weaknesses": "- Poor presentation: I personally found the presentation very confusing. In particular, section 3.2 seemed to add more confusion to the paper. I think and outline of the method (3.3) should have been presented before this theory section. In particular I found the idea of the parameterized transfer function $g_\\omega$ to have been irrelevant until after reading section 3.3. In regards to presentation, I think it needs to be more clearly stated that $\\omega$ is adapted in training, and that you are learning the initialization of $\\omega$, because section 2.3 gives the impression that $\\omega$ is independent of $X_t$ (line 136 - \"without any dependence on $X_t$\"), which is misleading since it updated using $X_t$. I think discussing this in comparison to MAML would be useful since readers are probably familiar with MAML.\n\n- Limited evaluation in terms of datasets. As a major selling point of a meta-approach to dataset distillation is being able to quickly adapt to unseen datasets, it seems necessary to test more than just CIFAR-10 and CIFAR-100. I understand that this is difficult as most dataset distillation algorithms don't benchmark on a lot of datasets (for example apart from CF-10/100, MNIST/F-MNIST, CUB-200 and tinyimagenet/imagenet subsets seem to be the only other common ones), and since this algorithm meta-trains on imagenet, some of these are eliminated. However, I think it is particularly important for this paper to include more datasets. Even mixed datasets (i.e. images from collections of different datasets) may be sufficient\n\n- More discussion of meta-training costs. In particular, how long does meta-training take? It also seems that $\\omega$ is optimized for several hundred iterations, and this sounds like in order to store this is memory would be quite costly. I see that in lines 200-205, it is mentioned that a A40 (a singificantly stronger GPU) is used for meta-training. This cost should be stated more clearly in the limitations\n\n-Minor:\n - I think there are typos in eq (2) (line 125). I'm not sure $\\theta^*$ is meant to appear here.\n- Missing citation. Lines 129-130: the method of solving for the optimal labels was first proposed as \"label solve\" in [1]\n\n[1] Timothy Nguyen, Zhourong Chen, & Jaehoon Lee (2021). Dataset Meta-Learning from Kernel Ridge-Regression. In International Conference on Learning Representations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for the pertinent comments and suggestions of Reviewer pHTr. We are glad that the reviewer finds our method novel and valuable, improvements in efficiency dramatic, and cross-IPC generalization impressive. The concerns, which withhold the reviewer's rating, are addressed as follows.\n\n* **Presentation.**\n\n  We thank the reviewer's feedback from a reader's perspective. Previously we organized Sec. 3 as this form because Sec. 3.2 and Sec. 3.3 correspond to the two steps in our method **in sequence**: we first get the synthetic label and then use the generator to generate synthetic samples. We agree with the reviewer that it may result in some confusion, especially for the theoretical analysis part. Thus, in the next version, we plan to adjust the organization as follows. Sec. 3.2 and Sec. 3.3 will still introduce the technical methods of the two steps respectively. Theorem 1, which appears in Sec. 3.2 at present, will be moved to a new sub-section 3.4. All the details mentioned by the reviewer will be clarified. We hope that the new organization can improve the presentation of our method.\n\n* **More datasets.**\n\n  We thank the reviewer for the pertinent suggestion and agree that involving more datasets could better demonstrate the effectiveness of our method. Thus, we further add evaluations on PACS, PathMNIST, BloodMNIST, and CUB200 datasets here. For some datasets, like Art painting, Cartoon, and Sketch in PACS, and PathMNIST and BloodMNIST for medical image classification, there are large domain gaps with the pre-training dataset ImageNet. We also evaluate the method on mixed datasets as the reviewer suggested: *P+A+C+S* indicates that we mix the 7 classes from the 4 domains in PACS and form a 28-class dataset, and *PathBloodMNIST* indicates that we mix the 9 classes in PathMNIST and 8 classes in BloodMNIST and form a 17-class dataset. The results of our method, our method without the meta generator, are shown in Tab. 2 of the rebuttal pdf. The results suggest that the meta generator can be generalized to unseen datasets successfully and is robust to domain variations. \n\n* **More discussion of meta-training costs.**\n\n  Thanks for pointing this out. Some details of the meta training are shown in Tab. 1 of the supplement. The time and GPU memory costs of meta training are 43 hours and 15.8 GB respectively. The memory would not increase with more meta-training steps since we use the first-order MAML (FOMAML) [b]. We will clarify these details clearly in the final version.\n\n* > I think there are typos in eq (2) (line 125). I'm not sure $\\theta^*$ is meant to appear here.\n\n  Thanks for the detailed inspection and $\\theta^*$ should be $\\theta$ here. We will correct this in the revision.\n\n* > Missing citation. Lines 129-130: the method of solving for the optimal labels was first proposed as \"label solve\" in [1].\n\n  We thank the reviewer for bringing this to our attention and acknowledge that [1] first proposed a simple yet effective label solving method. We will definitely add proper discussions with [1] mentioned by the reviewer in the revision, and would like to mention here that there is a small difference in equations: $X_sX_s^\\top(X_tX_s^\\top)^+Y_t$ in [1] v.s. $X_sX_t^+Y_t$ in ours. We empirically find that the latter works slightly better in practice, *e.g.*, 57.8% v.s. 58.9% accuracy on CIFAR10 with 10 IPC and 1k steps.\n\n* **Questions about Theorem 1.**\n\n  Thanks for the insightful questions. On the one hand, Theorem 1 is used to intuitively demonstrate the primary feasibility of our two-step pipeline: solving labels and transferring samples. Even if $\\omega$ is not dependent on $X_t$, we can obtain an upper-bounded error. In practice, if we further have the information of $X_t$, the error is expected to be lower. Please refer to the discussion between Lines 146 to 152.\n\n  On the other hand, for the sake of better clarity, we would like to modify the discussion in Eq. 6 in the revision and use the result in Eq. 5 as the error bound. In fact, Eq. 5 is valid for any transfer function $g_\\omega$. Since 1) $\\theta^*$ is unknown in practice, and 2) it is challenging to solve the optimal $g_\\omega$ by minimizing the first term of the right-hand side of Eq. 5 even if $\\theta^*$ is known, in this paper we approach $g_\\omega$ through adaptation from a meta model encoding some meta knowledge. \n\n* > What is the effect of using a larger learning rate for FRePo in figure 4?\n\n  Thanks for the question. In Fig. 1 of the rebuttal pdf, we try larger learning rates for FRePo. Using larger learning rates for FRePo benefits the initial convergence. However, the performance may get saturated too early, and it would not increase, or even decrease, in later adaptation stages. Our method still demonstrates the advantages in terms of overall performance.\n\n* > Why do you base $Y_s^*$ on the initialization images $X_s$ instead of the transformed images $g_{\\omega}(X_s)$ in Alg. 1 line 5?\n\n  Thanks for pointing out an alternative solution that first obtains $g_\\omega(X_s)$ and then solves the label in some space parameterized by $\\theta$. In fact, label solving can find the optimal label for some given $\\theta$. However, our final goal is to minimize the error on $\\theta^*$, which is not known and is approximated through iterative adaptation. Thus, it is better to place the adaptation stage in the last step before the label solving, which aligns with our final goal $\\theta^*$ rather than $\\theta$. The experiment results indeed verify this, *e.g.*, 54.0% v.s. 58.9% accuracy on CIFAR10 with 10 IPC and 1k steps.\n\nWe would like to thank Reviewer pTHr again for the in-depth reviews. We would definitely love to further interact with the reviewer if there are any further questions.\n\n[a] Dataset Distillation Using Neural Feature Regression, Zhou et al., NeurIPS 2022.\n\n[b] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Finn et al., ICML 2017."}, {"review_id": "X0VFLLRHGE", "reviewer": "Reviewer_CHTg", "summary": "The reviewed paper makes a clear and novel contribution to the field of dataset distillation, which is expected to have a beneficial impact. The paper is well written, demonstrating a structured and easily understandable presentation. Notably, the empirical evaluation is exceptional, covering various aspects of the proposed method&rsquo;s effects, including evaluation of a potential application and examination of its combination with orthogonal methods.\n", "questions": "-   What are the runtime costs of training the meta-generator?\n-   Line 227: &ldquo;Moreover, we report the cost of time to derive synthetic datasets.&rdquo; -> Can you be more specific here? What includes &ldquo;derive synthetic datasets&rdquo;? Is this the adaptation procedure or does this inlcude the meta-training?\n-   Line 200: &ldquo;For computation efficiency, the generator processes each sample independently&rdquo; -> Why should this be more efficient than batch-wise processing?\n-   Line 202: &ldquo;The meta generator is trained by the Adam optimizer [15]&rdquo; -> In Algorithm 1 it is plain GD. Is this just out of simplicity in Algorithm 1 or am I missing something?\n-   Line 238: &ldquo;On the contrary, the results of our method are produced by a generator in a one-stop fashion&rdquo; -> What do you mean by &ldquo;one-stop&rdquo; fashion? The meta-generator needs to be adapted to the target dataset which includes a certain number of forward-backward passes of $g_{\\omega}$.\n", "limitations": "Limitations are addressed in the appendix. One main limitation which the work generally glances over is the meta-training procedure and its associated cost. All of the experimental evaluations have focused on the adaptation to a target dataset, but a core part of the proposed method lies in the meta-training of $g_{\\omega}$.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The reviewed paper makes a clear and novel contribution to the field of dataset distillation, which is expected to have a beneficial impact. The paper is well written, demonstrating a structured and easily understandable presentation. Notably, the empirical evaluation is exceptional, covering various aspects of the proposed method&rsquo;s effects, including evaluation of a potential application and examination of its combination with orthogonal methods.", "weaknesses": "### Main Claim of being Backprop-Free\n\nI disagree with the main claim of this work being backpropagation-free. The core mechanism of the *distillation* procedure in this work is the training of the meta-generator $g_{\\omega}$ and its adaptation to a new target dataset. Both procedures rely on backpropagation. Yes, the actual generation of synthetic samples does not include backpropagation since it is a forward pass of a generative model with real samples as input. But that is like saying GANs are backprop-free &#x2013; sure, the generation of new samples does not require backpropagation, nevertheless the GAN needs to be trained, i.e. its parameters optimized, which relies on backpropagation for the generation of new samples to actually match the target data distribution. There is one, similar in concept, concurrent work to this paper: &ldquo;DiM: Distilling Dataset into Generative Model&rdquo; [1] which also distills the dataset into generative model to then later produce synthetic samples from this model &#x2013; similar to how in this work the meta-generator is trained to then produce synthetic samples. As explained in DiM and applicable to this work as well, the difference to standard dataset distillation methods is that the medium of information (into which the dataset is distilled) are not images but a parameterized model (from which one can then infer/sample images).\n\nI am bit unsure of what to do now. I really like this work, it is novel, of high quality, easy to comprehend, and the evaluation is remarkable. But I disagree with the main claim of this being backprop-free, which is also part of the title. If this claim is removed (which I do not know if it is possible or if this would be too much of a change), I am happy to change my rating to accept. I am looking forward to discuss this issues with the authors and the other reviewers.\n\n\n### Runtime Performance\n\nIn line 38-41 the authors state &ldquo;for FRePo, the method with the best performance requires above 3 hours to obtain a synthetic dataset with 1 image per class for CIFAR100&rdquo;. Where do these &ldquo;3 hours&rdquo; come from? Your Table 1 lists about 3 hours for FRePo on 1-IPC on CIFAR100 but this does not align with what is reported in the FRePo paper: &ldquo;When learning one image per class on CIFAR100, FRePo reaches a similar test accuracy (23.4%) to the second-best method (24.0%) in 38 seconds [&#x2026;]. Moreover, FRePo achieves 92% of its final test accuracy (26.4% out of 28.7%) in only 385 seconds.&rdquo;. I couldn&rsquo;t find the runtime for their &ldquo;final test accuracy&rdquo;, but even only taking their 26.4% result in 385 seconds (1-IPC CIFAR100) into consideration, FRePo beats the proposed method in terms of accuracy (24.66%) and is 1.3x faster than the proposed method (505s) &#x2013; opposed to x22 slower as claimed in the paper. Furthermore, the hardware used in FRePo (Quadro RTX 6000) is worse in terms of performance than the one used in this work (RTX 3090 for adaptation) which should only widen this discrepancy even more. Furthermore, the meta-training cost is not taken into account. Maybe I got something completely wrong here and I&rsquo;m happy to be corrected if I did, but since the x22 statement is repeated multiple times in this work I wanted to look into the details here.\n\n\n\n### Minor Suggestions for Improvement\n\n-   Figure 3-5: For improved visual consistency, you could try to make these figures the same height and width.\n-   Paragraph of Line 296: I was shortly confused about which figure you were talking about &#x2013; another reference to Figure 3 would help here.\n\n[1] Wang, K., Gu, J., Zhou, D., Zhu, Z.H., Jiang, W., & You, Y. (2023). DiM: Distilling Dataset into Generative Model. ArXiv, abs/2303.04707.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are more than grateful to Reviewer CHTg for the positive feedback, valuable comments, and frank discussions regarding the manuscript. We are very delighted that the reviewer found our contribution novel, paper well written, and evaluation exceptional. For the concerns and questions, we would like to have the following discussions. \n\n* > Main claim of being backpropagation-free.\n  \n  We thank the reviewer for pointing this out and fully understand the concern that both meta learning and fast adaptation stages require back propagation. Previously, we titled the method to be backpropagation-free since the meta learning is required only once for an infinite number of datasets and the adaptation is required only once for one dataset. Once adapted, the generator works in a feed-forward and backprop-free manner for different distillation requests, *e.g.*, different IPC and different initialization as shown in the experiment part, which is one of the most distinguishable features in this paradigm.\n\n  We do agree with the reviewer that the method is not purely backprop-free. Thus, to highlight the differences with prior works and the features in our method more clearly, we will change our title to \"*Feed-Forward Dataset Distillation via Learning to Fast Adapt*\", since 1) synthetic data in our approach are generated indeed in a feed-forward way although not purely backprop-free, and 2) the primary goal of this paper is exactly to find a meta generator that can be adapted to downstream datasets efficiently.\n\n  The reviewer mentioned a concurrent work DiM, which also uses generative models to produce synthetic data. However, the motivations, core methods, and functionalities are largely different. We will discuss more in the revision.\n\n* **Concerns in runtime performance.**\n\n  Thanks for pointing this out. The running time performance in Tab. 1 is from the time required for running the 500,000 iterations in the official code of FRePo [a] on our test environment, as mentioned in Line 237. Note that Tab. 1 is used for comparison with the final and official performance. In the following sub-sections, we would focus on detailed comparisons with it where **the number of iterations** are controlled as the same.\n\n  The reviewer mentioned a case that the baseline FRePo achieves better performance in less time.  The reason here is on the libraries used for implementation. Specifically, the official implementation of FRePo uses JAX, while our implementation uses PyTorch. There is a large gap in the efficiency of the two libraries. In general, for the baseline, the JAX version runs ~2 times faster than the PyTorch counterpart. In addition, as shown in Fig.2 of the rebuttal pdf for the mentioned CIFAR100 with 1 IPC, we find that the PyTorch counterpart of the baseline often results in inferior performance compared with the JAX version and the official JAX version is better to be adopted for detailed comparisons. To eliminate the effects of different libraries, we do not apply the running time as the metric and focus on the performance under the same number of iterations instead.\n\n  Moreover, we find that in many cases even though the running time is applied as the metric, where our method runs fewer steps, our method can still yield performance gain as shown in Tab. 1 of the rebuttal pdf. The results align with our primary goal: to enhance the performance of DD in a limited number of iterations. \n\n* > Figure 3-5: For improved visual consistency, you could try to make these figures the same height and width. \n\n* > Paragraph of Line 296: I was shortly confused about which figure you were talking about – another reference to Figure 3 would help here.\n\n  Thanks for the suggestions. Sure, we will fix them in the revision. \n\n* > What are the runtime costs of training the meta-generator?\n\n  Thanks for the valuable question. The time cost of training the meta-generator for 200,000 iterations on an A40 GPU is 43 hours. We will clarify this in the revision. \n\n* > Line 227: “Moreover, we report the cost of time to derive synthetic  datasets.” -> Can you be more specific here? What includes “derive synthetic datasets”? Is this the adaptation procedure or does this include the meta-training?\n\n  Thanks for the question. The time here includes the label-solving step, the adaptation procedure, and one feed-forward step. Meta-training is not included. We will be more specific in the final version. \n\n* > Line 200: “For computation efficiency, the generator processes each sample independently” -> Why should this be more efficient than batch-wise processing?\n\n  Thanks for the question. We indeed process samples batch-wisely. Here the \"independently\" means that the generator applies the same transformation function for all samples, and there is no interaction among different samples. \n\n* > Line 202: “The meta generator is trained by the Adam optimizer [15]” -> In Algorithm 1 it is plain GD. Is this just out of simplicity in Algorithm 1 or am I missing something?\n\n  Thanks for the question. This is just out of simplicity in Algorithm 1. We will clarify this in the revision. \n\n* > Line 238: “On the contrary, the results of our method are produced by a generator in a one-stop fashion” -> What do you mean by “one-stop” fashion? The meta-generator needs to be adapted to the target dataset which includes a certain number of forward-backward passes of $g_\\omega$.\n\n  Thanks for the question. Here the \"one-stop\" means that once adapted the synthetic data can be generated in a single feed-forward time. We agree with the reviewer that it may result in some confusion. In the future, we will replace \"a one-stop fashion\" with \"a limited number of adaptation steps\" for better clarity.\n\nThanks again for the insightful review. Hope our response can clear the reviewer's concerns and we are more than happy to have a discussion if there are more questions. \n\n[a] Dataset Distillation Using Neural Feature Regression, Zhou et al., NeurIPS 2022."}, {"review_id": "ymNy1BE5c9", "reviewer": "Reviewer_Bfcg", "summary": "In this paper, the authors address the efficiency concerns associated with dataset distillation. Initially, the authors identify the challenges arising from the dependencies within the intensive loop of forward-backward propagation across numerous neural networks, as well as the repetitive training procedures necessary when working with synthetic datasets of varying sizes. Motivated by these challenges, the authors propose an innovative approach to dataset distillation that eliminates the need for backpropagation. In this approach, synthetic labels are generated by analytically solving a least-squares problem in a feature space, while synthetic images are produced using a pre-learned meta generator. The authors provide theoretical evidence supporting the existence of a conditional generator capable of transferring synthetic samples from their original space to the desired space with an upper-bounded error. Experimental results demonstrate that the proposed backprop-free dataset distillation approach achieves comparable performance to state-of-the-art methods, while also providing a 22× acceleration. Additionally, the approach exhibits the flexibility of cross-size generalization.", "questions": "1. What is the physical adaption time for the results reported in the experiments?\n2. How long does it require to train the meta generator on ImageNet1k?\n", "limitations": "The authors have thoroughly addressed the limitations in Section C of the supplementary material.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I see the following positive traits:\n\n+ This paper effectively tackles a crucial challenge in dataset distillation by addressing the cumbersome forward-backward propagation process required for effective distillation. The authors present an elegant solution rooted in meta-learning, which not only achieves an impressive speed improvement of 22 times but also maintains competitive performance levels.\n\n+ The proposed method is supported by robust theoretical analysis, as evidenced by the proof of Theorem 1, where an upper-bounded error is derived, demonstrating the existence of a conditional generator capable of transferring the synthetic dataset from its original space to the desired feature space with an upper-bounded error.\n\n+ The paper offers comprehensive algorithmic workflows, which are presented in both the main paper (Section 3.3, Algorithm 1) and the supplementary material (Section A: More Details, Algorithm 1) for handy reproducibility. \n\n+ The paper is very well-written and easy to follow. The authors try to elaborate every detail while providing a clear overall picture.\n\n+ The supplementary material of the paper offers an abundance of extensive details and additional experimental results including the visualization results, further validating the effectiveness of the proposed backprop-free dataset distillation method.\n", "weaknesses": "In general, this paper focuses on a significant problem, proposes an elegant solution, and the experimental results convincingly validate the effectiveness of the proposed approach. I only have some minor concerns as follows.\n\n- The authors mention that the estimated acceleration factor includes the adaptation time. Given that the main emphasis of this paper is on time complexity, I would suggest the authors present the adaption time and feed-forward time separately in the experiments, to effectively illustrate the time cost involved in the process.\n\n- I wonder how long it takes to train the meta generator on ImageNet1k.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Bfcg for the positive feedback on the manuscript and are very excited that the reviewer mentioned the strengths of effectiveness, theoretical analysis, algorithmic workflows, writing, and experiments. The questions are addressed below. \n\n* > What is the physical adaption time for the results reported in the experiments?\n\n  Thanks for the valuable question. The time reported includes the label-solving step, the adaptation procedure, and one forward propagation step. For the example of 1k steps on CIFAR10 with 10 IPC, the three terms take 7s, 56s, and 3ms respectively. The adaptation stage takes the major percentage, and the other two terms would not increase much with the size of synthetic datasets. The detailed analysis of the physical adaptation time reported in Tab. 1 of the main paper is as follows.\n  \n  |      Setting      | Label Solving (s) | Adaptation (s) | Forward Propagation (ms) | Total Time (s) |\n  | :---------------: | :---------------: | :------------: | :----------------------: | :------------: |\n  |  CIFAR10 - 1 IPC  |         7         |      113       |           1.5            |      120       |\n  | CIFAR10 - 10 IPC  |         7         |      498       |           3.0            |      505       |\n  | CIFAR10 - 50 IPC  |         7         |      1388      |           11.0           |      1395      |\n  | CIFAR100 - 1 IPC  |         7         |      498       |           3.0            |      505       |\n  | CIFAR100 - 10 IPC |         7         |      2997      |           20.2           |      3004      |\n  \n  We will include these details in the final version. \n\n* > How long does it require to train the meta generator on ImageNet1k?\n\n  Thanks for the insightful question. It requires 43 hours to train the meta-generator for 200,000 iterations on an A40 GPU. We will clarify this in the revision. \n\nHope this will address the reviewer's questions, and thanks for the time, efforts, and positive review again. "}]

Author rebuttal:
We would like to express our heartfelt gratitude to all reviewers for their constructive reviews. The questions and concerns are addressed in the rebuttal under the Official Review block of each reviewer. We would like to upload a pdf file here, denoted as "rebuttal pdf" in the following parts. The file contains the following tables and figures that cannot fit into the rebuttal blocks:

* Fig. 1: Comparisons with the baseline FRePo using different learning rates. (Reviewer pTHr)
* Fig. 2: Difference between JAX and PyTorch implementations of the baseline FRePo. (Reviewer CHTg)
* Tab. 1: Comparisons with the baseline FRePo using running time as the metric. (Reviewer CHTg)
* Tab. 2: Evaluations on more benchmarks. (Reviewer pTHr)

We are more than happy to discuss with the reviewers if there are more questions. And thanks for the time and effort in reviewing this article again!

Best Regards,
Authors of Submission 5629

Meta-review: This paper proposes a novel dataset distillation scheme that involves a meta-generator capable of generating synthetic datasets for a new dataset by adapting to it with only a few gradient update steps. Specifically, the generator is conditioned on the initially distilled dataset and generates synthetic data from it, while obtaining labels for the generated data points by solving a least-square problem in a feature space. The authors provide theoretical analysis demonstrating the existence of a conditional generator capable of transferring the synthetic dataset from its original space to a desired feature space with an upper-bounded error. The experimental results show that the proposed method can achieve significant improvements in computation time and generalization performance.

All reviewers unanimously express positivity about accepting the paper. They find the idea of adapting a meta-generator to each dataset novel and effective, the theoretical analysis sound, and the experimental results strong. The majority of the reviewers also find the paper well-written and mostly clear, while a few have concerns regarding the organization and clarity in certain sections. The reviewers, as well as the area chair, unanimously agree that the proposed scheme is novel and solves the practical challenge of high computation cost when applying dataset distillation to real-world scenarios. Therefore, the proposed work makes a clear and novel contribution.

However, there were some concerns regarding the misleading title (i.e., "backprop-free" while the generator should still be trained with backpropagation), the lack of discussion on the meta-training cost, and some unclear parts in the report of the runtime performance. Yet, the authors' rebuttal has addressed most of the concerns. The authors are strongly advised to incorporate the discussions with the reviewers into the final revision of the paper
Decision: Accept (spotlight)