Title: MGDD: A Meta Generator for Fast Dataset Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset Distillation

Dataset Condensation

Efficient Learning

Conditional Generation

Meta Learning
Abstract: Existing dataset distillation (DD) techniques typically rely on iterative strategies to synthesize condensed datasets, where datasets before and after distillation are forward and backward through neural networks a massive number of times. Despite the promising results achieved, the time efficiency of prior approaches is still far from satisfactory. Moreover, when different sizes of synthetic datasets are required, they have to repeat the iterative training procedures, which is highly cumbersome and lacks flexibility. In this paper, different from the time-consuming forward-backward passes, we introduce a generative fashion for dataset distillation with significantly improved efficiency. Specifically, synthetic samples are produced by a generator network conditioned on the initialization of DD, while synthetic labels are obtained by solving a least-squares problem in a feature space. Our theoretical analysis reveals that the errors of synthetic datasets solved in the original space and then processed by any conditional generators are upper-bounded. To find a satisfactory generator efficiently, we propose a meta-learning algorithm, where a meta generator is trained on a large dataset so that only a few steps are required to adapt to a target dataset. The meta generator is termed as MGDD in our approach. Once adapted, it can handle arbitrary sizes of synthetic datasets, even for those unseen during adaptation. Experiments demonstrate that the generator adapted with only a limited number of steps performs on par with those state-of-the-art DD methods and yields $22\times$ acceleration.

MGDD: A Meta Generator for Fast Dataset Distillation

Abstract

Existing dataset distillation (DD) techniques typically rely on iterative strategies to synthesize condensed datasets, where datasets before and after distillation are forward and backward through neural networks a massive number of times. Despite the promising results achieved, the time efficiency of prior approaches is still far from satisfactory. Moreover, when different sizes of synthetic datasets are required, they have to repeat the iterative training procedures, which is highly cumbersome and lacks flexibility. In this paper, different from the time-consuming forwardbackward passes, we introduce a generative fashion for dataset distillation with significantly improved efficiency. Specifically, synthetic samples are produced by a generator network conditioned on the initialization of DD, while synthetic labels are obtained by solving a least-squares problem in a feature space. Our theoretical analysis reveals that the errors of synthetic datasets solved in the original space and then processed by any conditional generators are upper-bounded. To find a satisfactory generator efficiently, we propose a meta-learning algorithm, where a meta generator is trained on a large dataset so that only a few steps are required to adapt to a target dataset. The meta generator is termed as MGDD in our approach. Once adapted, it can handle arbitrary sizes of synthetic datasets, even for those unseen during adaptation. Experiments demonstrate that the generator adapted with only a limited number of steps performs on par with those state-of-the-art DD methods and yields 22 × acceleration.

1 Introduction

Dataset distillation (DD) introduced by Wang et al. [51] aims to compress an original dataset T into a much smaller synthetic set S , such that the performance of a neural network, trained with the condensed dataset S , is similar to the network trained with T . The derived synthetic datasets not only save the cost of storage and transmission but also significantly reduce the computational resources and time required by training models using original datasets. As such, DD finds its application across a wide spectrum of domains and is receiving increasing attention from the community.

The typical paradigm of DD is to optimize S in an iterative loop, as shown in Fig. 1(a). In each iteration, a new network is sampled, leveraged by which a matching metric is calculated for S and T , and the matching loss is then back-propagated through the network to update S . Recently, a large number of approaches have been dedicated to exploring advanced matching objectives to improve the training performance of distilled datasets, including matching training performance via meta learning [51, 7], matching feature regression performance [37, 38, 61, 32, 33], matching training gradients [60, 58, 20], matching training trajectories [2, 8, 4], and matching feature statistics [59, 50].

Although impressive results have been achieved and it has been demonstrated that neural networks trained by a synthetic dataset with even only 1 image per class can yield reasonable performance

Match

Loss

Backprop

(a) Previous DD Paradigm

𝒯

𝒮

×

N Times

Analytical

Label

Generator

𝒯

Initialized

𝒮

Random

Select

Final

𝒮

(b) MGDD Paradigm

Solve

on real data, the iterative optimization process adopted in existing works results in significant computational overhead. As shown in Fig. 2, for FRePo [61], the method with the best performance requires above 3 hours to obtain a synthetic dataset with 1 image per class for CIFAR100 [23], let alone RTP [7], which is a back-propagation-through-time method and requires over 9 days for optimization. Such a dramatic latency makes existing approaches hard to be applied in scenarios requiring high efficiency, like handling data streams. Moreover, when the memory budget for a synthetic dataset changes, existing methods have to repeat the time-consuming optimization for the different sizes of S , which lacks flexibility.

To alleviate the drawbacks of the conventional iterative forward-backward process, in this paper, we propose a generative dataset distillation approach, where the optimization loop is replaced by a single feed-forward propagation, as shown in Fig. 1(b). Specifically, given an initialization of synthetic samples in S , we first obtain the corresponding synthetic labels by analytically solving a least-squares problem in a feature space. Then, a generator is adopted to transfer the initialized samples to the final ones. Our theoretical analysis indicates that S solved in an original space can be transferred to the final result by any conditional generator with an upper-bounded error, which validates the feasibility of this pipeline.

Then, the key problem of our framework lies in finding a suitable generator as quickly as possible for the feed-forward generation process of S . To this end, we propose a method called MGDD , where a meta generator is learned with a meta learning algorithm on a large database like ImageNet [6]. Trained in a learning-to-learn fashion, the meta generator is optimized such that only a small number of adaptation steps are required for a target dataset unseen in the meta learning. Experiments demonstrate that our approach yields 22 × acceleration 2 and comparable performance with existing state-of-the-art methods, as shown in Fig. 2. Beyond that, the generator once adapted can also handle unseen sizes of S during adaptation, which improves the flexibility of cross-size generalization in existing DD methods significantly. We also validate that MGDD gets competitive performance on target datasets with large domain shifts from those seen in the meta learning.

Our contributions can be summarized from the following three aspects:

We propose an innovative feed-forward generation fashion for DD without backward propagation after adaptation which significantly improves the efficiency of existing methods;

We introduce MGDD which uses a meta-learning algorithm to learn a meta generator and helps the generator adapt to a target dataset rapidly;

The proposed method achieves significant acceleration and improvement in the flexibility of cross-size generalization for existing DD approaches with comparable performance.

2 Related Works

Unlike conventional efficient learning schemes that mainly focus on lightening models [9, 34, 55, 54, 10, 18, 19, 17], dataset distillation (DD) looks into data compression: given a real large dataset T , DD aims at a smaller synthetic dataset S which can match the training performance of T . The seminal work by Wang et al. [51] proposes a meta learning approach to model this objective: in meta training, a network is trained with the current S for multiple times, while in meta test, the loss for the updated network is evaluated on T , which is then back-propagated through the bi-level optimization

to update S . The following work by Deng et al. [7] improves performances by adopting momentum during meta training.

Considering the concerns on memory and time complexity of unrolling the computational graph in meta learning, a variety of works introduce various surrogate matching objectives to update S . Zhao et al. [60] propose to match training gradients of S with those of T , and following researches [20, 27, 16, 57, 45] focus on improving the classical gradient-matching objective. Beyond the single-step gradient, Cazenavette et al. [2] and subsequent works [4, 8] consider regulating multistep training effects and propose matching training trajectories. Without the necessity of calculating higher order gradients, the distribution matching methods [59, 50] minimize the distance between feature statistics of S and T , and result in satisfactory computational efficiency. Another branch of methods [37, 38, 61, 32, 33] turn to kernel ridge regression (KRR) model to improve the efficiency of the seminal meta learning based solution, since KRR enjoys the analytical form of solution, which gets rid of the meta-training process and yields best trade-off between performance and efficiency.

Different from works focusing on objectives of DD, some other researches explore methods of synthetic data parameterization, where synthetic samples can be stored in some other formats except the raw one to improve the data efficiency, and raw samples are recovered via some functions during down-stream training, e.g. , data augmentation [58], up-sampling [20], linear transformation [7], and neural networks [30, 26, 3, 49].

For a thorough introduction to dataset distillation, we refer readers to the recent surveys for this area [56, 13, 43, 28]. For existing methods, no matter what objectives and parameterizations are adopted, they all rely on an intensive loop of forward-backward propagation through a massive number of neural networks. Although a concurrent work [31] also adopts a similar pre-training and adaptation pipeline, it still relies on an iterative loop to solve an initialization of synthetic data.

10 2

10 3

10 4

10 5

10 6

Log-Scale Running Time (sec.)

12

14

16

18

20

22

24

26

28

Accuracy (%)

DC

DSA

DM

MTT

FRePo

RTP

IDC

Ours

Ours-DS

Different from prior works, we innovatively introduce a feed-forward fashion for dataset distillation in this paper. In fact, our method is orthogonal to different training objectives and data parameterizations, and can be built upon any combination of them. In this paper, we consider the KRR-based objective thanks to its favorable performance and computational efficiency and experiment with both raw and down-sampled parameterizations.

3 Methods

In this section, we elaborate on the technical methods of the proposed MGDD pipeline. We first introduce in Sec. 3.1 some preliminary information related to the matching objective. Then, for the main method, according to the overview of the whole pipeline in Fig. 1(b), the final synthetic labels and samples are derived by solving a least-squares problem and a conditional generator given initial synthetic data, which would be illustrated in detail in Sec. 3.2 and 3.3 respectively. Finally in Sec. 3.4, we analyze the feasibility of this two-step pipeline theoretically.

3.1 Preliminary

Let us denote datasets before and after distillation as T = ( X t , Y t ) and S = ( X s , Y s ) respectively, where X t ∈ R n t × d , Y t ∈ R n t × c , X s ∈ R n s × d , Y s ∈ R n s × c , n t and n s are number of real and synthetic data respectively, d and c are input and output dimensions respectively. Typically, for the RGB image classification task, d is equal to h × w × 3 , c is the number of classes, and Y t is organized in the one-hot format. For the objective of DD, in this paper, we mainly consider the KRR-based methods in neural feature spaces [61, 32] due to its overall superior performance in terms of accuracy and efficiency. Specifically, assume there is an optimal neural network f θ ∗ to projects X t and X s to a feature space with p dimensions and n s ≪ n t < p . The prediction error on T for the optimal KRR

parameter solved by S , denoted as W θ ∗ s is adopted as the loss function:

In practice, since the optimal parameter θ ∗ is unknown, it is approximated by different random initializations [32] or alternately optimization with S [61].

3.2 Solving Synthetic Labels

Through Eq. 1, we can find that the loss in a neural space θ is upper-bounded by the distance between parameters solved by S and T :

where † denotes the pseudo-inverse of a matrix. In our MGDD framework, synthetic samples X s are initialized as some random real samples in X t . Given a fixed X s and a random network f θ , the upper bound in Eq. 2 forms a least-squares problem with respect to synthetic labels Y s , which can be minimized by an analytically optimal solution:

Y ∗ s obtained with Eq. 3 serves as final synthetic labels. In the next subsection, we will introduce the generation of synthetic samples conditioned on their initialization.

3.3 Learning a Synthetic Sample Generator

Conditioned on initialized synthetic samples X s , a generator g ω is adopted to predict the final synthetic data X ∗ s , where the parameter ω can encode useful information of the target dataset T and the optimal neural space parameterized by θ ∗ . We expect that the generator can acquire such knowledge through a fast learning process within a limited number of training steps. To this end, we propose a learning-to-learn algorithm based on MAML [12], where a meta generation network is learned to optimize the performance of the network adapted for a few steps from the meta one.

Specifically, to ensure the generality of the meta generator for different target datasets, we perform the training algorithm on ImageNet1k [6], a large-scale dataset for image classification. In each training iteration, a subset of all classes is randomly sampled from it to mimic different target datasets that the generator may encounter in practice. And the meta generator is learned in a bi-level learning framework including a meta-training loop and a meta-testing step, and the meta-testing loss is backpropagated through the computational graph of meta-training steps to update the parameter of the meta generator.

In each meta-training and meta-testing step, from the selected classes, we randomly sample a batch of real images as T and initialize synthetic data X s with part of them. With a random and fixed neural network as f θ , the synthetic labels Y ∗ s are solved via Eq. 3. Then, the final synthetic samples X ∗ s are predicted by the current generator in a forward pass, and S = ( X ∗ s , Y ∗ s ) is evaluated by the loss L ( S ; θ ∗ ) in Eq. 1. In this paper, following Loo et al. [32], we approximate the optimal neural parameter θ ∗ via random sampling in different steps from the distribution for initialization. The loss signal is back-propagated to the current generator and the meta generator to update parameters in meta-training and meta-testing respectively. It is worth noting that in different meta-training and metatesting times, we use different sizes of synthetic data, which enhances the cross-size generalization ability on target datasets since the meta-testing losses on sizes probably unseen during meta-training are optimized. The main algorithm is summarized in Alg. 1. Given a trained meta generator, a limited number of adaptation steps are performed for a target dataset. The procedure of adaptation is similar to the meta-training loop in Alg. 1.

As for the architecture of the generator, in this paper, we adopt a simple encoder-decoder model, where the encoder consists of three convolutional blocks with two average pooling layers while the decoder has a symmetric structure. Notably, we observe in practice that it is beneficial for different sizes of synthetic datasets to adopt different transfer functions. Taking various sizes into consideration, we concatenate additional size-embedding channels to the bottle-necked layer of the

Algorithm 1 MGDD: Meta-Learning Framework for Training a Synthetic Sample Generator

generator, inspired by the design of the position embedding in Transformer models [47] and the time-step embedding in diffusion models [15, 39, 41]. Please refer to the appendix for detailed architecture configurations.

3.4 Theoretical Analysis

There are two key steps in the proposed MGDD framework: solving the optimal synthetic labels Y ∗ s as introduced in Sec. 3.2 and generating the corresponding synthetic samples X ∗ s as introduced in Sec. 3.3. Define the error L ( S ; θ ) in Eq. 1 using the fixed X s and the optimal Y s in Eq. 3 with the projection function of f θ as ϵ . The reason we pursue the optimal Y s is that the final X ∗ s is transferred from the initial X s , whose error in the optimal neural space parameterized by θ ∗ is upper-bounded by ϵ , as derived in the following theorem.

Theorem 1. Given f θ : R d → R p , f θ ∗ : R d → R p , X t ∈ R n t × d , Y t ∈ R n t × c , X s ∈ R n s × c , d < p , Y ∗ s obtained by Eq. 3 with the optimal L (( X s , Y ∗ s ); θ ) denoted as ϵ , and an arbitrary transfer function g ω parameterized by ω taking X s as input, the transferred g ω ( X s ) yields an upper-bounded loss L (( g ω ( X s ) , Y ∗ s ); θ ∗ ) .

Proof. We first rewrite the given condition:

Then, we have:

where the inequality is based on the triangle inequality and the last equation is due to Eq. 4.

Theorem 1 indicates that to achieve feed-forward dataset distillation, we do not need to design a neural network taking the whole original dataset T as input. We can in fact adopt a conditional

generation function that transfers the initial synthetic samples to the desired ones, which has an error upper bound as shown in Theorem 1. Since the upper bound is related to the error in the original space parameterized by θ , it is crucial to solve the optimal synthetic labels with respect to θ in the first step. Also, we notice from Eq. 5 that the optimal generator is dependent on T and θ ∗ . Given that θ ∗ is intractable and can only be approximated by iterative sampling, we build a meta learning algorithm in Alg. 1 for the MGDD framework to enforce an efficient process to model this dependency in only a few steps.

4 Experiments

4.1 Implementing Details

As discussed in the previous section, there are 3 stages in the proposed MGDD, including training, adaptation, and inference stages. In the training stage, we aim at a meta generator g ω and adopt Alg. 1 to train g ω on a large dataset. In this paper, to ensure that the meta generator can acquire general knowledge for fast adaptation on a target dataset, we use ImageNet1k [6], a large-scale image recognition dataset popular in the computer vision and machine learning communities, as the dataset for meta learning. There are roughly 1,280,000 images in a total of 1,000 classes. We resize all images to the 32 × 32 resolution. In each outer loop of Alg. 1, we randomly select 100 classes at most and a batch of data with 2,000 images in maximal from the selected class as a current target dataset T , with which we initialize a synthetic dataset S with 1,000 samples at most in each inner step. The training objective is based on Eq. 1 and the implementation follows the open-source code of FRePo [61]. For computational efficiency, the generator processes each sample independently and the configuration of the architecture can be found in the appendix. The meta generator is trained by the Adam optimizer [21] and the learning rate β is set as 10 -5 . The learning rate in meta-training is set as 10 -4 and the number of meta-training steps T is 5 . The meta generator is trained with 200,000 meta-testing iterations. We use a cloud server with a single A40 GPU for meta learning and a workstation with a single 3090 GPU for the subsequent adaptation. The meta learning takes roughly 2 days while the time cost of adaptation is analyzed in Tab. 1.

IPC Steps

Methods

P

A

C

S

PACS

Path

Blood

PathBlood

CUB200

Baseline

53.54

±

0.99 33.53

±

3.19 51.69

±

1.25 39.76

±

1.74 38.03

±

0.60 54.23

±

3.12 68.38

±

0.55 55.11

±

1.26

4.84

±

0.19

±

1.71 40.29

±

0.68 60.45

±

0.32 35.87

±

0.07 42.73

±

0.25 57.06

±

1.36 69.14

±

1.86 58.03

±

0.51

6.73

±

0.13

1,000

Ours

57.31

±

1.41 41.18

±

0.48 60.71

±

1.02 43.98

±

0.55 43.46

±

0.89 57.60

±

1.19 70.83

±

0.81 59.86

±

0.66

7.40

±

0.20

Baseline

55.80

±

1.16 40.87

±

1.47 56.40

±

2.57 44.26

±

0.52 38.53

±

0.80 54.70

±

1.07 68.55

±

1.39 60.62

±

1.14

5.65

±

0.26

w/o Meta 57.37

±

1.38 42.15

±

0.89 62.84

±

1.19 41.47

±

1.05 43.89

±

0.62 58.33

±

1.76 71.28

±

1.61 60.72

±

0.17

7.25

±

0.09

2,000

Ours

60.18

±

1.99 42.50

±

0.09 63.96

±

0.80 46.47

±

0.96 44.92

±

0.48 59.06

±

1.02 72.72

±

0.90 62.22

±

0.27

7.68

±

0.18

Baseline

68.64

±

0.52

41.59

±

1.53 59.38

±

0.93 52.85

±

0.51 44.87

±

0.28 64.74

±

1.10 71.17

±

0.72 64.57

±

1.63 12.41

±

0.20

1

Full

Ours

66.77

±

1.81

44.24

±

0.36 64.48

±

0.68 53.40

±

1.58 48.39

±

0.49 70.25

±

0.11 72.93

±

1.70 65.30

±

0.44 12.51

±

0.27

Baseline

76.33

±

0.31 47.20

±

1.61 70.92

±

1.02 63.26

±

1.26 59.54

±

0.34 76.18

±

0.87 80.09

±

0.82 74.46

±

0.91 10.33

±

0.64

w/o Meta 77.28

±

0.44 54.57

±

0.75 72.81

±

0.10 69.11

±

0.17 58.33

±

0.37 75.00

±

0.74 79.26

±

0.17 74.45

±

0.39 13.69

±

0.09

1,000

Ours

79.40

±

0.29 56.18

±

0.34 74.49

±

0.35 70.63

±

0.33 61.41

±

0.13 76.21

±

0.10 80.20

±

0.41 75.78

±

0.69 13.98

±

0.09

Baseline

77.59

±

0.41 51.18

±

2.31 72.36

±

0.44 67.68

±

0.86 59.86

±

0.21 76.96

±

0.70 81.73

±

0.37 75.01

±

0.51 11.25

±

0.51

w/o Meta 78.24

±

0.42 55.52

±

0.63 74.89

±

0.23 71.09

±

0.63 60.69

±

0.33 77.06

±

0.18 81.71

±

0.37 75.26

±

0.07 13.83

±

0.24

2,000

Ours

80.78

±

1.05 56.82

±

0.51 76.12

±

0.63 72.11

±

0.48 62.33

±

0.15 77.44

±

0.11 82.31

±

0.45 76.32

±

0.14 14.69

±

0.37

Baseline

85.40

±

0.30 60.64

±

1.64

78.10

±

0.54 76.87

±

1.09 65.38

±

0.48 77.44

±

1.53

85.81

±

0.56

76.58

±

0.25

16.84

±

0.12

10

Full

Ours

82.27

±

0.89 59.47

±

1.20

78.88

±

0.15 76.91

±

0.25 65.76

±

0.26 78.21

±

1.42

84.47

±

0.23

77.07

±

0.98

15.05

±

0.37

In the adaptation stage, we load the meta generator obtained by the training stage and try to adapt it for a target dataset. The optimization is similar to that in the meta-training step of Alg. 1 and the difference is that we use the Adam optimizer to update the parameters instead of pure gradient decent. Notably, there are two ways to initialize X s in each adaptation step, namely single-initialization and multi-initialization modes. In the single-initialization mode, X s is set as the same group of random real samples in the real dataset for all adaptation steps, while in the multi-initialization mode, it can be set differently, with various random seed or/and various sizes of synthetic datasets. We experiment with both ways in this section.

4.2 Comparisons with State of the Arts

In this part, we compare the proposed MGDD with existing state-of-the-art DD methods on standard benchmarks, including CIFAR10 and CIFAR100 [23] datasets. There are 50,000 training images and 10,000 testing images in the 32 × 32 resolution for both datasets, and the numbers of classes are 10 and 100 respectively. For CIFAR10, we compare our method with state of the arts on settings of 1, 10, and 50 images per class (IPC) in synthetic datasets and for CIFAR100, IPCs are 1 and 10. The candidates are three gradient-matching methods DC [60], DSA [58], and IDC [20], one trajectory-matching method MTT [2], one distribution-matching method DM [59], one back-propagation-through-time (BPTT) method RTP [7], and one kernel-ridge-regression-based method FRePo [61]. We obtain synthetic datasets using each algorithm on a 3-layer ConvNet whose architecture is the same as that in [61] and evaluate the accuracy on testing datasets for networks with the same architecture trained on synthetic datasets. For our method, we adapt the generator from the meta model on each comparison setting for 10,000 steps, and for each setting, we report mean and standard deviation over the accuracy of 3 networks initialized from scratch. Moreover, we report the cost of time to derive synthetic datasets.

As shown in Tab. 1, our method achieves comparable performance with DD methods based on heavy iterative loops and yields significant acceleration. For gradient-matching-based based solutions, they typically rely on computing second-order derivatives over a large amount of neural networks to back-propagate the matching loss to synthetic data, which results in high computation overhead. Even worse, for trajectory-matching and BPTT methods, higher-order derivatives are necessary. As a result, their time cost for optimizing synthetic datasets is formidable, and our method achieves up to 6800 × acceleration compared with BPTT. For DM and FRePo, although they do not rely on high-order gradients, they typically require a massive number of forward-backward iterations to update synthetic datasets, e.g. , the FRePo baseline requires 500,000 iterations for optimization, and the efficiency is still unsatisfactory. On the contrary, the results of our method are produced by a generator in a one-stop fashion, which is trained by a meta-learning algorithm and learns to adapt to a target dataset rapidly. It thus enjoys the superior running efficiency of DD. More analysis of the accuracy performance given the same running time can be found in the appendix.

Notably, although the generator is only trained under 32 × 32 RGB images in meta-learning, given that it has a fully-convolutional structure and maintains the same resolution for input and output, it

is applicable to datasets with larger resolutions. It can also be adapted for datasets with different numbers of input channels with minor modifications. In the appendix, we provide more experimental results of such cross-resolution and cross-channel-number generalization cases.

4.3 Empirical Studies

In this part, we focus on some interesting properties of the proposed MGDD method, including crossdataset, cross-objective, cross-architecture, cross-synthetic-initialization, cross-parameterization, and cross-IPC settings. More studies including cross-resolution, cross-channel-number, and cross-classnumber that cannot fit into the main content are put in the appendix.

Cross-Dataset Generalization: MGDD proposed in this paper is expected to be generalized to any downstream target datasets, including those unseen and with large domain shifts from datasets used in the meta learning. To evaluate the cross-dataset generalization performance of MGDD, we conduct experiments on more datasets including one domain generalization dataset PACS [29], two medical classification datasets PathMNIST and BloodMNIST [53], and one fine-grain image classification dataset CUB200 [48]. PACS contains 9,991 images from 7 classes and 4 domains: Photo (P), Art Painting (A), Cartoon (C), and Sketch (S). The style variations across the 4 domains are drastic. We perform dataset distillation on each domain both independently and jointly, which formulates a 28-class dataset. PathMNIST and BloodMNIST contain 107,180 images of 9 classes and 17,092 images from 8 classes respectively. We also combine them together to form a 17-class dataset denoted as PathBloodMNIST. CUB200 contains 6,000 images of 200 bird species. We process all the images to the 32 × 32 resolution in the RGB format and compare the performance with the FRePo baseline [61] and a generator from scratch instead of the meta generator, under 1,000 and 2,000 steps as well as full convergence. The quantitative results shown in Tab. 2 validate the robustness of MGDDunder various datasets and domain shifts.

Cross-Objective Generalization: By default, both the meta-learning and adaptation objectives used in this paper for MGDD are the KRR objective in Eq. 1 following FRePo [61]. Empirically, we find that it is also feasible to adopt different objectives for adaptation. Here, we switch the adaptation objective to DC [60] and DM [59] respectively. The optimization steps for both baselines and our method are set as 2,000. As shown in Tab. 3, our methods yield consistent improvement over different baselines in a limited number of optimization steps.

Cross-Architecture Generalization: Adapted on an original architecture, a satisfactory generator is expected to produce results that are also valid to train networks with different structures, namely cross-architecture generalization. Tab. 4 shows the performance on CIFAR10, where ConvNet is used in adaptation while AlexNet [24], VGG11 [46], and ResNet18 [14] are used as unseen architectures for evaluation. The results indicate that the accuracy improvement on the original model still holds for unseen structures.

Cross-Initialization Generalization: We find that the single-initialization scheme used in Tab. 1 may lead to over-fitting of the generator to the adopted single initialization. As shown in Tab. 5, if we change the initialization of synthetic data, the performance would drop dramatically. Fortunately, multi-initialization is an alternative to account for this drawback, wherein each adaptation step, a new initialization of the synthetic dataset is sampled from the original dataset. Tab. 5 indicates that multi-initialization typically requires more adaptation steps for convergence and can perform on par with single-initialization. It is useful when samples in the synthetic dataset are concerned with user privacy, given that visualization results produced by the KRR objective are somehow realistic, as illustrated in Fig. 6 and [61]. In such cases, replacing the current dataset with the results of another

10

20

30

40

50

100

200

IPC

40

45

50

55

60

65

70

Test Accuracy (%)

w/o Embed

Random

Ours

100

200

300

400

500

1000 2000

Adaptation Steps

40.0

42.5

45.0

47.5

50.0

52.5

55.0

57.5

60.0

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

2

4

6

8

10

CL Task Index

25

30

35

40

45

50

55

60

65

Test Accuracy (%)

Ours from Meta

Ours Adapt Once

Baseline

Ours from Previous

initialization would help solve the problem efficiently, without the necessity to re-run the whole optimization loop of existing methods.

Cross-Parameterization Generalization: Beyond different training objectives, the proposed MGDD is also orthogonal with different synthetic data parameterization tricks. In Tab. 6, we consider storing 2 × down-sampled synthetic images instead of the raw ones. Thus, 4 × synthetic samples can be stored given the same storage budget. We find that the simple strategy can lead to additional performance gain for relatively small budgets. The observation is consistent with previous works [20, 30, 7].

Cross-IPC Generalization: One crucial benefit of the MGDD is the cross-IPC generalization. Once the generator is adapted for a target dataset, when the storage budget changes, we do not need to perform the optimization again, unlike previous solutions relying on iteratively updating synthetic datasets. To demonstrate the cross-IPC generalization performance, we conduct experiments on CIFAR10 and adapt the meta generator using multi-initialization, with 10 and 50 IPCs for 2,000 steps. The adapted generator is evaluated on unseen IPCs 20, 30, and 40. The results are shown in the red curve of Fig. 3, where the generator produces satisfactory synthetic datasets with unseen IPCs.

To make the generator aware of the sizes of synthetic datasets, we concatenate size embedding channels to features at the middle layer of the generator. To understand how the embedding works, we remove these channels and conduct the same evaluation. As shown in the yellow curve of Fig. 3, the performance degrades without size embedding. In Fig. 6, we visualize some samples before and after the adapted generator on CIFAR10 with 1 and 50 IPCs. We can observe that the results are quite different: for small sizes, the generated results are vague, while for large sizes the results are more realistic, and their styles are also different. Thus, it is reasonable to take different sizes of synthetic datasets into consideration in the inference stage.

We also try training the generator from scratch instead of the meta model on the target dataset. As shown in Fig. 3, the worse performance in the green curve indicates that meta-learning is crucial for finding a satisfactory initial checkpoint for downstream adaptation.

Moreover, we also evaluate the performance on higher IPCs like 100 and 200, and the results are still encouraging compared with random real samples, which indicates that our method can serve as an alternative when the computational resource cannot support optimization for larger IPCs directly.

Various Adaptation Steps: In Fig. 4, we visualize the accuracy under different adaptation steps on CIFAR10 with 10 IPC as the red curve. Compared with training from scratch and the baseline FRePo [61], as shown in the yellow and green curves respectively, our method results in significantly faster adaptation convergence, which would be attributed to the good initial generator found by the meta-learning algorithm. Thus, our method is more applicable in scenarios requiring high efficiency, like processing streaming data.

50 IPC before Generator

1 IPC before Generator

dass_0

Furthermore, to demonstrate the effectiveness of analytical labels, we replace them with vanilla one-hot labels in synthetic datasets and the performance is shown in the blue curve. The considerable performance drop indicates the importance of minimizing the error in an original space via analytical labels, which is consistent with the theoretical analysis in Theorem 1.

4.4 Application: Continual Learning

Continual learning (CL) aims to learn from a stream of data, where historical and future data are unavailable when learning the current data batch. One important issue is catastrophic forgetting [22]: a model tends to forget knowledge acquired in previous data when learning on newly-coming data. Focusing on this drawback, many works introduce a buffer with limited memory to store core data and knowledge of past experience for future use [40, 1]. Dataset distillation benefits this field by generating informative samples [36, 5, 42, 35, 44] to prevent forgetting as much as possible.

In this paper, we evaluate the CL performance of the proposed MGDD on CIFAR100, following the same protocol of [61, 59], where all 100 classes are divided into 10 tasks randomly with 10 classes for each task. For each task, a buffer with 20 images for each class is allowed for synthetic data. We first try adapting the generator on each new task from the meta model for 2,000 steps and the performance is shown in the yellow curve in Fig. 5. Alternatively, we can choose to adapt the generator from the checkpoint of the previous task, which has already learned some global knowledge of full data and yields better performance, as shown in the red curve. Notably, it is also feasible to only adapt the generator on the first task and the remaining tasks directly adopt this generator to output synthetic data. As shown in the blue curve, with the most significant flexibility, the performance is still comparable with the FRePo baseline [61] shown in the green curve, which suggests great practical value for processing data streams.

5 Conclusion

In this paper, we propose MGDD, a novel feed-forward paradigm for dataset distillation (DD). Specifically, in our pipeline, synthetic labels are obtained by solving a least-squares problem equipped with an analytical solution, and synthetic samples are transferred from their initial results by a conditional generator instead of taking the whole original training dataset as input. Theoretical derivation indicates an error upper bound of the proposed framework. On the one hand, unlike existing DD approaches requiring time-consuming forward-backward iterations through a massive number of networks, MGDD generates distilled results with a generator adapted rapidly from a meta generator, which improves the efficiency of DD significantly. On the other hand, existing techniques have to repeat the whole iterative algorithms for different sizes of synthetic datasets, while MGDDcan perform inference flexibly on various sizes once adapted to the target dataset. Focusing on the efficiency of adaptation on target datasets, we propose a meta-learning algorithm to train a meta generator, such that it can acquire knowledge of target datasets sufficiently in only a few steps. Experiments demonstrate that the proposed MGDD performs on par with existing state-of-the-art DD baselines under 22 × acceleration. It also exerts strong cross-size generalization ability even on sizes of synthetic datasets unseen during adaptation. Future works may explore advanced feed-forward fashions of DD, focusing on generation pipelines, training algorithms, and network architectures, making improvements on the cross-dataset, cross-size, and cross-architecture generalization.

Acknowledgment

This work is supported by the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant (project number: A0005947-21-00, project reference: ECT-RP2), and the Singapore Ministry of Education Academic Research Fund Tier 1 (WBS: A0009440-01-00).

References

Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking experience replay: a bag of tricks for continual learning. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 2180-2187. IEEE, 2021.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. arXiv preprint arXiv:2203.11932 , 2022.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. arXiv preprint arXiv:2305.01649 , 2023.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. arXiv preprint arXiv:2211.10586 , 2022.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):3366-3385, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206.02916 , 2022.

Jiawei Du, Yidi Jiang, Vincent TF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. arXiv preprint arXiv:2211.11004 , 2022.

Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural Pruning for Diffusion Models. In Advances in Neural Information Processing Systems , 2023.

Fastai. Fastai/imagenette: A smaller subset of 10 easily classified classes from imagenet, and a little more french.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. arXiv preprint arXiv:2305.01975 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.00311 , 2022.

Yongcheng Jing. Efficient Representation Learning With Graph Neural Networks . PhD thesis, 2023.

Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Amalgamating knowledge from heterogeneous graph neural networks. In CVPR , 2021.

Yongcheng Jing, Chongbin Yuan, Li Ju, Yiding Yang, Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In CVPR , 2023.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. arXiv preprint arXiv:2205.14959 , 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.00719 , 2022.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. arXiv preprint arXiv:2202.02916 , 2022.

Shiye Lei and Dacheng Tao. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.05603 , 2023.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision , pages 5542-5550, 2017.

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Songhua Liu and Xinchao Wang. Few-shot dataset distillation via translative pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 18654-18664, October 2023.

Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755 , 2023.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. In Advances in Neural Information Processing Systems , 2023.

Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing , 469:28-51, 2022.

Wojciech Masarczyk and Ivona Tautkute. Reducing catastrophic forgetting with learning on synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Workshop , 2020.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:2011.00050 , 2020.

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34, 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.

Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. arXiv preprint arXiv:2103.15851 , 2021.

Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint arXiv:2301.04272 , 2023.

Mattia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2022.

Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Loss-curvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics , pages 8606-8628. PMLR, 2023.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.04707 , 2023.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. arXiv preprint arXiv:2203.01531 , 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR , abs/1708.07747, 2017.

Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data , 10(1):41, 2023.

Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision , 2022.

Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. In Advances in Neural Information Processing Systems , 2022.

Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Xu Dongkuan. Accelerating dataset distillation via model augmentation. arXiv preprint arXiv:2212.06152 , 2022.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , pages 12674-12685. PMLR, 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929 , 2020.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719 , 2022.

In this part, we include more details about the technical methods, more experimental results of our MGDDmethod, and more discussion on limitations and future works, which cannot be accommodated in the main paper due to the page limit. Our method first trains a meta generator to generate synthetic samples and then an adaptation stage is executed for a target dataset. We provide algorithmic details of the adaptation stage, a summary of hyper-parameters, and configurations of our generator architecture. Then, we conduct more evaluations on the cross-number-of-channel, cross-resolution, cross-ipc, and cross-number-of-classes performance of our method. More discussions of the adaptation performance and more qualitative examples are also included. Finally, we discuss limitations of the proposed method and potential future works.

A More Details

Adaptation Algorithm: Alg. 1 demonstrates the procedure of meta learning to obtain a meta synthetic sample generator. On a downstream target dataset, the meta network is adapted to a specific network with a limited number of steps. The adaptation algorithm is similar to the meta-training step of the meta learning algorithm. Here, we present the full details in Alg. 2. We can prepare multiple initialization of synthetic samples through randomly sampling from the target dataset. Recall that the main pipeline of our algorithm is to first obtain analytical synthetic labels in a random neural space θ : Y ∗ s = f θ ( X s ) W θ t . Here, the optimal kernel-ridge-regression parameters of the target dataset W θ t can be computed by W θ t = f θ ( X t ) ⊤ ( f θ ( X t ) f θ ( X t ) ⊤ ) -1 Y t , if the number of real samples n t is smaller than the feature dimension p . Otherwise, W θ t = ( f θ ( X t ) ⊤ f θ ( X t )) -1 f θ ( X t ) ⊤ Y t .

After the calculation of analytical labels, we fix them and train the synthetic sample generator initialized by parameters of the meta generator for some steps. The optimization objective is similar to those in Zhou et al. [61] and Loo et al. [32]. The difference is that the optimization target is parameters of the generator instead of synthetic samples.

Summary of Hyper-Parameters: For a clear view, we summarize the hyper-parameters and their values in both meta learning and adaptation stages as shown in Tab. 7. All experiments follow these default settings of hyper-parameters if not specified. Other configurations unmentioned follow the settings of the baseline FRePo [61].

Generator Architecture: We illustrate the detailed configurations of our generator architecture in Fig. 7. It essentially adopts an encoder-decoder structure with 3 Conv-BatchNorm-ReLU blocks and 2 AvgPool layers for down-sampling for the encoder and a symmetric structure for the decoder. Notably, to make the network aware of different sizes of synthetic datasets, we concatenate the size embedding to bottle-necked features after the encoder. Inspired by the positional embedding in Transformer models [47] and the time-step embedding in diffusion models [15, 39], we encode the size by sinusoidal signals and a learnable non-linear transformation function. Embedding features are replicated and expanded along the spatial axes before concatenation with features from the encoder.

AvgPool

ReLU

BN

Conv 128

ReLU

BN

Conv 512

Conv 3

AvgPool

ReLU

BN

Conv 256

ReLU

BN

Conv 256

UpSample

Conv 128

BN

ReLU

Linear

ReLU

Linear

Expand

Concat.

～

Sinusoid

B More Results

Cross-Number-of-Channel Generalization: In the meta learning stage, a meta generator is trained taking RGB images as input and output. Here, we demonstrate that it is also be feasible for the meta generator to be adapted for target datasets that have different numbers of channels. Specifically, we additionally train convolution layers for channel adaptation to map the number of channels from the original number to 3 and from 3 to original number at the beginning and the ending positions of the generator, respectively. The parameters of these adaptors are initialized from a uniform distribution and are optimized jointly with parameters of the generator.

Here, we conduct experiments on MNIST [25] and FashionMNIST [52] datasets. Both of them contain 10 classes with 60,000 single-channel images. Results are shown in Tab. 8 following the same comparison protocols as Tab. 1, where the generator in our method is adapted for 10,000 steps in each setting. Experiments demonstrate that our method can achieve comparable performance with those state-of-the-art ones in a significantly shorter period of time. The conclusion is the same as that in the main paper.

Cross-Resolution Generalization: Although the meta generator is trained under 32 resolution, it is possible for it to be adapted for datasets with different resolutions, thanks to the fully-convolutional architecture of the generator. We demonstrate the cross-resolution generalization performance on ImageNette [11], which contains 10 classes and 9,469 images. Following the FRePo baseline [61], we conduct experiments on 1 and 10 IPCs under 128 resolution. Results in Tab. 9 demonstrate the feasibility of such cross-resolution generalization.

Cross-Number-of-Class Generalization: Here, we conduct experiments on CIFAR100 subsets with random 20 and 50 classes respectively and compare the performance with the FRePo baseline [61]. Results in Tab. 10 demonstrate that the meta generator performs robustly on datasets with various numbers of classes.

Cross-IPC Generalization: For existing methods, when budgets for synthetic datasets change, they have to either repeat the time-consuming training loop of dataset distillation, which is inconvenient if not infeasible at all, or prune some synthetic data heuristically, which leads to inferior performance. For example, as shown in Tab. 11, on CIFAR10, if the original synthetic IPC is 50 and the new IPC becomes 20 or 5, random pruning would lead to unsatisfactory performance for existing methods. By contrast, the generator in our MGDD can work for arbitrary sizes of synthetic datasets once adapted, which makes it handle such scenarios better. We present another example on CIFAR100, the original IPC is 10 and the new IPC is 5 or 2.

100

300

500

1000

2000

Adaptation Steps (1 IPC)

50

60

70

80

90

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (10 IPC)

95.0

95.5

96.0

96.5

97.0

97.5

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (50 IPC)

98.2

98.3

98.4

98.5

98.6

98.7

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

Comparisons under the Same Steps: To better demonstrate the superiority of the proposed method, we compare our method with state of the arts with the number of training/adaptation steps controlled the same. As shown in Tab. 12, under 1000 steps, our method outperforms others significantly especially on relatively challenging datasets with more patterns, like CIFAR10 and CIFAR100. Furthermore, in Fig. 8, 9, 10, and 11, we visualize the performance of generators in each setting with different adaptation steps on MNIST, FashionMNIST, CIFAR10, and CIFAR100 datasets respectively as supplements to Fig. 4. It can be shown that our method can achieve the most satisfactory performance with only a limited number of adaptation steps compared with the baseline FRePo and generators from scratch, which indicates that the proposed method is more suitable for scenarios requiring high efficiency, like processing data streams. Note that for 1 IPC, we observe that using analytical labels would often lead to inferior performance compared with vanilla one-hot labels. We speculate that it is because soft labels by the analytical solution are relatively not good at leading the generator to synthesize class-discriminative patterns when the size of synthetic dataset is small. Thus, we do not use analytical labels for 1 IPC by default.

Qualitative Results: In Fig. 12, we supply qualitative visualization of initialized synthetic samples and results by generator under 1 and 10 IPC on CIFAR10 and 1 IPC on CIFAR100, as supplements to Fig. 6.

C Limitations and Future Works

Our MGDD method mainly focuses on the efficiency issue in existing methods. Although it can be demonstrated that our method can result in better performance in only limited time, it does not reduce the time and memory complexity of computing the matching metrics since we adopt the same objectives as previous approaches. When adapting for large synthetic datasets, it may still face the issue on GPU memory in existing works. Nevertheless, it is possible for our method to adapt on some small IPCs and then generalize to large synthetic datasets, as discussed in the main paper, which can serve as a remedy to this limitation. Besides, initialized samples of synthetic datasets come from real data, and results by generator still look somehow realistic, which may potentially make the method vulnerable to privacy attack, especially for data like personal information. Also, in scenarios like storing synthetic samples of human faces, the generator may break the integrity of faces and lead to an infringement of portrait rights if being misused.

Future works may focus on more effective training objective, training pipeline, and architecture of the generator in meta learning or/and adaptation stages to further improve the cross-dataset, cross-ipc, and cross-architecture generalization. It would also be valuable to extend the MGDD to other tasks and modalities beyond image classification and explore advanced input and output parameterizations of the generator.

100

300

500

1000

2000

Adaptation Steps (1 IPC)

55.0

57.5

60.0

62.5

65.0

67.5

70.0

72.5

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (10 IPC)

78

79

80

81

82

83

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (50 IPC)

82

83

84

85

86

87

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (1 IPC)

20

25

30

35

40

45

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (10 IPC)

40.0

42.5

45.0

47.5

50.0

52.5

55.0

57.5

60.0

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (50 IPC)

47.5

50.0

52.5

55.0

57.5

60.0

62.5

65.0

67.5

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (1 IPC)

8

10

12

14

16

18

20

22

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

100

300

500

1000

2000

Adaptation Steps (10 IPC)

18

20

22

24

26

28

30

32

Test Accuracy (%)

w/o Ana. Label

Baseline

Ours

Before Generator

After Generator

1 IPC, CIFAR10

Before Generator

After Generator

10 IPC, CIFAR10

Before Generator

After Generator

1 IPC, CIFAR100

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the efficiency concerns associated with dataset distillation. Initially, the authors identify the challenges arising from the dependencies within the intensive loop of forward-backward propagation across numerous neural networks, as well as the repetitive training procedures necessary when working with synthetic datasets of varying sizes. Motivated by these challenges, the authors propose an innovative approach to dataset distillation that eliminates the need for backpropagation. In this approach, synthetic labels are generated by analytically solving a least-squares problem in a feature space, while synthetic images are produced using a pre-learned meta generator. The authors provide theoretical evidence supporting the existence of a conditional generator capable of transferring synthetic samples from their original space to the desired space with an upper-bounded error. Experimental results demonstrate that the proposed backprop-free dataset distillation approach achieves comparable performance to state-of-the-art methods, while also providing a 22× acceleration. Additionally, the approach exhibits the flexibility of cross-size generalization.\n\nSTRENGTHS:\nI see the following positive traits:\n\n+ This paper effectively tackles a crucial challenge in dataset distillation by addressing the cumbersome forward-backward propagation process required for effective distillation. The authors present an elegant solution rooted in meta-learning, which not only achieves an impressive speed improvement of 22 times but also maintains competitive performance levels.\n\n+ The proposed method is supported by robust theoretical analysis, as evidenced by the proof of Theorem 1, where an upper-bounded error is derived, demonstrating the existence of a conditional generator capable of transferring the synthetic dataset from its original space to the desired feature space with an upper-bounded error.\n\n+ The paper offers comprehensive algorithmic workflows, which are presented in both the main paper (Section 3.3, Algorithm 1) and the supplementary material (Section A: More Details, Algorithm 1) for handy reproducibility. \n\n+ The paper is very well-written and easy to follow. The authors try to elaborate every detail while providing a clear overall picture.\n\n+ The supplementary material of the paper offers an abundance of extensive details and additional experimental results including the visualization results, further validating the effectiveness of the proposed backprop-free dataset distillation method.\n\nWEAKNESSES:\nIn general, this paper focuses on a significant problem, proposes an elegant solution, and the experimental results convincingly validate the effectiveness of the proposed approach. I only have some minor concerns as follows.\n\n- The authors mention that the estimated acceleration factor includes the adaptation time. Given that the main emphasis of this paper is on time complexity, I would suggest the authors present the adaption time and feed-forward time separately in the experiments, to effectively illustrate the time cost involved in the process.\n\n- I wonder how long it takes to train the meta generator on ImageNet1k.\n\nQUESTIONS:\n1. What is the physical adaption time for the results reported in the experiments?\n2. How long does it require to train the meta generator on ImageNet1k?\n\nLIMITATIONS:\nThe authors have thoroughly addressed the limitations in Section C of the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe reviewed paper makes a clear and novel contribution to the field of dataset distillation, which is expected to have a beneficial impact. The paper is well written, demonstrating a structured and easily understandable presentation. Notably, the empirical evaluation is exceptional, covering various aspects of the proposed method&rsquo;s effects, including evaluation of a potential application and examination of its combination with orthogonal methods.\n\nSTRENGTHS:\nThe reviewed paper makes a clear and novel contribution to the field of dataset distillation, which is expected to have a beneficial impact. The paper is well written, demonstrating a structured and easily understandable presentation. Notably, the empirical evaluation is exceptional, covering various aspects of the proposed method&rsquo;s effects, including evaluation of a potential application and examination of its combination with orthogonal methods.\n\nWEAKNESSES:\n### Main Claim of being Backprop-Free\n\nI disagree with the main claim of this work being backpropagation-free. The core mechanism of the *distillation* procedure in this work is the training of the meta-generator $g_{\\omega}$ and its adaptation to a new target dataset. Both procedures rely on backpropagation. Yes, the actual generation of synthetic samples does not include backpropagation since it is a forward pass of a generative model with real samples as input. But that is like saying GANs are backprop-free &#x2013; sure, the generation of new samples does not require backpropagation, nevertheless the GAN needs to be trained, i.e. its parameters optimized, which relies on backpropagation for the generation of new samples to actually match the target data distribution. There is one, similar in concept, concurrent work to this paper: &ldquo;DiM: Distilling Dataset into Generative Model&rdquo; [1] which also distills the dataset into generative model to then later produce synthetic samples from this model &#x2013; similar to how in this work the meta-generator is trained to then produce synthetic samples. As explained in DiM and applicable to this work as well, the difference to standard dataset distillation methods is that the medium of information (into which the dataset is distilled) are not images but a parameterized model (from which one can then infer/sample images).\n\nI am bit unsure of what to do now. I really like this work, it is novel, of high quality, easy to comprehend, and the evaluation is remarkable. But I disagree with the main claim of this being backprop-free, which is also part of the title. If this claim is removed (which I do not know if it is possible or if this would be too much of a change), I am happy to change my rating to accept. I am looking forward to discuss this issues with the authors and the other reviewers.\n\n\n### Runtime Performance\n\nIn line 38-41 the authors state &ldquo;for FRePo, the method with the best performance requires above 3 hours to obtain a synthetic dataset with 1 image per class for CIFAR100&rdquo;. Where do these &ldquo;3 hours&rdquo; come from? Your Table 1 lists about 3 hours for FRePo on 1-IPC on CIFAR100 but this does not align with what is reported in the FRePo paper: &ldquo;When learning one image per class on CIFAR100, FRePo reaches a similar test accuracy (23.4%) to the second-best method (24.0%) in 38 seconds [&#x2026;]. Moreover, FRePo achieves 92% of its final test accuracy (26.4% out of 28.7%) in only 385 seconds.&rdquo;. I couldn&rsquo;t find the runtime for their &ldquo;final test accuracy&rdquo;, but even only taking their 26.4% result in 385 seconds (1-IPC CIFAR100) into consideration, FRePo beats the proposed method in terms of accuracy (24.66%) and is 1.3x faster than the proposed method (505s) &#x2013; opposed to x22 slower as claimed in the paper. Furthermore, the hardware used in FRePo (Quadro RTX 6000) is worse in terms of performance than the one used in this work (RTX 3090 for adaptation) which should only widen this discrepancy even more. Furthermore, the meta-training cost is not taken into account. Maybe I got something completely wrong here and I&rsquo;m happy to be corrected if I did, but since the x22 statement is repeated multiple times in this work I wanted to look into the details here.\n\n\n\n### Minor Suggestions for Improvement\n\n-   Figure 3-5: For improved visual consistency, you could try to make these figures the same height and width.\n-   Paragraph of Line 296: I was shortly confused about which figure you were talking about &#x2013; another reference to Figure 3 would help here.\n\n[1] Wang, K., Gu, J., Zhou, D., Zhu, Z.H., Jiang, W., & You, Y. (2023). DiM: Distilling Dataset into Generative Model. ArXiv, abs/2303.04707.\n\nQUESTIONS:\n-   What are the runtime costs of training the meta-generator?\n-   Line 227: &ldquo;Moreover, we report the cost of time to derive synthetic datasets.&rdquo; -> Can you be more specific here? What includes &ldquo;derive synthetic datasets&rdquo;? Is this the adaptation procedure or does this inlcude the meta-training?\n-   Line 200: &ldquo;For computation efficiency, the generator processes each sample independently&rdquo; -> Why should this be more efficient than batch-wise processing?\n-   Line 202: &ldquo;The meta generator is trained by the Adam optimizer [15]&rdquo; -> In Algorithm 1 it is plain GD. Is this just out of simplicity in Algorithm 1 or am I missing something?\n-   Line 238: &ldquo;On the contrary, the results of our method are produced by a generator in a one-stop fashion&rdquo; -> What do you mean by &ldquo;one-stop&rdquo; fashion? The meta-generator needs to be adapted to the target dataset which includes a certain number of forward-backward passes of $g_{\\omega}$.\n\nLIMITATIONS:\nLimitations are addressed in the appendix. One main limitation which the work generally glances over is the meta-training procedure and its associated cost. All of the experimental evaluations have focused on the adaptation to a target dataset, but a core part of the proposed method lies in the meta-training of $g_{\\omega}$.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposing a meta-learning version of dataset distillation (DD)\n- The main idea is that dataset distillation is slow, and meta-learning can be used to amortize the dataset distillation procedure\n- The goal a function $g_\\omega (X_{init}) = X_S$, which changes an initialized distilled dataset to the final one\n- The labels are learned to minimize the KRR loss, using the feature space of randomly initialized networks\n- Compared to MAML, in this case $\\omega_{init}$ is functionally equivalent to $\\theta_{init}$ in MAML, and $\\omega$ is updated for a few parameter steps. In this sense the goal is to learn an optimal $\\omega_{init}$\n\nSTRENGTHS:\n- A meta-learning approach to DD is new, and is valuable considering that current dataset distillation algorithms are very slow (and often the cost of doing DD is much much greater than the cost of training networks on the full data, often negating their value)\n\n- The improvements in computation time are very dramatic, and the cost in terms of performance on the reported baselines is low\n\n- I am very impressed with the cross-IPC generalization (line 281). Being able to growing/shrink a distilled dataset in response to changing needs without completely retraining a network is very desirable, and is lacking with current DD algorithms\n\nWEAKNESSES:\n- Poor presentation: I personally found the presentation very confusing. In particular, section 3.2 seemed to add more confusion to the paper. I think and outline of the method (3.3) should have been presented before this theory section. In particular I found the idea of the parameterized transfer function $g_\\omega$ to have been irrelevant until after reading section 3.3. In regards to presentation, I think it needs to be more clearly stated that $\\omega$ is adapted in training, and that you are learning the initialization of $\\omega$, because section 2.3 gives the impression that $\\omega$ is independent of $X_t$ (line 136 - \"without any dependence on $X_t$\"), which is misleading since it updated using $X_t$. I think discussing this in comparison to MAML would be useful since readers are probably familiar with MAML.\n\n- Limited evaluation in terms of datasets. As a major selling point of a meta-approach to dataset distillation is being able to quickly adapt to unseen datasets, it seems necessary to test more than just CIFAR-10 and CIFAR-100. I understand that this is difficult as most dataset distillation algorithms don't benchmark on a lot of datasets (for example apart from CF-10/100, MNIST/F-MNIST, CUB-200 and tinyimagenet/imagenet subsets seem to be the only other common ones), and since this algorithm meta-trains on imagenet, some of these are eliminated. However, I think it is particularly important for this paper to include more datasets. Even mixed datasets (i.e. images from collections of different datasets) may be sufficient\n\n- More discussion of meta-training costs. In particular, how long does meta-training take? It also seems that $\\omega$ is optimized for several hundred iterations, and this sounds like in order to store this is memory would be quite costly. I see that in lines 200-205, it is mentioned that a A40 (a singificantly stronger GPU) is used for meta-training. This cost should be stated more clearly in the limitations\n\n-Minor:\n - I think there are typos in eq (2) (line 125). I'm not sure $\\theta^*$ is meant to appear here.\n- Missing citation. Lines 129-130: the method of solving for the optimal labels was first proposed as \"label solve\" in [1]\n\n[1] Timothy Nguyen, Zhourong Chen, & Jaehoon Lee (2021). Dataset Meta-Learning from Kernel Ridge-Regression. In International Conference on Learning Representations.\n\nQUESTIONS:\n- I'm not sure if Theorem 1 is particular relevant to the method. As mentioned in the weaknesses, this theorem takes $\\omega$ to be independent of $X_t$ but in reality it is not because of meta-adaptation. Furthermore, there is not really interpretation of what the upper bound means. What does the right hand side term in eq. 6 mean? In particular why does it make sense to consider the product $f_{\\theta^*}(X_s)W^{\\theta}_s$, considering that one is based on the feature space $\\theta$ and the other based on $\\theta^*$? While the feature spaces are expected to induce similar kernels, multiplying the features together directly does not make sense because they could be randomly permuted versions of one another, for example\n\n- What is the effect of using a larger learning rate for FRePo in figure 4?\n\n- Why do you base $Y_s^*$ on the initialization images $X_{s}$ instead of the transformed images $g_\\omega(X_s)$ in Alg. 1 line 5?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
This paper proposes a meta-learning conditional generator method that takes some initialization of synthetic images and quickly learns to render it for Dataset Distillation purposes. The authors extensively verified the performance and show that the proposed method is substantially faster and achieves good performance.

STRENGTHS:
+ This paper is well motivated and a good contribution to the Dataset Distillation community
+ The proposed algorithm is interesting. The algorithm is almost like treating the conditional generator as an updater/optimizer that learns to iteratively refine the synthetic set for a customized loss
+ The experimental results support the claims

WEAKNESSES:
- From figure 6, it looks like the updated images aren't moving a lot after the refinement by generator. Could the authors explain the potential reasons?
- The performance of DD are eventually not better than previous results.
- The method requires an additional neural network component for update/refinement

Misc. line 40 RTP -> RTP[7], missing citation number.

QUESTIONS:
See above.

LIMITATIONS:
Didn't find the equivalent paragraph for limitation. The authors should add and highlight it?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11