Dataset Distillation using Neural Feature Regression
Authors: Yongchao Zhou
Ehsan Nezhadarya
Jimmy Ba
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Learning
Dataset Distillation
Bi-Level Optimization
Meta Learning
Continual Learning
Privacy-Preserving

Abstract: Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data. Meta-gradient computation is one of the key challenges in this formulation, as differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achieving the state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. The proposed algorithm is analogous to truncated backpropagation through time with a pool of models to alleviate various types of overfitting in dataset distillation. FRePo significantly outperforms the previous methods on CIFAR100, Tiny ImageNet, and ImageNet-1K. Furthermore, we show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense. Please check out our webpage at https://sites.google.com/view/frepo.

Dataset Distillation using Neural Feature Regression
Yongchao Zhou Department of Computer Science University of Toronto
Ehsan Nezhadarya Toronto Al Lab
LG Electronics Canada ehsan www...eehaaara@@@le.ccomm
yongchao zhou@mail www.onto.ccomm ca
Jimmy Ba Department of Computer Science University of Toronto j jbaccs. toronto.......... edu
Abstract
Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated bi-level meta-learning problem where the outer loop optimizes the meta- dataset and the inner loop trains a model on the distilled data. Meta-gradient computation one of the key challenges in this formulation, differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achieving the state-of-the- performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. The proposed algorithm is analogous to truncated backpropagation through time with a pool of models to alleviate various types of overfitting in dataset distillation. FRePo significantly outperforms the previous methods on CIFAR100, Tiny ImageNet, and ImageNet- IK. Furthermore, we show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense. Please check out our webpage at wethhttt::/// //sites. google. com/voo//v/ooooo wevvvviefffrrrre
Introduction
Knowledge distillation is technique in deep learning to compress knowledge for easy deployment. Most previous works focus on model distillation [2, 3 where the knowledge acquired by a large teacher model transferred to small student model. contrast, dataset distillation 4 aims to learn small set of synthetic examples preserving most of the information from a large dataset such that a model trained on it can achieve similar test performance as one trained on the original dataset. Distilled data accelerate model training and reduce the cost of storing and sharing dataset. Moreover, its highly condensed and synthetic nature can also benefit various applications, such as continual learning (5H)), neural architecture search [5] [ and privacy-preserrve
tasks (9 10].
Dataset distillation was first studied by Maclaurin et rameter optimization and subsequently Wang et al. 4 formally proposed dataset distillation as a new task. Dataset distillation can be naturally formulated as bi-level meta-learning problem. The inner loop optimizes the model parameters on the distilled data (meta-pane mnraa-parameters) while the outer loop refines the distilled data with meta-gradies updates.
in the context of gradient- cradient-based hyperpa-
One key challenge in dataset distillation is computing the meta-gradiem Several methods 4НН compute by back-propagating through the unrolled computation graph, but they often suffer from
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) CIFAR100
(b) Tiny ImageNet
(c) ImageNette and ImageWoof
Figure I: Example distilled images from 32x32 CIFARI00, 64x64 Tiny ImageNet, and 128x 128 ImageNet Subset. The images look real and transfer well to different architectures. They can be used for various downstream applications, such continual learning and membership inference defense.
huge compute and memory requirement 14.. training instability 15 16]
and truncation bias [7
To avoid unrolled optimization, surrogate objectives are used to derive the meta-gradient, such as gradient matching (5) 77 [8), feature aligmment [8] 19]. and training trajectory matching [20].
Nevertheless, a surrogate objective may introduce its own bias [9], and thus, may not accurately
reflect the true objective. An alternative is using kernel methods, such as Neural Tangent Kernel (NTK) (1], to approximate the inner optimization modern neural network can be extremely expensive 22 23
22 23 However, computing analytical NTK for
Even with accurate meta-gradient, dataset distillation still suffers from various types of overfitting.
For instance, the distilled data can easily overfit to a particular learning algorithm 4 13 20 certain stage of optimization
K [9]. or a certain network architecture 55 7 20 22 23 Meanwhile,
the model can also overfit the distilled data during training, which is the most common cause of overfitting when we train on small dataset. All these kinds of overfitting impose difficulties on the training and general-purposs. use of the distilled data.
We propose an efficient meta-gradient computation method and "model pool" to address the overfitting problems. The bottleneck in meta meta-gradient computation arises due to the complexity of inner optimization,
need to know how the inner parameters vary with the outer parameters (24). However, the inner optimization can be pretty simple if we only train the last layer of a neural network convergence while keeping the feature extractor fixed. In this case, computing the prediction on the real data using the model trained on the distilled data can be expressed as a kernel ridge regression (KRR) with respect to the conjugate kernel (25). Hence, computing the meta-gradient simply back- propagating through the kernel and a fixed feature extractor. To alleviate overfitting, we propose to maintain a diverse pool of models instead of periodically training and resetting a single model as in prior work [7] [3] Intuitively, our algorithm targets the following question: what is the best data to train the linear classifier given the current feature extractor? Due to the diverse feature extractors we use, the distilled data generalize well to wide range of model distributions.
Summary of Contributions:
We propose effective method for dataset distillation. Our method, named neural Feature Regression with Pooling (FRePo), achieves state-of-the-an art results on various benchmark datasets with a 100x reduction in training time and 10x reduction in GPU memory require- ment. Our distilled data looks real (Figure I and transfers well to different architectures.
We show that FRePo scales well to datasets with high-resolution images or complex label space. We achieve 7.5% topl accuracy on ImageNet- IK (26) using only one image per class. The same classifier obtains only 1. 1.1% accuracy from a random subset of real images. The previous methods struggle in this task due to large memory and compute requirements.
We demonstrate that high-quality distilled data can significantly improve various downstream applications, such as continual learning and membership inference defense.

---

Derilled
Porwed
Xxx, Xxx.
((X.) (AA)
X.
60 FRIPO
00 Urrodedd Opramention
Figure 2: Comparison of FRePo and Unrolled Optimization. S, X Y, are the distilled dataset, images and labels. C is the meta-training loss and ((k), g() are the model parameter and gradient at step k. fX) is the feature for input X and K is the Gram matrix of X, and X,, FRePo is analogous to 1-step TBPTT as computes the meta-gradient at each step while performing the online model update. However, instead of backpropagating through the inner optimization, FRePo computes the meta-gradient through kernel and feature extractor.
Method
2.1 Dataset Distillation as Bi-level Optimization
Suppose we have a large labeled dataset T = yi), -(xx..))....x((77)1 with [7] image and label pairs. Dataset distillation aims to learn a small synthetic dataset S YI), (xis) Yis) : that preserves most of the information in T. We train several neural networks parameterized by 9 on the dataset S and then compute the validation loss [(Alg S), 7) on the real dataset T. where Alg (0, S) the neural network parameters optimized by learning algorithm Alg with the model initialization 0 and distilled dataset S as its inputs. The validation loss L(Alg (0, .). 7) is noisy objective with the stochasticity coming from random model initialization and inner learning algorithm. Thus, we are interested in minimizing the expected value of this loss, which we denote it as F(S). We formulate the dataset distillation as the following bi-level optimization problem.
outer-level
inner-level
S* argmin F(S), where F(S) Eg Po C Alg (0, 7)
(1)
In this bi-level setup, the outer loop optimizes the distilled data to minimize F(S), while the inner loop trains neural network using the learning algorithm, Alg, to minimize the training loss on the distilled data S. From the meta-learning perspective, the task is defined by the model initialization 8, and we want to learn meta- parameter S that generalizes well different models sampled from the model distributions Pg. During learning, we optimize the meta-parane parameter S by minimizing the meta-training loss F(S). In contrast, at meta-test time, train evaluate the trained model on a held-out real dataset. This meta-test performance reflects the quality of the distilled data.
model from scratch S and
2.2 Dataset Distillation using Neural Feature Regression with Pooling (FRePo)
The luter-level problem can be solved using gradient- based methods of the form S S-a VsF(S), where is the learning rate for the distilled data and V.s F(S) is the meta-gradient 27 For particular model 0, the meta-gradient can be expressed as v (Alg (0, S), T). Computing this meta-gradient requires differentiating through inner optimization. If Alg is an iterative algorithm like gradient descent, then backpropagating through the unrolled computation graph [4] can be solution. However, this type of unrolled optimization introduces significant computation and memory overhead, as the whole training trajectory needs to be stored in memory (Figure ((b)).
Traditionally, these issues are alleviated with truncated backpropagation through time (TBPTT) 30 Instead of backpropagating through an entire unrolled sequence, TBPTT performs backprop- agation for each subsequence separately. It is efficient because its time and memory complexity scale linearly with respect to the truncation steps. However, truncation may yield highly biased gradients that severely impact training. To mitigate this truncation bias [4], we consider training only the top layer of a network to convergence. The key insight is that the data helpful for training the output
layer can also help train the whole network. Thus, we decompose the neural network into a feature
3

---

Algorithm 1 Dataset Distillation using Neural Feature Regression with Pooling (FRePo)
Require: T: a labeled dataset; o: the learning rate for the distilled data
Initialization: Initialize labeled distilled dataset S (X Y.). Initialization: Initialize model pool M with m models { randomly initialized from Po.
1: while not converged do
2: P Sample a model uniformly from the model pool: 0; ~ M.
3: > Sample a target batch uniformly from the labeled dataset: (X,, Yt) T.
4: P Compute the meta-training loss L using Eq. 2
5: > Update the distilled data S: X, X, a V x. C, and Y,, - Y, a avy
6: b Train the model 0g on the current distilled data S for one step.
7: > Reinitialize the model 0,- Po if 02 has been updated more than K steps.
8: end while
Output: Learned distilled dataset S = (X Yx)
extractor and a linear classifier. We fix the feature extractor at each meta- geotd-gr computation and train the linear classifier to convergence before updating S. After that, we adjust the feature extractor by training the whole network the updated distilled data. We note that similar two-phase procedure has been studied in the context of representation learning 31.
Meta-Gradiens Computation: If we consider the mean square error loss, then the optimal weights for the linear classifier have closed-form solution. Moreover, since the feature dimension is typically larger than the number of distilled data, we can use kernel ridge regression (KRR) with conjugate
kernel (25 rather than solving the weights explicitly [12]. The resulting meta-training loss (Eq. 2 is
similar to that used in KIP
[2 23 but we use more flexible kernel rather than NTK.
L (Alg (0,8),
マ Y, -Y, (K2 + - 2
(2)
where (X,,, ()) and (X,,, Y2) are the inputs and labels of the real data T and distilled data S respec- tively. The Gram matrix between real inputs and distilled inputs is denoted as K x,,,, E RR Tixis), while the Gram matrix between distilled inputs is denoted as K E [RI À controls the regu- larization strength for KRR. Let denote the neural network feature for given input X and model parameter 0 as fXX, 0) E Nxd where N is the number of input and d is the feature dimension The conjugate kernel is defined by the inner product of the neural network features. Thus, the two Gram matrices are computed as follows:
K f(X,, 0)5(X,,0) K².x. [(X..)
(3)
Now, computing the meta-gradient v. (0, S) TT) is just back paok-propaggttng through the con- jugate kernel and fixed feature extractor, which very efficient and takes even fewer operations than computing the gradient for the network's 's weights. Moreover, we decouple the meta-gradient computation from the model online update. Hence, we can train the online model using any optimizer, and the distilled data will be agnostic to the specific learning algorithm choice. Our proposed method similar 1-step TBPTT in that we compute the meta meta-gradient at each step while performing the online model update. Unlike the conventional 1-step TBPTT, we compute the meta- meta-gradient using KRR output layer to mitigate truncation bias, illustrated in Figure
As discussed in Section I there are various types of overfitting in dataset distillation. Several techniques have been proposed to alleviate such problem, such as random initialization [4) periodic reset [13 5 77. and dynamic bi-level optimization [19]. These techniques share the same underlying principle: the model diversity matters. Thus, we propose to maintain a "model pool" filled with diverse set of parameters obtained from different number of training steps and different random initializations. Unlike the previous methods that periodically training and resetting a single model, FRePo randomly sample a model from the pool at each meta- -ttadiegt computation and update using the current distilled data. However, a model has been updated more than K steps, we reinitialize it with new random seed. From the meta-learning perspective, we maintain diverse
Model Pool:
In practice, we use all the synthetic data and sample minibatch from the real dataset to compute the meta-gradient (Algorithm I).

---

set of meta- tata- to sample from and avoid sampling very similar tasks at each consecutive gradient computation avoid overfitting to a particular setup.
Pool Diversity: We can increase the regularization strength by increasing the diversity of the model pool by setting a larger K, using data augmentation when training the model on the distilled data, or using models with different architectures. To keep our method simple, we use the same architecture for all models in the pool and do not use any data augmentation when training the model on the distilled data. Thus, our model pool only contains models with different initialization, at different optimization stages, and trained different time-step of the distilled data.
3 Related Work
Unrolling in Bi-Level Optimization: One way to compute the meta-gradient is to differentiate
through the unrolled inner optimization [1] However, this approach inherits several difficulties
of the unrolled optimization, such as: 1) large computation and memory cost [4] 2) truncation
bias with short unrolls [17] 3) exploding or vanishing gradients with long unrolls [5] 4) chaotic
and poorly conditioned loss landscapes with long unrolls
[16]. In contrast, our method considers approximating the inner optimization with kernel ridge regression instead of unrolled optimization.
Surrogate Objective: To avoid unrolled optimization, several works turn to surrogate objectives. DC [.. DSA 77, and DCC [8 formulate the dataset distillation as a gradient matching problem between the gradients of neural network weights computed on the real and distilled data. In contrast, DM 8 and CAFE [19 consider the feature distribution alignment between the real and distilled data. Moreover, MTT [20] shows that knowledge from many expert training trajectories can be distilled to dataset by using a training trajectory matching objective. Nevertheless, surrogate objectives may introduce new biases and thus, may not accurately reflect the true objective. For example, gradient matching approaches [S] 7 [8] only focus on short-range behavior and may easily overfit to a biased set of samples that produce dominant gradients [9 20
Closed-form Approximation: An alternative way circumvent unrolled optimization is to find closed-form approximation to the inner optimization. Based on the correspondence between infinitely- wide neural networks and kernel methods, KIP
22 23 approximates the inner optimization with NTK [2T] In this case, the meta- pett-ggrddeet can be computed by back-propagating through the NTK. However, computing NTK for modern neural networks is extremely expensive. Thus, using NTK for dataset distillation requires thousands of GPU hours and sophisticated implementation of the distributed kernel computation framework (33. Similar to ours, Bohdal et al. [2] also decomposes the neural network as a feature extractor and linear classifier. However, they only learn the label and explicitly solve for the optimal classifier weights rather than perform KRR.
Dataset Distillation
4.1 Implementation Details
We compare our method to four state-of-the-ar dataset distillation methods 8 20 23 on various benchmark datasets [26 B7. We train the distilled data using Algorithm I with the same set of hyperparameters for all experiments except stated otherwise. Unlike prior work 7 g 20 we do not apply data augmentation during training. However, we apply the same data augmentation 7 20) during evaluation for fair comparison. We preprocess the data in similar way as in previous works 20 23) but use a wider architecture than previous works 77 8 20 because the KRR component does not behave well when the feature dimension is low, resulting method. Results on the original architecture are included in Appendix ??. We evaluate each distilled data using five random neural networks and report the mean and standard deviation. For the bascline method, we report the best of the reported value in the original paper and our reproducing results.
significant performance drop for our
For the sake of brevity, we provide implementation details about data preprocessing, distilled data initialization, and hyperparameters in Appendix ?? and various ablation studies regarding the model pool, batch size, distilled data initialization, label learning, and model architectures in Appendix ??. More distilled image visualizations can be found in Appendix ??. Our code is available at https:/ // /gi thub. comm//ooggccaao7//FRR
5

---

Table 1: Test accuracies models trained on the distilled data from scratch. 1 denotes performance better than the original reported performance. KRR preformance is shown in bracket. FRePo performs extremely well for one image per class setting on CIFARIOM Tiny ImageNet and CUB-200.
Img/Cls
DSA 7
DM g
KIP 23
MTT 20
FRePo
88.7 0.6
89.9 + 0.8
90.1 + 0.1
91.4 0.9*
93.0 + 0.4 (92.6 0.4)
MNIST
10
97.9 + 0.1*
97.6 + 0.11
97.5 0.0
97.3 + 0.11
98.6 + 0.1 (98.6 0.1)
50
99.2 + 0.1
98.6 0.1
98.3
98.5 0.1
99.2 : 0.0 (99.2 + 0.1)
1
70.6 10.6:000 0.6
71.5 0.5 0.5
73.5 + 0.5
75.1 0.9
75.6 i 0,3 (77.1 + 0.2)
F-MNIST
10
81.8
83.6 0.21
86.8 + 0.1
87.2 + 0.31
86.2 + 0.2 (86.8 : 0.1)
50
88.8 88.8
88.2 1 0.1
88.0 0.1
88.3 1 0.11
89.6 : 0.1 (89.9 0.1)
3.87
1
49,9 1 0.2
46.3 0.8
46.8 1 0.7 (47.9 0.6)
CIFARIO
10
49.2+ 5.82+
62.7 0.3
65.3 + 0.7
65.5 + 0.4 (68.0 0.2)
50
66.8 06.8+0.. 0.4!
63.7 0.5!
68.6 + 0.2
71.6 0.2
71.7 0.2
(74.4
0.1)
1
16.8
1.42
15.7 + 0.2
24.3 + 0.3
28.7 : 0.1 (32.3 0.1)
CIFAR100
10
32.3
29.7+
28.3 + 0.1
40.1 + 0.4
42.5 + 0,2 [44.9
50
42.8 0.4
43.6 0.4
47.7 0.2
44.3 + 0.2
(43.0+
6.6 + 0.2
3.9 0.2
8.8 0.3
15.4 + 0.3 (19.1 0.3)
T-ImageNet
10
12.9 + 0.4
23.2 + 0.2
25.4 0.2 (26.5 0.1)
1
1.3 13
1.6 +: 0.11
2.2 + 0.1
12.4 + 0,2 (13.7 = 0.2)
CUB-200
10
4.5 + 0.3"
4.4 0.2
16.8
0.1 (16.1 0.3)
0101
W0l Clakk
WilCllekk Timw
(a) Test Acc
(b) Test Acc (Log Scale)
(c) Time Per Step
(d) GPU Memory Usage
Figure 3: (a.b) Training efficiency comparison when learning Img/Cls on CIFAR100. (c.d) Time per iteration and peak memory usage we increase the model size. FRePo is significantly more efficient than the previous methods, almost two orders of magnitude faster than the second-best method (i.e.. MTT), with only 1/10 of the GPU memory requirement.
4.2 Standard Benchmarks
Distillation Performance: We first evaluate our method on six standard benchmark datasets. We learn 1, 10, and 50 images per class for datasets with only ten classes, while we learn and 10 images per class for CIFAR100 [4 with 100 classes, Tiny ImageNet [35] with 200 classes, and CUB-200 (37] with 200 fine-grat ine-ggrinedd classes. As shown in Table in
achieve the state-of-the-art performance in most settings despite the hyperparameter may be suboptimal. Our method performs exceptionally well datasets with complex label space when learning few images per class. For example, we improve the CIFAR100 Tiny ImageNet, and CUB-200 in one image per class setting from 24.3%, 8.8%, and 2.2% to 28.7%. 15.4%. and 12.4%. respectively. Figure shows that our distilled images look real and natural though we do not directly optimize for this objective. We observe strong correlation between the accuracy and image quality: the better the image quality, the higher the test accuracy. Our results suggest that a highly condensed dataset does not need to be very different from the real dataset as it may just reflect the most common pattern in dataset. We also report the KRR predictor's 's test accuracy using the feature extractor trained on the distilled data. When the dataset is as simple as MNIST (32), the KRR predictor achieves similar performance as the neural network predictor. In contrast, for more complex datasets, the KRR predictor consistently outperforms the neural network predictor, with the most significant gap being 3.7% for Tiny ImageNet in the one image per class setting.

---

Table 2: Cross- aross-archieectuee transfer performance on CIFARIO with 10 Img/Cls. Despite being trained for a specific architecture, our distilled data transfer well to various architectures unseen during training. Conv is the default evaluation model used for each method. NN, DN, IN, and BN stand for no normalization, default normalization, Instance Normalization, Batch Normalization respectively.
Evaluation Architecture
Train Arch
Conv
Conv-NN
ResNet-DN
ResNet-BN
VGG-BN
AlexNet
DSA
7
Conv-IN
53.2 + 0.8
36.4 1.5
42.1 0.7
34.1 + 1.4
46.3 1 1.3
34.0 2.3
DM
8
Conv-IN
49.2 + 0.8
35.2
1 0.5
36.8
1 1.2
35.5
+ 1.3
11.2 + 1.8
34.9 1 1.1
MTT
20
Conv-IN
61.4 0.9
41.6 1.3
19.2
+ 1.1
42.9
+ 1.5
16.6 2.0
31.2
+ 2.6
KIP
23
Conv-NTK
62.7 0.3
58.2
1 0.4
49.0
+ 1.2
45.8
+ 1.4
30.1 1.5
57.2
2 0.4
FRePo
Conv-Bi BN
65.5 + 0.4
65.5
- 0.4
58.1
0.6
57.7
0.7
59.4 + 0.7
61.9
0.7
FRePo
(b) MTT 20
(c) DSA
(d) Label Visualization
Figure 4: (a,b,c) Distilled I img/cls from CIFARIO0 using FRePo, MTT, and DSA. High quality images also produce high test accuracy. (d) Three categories learned labels. (Top) High confidence, large margin; (Middle) High confidence, small margin; (Bottom) Low confidence, small margin.
Label Learning: similar trend can also be observed for label learning. When the dataset is simple and has only a few classes, label learning may not be necessary. However, it becomes crucial for complex datasets with many labels, such as CIFARI00 and Tiny-ImageNet (See more details in Appendix ??). Similar to the teacher label in the knowledge distillation [I] we observe that the distilled label also encodes the class similarity. We identify three typical cases in Figure [d] The first group consists of highly confident labels with much higher value for one class than other classes (large margin), such as sunflower, bicycle, and chair. In contrast, the distilled labels in the second group are confident but may get confused with some closely-related classes (small margin). For instance, the learned label for "girl" has almost equally high values for the girl, woman, man, boy, and baby, suggesting that these classes very similar and may be difficult for the model to distinguish them apart. The last group contains distilled labels with low values for all classes, such as bear, beaver, and squirrel. It is often hard for humans to recognize the distilled images in such group, suggesting that they may be the challenging classes in a dataset.
Training Cost Analysis: Figure 3a shows that our method is significantly more time-efficient
than the previous methods. When learning one image per class on CIFAR100, FRePo reaches a similar test accuracy (23,4%) to the second-best method (24.0%) ) in 38 seconds, compared to 3805 seconds for MTT, which is roughly two orders of magnitude faster. Moreover, FRePo achieves 92% of its final test accuracy (26.4% out of 28.7%) in only 385 seconds. As shown in Figure 33 our algorithm takes much less time to perform one gradient step on the distilled data. Thus, we can perform more gradient steps in fixed time. Furthermore, Figure 3d suggests that our algorithm has much less GPU memory requirement. Therefore, we can potentially use a much larger and more complex model to take advantage of the advancement in neural network architecture.
Cross-Architecture Generalization: One desired property of our distilled data is that it generalizes well to architecture it has not seen during the training. Similar to previous works [,200, we evaluate the distilled data from CIFAR on a wide range of architectures which has not seen during training, including AlexNet 38 VGG R9) and ResNet [40] Table shows that our method outperforms previous methods on all unseen architectures. Instance Normalization (IN) 41, as the vital ingredient in several methods (DSA, DM, MTT), seems to hurt the arohttrccie transfer. The performance
7

---

Table 3: Distillation performance on higher resolution (128x128) dataset (i.e. ImageNette, Image- Woof) and medium resolution (64x64) dataset with a complex label space (i.e. ImageNet- FRePo scales to high-resolution images and learns the discriminate feature of complex datasets.
ImageNette (128x128)
Image Woof (128x128)
ImageNet (64x64)
Img/Cls
10
10
2
Random Subset
23.5 +: 4.8
17.7 + 2.4
14.2 + 0.9
27.0 = 1.9
1.1 + 0.1
1.4 = 0.1
MTT 20
17.7 + 0.9
63.0 + 1.3
28.6 + 0.8
35.8 = 1.8
FRePo
48,1 i 0.7
66.5 + 0.8
29.7 + 0.6 42.2 = 0.9 7,5 i
0.3
9.7 = 0.2
degrades lot when no normalization (NN) is applied (Conv-NN, AlexNet) or using different normalization, like Batch Normalization (BN) It suggests that the distilled data generated by those methods encode the inductive bias of particular training architecture. In contrast, our distilled data generalize well to various architectures, including those without normalization (Conv-NN, AlexNet). Note that Figure I 4 also indicate that our distilled data encode less architectural bias as the distilled images look natural and authentic. A simple idea to further alleviate the overfitting of a particular architecture not be stable as the meta- peetaggradien computed by different architectures can be very different.
include more architectures the model pool. However, the training may
4.3 ImageNet
High Resolution ImageNet Subset To understand how well our method performs high-resolution images, we evaluate it on ImageNette and ImageWoof datasets [36] with a resolution of 128x128. We learn and 10 images per class on both datasets and report the performance in Table 3 and visualize some distilled images in Figure L As shown in Table 3 we outperform MTT on all settings and achieve much better performance when we distill ten images per class on a more difficult dataset Image Woof. It suggests that our distilled data is better at capturing the discriminative features for each class. Figure shows that our distilled images look real and capture the distinguishable feature of different classes. For the easy dataset (i.e.. ImageNette), all images have clear different structures, while for Image Woof, the texture of each dog seems to be crucial.
Resized ImageNet-IK: We also evaluate our method on a resized version of ILSVRC2012 26 with a resolution of 64x64 to see how it performs on complex label space. Surprisingly, we can achieve 7.5% and 9.7% Top1 accuracy using only Ik and 2k training examples, compared to 1.1 1% and 1.4% using an equally-sized real subset.
5
Application
5.1 Continual Learning
Continual learning (CL) 43 aims to address the catastrophic forgetting problem 43 when model learns sequentially from a stream of tasks. commonly used strategy to recall past knowledge is based on replay buffer, which stores representative samples from previous tasks [46 49
Since
sample selection is an important component of constructing an effective buffer [48 51] we believe distilled data can be a key ingredient for a continual learning algorithm due to its highly condensed
nature. Several works 668, [2) have successfully applied the dataset distillation to the continual learning scenario. Our work shows that we can achieve much better results by using a better dataset distillation technique.
We follow Zhao and Bilen 8 that sets up the bascline based on GDumb [49 which greedily stores class-balanced training examples in memory and train model from scratch on the latest memory only. In that case, the continual learning performance only depends on the quality of the replay buffer. We perform and 10 step class-incremental learning [53] on CIFAR with an increasing buffer size of 20 images per class. Specifically. we distill 400 and 200 images at each step and put them into the replay buffer. We follow the same class split as Zhao and Bilen S and compare our method to random [49], herding [54, 55 DSA 77. and DM g. We use the default data preprocessing and default model for each method in this experiment as we find gives the best performance for each method. We use the test accuracy on all observed classes as the performance measure [8 48).
8

---

Table AUC of five attackers on models trained on the real and distilled MNIST data. The model trained on the real data is vulnerable to MIAs, while the model trained on the distilled data is robust to MIAs. Training on distilled data allows privacy preservation while retaining model performance.
Attack AUC
Test Acc (%)
Threshold
LR
MLP
RF
KNN
Real
99.2 + 0.1
0.99 + 0.01
0.99 + 0.00
1.00 + 0.00
1.00 + 0.00
0.97 = 0.00
Subset
96.8 + 0.2
0.52 + 0.00
0.50 = 0.01
0.53 + 0.01
0.55 0.00
0.54 = 0.00
DSA
98.5 + 0.1
0.50 0.00
0.51 0.00
0.54 0.00
0.54 0.01
0.54 0.01
DM
98.3 +
0.50 i 0.00
0.51
0.54
0.53 = 0.01
FRePo
98.5 + 0.1
0.52 i 0.00
0.51 0.00
0.53 1 0.01
0.52 - 0.01
0.51 0.01
30
135
252
500
16
124
513
25
14
Naxberel Casss
Namberel Cl
Traneg Sa0e
Thali Steg
(a) 5-step CL
(b) 10-step CL
(c) MIA: ACC
(d) MTA: AUC
Figure 5: (a.b) Multi-class accuracies across all classes observed up to certain time point. We perform significantly better than other methods in both and 10 step maas-inncremenal continual learning, (c,d) Test accuracy and attack AUC as we increase the number of training steps. AUC keeps increasing when training a model on the real data for more steps. In contrast, AUC keeps low when training on distilled data.
Figure 5 shows that our method performs significantly better than all previous methods. The final test accuracy for all classes for method (FRePo) and the second-best method (DM) are 41.6%, 33.9% in 5-step learning, and 38.0%. 34.0% in 10-step learning. However, we notice that for FRePo, distilling 2000 images in continual learning setup achieves similar test accuracy (41.6%) as distilling only 1000 images from the whole dataset (41.3% from Table In addition, performance drops as perform more steps. It suggests that FRePo considers all available classes to derive the most condensed dataset. Splitting the data into multiple groups and performing independent distillation may generate redundant information or fail capture the distinguishable features.
5.2 Membership Inference Defense
Membership inference attacks (MIA) aim to infer whether given data point has been used to train the model or not (56158). Ideally, we want a model to learn from the data but not memorize it to preserve privacy. However, deep neural networks are well-known for their ability to memorize all the training examples, even on large and randomly labeled datasets [59]. Several methods have been proposed to defend against such attacks by either modifying the training procedure [60] or changing the inference workflow [61]. This section shows that the distilled data contain little information regarding sample presence in the original dataset. Thus, instead of training the original datasets, training on distilled data allows privacy preservation while retaining model performance.
We consider three distilled data generated by DSA [7]. DM g and FRePo. We perform five popular "black box" MIA provided by Tensorflow Privacy (62) on models trained on the real data or the data distilled from it. The attack methods include threshold attack and four model-based attacks using logistic regression (LR), multi-layer perceptron (MLP), random forest (RF) and K-nearest neighbor (KNN). The inputs to those attack methods are ground-truth labels, model predictions, and losses. To measure the privacy vulnerability of the trained model, we compute the area under the ROC curve (AUC) of an attack classifier. Following prior work, [36,, 63]. we keep balanced set of training examples (member) and test examples (non-member) with 10K each to maximize the uncertainty of MIA. Thus, the random guessing strategy results in 50% MIA accuracy. We conduct experiments on MNIST and FashionMNIST with distillation size of 500. For space reasons, we provide more implementation details and results in appendix.

---

As shown in Table all models trained on the distilled data preserve privacy as their attack AUCs are closed to random guessing. However, we observe small drop test accuracy compared to the model trained on the full dataset, which is expected as we only distill 500 examples instead of 10,000 examples. Compared to the model trained on an equally sized subset of the original data, the model trained on distilled data results in much better test performance. Figure 5c 5d demonstrate the trade-off between test accuracy and attack effectiveness as measured by ROC AUC. It shows that early stopping can be an effective technique to preserve privacy. However, we will still be under high MIA risk if we perform early stopping by monitoring the validation loss. In contrast, training a model on the distilled data does not have this problem as the attack AUCs keep at very low level regardless of training steps.
6 Conclusion
We propose neural Feature Regression with Pooling (FRePo) to overcome two challenges dataset distillation: meta-gradient computation and various types of overfitting in dataset distillation. We obtain state-of-the-an performance on various datasets with 100x reduction in training time and 10x reduction in GPU memory requirement. The distilled data generated by FRePo looks real and natural and generalizes well to a wide range of architectures. Furthermore, we demonstrate two applications that take advantage of the high-quality distilled data, namely, continual learning and membership inference defense.
Broader Impact "Synthetic data", in the broader sense of artificial data created by generative models, can help researchers understand how an otherwise opaque learning machine "sees" the world. There have been concerns regarding the risk of fake data. This paper explores a new research direction in generating synthetic data only for downstream classification tasks. We believe this work can provide additional interpretability and potentially address the common concerns in machine learning regarding training data privacy.
Acknowledgments and Disclosure of Funding
We would like thank Harris Chan, Andrew Jung, Michael Zhang, Philip Fradkin, Denny Wu, Chong Shao, Leo Lee, Alice Gao, Keiran Paster, and Lazar Atanackovic for their valuable feedback. Jimmy Ba was supported by NSERC Grant [2020-06904), CIFAR AI Chairs program, Google Research Scholar Program and Amazon Research Award. This project was supported by LG Electronics Canada. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence.
References
[1] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015, URL htttt://aaxxiiii org/ abs////s..//////////////////aa/a 02531
[2] Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ram- abhadran. Efficient knowledge distillation from an ensemble of teachers. In Francisco Lacerda, editor, Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pages 3697-3701. ISCA, 2017. URL html
httt:://ww..cet
[3] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httnntts//opperrvvieeee wtt.://rwww..eeeffoorrrrrissosaa
[4] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. 1811. 10959
CoRR, abs/181 1. 100959, 2018. htll:ttt////xxxi.orrg
[5] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient match- ing. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event,
10

---

Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpps://pparrriev..tt///// id=mAAKLLYLSsI
[6] Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. CoRR, abs/2103.15851,
2021. URL https: httpp:///axxvv.ooo wwwg//aa/////// 15851
[7] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12674-12685. PMLR, 2021. URL HRRLhttp::// /tRL/attpp//roceediing proceedinges
mlr. wrw.prrsss/eeee v/////11/////// zhw.2la......... 1139///ao11/aaala html
[8] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching, CoRR,
abs/2110.0411 2021. URL https://arxxx wwrreriiiirrrroooooooo httggabb//110... 04181
[9] Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. CoRR, abs/2104.02857, 2021. https:t:s////xiiiv.o. org. hbtgg//bb/2104/ 02857
[10] Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. CoRR, abs/2008.04489, 2020. BtRLhhttps:// /ttpxx.xivvv////////////////.a org/ ab/////////// 2008. 04489.
[11] Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient- based hyperparameter optimization through reversible learning. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, Lille, France, 6-1 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2113- 2122. JoLR.org, 2015. httLhhtp:////ooooee
mlr. www.ppressvvvve /v37 mlI.ppress//37/mallaau mwt.///rnnaaaainnn5mmmm html
[12] Ondrej Bohdal, Yongxin Yang, and Timothy M. Hospedales. Flexible dataset distillatio Learn labels instead of images. CoRR, abs/2006.08572, 2020. URL httpss///arzvv..rgggg
[13] Ilia Sucholutsky and Matthias Schonlau. Improving dataset distillation. CoRR, abs/1910.02551,
2019. http:://t///: org/ abs 02551
[14] Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in un- rolled computation graphs with persistent evolution strategies. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pages 10553-10563. PMLR, 2021. URL URRL:ttt://proccedinng mlr www.mmers..mm......m..m.. press/v139/ vicol21a.1 html
[15] Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pages 1310-1318. JMLR.org, 2013. URL TtLp:ttp://// httpp://proceeednngs mlr .press/ v28/pascanu13.. html
[16] Luke Metz, Niru Maheswaranathan. Jeremy Nixon, C. Daniel Freeman, and Jascha Sohl- Dickstein. Understanding and correcting pathologies in the training of learned optimizens. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4556-4565. PMLR, 2019. URL http::/prrcceedingg .ml.eeddiiinmm....m...mmmm..m..d www.g//ssm.rrrr..s..emmmmmmmeeee pwwps/ressssee. /v7 wttps//////mmmttt//.m.a www.wet.......m html
[17] Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger B. Grosse. Understanding short-horizon bias in stochastic meta-optimization. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018.
[18] Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. CoRR, abs/2202.02916 2022. URI. https:tpps.. ///axivv org/abs/2202.0229

---

[19] Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang. Hakan Bilen, Xinchao Wang, and Yang You. CAFE: learning to condense dataset by aligning features. CoRR, abs/2203.01533 2022. doi: 1044850WarXxx22200.133311 URL. https:
//doi.org/1n
48550/arXiv 10ooo.orr//0.445//////.. 2203. .01311
[20] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Yun Zhu. Dataset distillation by matching training trajectories. CoRR, abs/2203.11932. 2022. doi:
10.48550/axx..2223..111
https://o// org/10. 41111.4855//rra/ 20t:500/aXiv..o htt.//aaivvv...0 2203. 11932
[21] Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pages 8570-8581, 2019. URL https: htttp:://proceedinggg neurips. wwtrriss.ccpaappee httpp/ppppp/////// 0d1a9651497 (tt.://880B81131114488886666 htw.cc444552oo--bsrrar....r Aww.raat.ttttttttaaaa
[22] Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge- regression. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreviee net/ forum?id PrrQrkoQR
[23]
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Mare' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 5186-5198, 2021. URL httttpp://pprceedingge www.nnng...neuirie www.wes..c//ppeeeeeeeeee pttppp/c//ppppe/22//aaaaal
79922229912222655544301eeee2.2.2 Abstraattt html
[24] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 1540- 1552. PMLR, 2020. URL htt::ttttp/pceeediiiig
mlr pttpsse/110/oorriee000000 html
[25] Radford M. Neal. Bayesian learning for neural networks. 1995.
[26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (JCV), 115(3):2111255) 252, 2015. doi: 10. (000//s11233-015-0866666
[27] Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Mcta-learning with implicit gradients. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc Buc, Emily B. Fox, and Roman Garnett, editors, Advances Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 113-124, 2019. URL https: pettps://pteee http://procee....e//ee/// neur courips.com wwtpiisss.ccppppeee htc/pappe/2019/haaa// 07765300116622442234434222eel Aawpeeeaaddd--stbttttt..e.eee
[28] P.J. Werbos. Backpropagation through time: what it does and how do it. Proceedings of the IEEE, 155100::550-1560, 1990. doi: 10.110955.5837.
[29] Ilya Sutskever. Training recurrent neural networks. University of Toronto Toronto, ON, Canada, 2013.
[30] Corentin Tallec and Yann Ollivier. Unbiasing truncated backpropagation through time. CoRR, abs/1705.08209, 2017. URL
[31] Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How one gradient step improves the representation. arXiv preprint arXiv:2205, 00.000...4,,00000
12

---

[32] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-2) 23224, 1998. doi: 10.1109/5.72671.
URI http:s//:////ii..... 10.0/00010001111155555 7767911
[33] Han Xiao, Kashif Rasul, and Roland Vollgraf. fushion-mnist novel image dataset for benchmarking machine learning algorithms. CoRR, hes/1708.07747, 2017. URL http://
arxiv.orge abs/ 1708. 07747
[34] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[35] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
[36] Jeremy Howard. A smaller subset of 10 casily classified classes from imagenet, and a little more french.
[37] Technical report.
[38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106-1114, 2012. URL https: / neurips cc/ htppp/apppp///a /tppppppppp//////// 2t////2122ahaaa// htw.://beee22aaccbbbbbbttaa.... Abstract. html
[39] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL /ttt:p://arxv....// /twp/xxiiivv././///aa///aaaaa org/ abs/ 1109/4499... 1556
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 37-30, 2016, pages 770-778, IEEE Computer Society, 2016. doi: 10.1 10.110/....2201...00 URL htttt::////oooo org/10. 2109/CCPR.2016..1..
[41] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607.08022,, 2016.
URL httphtttt///xxiirrrr/t/ abs/
1607.08022
[42] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 1 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448-456 JMLR.org, mlr press/v37/ioffel1 htmll
2015. URL httt::://proceeeeee
[43] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Over- coming catastrophic forgetting in neural networks. CoRR, abs/1612.00796. 2016. URL 16t/g//0ab//1112077666
[44] Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3:128-135, 05 1999. doi:
[45] Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 7 (2):123-14,, 1995. doi: 10. 10.1080005540009555000339 09540099550033331
https://s://dot www.ii....11100.1. 1080/
[46] Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking ex- perience replay: a bag tricks for continual learning. In 25th International Conference on Patlern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021, pages 2180-2187. IEFE, 2020. doi: 10. 10.1109/1CP488806.00 20219996622119111144
URL httpss
//doi orooioorg/ii 10. 1109 Iwt.//00RR....... 2021. 9412614
13

---

[47] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class- incremental learning. In IEEE Conference on Computer Vision and Patter Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 2544-2553. Computer Vision Foundation /
IEFE, 2021. URL Adaptive_ www.eeggtiinnnnneeooooooooo paper.htmn
thecvf com/ http///cnteee///PP///////aaa//// clr__classsse
[48] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. CoRR, abs/1611.00772 2016. URL http://arxiv org/abs/1611.0772
[49] Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II, volume 12347 of Lecture Notes in URL
Computer Science, pages 524-540. Springer, 2020. doi: 10. https://doi. org 10. --000990----000-----------------
[50] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence A'Accéé-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1188861-1122... 2019, / /pwce//dnngocce.e..cco...oo neur ips. wtppp.cc.ppp..cc....ppp...cop..e 26f2686077aa44666666661111166662 Abstract.bbbbtttt.......
[51] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett,, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11849-11860, 2019. URL https::////// htttts:://proceednngg nww.ipu..i.... wttpppp.cccppp////ppppppppppccpt 1 html
2019/hash/ htaaaa/sSS522aaeeeee333333333333 3
[52] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle. WA, USA, June 13-19, 2020, pages 12242-
12251. Computer Vision Foundation IEEE, 2020. doi: 10. /tC.C///RR22000220.0011111
Maciiuu__Mmmiiccc._em URL https: httpp::/ /tttp:://openaccess: thecvf.... com/ content wwwtonteettccer..__t 2020/ntmm/iii Training Burigg_Mullli Mwwtiuuliiii-i Class Incremental LaarllLLLaarnnnnnn
Aat.aaaaaa.aaaa.aaaaaaaaaaaaaaaa Frithout_Forgeetinn..0m CVPR_2020,
paper .tml
[53] Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. CoRR, abs/1904.07734, 2019. URL http:://aa/: hbs///bb///////////////
[54] Francisco M. Castro, Manuel J. Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Vittorio Ferrari, Martial Hebert, Cristian Smin- chisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XII, volume 11216 of Lecture Notes in Computer Science, pages 241 257. Springer, 2018 doi: 10. URL htt.https::///// htt.t:ttt/tp........ org/ 10. 1007 000000//7770 /0000/9978-3333 000-088-----0000----------8
[55] Yutian Chen, Max Welling, and Alexander J. Smola. Super-samples from kernel herding. In Peter Grünwald and Peter Spirtes, editors, UAI 2010, Proceedings of the Twenty-Sine Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, July 8-11, 2010, pages 109-116. AUAI Press, 2010. URL httttttss//sssiittttt org uai haala/iisspalrrrrreeeeeataaa
jsp?mmnu- wmm.:mmmmuuu....mm. 2at.am//aaarrrrcl.eeeeeeeeeeeeer 1d=26
[56] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pages 3-18. IEEE Computer Society, 2017. doi: 10.1109/SP.2017..0 URL https: //tp::::://o...o/ org/ 10. 1109/SP 2017. 41
14

---

[57] Yunhui Long, Vincent Bindschaedlerr Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A. Gunter, and Kai Chen. Understanding membership inferences on well well-generalized learning models. CoRR, abs/1 hes/180200489, 2018.
wrg. abs/ 18020 04889
[58] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019. San Diego. California. USA, February 24-27, 2019. The Internet Society, 2019.
[59] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Reprexentutions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. wttet//oormmmiddyyysseee
OpenReview.net, 2017. URL https::/ attpss://openreviews
[60] Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In David Lie. Mohammad Mannan, Michael Backes, and XiaoFeng Wang, editors, Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronto, ON, Canada, October 15-19, 2018, pages 634 ACM, 2018. doi: 10.1145/324339442243355 URL, https:///oi...... org 32438555
10/////1111 10. 1145/
[61] Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: Defending against black-box membership inference attacks via adversarial examples. In Lorenzo Cavallaro. Johannes Kinder, XiaoFeng Wang, and Jonathan Katz, editors. Proceedings the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS 2019, London, UK, November / 11-1,, 2019, pages 259-274. ACM, 2019. doi: 10. 10.11455/3195353333221. URL BttLpttss://do..oom org 10. 1111/1111//133 3363201
[62] lensorflow/prrvacy:: library for training machine learning models with privacy for training data, 2022. URL https: wwt.coooo/ooo. wtt.:/onoo//o//riaeeeeeoeeee
[63] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, CSF 2018, Oxford, United Kingdom, July 9-12, 2018, pages 268-282. IEEE Computer Society, 2018. doi: 10. 10.1109/CFF2011 201.0000...... URL. https: //doi. org 10. 1109/
CSF 2012.00027
15


Reviews:
[{"review_id": "p6q0_pnjIK7", "reviewer": "Reviewer_aP6S", "summary": "This paper proposes a new method of dataset distillation  called “Neural Feature Regression with Pooling.” This work avoids using a surrogate objective  like previous works (DSAm DM, MTT) while also circumventing the computational constraints of other methods that optimize the true objective (DD, KIP). Like other methods, FRePo trains teacher networks from which to obtain “meta-gradients.” Unlike other methods, FRePo obtains the meta gradient by finding the closed form solution of the Kernel Ridge Regression problem posed by the final layer (linear classifier) of the teacher network. This method shows state-of-the-art results, outperforming previous methods in nearly every setting. Two other applications in continual learning and membership inference defense are also explored.\n", "questions": "While my questions should be easy to answer, I think they are quite critical to the validity of this work. I will happily adjust my rating if they are addressed.\n\n1. How does FRePo perform on the original architecture? (I believe you call it DC-Conv in the appendix). Even if these results are significantly lower, they should be included. They can even be left to the appendix as long as it is made VERY clear in the body that a new, wider architecture is being used and the reasons for this change are properly explained. (You mentioned in the appendix that the reason is to ensure the KRR behaves well. This explanation makes sense, but it is still a limitation of this work.)\n\n2. Are the re-evaluations in Table 1 done using the \\emph{exact} same architecture used to get the FRePo numbers? (i.e., is the same normalization type being used?) If not, how do the other algorithms perform using BatchNorm.\n\n3. Regardless of the answer to the previous question, how does FRePo perform using InstanceNorm?\n\n4. How is FRePo’s cross-architecture performance when trained on Conv-IN like all the other methods in Table 2 (aside from KIP)?\n\n5. If FRePo is faster and uses less memory than MTT, DSA, and DM, why were results for CIFAR100 and T-ImageNet at 50 img/cls not included here when they were included in previous works?\n\n-----\n\nEdit: All my questions and concerns have been addressed.", "limitations": "The limitations on the size of the teacher model seem unique to this method among dataset distillation works and should be clearly addressed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is very well written with very few grammatical errors (e.g., subject-verb agreement). All figures and tables are formatted well and clearly communicate the authors’ ideas.\n\nAs for the method itself, FRePo clearly outperforms the other methods in the explored settings. Another dataset (ImageNet1k resized) is also introduced and will likely be a new evaluation metric for future dataset distillation works.\n\nExtensive visualizations are also included in the appendix.\n\n--------\n\nWeaknesses\n\nI was disappointed to see that the fact that a new model was used (with respect to previous dataset distillation works) was not mentioned until deep into the appendix. While advocating for the adoption of a new backbone model is fine, this should be made very clear in the body of the paper. I understand that the authors re-evaluated the previous methods using this new architecture in Table 1, but the fact that a new architecture was used was not clear at all from the body alone. The authors should have also included results from their method on the original architecture. If this yields poor results, then this is a significant limitation of the method and should be addressed as such (but by no means detracts from the merit of this new contribution).\n\nFurthermore, even given the appendix, it still remains unclear if these re-evaluations of previous methods use an architecture identical to the new one used by FRePo. The methods should have been re-evaluated with batch-norm, and results for FRePo using instance norm should have also been included. It is unclear how much of the improvement over previous state of the art is due to the algorithm, normalization type, or number of channels.\n\nSimilarly, Table 2 should also \\textit{at least} include an additional row for FRePo trained using Conv-IN.\n\nAlso, if FRePo is faster and uses less memory than MTT, DSA, and DM, why were results for CIFAR100 and T-ImageNet not included here when they were included in previous works?\n\nOne other tiny thing: your labels for “Samoyed” and “Golden Retriever” seem to be swapped in all of your ImageWoof figures :P", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ok8TDhqtIik", "reviewer": "Reviewer_p7tg", "summary": "This paper proposes to use Kernel Ridge Regression on the last layer of a slowly updated feature extractor to perform Dataset Distillation. There are several key contributions in this paper, including demonstrating KRR can work well in DD, using model pools, fast training speed and state-of-the-art results on DD benchmarks. The experiment section shows results from both DD and other applications including continual learning and membership inference attacks.", "questions": "The paper nicely contributes to the DD task. I have two questions:\n1. How does the algorithm compute the kernel matrix(see above)\n2. Have the authors considered using implicit differentiation[24] instead, which also has low memory requirements? What would be the advantage of using KRR?", "limitations": "This work has potential benefits in improving privacy and understanding how neural networks perceive a task. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper presents a practical method that performs well both in terms of performance and training time. \n\nStrengths:\n+ The paper contributed a new algorithm that uses KRR on the last layer and slowly updates the feature backbone to perform dataset distillation.\n+ The paper well presents the algorithm and the related techniques and tricks for training\n+ More stable performance on cross-architecture transfer is shown in table 2\n+ The authors show that the algorithm can be used for higher-resolution training such as 128x128 images and 64x64 ImageNet dataset.\n\nWeakness\n- I'm curious how the authors handled the kernel computation in equation 2. If not mistaken, it seems the algorithm needs to compute the kernel for the full real dataset? Do you use minibatches instead?\n\nMinor (no need to compare)\nThere are two recent new works on the same task, it might be worth adding them to the discussion/related work part.\n1. Dataset Condensation via Efficient Synthetic-Data Parameterization\n2. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jFMVFV-q7lA", "reviewer": "Reviewer_nCf8", "summary": "This paper proposes a new data distillation approach based on neural feature regression that is similar to a truncated backgprop through time using a pool of models. The approach sets a new state-of-the-art results both in terms of accuracy and training efficiency.  Ample experiments showcase the advantages of the proposed approach.", "questions": "Please see the Strengths and Weaknesses section", "limitations": "It would be helpful if the authors would have specified in the form where each content is addressed instead of simply stating Yes/No.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- State-of-the-art results while being significantly more efficient\n- Well written and structured with in depth ablation studies\n\nWeaknesses & Q:\n- L173, why no augmentation is applied during training? Shouldn't this prevent overfitting too?\n- How does this approach fair in comparison with state of the art when combined with few shot learning approaches (that were targeted for real data)? \n- Since the paper mentions the computational advantages, perhaps this information could be added in the tables too. Thought figure 3 addresses this in part.\n- Given the difficulty on modeling similar classes, it would be interesting to see how such method fair on fine grained classification (e.g. on CUB-200)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bTYbyNBujPl", "reviewer": "Reviewer_JSgu", "summary": "This paper presents an efficient method for dataset distillation -- the goal is to extract/synthesize a small set of images that represent a dataset, so that deep learning models trained on the small set can achieve high (as high as possible) classification accuracy. The core contribution of this ppaer is to simplify and accelerate the conventional two-stage optimization procedure with TBPTT, where the gradients need to be propagated through a long chain. Instead, a kernel is used for computing the loss function, minimizing which leads to a single-step optimization approach which is highly efficient. The method is evaluated over a series of popular datasets including ImageNet which was never studied before.", "questions": "1. What is the major difficulty that prevents the proposed method from being applied to large images, e.g. 256x256 images? If one uses a small resolution setting to extract synthesized images (e.g. 64x64) and using techniques like GAN to up-sample it larger (e.g. 128x128), does it bring good performance on large datasets (e.g. ImageNet)?\n\n2. What is the time and memory consumption for the ImageNet experiments, in particular, how long did the program run, and how many GPUs  (or other device) did it use? In addition, if one hopes to extract more training instances from ImageNet (e.g. 5, 10, 20, ... images per class), what is the computational cost w.r.t. the number of training instances?\n\n3. If one has extracted, say, 10 images per class, and hopes to add 5 images per class for better performance, does the algorithm need to start from the very beginning (to extract 15 images per class), or can it start from the current point? In other words, what is the incremental performance of the proposed approach?\n\n4. Are there any difficulties in applying the method to other vision problems, such as detection and segmentation? I do not mean to add experiments (though it would be great to do so), I just want to see the major difficulties of generalizing the method.", "limitations": "Please see the above comments.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The studied problem is important and interesting.\n\n2. The proposed approach is sound and effective. The idea is reasonable and the experimental results are good.\n\n3. The paper is well presented.\n\nWeaknesses\n\nI did not see significant weaknesses of this paper. Although the solution is still preliminary (in the context of the large CV community) and difficult to scale up to larger datasets, the proposed method is indeed a very good progress on this direction. There are some limitations that I hope the authors to address or explain, which I will write down in the following part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept