Self-Supervised Dataset Distillation for Transfer Learning
Authors: Dong Bok Lee
Seanie Lee
Joonho Ko
Kenji Kawaguchi
Juho Lee
Sung Ju Hwang
Venue: ICLR 2024 poster (2024)
Keywords: Dataset Distillation
Self-supervised Learning
Kernel Ridge Regression
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose self-supervised dataset distillation for facilitating various transfer learning scenarios.

Abstract: Dataset distillation aims to optimize a small set so that a model trained on the set achieves performance similar to that of a model trained on the full dataset. While many supervised methods have achieved remarkable success in distilling a large dataset into a small set of representative samples, however, 
they are not designed to produce a distilled dataset that can be effectively used to facilitate self-supervised pre-training. To this end, we propose a novel problem of distilling an unlabeled dataset into a set of small synthetic samples for efficient self-supervised learning (SSL). We first prove that a gradient of synthetic samples with respect to a SSL objective in naive bilevel optimization is \textit{biased} due to the randomness originating from data augmentations or masking for inner optimization. To address this issue, we propose to minimize the mean squared error (MSE) between a model's representations of the synthetic examples and their corresponding learnable target feature representations for the inner objective, which does not introduce any randomness. Our primary motivation is that the model obtained by the proposed inner optimization can mimic the \textit{self-supervised target model}. To achieve this, we also introduce the MSE between representations of the inner model and the self-supervised target model on the original full dataset for outer optimization. We empirically validate the effectiveness of our method on transfer learning. Our code is available at 
https://github.com/db-Lee/selfsup_dd

Published as a conference paper at ICLR 2024
SELF-SUPERVISED DATASET DISTILLATION FOR TRANSFER LEARNING
Dong Bok Lee', Lee'; Seanie "ee',,Jonnin Ko', Kenji Kawaguchi", Juho Lee', Sung Ju Hwang KAIST", National University of Singapore? {markhi, 1snfami Ly02, joonho. ko, juholee, sjawang82)ekaist. .cc.sttt.c...... jenjiecom comp........... nus. edu.sq sg
ABSTRACT
Dataset distillation aims optimize small set so that model trained on the set achieves performance similar to that of model trained on the full dataset. While many supervised methods have achieved remarkable success in distilling a large dataset into a small set of representative samples, however, they are not de- signed produce a distilled dataset that can be effectively used to facilitate self- supervised wre-training. To this end, we propose novel problem of distilling an unlabeled dataset into set small synthetic samples for efficient self-supervised learning (SSL). We first prove that a gradient of synthetic samples with respect to a SSL objective in naive bilevel optimization is biased due to the randomness origi- nating from data augmentations masking for inner optimization. To address this issue, we propose to minimize the mean squared error (MSE) between a model's ss representations of the synthetic examples and their corresponding learnable tar- get feature representations for the inner objective, which does not introduce any randomness. Our primary motivation is that the model obtained by the proposed inner optimization can mimic the self-supervised targel model. To achieve this, we also introduce the MSE between representations of the inner model and the self self-supervised target model on the original full dataset for outer optimization. We empirically validate the effectiveness of our method on transfer learning. Our code is available at https: / /ttpt/gtttuu//t com/ db-Lee/ / se selsselsssss Fsup_dd.
INTRODUCTION
As consequence of collecting large-scale datasets and recent advances in parallel data processing, deep models have achieved remarkable success in various machine learning problems. However, some applications such as hyperparameter optimization (Franceschi al., 2017), continual learn- ing [Lopez-Paz & Ranzato, 2017), or neural architecture search (Liu et al., 2019) require repetitive training processes. In such scenarios, it is prohibitively costly use all the examples from the huge dataset, which motivates the need to compress the full dataset into small representative set of exam- ples. Recently, many dataset distillation (or condensation) methods (Wang et al., 2018; Zhao al., 2021; Zhao & Bilen, 2021; Nguyen et al., 2021a;b; Cazenavette et al., 2022; Zhou et al., 2022; Loo al., 2022; Zhao Bilen, 2023) have successfully learned small number of examples on which we can train model to achieve performance comparable to the one trained on the full dataset.
Despite the recent success of dataset distillation methods, they are not designed to produce a distilled dataset that can be effectively transferred to downstream tasks (Figure I-(a)). In other words, may not achieve meaningful performance improvements when pre-training a model on the distilled dataset and fine-tuning it the target dataset. However, condensing general- -urposs datasets into a small set for transfer learning is crucial for some applications. For example, instead of using large pre-trained model, we may need to search hardware-specific neural architecture due to constraints on the device (Lee et al., 2021). To evaluate the performance of an architecture during the search process, we repeatedly pre-train a model with the architecture on large unlabeled dataset and fine-tune it on the target training datast, which is time consuming and expensive. If we distill the pre-training dataset into small dataset at once, we can accelerate the architecture search by pre-training the model on the small set. Another example is target data-fre free knowledge distillation
"Equal contribution

---

Published as a conference paper at ICLR 2024
Transfer Learning
0 FIG 2
Large-scale Unlabeled Dataset
Effcient Pre-training
SVHN
condense
3
4 5
66
8
conde Se
Aircraft
fine bune
target
datasot
Architecture Search
CIFAR10
condense
frag
horse
PloWers
(a) Previous Dataset Distillation
(b) Proposed Self-supervised Dataset Distillation
Figure 1: (a): Previous supervised dataset distillation methods. (b): Our proposed method that distills unla- beled dataset into a small set that can be effectively used for pre-training and transfer to target datasets.
(KD) (Lopes et al., 2017; Raikwar & Mishra, 2022), where we aim to distill a teacher into a smaller student without access to the target training data due data privacy or intellectual property issues. Instead of the target dataset, we can employ a condensed surrogate dataset for KD (Kim et al., 2023).
To obtain a small representative set for efficient pre-training, as illustrated in Figure 1-(b), we pro- pose a supervid dataset distillation framework which distills an unlabeled dataset into a small set on which the pre-training will be done. Specifically, we formulate the unsupervised dataset dis- tillation as a bilevel optimization problem: optimizing small representative set such that model trained on the small set can induce a latent representation space similar to the space of the model trained on the full dataset. Naively, we can replace the objective function of existing bilevel opti- mization methods for supervised dataset condensation with a SSL objective function that involves some randomness, such as data augmentations or masking inputs. However, we have empirically found that propppropgg through data augmentation or masking is unstable. Moreover, we prove that gradient of the self-supervised learning (SSL) loss with randomly sampled data augmentations or masking is a biased estimator of the true gradient, explaining the instability.
Based on this insight, we propose use mean squared error (MSE) for both inner and outer objective functions, which does not introduce any randomness due to the SSL objectives and thus contributes to stable optimization. First, we parameterize a pair of synthetic examples and target representations of the synthetic ones. For inner optimization, we train a model to minimize the MSE between the target representations and model's representations of the synthetic examples. Then, we evaluate the MSE between the original data representation of the model trained with the inner optimization and that of the model pre-trained on the original full dataset with a SSL objective. Since we do not sample any data augmentations or input masks, we can avoid the biased gradient of the SSI. loss. Lastly, similar to Zhou al. (2022), we simplify the inner optimization to reduce computational cost. We decompose the model into feature extractor and linear head, and optimize only the linear head with kernel ridge regression during the inner optimization while freezing the feature extractor. With the linear head and the frozen feature extractor, we compute the meta-gradient of the synthetic examples and target representations with respect to the outer loss, and update them. To this end, we dub our proposed self- euf-suuereeeed dataset distillation method for transfer learning as Kernel Ridge Regression on Self-supervised Targel (KRR-ST).
We empirically show that our proposed KRR-ST significantly outperforms the supervised dataset distillation methods in transfer learning experiments, where we condense a source dataset, which is either CIFAR100 (Krizhevsky al., 2009). TinyImageNet (Le & Yang, 2015), or ImageNet (Deng et al., 2009), into a small set, pre-train models with different architectures on the condensed dataset, and fine-tune all the models on target labeled datasets such as CIFARIO, Aireraft (Maji al., 2013), Stanford Cars (Krause et al., 2013), CUB2011 (Wah et al., 2011), Stanford Dogs (Khosla et al., 201 and Flowers (Nilsback & Zisserman, 2008). Our contributions are as follows:
We propose a new problem of self sepeesuueee dataset distillation for transfer learning, where we distill an unlabeled dataset into small set, pre-train model on it, and fine-tune it on target tasks.
We have observed training instability when utilizing existing SSL objectives in bilevel optimiza- tion for self-supervised dataset distillation. Furthermore, we prove that a gradient of the SSL objectives with data augmentations or masking inputs is a biased estimator of the true gradient.
To address the instability, we propose KRR-ST using MSE without any randomness at an inner loop. For the inner loop, minimize MSE between model representation of synthetic samples and target representations. For an outer loop, we minimize MSE between the original data repre- sentation of the model from inner loop and that of the model pre-trained the original dataset.
2

---

Published as a conference paper at ICLR 2024
We extensively validate our proposed method on numerous target datasets and architectures, and show that ours outperforms supervised dataset distillation methods.
RELATED WORK
Dataset Distillation (or Condensation) To compress large dataset into small set, instead of selecting coresets (Borsos et al., 2020; Mirzasoleiman et al., 2020), dataset condensation optimizes a small number of synthetic samples while preserving information of the original dataset to effectively train high-performin deep learning models. Wang et al. (2018) propose the first dataset distillation (DD) method based on bilevel optimization, where the inner optimization is simply approximated by one gradient-step. Instead of one-step approximation, recent works propose paok-propagation through time for full inner optimization steps (Deng & Russakovsky, 2022), or implicit gradient based on implicit function theorem (Loo et al., 2023). This bilevel formulation, however, incurs expensive computational costs and hinders scaling to large datasets. To overcome this, several pa- pers propose surrogate objective alternativss to bilevel optimization. Specifically, DSA (Zhao al., 2021; Zhao & Bilen, 2021) minimizes the distance between gradients of original and synthetic sam- ples for each training step. MTT (Cazenavette et al., 2022) proposes match the parameter obtained on real data and the parameter optimized on the synthetic data. DM (Zhao & Bilen, 2023) matches the first moments of the feature distributions induced by the original dataset and synthetic dataset. As another line of work, Kernel Inducing Points (Nguyen et al., 2021a;b) propose DD methods based on kernel ridge regression, which simplifies inner optimization by baoo-proggg through Neural Tangent Kernel (NTK) (Lee et al., 2019). Due to the expensive cost of computing NTK for neural networks, FRePo (Zhou et al., 2022) proposes kernel ridge regression on neural features sampled from a pool, and RFAD (Loo et al., 2022) proposes random feature approximation of the neural network Gaussian process. Despite the recent advances in DD methods, none of them have tackled unsupervised//sll-sppprre
DD for transferable learning.
Self-Supervised Learning A vast amount of works have proposed supervipe learning (SSL) methods. A core idea of SSL is that we use large-scale unlabeled data to train a model to learn meaningful representation space that can be effectively transferred to downstream tasks. We intro- duce few representative works. SimCLR (Chen et al., 2020a) is one of the representative con- trastive learning methods. maximizes the similarity between two different augmentations of the same input while minimizing the similarity between two randomly chosen pairs. MOCO (He et al., 2020) constructs a dynamic dictionary using a moving average encoder and queue, and minimizes contrastive loss with the dictionary. On the other hand, several non-contrastive works achieve markable performance. BYOL (Grill et al., 2020) encodes two different views of an input with a student and teacher encoder, respectively, and minimizes the distance between those two represen- tations. Barlow Twins (Zbontar et al., 2021) constructs a correlation matrix between two different views of batch of samples with encoder and trains the encoder to enforce the correlation matrix to the identity matrix. Lastly, MAE (He et al., 2022) learns a meaningful image representation space by masking an image and reconstructing the masked input. In this paper, we utilize Barlow Twins as an SSL framework to train our target model.
3 METHOD
3.1 PRELIMINARIES
Problem Definition Suppose that we are given an unlabeled dataset X, [x].... [x,]] E RRuxd where each row x; _R is an i.i.d sample. We define the problem of self-supervised dataset distillation as the process of creating a compact synthetic dataset X, = [&] ... xm]] E Rg that preserves most of the information from the unlabeled dataset X, for pre-training any neural net- works, while keeping m < n. Thus, after the dataset distillation process, we can transfer knowledge embedded in the large dataset to various tasks using the distilled dataset. Specifically, our final goal is to accelerate the pre-training of a neural network with any architectures by utilizing the distilled dataset X, in place of the full unlabeled dataset X, for pre-training. Subsequently, one can evaluate the performance of the neural network by fine-tuning it on various downstream tasks.
Bilevel Optimization with SSL Recent success of transfer learning with self-supervised learning (SSL) (Chen et al., 2020a; He et al., 2020; Grill et al., 2020; Zbontar et al., 2021) is deeply rooted in the ability to learn meaningful and task-agnostic latent representation space. Inspired by the SSL,,
3

---

Published as a conference paper at ICLR 2024
want to find a distilled dataset X, such that model 0o R + Rdv, trained X, with SSL objective, achieves low SSL loss on the full dataset X, Here, denotes the parameter of the neural network 9o. Similar to previous supervised dataset condensation methods (Wang al., 2018; Zhao et al., 2021; Cazenavette et al., 2022; Deng & Russakovsky, 2022), estimation of X, can be formulated bilevel optimization problem:
minimize Lssi (X.); where 8 (X,) arg min (s x,).
X
Here, LysL(0: (8; X²) denotes a SSL loss function with ge evaluated on the dataset X,. The bilevel optimization can be solved by iterative gradient-based algorithms. However, is computationally expensive since computing gradient with respect to X, requires back-propagating through unrolled computational graphs of inner optimization steps. Furthermore, we empirically find out that back- propagating through data augmentations involved in SSL is unstable and challenging.
3.2 KERNEL RIDGE REGRESSION ON SELF-SUPERVISED TARGET
Motivation We theoretically analyze the instability of the bilevel formulation for optimizing a condensed dataset with an SSL objective and motivate our objective function. Define dg by A R:
Let us write (0 X.) = BE=DIEE ( (0. where & ~ D is the random variable corresponding to the data augmentation (or input mask), and Ee is SSL loss with the sampled data augmentation (or input mask) E. Define ((X,) arg ming ESSL (0; X,) where LSsI X,) = : - ls (0,X,)
and Si ~ D. In practice, we compute 0C (0(X xx to update X.. The use the empirical estimate LSsL(0: Xx) in the place of the true SSL loss CSsL(O: Xx)) is justified in standard SSL without bilevel optimization because its gradient is always unbiased estimator of the true gradient: i.e., [E{]
Ux.
(0:xx) where Ç = (4))==1 However, the following theorem shows that this is not the case for bilevel optimization. This explains the empirically observed instability of the
[8.X.) (E__
86
SSL loss in bilevel optimization. proof is deferred to Appendix A.
(. (((x)):X)
0.su (9" (9 (X.):x)
Theorem 1. The derivative
a biased estimator of
aX. @C 10sm100( (X.)x.)) 0x,
ax,
i... Ecl
Ox.
A
unless
( (0:X,)
(X.) 00(X,) 0=0(x,))
06
(=0*(X))
00 (X.)
O(X.)ss
Ecl
(P.X.. 1o-0xx,) [
La 00(X.)
+
Eie Covçl
BLax (9:X.)
for all
00
00.
(i.) (1, m} x da, J.
m]
Regression on Self-supervised Target Based on the insight of Theorem 1, we propose to replace the inner objective function with mean squared error (MSE) by parameterizing and optimizing both synthetic examples X, and their target representations Y. = [ý1 ... "m]" E Rmmxy as:
Cinner (0; X., Y,) = 2 ||Y, - (x
(2)
which avoids the biased gradient of SSL. loss due to the absence of random variables Ç corresponding to data augmentation (or input mask) in the MSE. Here, 11-11 denotes a Frobenius norm. Similarly, we replace the outer objective with the MSE between the original data representation of the model trained with Cimer X,, Y²) and that of the target model gø Rdd + Rdy trained on the full dataset X, with the SSL objective follows:
1 2 (1,)(Xx)) go- (X....) (x.))l where 0r 0'(X, Y.) arg min Linner (0; (o, Y.).
minimize X..Y.
(3)
Note that first pre-train the target model go the full dataset X, with the SSL objective, i.e. á = arg ming CSsu X.). After that, (X,) ) is fixed target which is considered to be con- stant during the optimization of X, and Y.. Here, (X,) (x1) Go- (x..X..V.) ( (Xe)
(x.)) E Raxd, and Raxd, The intuition behind the objec- tive function is as follows. Assuming that a model trained with an SSL objective on a large-scale dataset generalizes to various downstream tasks (Chen et al., 2020b), we aim ensure that the rep-
1ô0- (x1) ... (x.(xx,,.) (x,))
resentation space of the model Úø- (X,,)), trained on the condensed data, is similar to that of the self- supe-supe target model 94-
Again, one notable advantage of using the MSE is that it removes the need for data augmentations or masking inputs for the evaluation of the inner objective. Furthermore, can easily evaluate
4

---

Published as a conference paper at ICLR 2024
Algorithm| 1 Kernel Ridge Regression on Self-supervised Target (KRR-ST)
1: Input: Dataset X₁, batch size b. learning rate n, SSL objective LSsL. and total steps T.
2: Optimize 96 with SSL loss on X₃ using data augmentation: & +- arg ming LSSL (d; Xt).
3: Initialize X, = [8] ... xm]_ with a random subset Xt.
4: Initialize Y. = [ý1- 1.1..:100 -mm with y; = (x,)) for = =1,...... =I....,,. m.
5: Initialize a model pool .M ((1.WW..tt) ..... (wi, Wi, W() using X, and Y
6: while not converged do
7: Uniformly sample mini batch Xx [x1 .1....50 X6) from the full dataset X,.
8: Uniformly sample an index i from (1..... 1} and retrieve the model (w;,W, 44) E M.
9:
Compute the outer objective Louter Couter(X,, (X., Y,) with fw; in equation 4.
10:
Update X₂ and Y;: X, t X, - a VVx. x, Loweer (Xmmer((s,) (X,,Y.), Yo f Y.- - vV Louter Y,).
11:
if T then
12:
Set t, < t + and evaluate MSE loss MSSE + 1 I - -hw. o fmn (x.))l Update wi and W; with wi c W, 1) V/V,, MMSE. W. s- W, 16-11 1/ w; CMSE-
13:
14:
else
15:
Set t4 < and randomly initialize wa and W.. end if
16:
17:
end while
18:
Output: Distilled dataset (X,.Y.)
the inner objective with full batch X, since the size of X, (i.e., m) is small enough and we do not need m x pairwise correlation matrix required for many SSL objectives (Chen et al., 2020a; He et al., 2020; Zhontar et al., ;0 Bardes et al., 2022). Consequently, the elimination of randomness enables to get an unbiased estimate of the true gradient and contributes stable optimization.
Kernel Ridge Regression Lastly, following Zhou et al. (2022), we simplify the inner optimization to reduce the computational cost of bilevel optimization in equation 3. First, we decompose the function go into a feature extractor fw : IRd - Rd and a linear head hw : v E Rd. + v +v e Rdy, where W E Rda and 0 = (w,W) (i.e., 9o hw o fw). Naively, we can train the feature extractor and linear head on X, and Y, during inner optimization and compute the meta-gradient of X, and Y, w.r.t the outer objective while considering the feature extractor constant. However, previous works (Cazenavette et al., 2022; Zhou et al., 2022: Zhao et al., 2023) have shown that using diverse models at inner optimization is robust overfitting compared to using single model.
W
Based on this insight, we maintain a model pool M consisting of l different feature extractors and linear heads. To initialize each o in the pool, we first sample t {l..... , 7'} and then optimize and W to minimize the MSE in equation 2 on randomly initialized X, and Y, with full-batch gradient descent algorithms for steps, where T the maximum number of steps. Afterward, we sample a feature extractor fw from M for each meta-update. We then optimize another head hw- on top of the sampled feature extractor fa which is fixed. Here, kernel ridge regression (Murphy, 2012) enables getting a closed form solution of the linear head as lw- V + v S=,(
(xx,,x, + Am) ) "Y, where A > 0 is hyperparameter for l regularization, Im E IR is an identity matrix, and K Kx,.x, = (X,) (X.) E mm with (X,) = [Sw (x1) ... [xm))
R
Then, we sample a mini-batch X, [x .....))) x,] E Rbxd, from the full set X, and compute a meta meta-gradient of X, and Y, with respect to the following outer objective function:
(4) where (Xee = [xo( ((x))))---9(x)))) (T+)) E Roxdy and (X,) = (x, ... (xxx)) (Xx)) E gbxds, Finally, we update the distilled dataset X, and Y, with gradient descent algorithms. After updating the distilled dataset, we update the selected feature extractor fa and its corresponding head hw with the distilled dataset for one step. Based on this, we dub our proposed method as Kernel Ridge Regression on supervuee Target (KRR-ST), and outline its algorithmic design in Algorithm 1.
1 Louter (X,, = (X.)
- (X.) (Kx..,
((Kx.. + Amm) -Y,l
Transfer Learning We now elaborate how we deploy the distilled dataset for transfer learning scenarios. Given the distilled dataset (x).Y.) we first pre-train a randomly initialized feature extractor fa and head hw v Rdd + v T W Rd, the distilled dataset to minimize either MSE for our KRR-ST, KIP (Nguyen et al., 2021a), and FRePO (Zhou et al., 2022), or cross-entropy loss for DSA (Zhao& & Bilen, 2021), MTT (Cazenavette et al., 2022). and DM (Zhao & Bilen, 2023):
minimize 1 [X,,
w.W 2 - -Y, Y or minimize W.W
(y,, sfftmax(((((() (W))) (5)
5

---

Published as a conference paper at ICLR 2024
where q) - - Ps log q: for p (p1..... (P1.....PP) Pc). 9 (q1,...,9e) 9c) E - and A is simplex over R Then, we discard hy and fine-tune the feature extractor fw with a randomly initialized task- head fig to minimize the cross-entropy loss f. Here, Q E Ie and is the number of classes. Note that can use any loss function for fine-tuning, however, we only focus on the classification in this work.
R + softmax(y Q) A s-11 on a target labeled dataset
4 EXPERIMENTS
In this section, we empirically validate the efficacy of our KRR-ST on various applications: transfer learning, architecture generalization, and target data-fres knowledge distillation.
4.1 EXPERIMENTAL SETUPS
Datasets We use either CIFAR 100 (Krizhevsky et al., 2009), TinyImageNet (Le & Yang. 2015), or ImageNet (Deng et al., 2009) source dataset for dataset distillation, while evaluating the distilled dataset on CIFAR 10 (Krizhevsky et al., 2009), Aircraft (Maji et al., 2013), Stanford Cars (Krause al., 2013), CUB2011 (Wah al., 2011), Stanford Dogs (Khosla et al., 201 1), and Flowers (Nils- back & Zisserman, 2008). For ImageNet, we resize the images into a resolution of 64 x 64, following the previous dataset distillation methods (Zhou et al., 2022; Cazenavette et al., 2022). We resize the images of the target dataset into the resolution of the source dataset, e.g., 32 x 32 for CIFAR100 and 64 x 64 for TinylmageNet and ImageNet, respectively.
Baselines We compare the proposed method KRR-ST with different baselines including simple baseline without pre-training, 5 representative supervised baselines from the dataset condensation benchmark (Cui et al., 2022). and kernel ridge regression baselines as follows:
1) w/o pre: We train a model on solely the target dataset without any pre-training.
2) Random: We pre-train model randomly chosen images of the source dataset.
3) Kmeans (Cui et al., 2022): Instead of 2) Random choice, we choose the nearest image for each centroid of kmeann--lustering (Lloyd, 1982) in feature space of the source dataset.
4-6) DSA (Zhao Bilen, 2021), DM (Zhao & Bilen, 2023), and MTT (Cazenavette et al., 2022):
These are representative dataset condensation methods based surrogate objectives such as gradient matching, distribution matching, and trajectory matching, respectively.
7) KIP (Nguyen et al., 2021a;b): Kernel Inducing Points (KIP) the first proposed kernel ridge regression method for dataset distillation. For transfer learning, we use the distilled datasets with standard normalization instead of ZCA-whitening.
8) FRePo (Zhou al., 2022): Feature Regression with Pooling (FRePo) relaxed version of bilevel optimization, where inner optimization is replaced with the analytic solution of ker- nel ridge regression on neural features. Since FRePo does not provide datasets distilled with standard normalization, we re-run the official code of Zhou et al. (2022) with standard normal- ization for transfer learning.
Implementation Details Following Nguyen et al. (2021a;b); Zhou al. (2022), we use convo- lutional layers consisting of batch normalization (Ioffe & Szegedy, 2015), ReLU activation, and average pooling for distilling dataset. We choose the number of layers based on the resolution of images, i.e., 3 layers for 32 x 32 and 4 layers for 64 x 64. respectively. We initialize and maintain I 1= 10 models for the model pool M, and update the models in the pool using full- gradient de- scent with learning rate, momentum, and weight decay being set to 0.1. 0.9, and 0.001, respectively. The total number of steps T is set to 1, 1,000. We meta-update our distilled dataset for 160,000 iter- ations using AdamW W optimizer (Loshchilov & Hutter, 2019) with initial learning rate of 0.001, 0.00001, and 0.00001 for CIFAR100, TinylmageNet, and ImageNet, respetively. The learning rate is linearly decayed. We use ResNet18 (He et al., 2016) self- self-supervised target model gs which is trained on a source dataset with Barlow Twins (Zbontar et al., 2021) objective.
After distillation, we pre-train model on the distilled dataset for 1,000 epochs with mini-bater size of 256 using stochastic gradient descent (SGD) optimizer, where learning rate, momentum, and weight decay are set to 0.1, 0.9, and 0.001, respectively. For the baselines, we follow their original

---

Published as a conference paper at ICLR 2024
Table I: The results of transfer learning with CIEARI00. The data compression ratio for source dataset is 2%. ConvNet3 is pre-trained on condensed dataset, and then fine-tuned on target datasets. We report the average and standard deviation over three runs. The best results are bolded.
Source
Target
Method
CIFARI00
CIFARIO
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o pre
64.95
87.3440.13
34.6620.8 39
18.4620.11 19.43
22.31
58.75+0.44 41
Random
65.2320.11
87.5520.11 19
33.9910.45 45
19.77:0.21
18. 21
21.69=0.16
59.31=00.2
Kmcans
65.67=0.22
87 67.0009
35.0820.69
20.02:10441
18. 15
21.8620.18
59.5820.11
DSA
65.48+0.21
87 2 21
34.38
19.59+a.25
18.08+0.33
21.90=0.24
58.50+0.04
DM
65.17=4.122
87 0.020.00
35.24+0.64
20.13+0.33
18.68+0.33
21.87=0.23
59.89+0.57
MTT
65.92=a0.18
87.87 os
36.11+0.27
21.42:+0.03
18.94+9..41
22.82 22.82=0.02
60.88+0.45
KIP
65.97
07.99:10.19
30.67
23. 12:10.6
20. 00
23.83:10.25
63.0410.33
FRePo
65.64:0.40
87 67 2:022
35.34 8..85
21.05:1..08
18.88
22.90
60.3511007
KRR-ST
66.81 =0.11
88. 08.722000
41.54 0.37
28.68
25.30 0.33 26. 39 20.08
67. 10.18
Table 2: The results of transfer learning with Tiny Imageeet.... The data compression ratio for source dataset is 2%. ConvNet4 is pre-trained condensed dataset, and then fine-tuned target datasets. We report the average and standard deviation over three runs. The best results are bolded.
Source
Target
Method
TinyImageNet
CIFARIO
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o pre
49.57=0.18
88.74=0.10
40.81
23.
22. 22.119-s 19
24.74=0.49
59. 18..37
Random
50.49
88.46
43.17
24.74:0.36
21.9110.08 08
25.17=0.22
62.49=0.40
Kmeans
50.69
88.58
43.77
26.26:10.33
22.52:0.37
25.85
03.63
DSA
50.42:00.5
88. 20.17
13.63
26.02:00.44
22.98 48
26.52=0.33
63. 63.9820.42
DM
49.81 10
88.48
42.14
25.68=0.44
22.48
25.05=0.22
63. 15.45500.14
MTT
50.92=0.18
89.20=0.0s
20.21
30.35=0.22
25.95=0.14
28.53=0.28 26
66.07=0.18
FRePo
49.71=0.06
28.28=0.08
47.59=0.59
29.25=a.41
24.81
27.68=0.37
20.47
KRR-ST
51.86=0.24
189.31
+0.08
58.83+0.30
49. 49.26=0.64
35.78
71.16#0.
Table 3: The results of transfer learning with ImageNet. The data compression ratio for the source dataset is =00.8%% ConvNet4 is pre-trained on a condensed dataset and then fine-tunce on target datasets. We report the average and standard deviation over three runs. The best results are bolded.
Method
CIFARIO
CIFARI00
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o pre
88.66.20.00 0.09
66.62.1038
42.45 46
23.62:10.7
22.00.0.09
24.59.0.46
59.39 29
Random
88.4620.00
40.0920.44 65.97
46
20.92:0.42
19.41 20.28
23.08=0.40
56.8140.44
FRePo
87. 20.20
65.2340.47
39.0320.35
20.00
19. 19.2620.21
22.05=0.45
52.50+0.51
KRR-ST 89 19.33=+0.15 68.04
24
57.17
00.16
46.95
35.66=0.54
35.51 #0.45 70.45 #0.34
experimental setup to pre-train a model on their condensed dataset. For fine-tuning, all the experi- mental setups are fixed as follows: we use the SGD optimizer with learning rate of 0.01, momentum of 0.9 and weight decay of 0.0005. We fine-tune the models for 10,000 iterations (CIFARIO0 100, CI- FARIO, and Tinylmagenet), or 5,000 iterations (Aircraft, Cars, CUB201 1, Dogs, and Flowers) with mini-batch size of 256. The learning rate is decayed with cosine scheduling.
4.2 EXPERIMENTAL RESULTS AND ANALYSIS
Transfer learning We investigate how our proposed KRR-ST can be effectively used for transfer learning. To this end, we pre-train a model on the distilled source dataset and fine-tune the model using a target training dataset. We report the average and standard deviation of the model's 's accu- racy on the target test dataset over three runs. First, we use ConvNet3 (3-layer CNN) to distill the CIFAR100 100 dataset into 1,000 synthetic examples, which is equivalent to 2% of the original dataset. After distillation, we pre-train the model with the synthetic samples and fine-tune it on the target training datasets. As shown in Table 1, KRR-ST outperforms all the baselines, including those using labels for distillation. Next, we distill TinylmageNet into 2,000 synthetic images, which con- stitute 2% of the original dataset. We pre-train ConvNet4 on the distilled dataset and fine-tune the model on the target datasets. As shown in Table 2, we observe that our unsupervised dataset distilla- tion method outperforms all the baselines by a larger margin than in the previous experiments with
7

---

Published as a conference paper at ICLR 2024
orgiial 51th data
(a) CIFARI00
(b) TinyImageNet
Figure 2: Visualization of the distilled images, their feature representation and corresponding distilled labels in the output space of the target model. All distilled images provided in Appendix B.
VOG11
A Aeltet
Mobiobie
RecNetiC
Wronn
CAre
CUR2011
Dogs
Fowers
Camt
GU5D011 Dogs
Fowers
Arrmm
Cart GUROOLL
Cogs
Fowers
Arow
Cart
OJ00011
Doge
Flwwrrs
pre
Random
Kmeens
DSA
FRePo
KRR-ST
Figure 3: The results of architecture generalization. ConvNet4 is utilized for condensing TinyImageNet into 2,000 synthetic examples. Models with different architectures are pre-trained on the condensed dataset and fine-tuned on target datasets. We report the average and standard deviation over three runs. The above results are reported as a tabular format in Appendix C.
the distilled CIFAR100 dataset. Lastly, we distill ImageNet into 1,000 synthetic samples which approximately 0.08% of the original full dataset using ConvNet4, and report experimental results in Table 3. For ImageNet experiments, FRePo is the only available supervised dataset distillation method since we can not run the other baselines due to their large memory consumption. These experimental results demonstrate the efficacy of our method in condensing an unlabeled dataset for pre-training a model that can be transferred to target datasets.
Visualization In this paragraph, we analyze how our method distills images and their correspond- ing target representations in both pixel and feature space. We first present the distilled images X, and visualize their representation go (X.) E 1Txx5122 along with the learned target representation Y, E Rm2552 and representations of the original full data gØ E R22512 where 9o is the target ResNet18 trained with SSL Barlow Twins objective on the original dataset X,. For visualization, we employ t-SNE (Van der Maaten & Hinton, 2008) to project the high-dimensional representations to 2D vectors. Figure 2 demonstrates the distilled images and corresponding feature space of CI- 100 and TinylmageNet. As reported in Zhou et al. (2022), we have found that distilled images with our algorithm result in visually realistic samples, which is well-known to be crucial factor for architecture generalization. Lastly, we observe that the distilled data points cover most of the feature space induced by the full dataset, even with either 1,000 or 2,000 synthetic samples which are only 2% of the full dataset. All distilled images are provided in Appendix B.
Architecture Generalization To examine whether our method can produce a distilled dataset that can be generalized to different architectures, we perform the following experiments. First, we use ConvNet4 as ÿe in equation 3 to condense TinylmageNet into 2,000 synthetic examples. Then, we pre-train models of VGG11 (Simonyan & Zisserman, 2015), AlexNet (Krizhevsky al., 2012), MobileNet (Howard et al., 2017), and ResNet10 (Gong et al., 2022) architectures on the condensed dataset. Finally, the models are fine-tuned on five target datasets Stanford Cars, Stanford Dogs, Aircraft, CUB2011, and Flowers dataset. We choose those architectures since they are lightweight and suitable for small devices, and pre-trained weights of those architectures for 64 x 64 resolution are rarely available on the internet. As shown in Figure 3 and Tables 5 to 8 from Appendix C, our method achieves significant improvements over baselines across different architectures except for setting (MobileNet the Aireraft dataset). These results showcase that our method can effectively distill the source dataset into a small one that allows pre- pre-trainin models with different architectures.
8

---

Published as a conference paper at ICLR 2024
Table 4: The results target data-free KD on CIFARIO. ConvNet4 is utilized for condensing TinylmageNer into 2,000 synthetic examples. Models with different architectures are pre-trained on the condensed dataset and fine-tuned on CIFARIO using KD loss. We report the average and standard deviation over three runs.
Method
ConvNet4
VGG11
AlexNet
MobileNet
ResNet10
Gaussian
32.45+0.65
33.25=1.33
30.58+0.56
23.96+0.94
24.83+1.88
Random
49.98+0.78
51.47=1.10
51.16+0.22
44.27 +0.92
40.63
Kmcans
52.00=1.10
53.86=0.92
53.31
48. 18.10
40.90=2.42
DSA
1..64::..8
10.97
47.4210.m 70
39.6710.92
37.33=1.70
DM
46.90=0.18
48.41
48.6110.25
1..09:
39.35=10:40
MTT
19.62=t0.90
53.18=0.72
51.22:+0.5
41.48:11.44
38.75=0.47
FRePo
45.25=0.440
50.51=0.60
17.24+0.21
42.98+0.77
42. 00..81
KRR-ST
58.31:0.94 62.15
59.80.10.99
54. 13:049
52.68 1..47
Target Data-Free Knowledge Distillation One of the most challenging transfer learning scenar- ios is data-free knowledge distillation (KD) (Lopes et al., 2017; Yin et al., 2020; Raikwar & Mishra, 2022), where we aim to distill the knowledge of teacher into smaller student models without a target dataset due to data privacy or intellectual property issues. Inspired by the success of KD with a sur- rogate dataset (Orekondy et al., 2019; Kim et al., 2023), we utilize distilled TinylmageNe dataset X₂ as a surrogate dataset for KD instead of using the target dataset CIFAR1O. 10. Here, we investigate the efficacy of each dataset distillation method on this target data-free KD task. First, we are given a teacher model Tg. Rd= + A--1 which is trained on the target dataset CIFARIO, where c = 10 is the number of classes. We first pre-trate pre-train feature extractor fw. demonstrated in equation 5. After that, we randomly initialize the task head ha v E Rd Y softmax(v Q) E A with Q E Rd Xc, and fine-tune w and Q with the cross-entropy loss using the teacher Ty as a target:
minimize wiQ
- Σ e (x) Q))-
(6)
In preliminary experiments, we have found that direct use of distilled dataset X, for KD is not bene- ficial due to the discrepancy between the source and target dataset. To address this issue, we always use mean and standard deviation of current mini-batch for batch normalization in both student and teacher models, even at test time, as suggested in Raikwar & Mishra (2022). We optimize the param- eter w and Q of the student model for 1,000 epochs with mini-batch size of 512, using AdamW optimizer with a learning rate of 0.0001. Besides the supervised dataset distillation baselines, we introduce another baseline (Raikwar & Mishra, 2022) referred as "Gaussian", which uses Gaus- sian noise as an input to the teacher and the student for computing the KD loss in equation 6, i.e. &; (0, Ja.). Table 4 presents the results of target data-free KD experiments CIFARIO. 10. Firstly, we observe that utilizing condensed surrogate dataset is more effective for knowledge dis- tillation than using a Gaussian noise. Moreover, supervised dataset distillation methods (DSA, DM, MTT, and FRePO) even perform worse than the baseline Random. On the other hand, our proposed KRR-ST consistently outperforms all the baselines across different architectures, which showcases the effectiveness of our method for target data-free KD.
5 CONCLUSION
In this work, we proposed a novel problem of unsupervised dataset distillation where we distill an unlabeled dataset into a small set of synthetic samples on which we pre-train model on, and fine-tune the model on the target datasets. Based on a theoretical analysis that the gradient of the synthetic samples with respect to existing SSL loss in naive bilevel optimization biased, we pro- posed minimizing the mean squared error (MSE) between a model's 's representation of the synthetic samples and learnable target representations for the inner objective. Based on the motivation that the model obtained by the inner optimization is expected to imitate the self-supervised target model, we also introduced the MSE between representations of the inner model and those of the self-supervised target model on the original full dataset for outer optimization. Finally, we simplify the inner op- timization by optimizing only a linear head with kernel ridge regression, enabling us to reduce the computational cost. The experimental results demonstrated the efficacy of our self-supervised data distillation method in various upplications such as transfer learning, architecture generalization, and target data-free knowledge distillation.

---

Published as a conference paper at ICLR 2024
Reproducibility Statement We use Pytorch (Paszke et al., 2019) to implement our self self-superrvsed dataset distillation method, KRR-ST. First, we have provided the complete proof of Theorem in Appendix A. Moreover, have detailed method in Algorithm and specified all the implementation details including hyperaparameters in Section 4.1. Our code is available at https: //g1 thub. wwt.ccommmbmbee sel fsup.dd.
Ethics Statement Our work is less likely bring about any negative societal impacts. However, we should be careful about bias in the original dataset, as this bias may be transferred to the distilled dataset. On the positive side, we significantly reduce the search cost of NAS, which, in turn, can reduce the energy consumption when running GPUs.
Acknowledgement This work was supported by Institute of Information & communications Tech- nology Planning & Evaluation (IITP) grant funded by the Korea goVernmen((MIIIT) (No.2019. 0-00075, Artificial Intelligence Graduate School Program(KAIST)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govern- MeST(MSIT) (No.2020-0-001 and N0.2022-0)00711) KAIST-NAVER Hypercreative Al Center, Samsung Electronies (I0201214-00) 14-00145-00),, the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: ALSG2-TC-2023-000--3
and the Singapore Min- istry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207).
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variane-iwarrane--vvriane regular- ization for self-supervised learning. In International Conference Learning Representations, 2022.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 13:14799-1449,, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexci A Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong cemi-supervised learners. Advances in neural information pro- cessing systems, 33:22243-22255 2020b.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation benchmark. Advances in Neural Information Processing Systems, 15:810-822, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009.
Zhiwei Deng and Olga Russakovsky, Remember the past: Distilling datasets into addressable mem- ories for neural networks. In Advances in Neural Information Processing Systems, 2022.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference Machine Learning, pp. 1165-1177 PMLR, 2017.
Jiaming Gong, Wei Liu, Mengjie Pei, Chengchao Wu, and Liufei Guo. Resnet10: A lightweight residual network for remote sensing image classification. In International Conference on Mea- suring Technology and Mechatronics Automation (ICMTMA), pp. 975-978, 2022.
10

---

Published as a conference paper at ICLR 2024
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1e0n::4440.881111 2017.
Jeremy Howard. Imagenette: A smaller subset of 10 casily classified classes from imagenet. https:/giihhb..om/issstammmgggg
March 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Puttern Recognition, Colorado Springs, CO, June 2011.
Byungjoo Kim, Suyoung Lee, Seanie Lee, Sooel Son, and Sung Ju Hwang. Margin-based neural network watermarking. In International Conference on Machine Learning. 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. Hardware-adaptive efficient latency prediction for NAS via meta-learning. In Advances in Neural Information Processing Systems, 2021.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Hanxiao Liu, Karen Simonyan. and Yiming Yang, DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
Stuart Lloyd. Least squares quantization in pcm. JEEE transactions on information theory. 28(2):
129-137, 1982.
11

---

Published as a conference paper at ICLR 2024
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 35:13877- 13891, 2022.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755, 2023.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv: 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernll ridge-
regression. In International Conference on Learning Representations, 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances Neural Information Processing Systems, 34:5186- 5198, 2021b.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp- 722-729. IEEE, 2008
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2019. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Piyush Raikwar and Deepak Mishra. Discovering and overcoming limitations of noise-engineered data-free knowledge distillation. Advances in Neural Information Processing Systems, 35:4902- 4912, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, ICLR 2015, 2015.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds hirds-200-2011 dataset. Technical Report ENS-TR-2011-001, California Institute of Technology, 2011.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 1811 1. 10959, 2018
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8715-8724, 2020.
12

---

Published as a conference paper at ICLR 2024
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Internationall Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2021.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. Advances in Newral Information Processing Systems, 35:9813-9827, 2022.
13

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM
Proof. Let (i-)) E (1,...,,m) x (I,...,dz) ,z,). By the chain rule,
(0 (X,);;
(
(aLssu(B: (8; Xx)
00*
O(x))s)
00
(=0)
a(Xa)is
Similarly,
ŌCss @Cssi(O(X,) (O(x)): a(X))ij
(
[DSss) (0:
(0()))
09
(-0(x)))
((X))s
By taking expectation and using the definition of the covariance,
(
(8;
(xx))
Ex
E8g
O(Xa)is
00
(=A(X.)
((Xa)is
du
ÖLsul (0; X.)
00(X.)s
Eg
00p
(=0(X.)
a(Xa)is
dy
ÖLsul (0; X1)
00(x))s
Ex
00k
0 ((x))
((Xa)is
k=1
do
do
Eelua|Ecaas +
Covelux.aul
k=1
k=
where vn 0Css (0:X.)
(xx) and on
2b(x.). ) 8(X)),
By defining the vectors U2,.,,,.....
86%
Rd and a (01,,.2........... Otdg
RRd,
ØLssI. (0l (0(X,); X,)
do Eclua] Ee [ax]
de
Eg
+
Cova[uk.ass]
O(X.)s5
1=1
Ecc [a1]
dg Covelun,as]
[c[11]
[Eclaa
1=1
an
dy Cove [un
= E5 [[0]
vdol] R
+
or
do
= Bg [w]
+ Eg [a]]
Covglex,an)
Therefore, E, tion 7 proves the statement.
0Lssa (6(.)) 8[X.)
Eg [u]
E2 [a]
Σ de 1=
Covglux,as) Comparing this with equa-
14

---

Published as a conference paper at ICLR 2024
B VISUALIZATION OF DISTILLED IMAGES
Figure 4: Visualization the synthetic images distilled by our method CIFARIO0.
Figure 5: Visualization of the synthetic images distilled by our method TinylmageNet.
Figure 6: Visualization of synthetic images distilled by our method in ImageNet.
15

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL RESULTS OF ARCHITECTURE GENERALIZATION
Table The results of transfer learning using VGG11. Conv4 is utilized for condensing TinylmageNett into 2,000 synthetic examples. We report the average and standard deviation over three runs.
Method
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o pre
59.17 a9.
32.41+0.44
32.62-u.11
32.24+0.34
60.88
Random
55.67+to.27
05.70
33.99-u.41
33.55+0.21
65. 65.7540.25
Kmeans
57.5010.65
38.6610.46
34.20=0.10
34.5010.33
66.79:1055
DSA
55.26110.39
37.22:0.54
34.75=0.98
33.0110.46
67 07.07:10.90
DM
56.40.10.1
39. 19.199
36.01=0.12
34.65
68.82:10.00
MTT
61.5/4..511
16.4420.42
37.95_0.33
38.35+0024
70.97:0.50
FRePo 19.13+1.23
30.72#0.51
31.57=0.92
30.93
611.6:#0.18
KRR-ST
66.35
54.43 20.17
39.22 0.28
38.54 H0.31
71.46::0.70
Table The results of transfer learning using AlexNet. Conv4 is utilized for condensing TinyImageNet into 2,000 synthetic examples. We report the average and standard deviation over three runs.
Method
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o pre
52.29++0.a
21.83::0.29
24.69=u.31
23.95
55.46++..3
Random
50.56:10.53
22.77:0.34
25.20
24.09
59.8110.62
Kmeans
50.7211.26
24.23:10.05
25.29-0.19
24.11
61. 01.20440.5
DSA
49.6710.38
24.78:2.22
25.77=0:38
24.3010.38
60.8010.22
DM
48.0110.43
22.84.10.46
23.88=0.14
23.04:10112
59.33110.3
MTT
54.21+0.46
29.93:t0.48
28.59=0.17
27.74t0.34
6/1.10+0.35
FRePo
51.39+0.31
21.86:11.om
25. 23.19-0.72
24.68
61.08+1.58
KRR-ST
58.7610.45
41.60.10.0
33.69 00.22
31.55 10026
65. 50:10.29
Table 7: The results transfer learning using MobileNet. Conv4 utilized for condensing TinylmageNet into 2,000 synthetic examples. We report the average and standard deviation over three runs.
Method
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o pre
4..90
33.96+2.00
30.73-um
32.21 +0.51
32,21
52.67+1.11
Random
58.7110.75
40.69
30.03-1.10
32.5220.27
56. 1740.92
Kmeans
59.59+1.02
42.90
30.82_0.24
33.09
55.5240.38
DSA
60.3210.47
11.44.11.29
32.10=o.1r
33.5810.s8
57.2740030
DM
60.29+0.97
12.63+11.0
31.66=0.s1
34.39
56.23+1.70
MTT
64.5440..56
50.84+1.13
36.31=a.61
37.97
60.25+1.68
FRePo
57.48+1.05
11.85#0.76
35.03=0.22
35.05+0.83
61.29+0.40
KRR-ST
63.08
54.89.0.00
97.65:2.44
40.8110.14
87.03:11 1.00
Table The results transfer learning using ResNet10, Conv4 utilized for condensing TinylmageNet into 2,000 synthetic examples. We report the average and standard deviation over three runs.
Method
Aireraft
Cars
CUB2011
Dogs
Flowers
w/o pre
6.34
4.16+0.11
5.19
6.69
43.38-1..9
Random
39.96+1-48
28. 33
18.55_0.33
29.3420.61
57.29410.0
Kmeans
41.67400.8
30. 12..122
18.63-0.56
29.34.0.11
57.46++018
DSA
42.43+11.6
31.66++..7
22.17=u.62
31.58+1.12
59.47+00.54
DM
43.8644.47
33.1774..81
22.83=0.18
32.42+0.a9
60.8544.76
MTT
46.62+b..a
36. t03.
24.65=0.08
33.42+0.14
61.4740..5
FRePo
30.70+0.52
14.82.t0.2
17.41=0.29
25.37
55.09
KRR-ST ST
59.9111.24
$1.55520.90
32.55+1.58
40.05
66.61:1.88
16

---

Published as a conference paper at ICLR 2024
D IMAGENETTE EXPERIMENTS
Table 9: The results of transfer learning using Conv5. Conv5 is utilized for condensing ImageNette (Howard, 2019) into 10 synthetic examples. Here, ResNet50 pre-trained ImageNet using Barlow Twins the target model ge for KRR-ST. We report the average and standard deviation of test accuracy over three runs.
Source ImageNette
Target CUB2011
Method
Aircraft
Cars
Dogs
Flowers
w/o pre
77.55
10.65
16.31
18.92
10.64
46,80
Random
6.122+3.12
21.15
5.77
4.03
10.10
22.31+1.34
DSA
63. 13..99
23.07 11.19
6.37
1.89
11.12
24.70
DM
67.6942.59
28.50
7.47:0.45
8.73
11.5620.39
26.23:2.43
KRR-ST
82.6420.62
54.34
16.6520.0s
22.15=0.01
23.29
47.8620.08
The results of transfer learning using ResNet18. Conv5 utilized for condensing Ima- geNette (Howard, 2019) into 10 synthetic examples. Here, we use ResNe150 pre-trained on ImageNet using Barlow Twins as the target model ge for KRR-st. We report the average and standard deviation of test accuracy over three runs.
Table 10:
Source ImageNette
Target CUB2011
Method
Aircraft
Cars
Dogs
Flowers
w/o
07.16=0.12
6.69
1.18
+..30
5.24
41.51=0.61
Random
79.16=0.48
12.47:0.21
5.3540.06
6.4410.08
9.94
45.50
DSA
78.94-0.08
8.2220.30
5.01
6.4740.13
7.8220.50
45.36=0.21
DM 79.13-0.16
8.79
5.21
6.71
8.9620.52
14.71
KRR-ST
86.94
33.9443.15
6.19
14.43
17.26
51.88+0.24
E ABLATION STUDY ON THE INITIALIZATION OF Y,
Table 11: The results of ablation study on the initialization of the target representation Y, with Conv3. w/o init denotes the initialization of Y, [51 [m] with ÿ: drawn from standard normal distribution (i.c., "m]
Y ld (0, Iay)) for 1, I...., ... mre, and w init denotes initializing with the target model (i.e., y, = goi (R.) for = m.) as done in Algorithm 1, respectively. We report the average and standard deviation of test accuracy over three runs.
Source
Target
Method
CIFAR100
CIFARIO
Aircraft
Cars
CUB2011
Dogs
Flowers
w/o init
65. 20.00
87.5620.1 19
25.5420.36
17.7520.0 07 23.92 23.9220.09
16.6820.18
20.82.0.30
54.9440.15
init
66.46
88.8720.05
22.9420.21
24.38
23.39+0.24
64.28
F FID SCORE OF CONDENSED DATASET
Table 12: The FID score (Heusel et al., 2017) of condensed dataset.
Method
CIFARI00
TinyImageNet
Random
34.49
20.31
DSA
113.62
90.88
DM
206.09
130.06
MTT
105.11
114.33
KRR-ST
76.28
47.54
17


Reviews:
[{"review_id": "MmERDBTopi", "reviewer": "Reviewer_59jF", "summary": "Dataset distillation is an important and interesting direction in dealing with data-redundancy settings. \nMost dataset distillation methods focus on dataset-specific tasks, overlooking the transfer ability of condensed datasets.   This paper tried to address the transfer ability in the unlabeled setting, via self-supervised in a bi-level optimization framework. The distillation experiments verified the proposed method.", "questions": "Please refer to the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting and novel problem setting:  self-supervised dataset distillation for transfer learning, which might produce task-agnostic condensed data and boost transferability.\n2. Theorem contribution: a gradient of the SSL objectives with data augmentations or masking inputs is a biased estimator of the true gradient. And provide detailed proof.\n3. Interesting experiments that outperform the supervised distillation method with self-supervised learning.", "weaknesses": "1. What is the motivation for minimizing MSE between the original data representation of the model from inner loop and that of the model pre-trained on the original dataset?\n\n2. Why self-supervised learning method is better than the supervised method in this problem? I only see the empirical results, could you provide more explanation?\n\n\nUpdate: I have read the responses and my concerns are partially addressed. The authors did not provide more explanation than empirical results (It might be a more difficult problem that is beyond the scope of this paper). However, I still vote to accept this paper regarding its setting and method novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fwle3YIDBr", "reviewer": "Reviewer_EboT", "summary": "This paper targets at a new problem branched from conventional dataset distillation (DD) ------- unsupervised DD, which aims to synthesize an informative small dataset that can be used for facilitating self-supervised pre-training. The experimental results show the potential of the proposed method in the application of transfer learning, architecture generalization and data-free knowledge distillation.", "questions": "See Above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The problem that this paper focused on is somewhat new to the dataset distillation community;\n2) The presentation and writing of this paper is coherent, the idea is easy to follow.", "weaknesses": "1) This paper dose NOT choose the state-of-the-art baselines in dataset distillation for comparison, such as IDC, IDM, etc.\n2) The authors only provide the experimental results of transfer learning, but did NOT provide the test accuracy of the model trained barely on the distilled dataset. This makes me wondering if the distilled images can keep enough information compared to the images distilled by other baselines, or is the proposed method only performs well in the scenario of transfer learning ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Omugw1HatD", "reviewer": "Reviewer_4KT8", "summary": "The paper considers dataset distillation, which aims at learning a small number of representative examples for a large data set. Unlike previous supervised methods, the authors target a self-supervised learning setup. To counter issues with naive bi-level optimization, they adapt the training loss to an MSE-based objective more akin to feature distillation. This allows for further simplifications such as casting one of the subproblems as kernel ridge regression. The method is evaluated in a transfer learning setting where the goal is to use the distilled data set to train a variety of networks which are then transferred to different data sets. A broad range of source data sets, target data sets and network architectures is considered.", "questions": "- For the kmeans clustering baseline, what does “kmeans-clustering in feature space of the source dataset” mean? What feature space is used?\n- Is the Barlow-Twin target model trained with augmentation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well written and easy to follow. The method is relatively well-motivated, with the scenario when one wants to train many different architectures to find the best one for mobile/resource constrained deployment. The experiments cover a broad range of source, and transfer data sets, and many different network architectures are considered. The method is novel as far as I can tell (although I’m not an expert on dataset distillation).", "weaknesses": "Despite the overall positive impression, I see several weaknesses:\n- In the experiments the authors distill the data sets into 1000-2000 examples, for self-supervised learning, without augmentation. The authors do not comment on augmentations when training on the distilled data. This approach might work for the small models and low resolution used in the experiments, but I’m not convinced that it generalizes to larger models, more complex data sets and higher resolution. Data augmentation is a central component in many SSL methods including Barlow Twins, which the authors use.\n- Unrelated to data augmentation, I feel it would be necessary to run the algorithm on a less small-scale setup, e.g. on 224x224 ImageNet, and on larger downstream models (ResNet18 or similar) to make a convincing case, in particular given the complexity of the algorithm. I know this requires some compute, but one such experiment would still be necessary in my opinion.\n- Some baselines might be weak; for example MobileNet and ResNet10 from scratch get < 4% accuracy on Cars.\n\nMinor comments:\n- The abstract might be hard to follow for readers unfamiliar with prior works on dataset distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwQq4AhRQg", "reviewer": "Reviewer_hYSv", "summary": "The authors introduce a novel problem within the domain of dataset distillation, focusing on the distillation of an unlabeled dataset into a compact set of small synthetic samples tailored for optimizing self-supervised learning (SSL) efficiency.\nIn this work, the authors address a significant challenge related to the bias introduced during the calculation of gradients for synthetic samples with respect to SSL objectives. This bias originates from the inherent randomness associated with data augmentation and masking techniques. To mitigate this issue, they present an innovative strategy. This approach centers on minimizing the mean squared error (MSE) between a model's representations of synthetic examples and the corresponding learnable target feature representations, effectively removing the randomness from the gradient computation.\nIn terms of computational efficiency, the authors propose a streamlined methodology that leverages a fixed feature extractor. They focus their optimization efforts on a linear head atop this stable feature extractor, resulting in substantial reductions in computational overhead. Importantly, this linear head optimization is thoughtfully formulated to yield a closed-form solution, employing kernel ridge regression for practical implementation.\nThe practical utility and impact of the proposed method are rigorously validated through empirical assessments spanning various applications, particularly those involving transfer learning. The experimental results are promising and demonstrate the efficacy of the proposed solution.", "questions": "Please refer to \"Weaknesses\" part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors introduce a novel problem within the context of dataset distillation, addressing the challenge of distilling an unlabeled dataset into synthetic samples optimized explicitly for self-supervised learning (SSL). As far as I know, this is the first work in this direction.\n\n2.  The authors identify and effectively tackle a significant bias issue that arises during gradient calculations for synthetic samples. By introducing a methodology that minimizes mean squared error (MSE) between model representations and target feature representations, they successfully eliminate randomness in gradient computations, enhancing the reliability of their approach.\n\n3. A notable strength lies in the authors' approach to computational efficiency. By assuming a fixed feature extractor and optimizing a linear head on top of it, they substantially reduce the computational burden. Furthermore, their formulation allows for a closed-form solution using kernel ridge regression, streamlining the implementation.\n\n4 The writing is clear and well-organized, making it easy for readers to understand the problem, approach, and results. The authors provide a structured narrative that guides the reader through their research.", "weaknesses": "1. Clarity Issue: The concept of \"target representation\" requires further elucidation for clarity and better understanding.\n\n2. Theoretical Analysis of Instability: It is good to conduct a theoretical analysis of the instability inherent in the bilevel formulation when optimizing a condensed dataset with a self-supervised learning (SSL) objective. My question is, does this instability observed in the bilevel formulation similarly apply to the supervised dataset condensation formulation?\n\n3. Rationale for Kernel Ridge Regression: The reason for selecting kernel ridge regression as the methodology in this work warrants clarification. Is there a specific rationale behind this choice within the proposed solution? Could alternative matching strategies such as distribution matching or gradient matching also be integrated into the proposed method?\n\n4. Stability Issues with SSL Data Augmentation: Furthermore, our empirical observations indicate that the process of back-propagating through data augmentations utilized in self-supervised learning (SSL) introduces instability and poses challenges. However, further explanation is needed to fully comprehend why data augmentation in SSL leads to these instabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ygY8ydBceR", "reviewer": "Reviewer_fRLb", "summary": "This paper explored a new direction of dataset distillation in the scenario of self-supervised learning, where the target dataset to be compressed were unlabeled. Specifically, the authors first analyzed the problem of naive bi-level formulation with a SSL objective and then proposed to replace it with a MSE loss to mimic the target model. Experiments were conducted to compare the proposed method KRR-ST with other counterparts via transfer learning and results demonstrated the effectiveness of KRR-ST.", "questions": "See questions and suggestions in the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The area of dataset distillation for self-supervised learning is under-explored and this paper made an attempt to compress those datasets without labels and provided important insights on future research.\n- Rigorous analysis about the biased gradient estimator of the inner optimization with a SSL objective was presented in the paper, and it motivated the proposed MSE loss in KRR-ST without impact of data augmentation.\n- Comprehensive experiments were conducted covering three different datasets (CIFAR100, TinyImageNet, and ImageNet) to showcase the advantages of KRR-ST for distillation in self-supervised learning settings.", "weaknesses": "- In Equation (2), both $X_s$ and $Y_s$ were trainable parameters and $\\theta$ was minimized in the inner optimization. The process looked a little weird to me although I understand that this design was to make inner and outer optimization consistent. More detailed analysis of $Y_s$ can be provided. For example, we know that $X_s$ was initialized with real images, then how about $Y_s$? Would initialization have great impact on the final performance?\n- An important comparison with the model pre-trained on the whole dataset was missing. These results should be included to indicate how far current dataset distillation methods were from the \"oracle\" model and to evaluate the practicality of KRR-ST in real world applications.\n- From the visualization, it seemed that the distilled images were not very different from original ones and looked exactly the same. Is it possible to report other metrics to measure the difference between synthetic examples and real ones apart from visualization?\n- It was not clear how dataset distillation methods such as DSA and DM were adapted to the self-supervised settings and it was a non-trivial process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)