Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation
Authors: Sheng-Feng Yu
Jia-Jiun Yao
Wei-Chen Chiu
Venue: ICLR 2025 Poster (2025)
Keywords: dataset distillation
self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Although larger datasets are crucial for training large deep models, the rapid growth of dataset size has brought a significant challenge in terms of considerable training costs, which even results in prohibitive computational expenses. Dataset Distillation becomes a popular technique recently to reduce the dataset size via learning a highly compact set of representative exemplars, where the model trained with these exemplars ideally should have comparable performance with respect to the one trained with the full dataset. While most of existing works upon dataset distillation focus on supervised datasets, \todo{we instead aim to distill images and their self-supervisedly trained representations into a distilled set. This procedure, named as Self-Supervised Dataset Distillation, effectively extracts rich information from real datasets, yielding the distilled sets with enhanced cross-architecture generalizability.} Particularly, in order to preserve the key characteristics of original dataset more faithfully and compactly, several novel techniques are proposed: 1) we introduce an innovative parameterization upon images and representations via distinct low-dimensional bases, where the base selection for parameterization is experimentally shown to play a crucial role; 2) we tackle the instability induced by the randomness of data augmentation -- a key component in self-supervised learning but being underestimated in the prior work of self-supervised dataset distillation -- by utilizing predetermined augmentations; 3) we further leverage a lightweight network to model the connections among the representations of augmented views from the same image, leading to more compact pairs of distillation. Extensive experiments conducted on various datasets validate the superiority of our approach in terms of distillation efficiency, cross-architecture generalization, and transfer learning performance.

Published as a conference paper at ICLR 2025
BoOST SELF-SUPERVISED DATASET DISTILLATION VIA PARAMETERIZATTON, PREDEFINED AUGMENTA- TION, AND APPROXIMATION
Sheng-Feng Yu Jia-Jiun Yao', Wei-Chen Chiu
National Yang Ming Chiao Tung University', Macronix International Co., Ltd.? robertyul@man com. tw, (tony.yaa yao. 10, walonjenyou. edu. tw
ABSTRACT
Although larger datasets are crucial for training large deep models, the rapid growth of dataset size has brought a significant challenge in terms of consid- erable training costs, which Dataset Distillation becomes a popular technique recently to reduce the dataset size via learning highly compact set of representative exemplars, where the model trained with these exemplars ideally should have comparable performance with respect the one trained with the full dataset. While most of existing works upon dataset distillation focus on supervised datasets, we instead aim to distill im- ages and their sulf--suey trained representations into a distilled set. This procedure, named as Self-Superviser Dataset Distillation, effectively extracts rich information from real datasets, yielding the distilled sets with enhanced cross- architecture generalizability. Particularly, in order preserve the key character- istics of original dataset more faithfully and compactly, several novel techniques are proposed: 1) we introduce innovative parameterization upon images and representations via distinct low-dimensional bases, where the base selection for parameterization is experimentally shown to play crucial role; 2) we tackle the instability induced by the randomness of data augmentation a key com- ponent in self-supervised learning but being underestimated in the prior work of self-supervisee dataset distillation by utilizing predetermined augmentations: 3) we further leverage lightweight network to model the connections among the representations of augmented views from the same image, leading to more com- pact pairs of distillation. Extensive experiments conducted on various datasets validate the superiority of our approach in terms of distillation efficiency, cross- architecture generalization, and transfer learning performance.
results in prohibitive computational expenses.
INTRODUCTION
In the realm of deep learning, the inexhaustible need for extensive datasets, e.g. ImageNet (Deng jet al., (2009) and LAION (Schuhmann et al.] (022), for model training is a double-edged sword. On one hand, large datasets are generally instrumental in training better-performing models; on the other hand, they introduce prohibitive training costs. Moreover, some learning scenarios even require repeated or iterative training processes, such continual learning and neural architecture search, thus further inflating the expense of training upon large datasets and leading to the financial burden of computing power. Hence, as the volume of data required for deep learning increases, so does the need to innovate the ways to curtail these expanding training expenses.
To this end, Dataset Distillation (DD) (Wagg et al. 2018 Zhao Bilen 202Tb Cazenavette al., 2022 Kim et al. (222)) and coreset selection (Coleman et al. (020)) emerge as two promising strate- gies for dataset reduction. Coreset selection focuses on identifying the most representative samples within dataset. These samples, named coresets, contribute more significantly to performance than the others, allowing models trained on coresets to achieve higher performance compared to those trained random subset of the same size. Conversely, DD aims create distilled dataset, which synthetic and optimized to maintain the model performance (i.e. the model trained upon distilled data should perform comparably the resulted from the full dataset). Despite the synthetic

---

Published as a conference paper at ICLR 2025
nature of the distilled data, which results in distribution that differs from the original data, DD has been shown to substantially improve training outcomes compared to coreset selection, albeit at the cost of longer processing times.
The concept of DD was introduced by Wang et al. (2018). It employs a bilevel optimization (com- posed of inner and outer loops). where the inner loop utilizes distilled data to train a model, and the outer loop optimizes the distilled data based on the model predictions on real data. Subsequent researches alter the optimization target, as seen in methods like DM (Zhao & Bilen (2023) and
MTT (Cazenavette et al., 2022). DM aims to ensure that the mean of the distilled data matches the mean of the real data, while MTT focuses on aligning the training trajectory between the real and distilled data. While earlier works on DD have achieved notable success in reducing dataset size without sacrificing model performance, most of these efforts have been limited to supervised datasets, named as supervised DD. Supervised DD, which relies heavily on labeled data, tends to emphasize dissriiieriientevve features while neglecting others. This narrow focus can result in overfitting to specific models, limiting cros-architecture generalization and reducing effectiveness in other tasks. Consequently, this approach often struggles with transferability to downstream ap- plications. In contrast, sulf-supe learning (Grill et al. 2020; Zbontar al. 2021 Eriesson et al.] 202T), has demonstrated its ability to map images to representations, that effectively trans- fer to a variety of downstream tasks. Thus, condensing these generalized representations offers the potential to create a distilled dataset which superior cross-architecture generalization and improved transferability.
A recent study, KRR-ST (Lee et al. 2024), proposes the first self-supervised DD framework for transfer learning, aiming distill set of image and representation pairs from an unlabeled image dataset. The distilled dataset can be used to train a new model that is encouraged to mimic the sule-supe model trained on the entire unlabeled dataset, where the resultant new model could act as good initialization for being transferred to another task through finetuning. Basically, the distillation framework in KRR-ST is as follows: 1) First, a teacher model is self-supervisedly trained on the unlabeled dataset; 2) A bilevel optimization process is then adopted, where the distilled image-representation pairs are used to train an inner model in the inner loop via minimizing the mean square error (i.e., given an input image, the representation extracted by the model should be close to its paired one), while the outer loop aims to update the distilled pairs by aligning the inner model representations with the teacher model. It is also worth noting that KRR-ST discovers that the random data augmentation (which typically is a crucial component in self-supervised learning algorithms) is incompatible with the bilevel optimization, where the gradient bias stemmed from random data augmentations would affect the optimization process, in results KRR-ST proposes to bypass the augmentation operations in its optimization to avoid such issue of incompatibility.
Although being the pioneer to tackle the self-supervised dataset distillation, KRR-ST still requires substantial storage space for the distilled images and their paired representations, which clearly are not compact enough thus constraining the overall distillation efficiency. To this end, in this work we highlight two key issues of KRR-ST and address them with our proposed techniques:
Each of the distilled images (respectively, their corresponding representations) in KRR- ST are stored independently, where the redundancy the regularity among images (re- spectively, representations) are not taken into consideration. As inspired by Deng & Rus- sskkossky] (2022), it has shown that learning the common bases shared among distilled im- ages not only benefits to compress them (as images are now represented by coefficients, i.e. parameterization) but also contributes improve performance, as well as the observation upon dimensional collapse found in Jing et al. (2022) and Hua et al. (022 where the rep- resentation trained from suf--sueer learning is distributed only in a -in-iimeniional subspace, we propose to separately parameterize both images and representations into two sets of bases (namely image bases and representation bases) with all distilled images and representations being reconstructed through their corresponding bases. In addition, we find that the initialization of the bases is the key to obtaining better model performance, hence we propose initializing the bases using the principal components of the given real dataset (i.e. the source of distillation).
2. As previously mentioned, KRR-ST skips the data augmentation during its optimization process, which is however a critical component in the realm of supervsee learning. In turn, we propose to keep leveraging the benefits of data augmentation via predefining all
2

---

Published as a conference paper at ICLR 2025
possible augmentations and storing all of the representations of augmented images avoid the randomness (which would lead to gradient bias in the bilevel optimization). Moreover, independently storing the representations of all augmented views of the same image consumes considerable storage space, we introduce the approximation networks (built upon multiple-layer perceptrons) which learn to predict the shifts in terms of representation from the original distilled image to its augmented views.
With our two proposed techniques described above, we only need to save the bases of distilled images and representations along with their coefficients, as well as the approximation networks (cf. Figure [ for an overview of our method).
We experiment our proposed method by condensing a source dataset (i.e. CIFAR100 (Krizhevsky 2009). TinyImageNet (Le & Yang| 22155, and ImageNet (Deng al. 2009) into small distilled dataset. Subsequently, to evaluate the quality of the distilled dataset, a feature extractor (whose archi- tecture could be different from the model used in the inner loop of the distillation procedure) is firstly pretrained on this distilled dataset, followed by performing linear evaluation of such feature extrac- tor on the souree dataset or the target datasets (e.g.. CIFARIO (Krizhevsky) (209).. CUB2011 (Wah) et al., 2011). Stanford Dogs (Khosla et al.) (011). Our proposed method produces superior dis- tilled dataset, demonstrating improved linear evaluation results across various target datasets and feature extractor architectures. Our key contributions are summarized as follows:
We introduce efficient parameterization for both distilled images and corresponding repre- sentations, coupled with a novel use of predefined data augmentations and approximation networks, further improve the compactness of the distilled dataset.
Our distilled dataset experimentally demonstrates better arosstarchitecture generalizability and the linear evaluation results across various image datasets.
Pretraining
Parameterization
Bilevel Optimization
Approximation
Unlubeled Dataset X,
Images
Outer Loop
Predict
B
T,IX,)
C"B
P.(CM)
Bulf-Buperiiselyy Training
Inner Loop
Teacher Model
B
Reconstruct Distilled Pairs
Teucher 9,
Bases
Augmentt
Representations
Coeff O
Train
Pair
Basns 179
X
7
Bases /2
Real Data + Reprosentation
Coeff Apgussizattion Net
Update and coufficients
Figure 1: The illustration of our proposed framework of self-supervised dataset distillation. Pre- training: teacher model ge is self-supervisedly trained on the unlabeled source data X, (i.e. the real dataset which we would like to distill). The initialization of the image bases 8 is based on the principal components of X,, and the principal components of gx(X)) (i.c. the representations of real data extracted by the teacher model ga) are adopted to initialize the representation bases 8 Parameterization: The distilled images X, are parameterized by the linear combination of image bases B with coefficients C',, and the corresponding representations are defined by the linear com- bination of the representation bases 2 with coefficients CV. As for the augmented images T. (X.), we parameterize its representation by creating new coefficient Cy Bilevel Optimization: The distilled data and its augmentations are used to train a feature extractor within the inner loop. In the outer loop, the feature extractor is encouraged to have a similar output as gø. Approximation: Approximation networks Qa are trained to predict the representation shift C - C" caused by the augmentation, and we get the approximate representation coefficient of the augmented image by summing the predicted shift (C*) and C". Reconstruct Distilled Pairs: For the future use, we can reconstruct the distilled data from image bases, representation bases, image coefficients, repre- sentation coefficients, and the approximation networks.
3

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Self-Supervised Learning.
While unsupervised learning refers to the general scheme of learning from data without human supervision, self-supervised learning (SSL), a type of unsupervised learning, uses tasks designed by the model itself to learn from the data. The learned features from SSL are usually good at generalizing and transferring to various downstream tasks (Ericsson et al. 2021). Early attempts of SSL often introduce pretext tasks of recognizing the transformation (e.g. image shuffling rotation) applied upon the input, effectively leading the model to acquire the capability of differentiating between images. In recent years, contrastive learning stands out particularly promising strategy of SSL. (Chen et al. 2020; Grill et al., 2020, Chen & He| 202T; [arrdes et al.; 2021] Zbonnar et al., 2021). where the model learns knowledge through distinguishing between similar (positive) and dissimilar (negative) pairs of data samples. For example. SimCLR (Chen et al. 2020) generates two versions of an image and teaches the model to bring similar pairs closer and push different pairs apart. However, it needs many negative (dissimilar) samples to improve accuracy. To circumvent this challenge, Barlow Twins (Zbontar et al.) (02T) offers a novel approach to SSL that eliminates the requirement for negative samples. It aims to reduce feature redundancy between differently augmented versions of the same image, aligning their correlation matrix with the identity matrix to ensure diverse and efficient representation learning. In this work, Barlow Twins is used as the SSL method to train the teacher model.
Dataset Distillation. The need of extensive training datasets for modern deep learning models has made training cost reduction a key research area. Dataset distillation (DD), initially introduced by /Wang et al. (2018), seeks to create compact informative samples that can effectively train deep learning image classification models. Within a constrained storage budget, it formulates DD as a bilevel meta learning problem where the outer loop optimizes the distilled images by minimizing the classification loss on real training data, and the inner loop trains a model on the distilled data. Subsequent studies have expanded on this concept by introducing I) various matching criteria, II) simplifying bilevel optimization, and III) image parameterization under a limited storage budget.
Matching criteria can be broadly classified into two categories. The first approach focuses on changes in the model's s parameters when trained on real data versus distilled data. Gradient match- ing techniques (Zhao & Bilen 2021b a) aim at mimicking the one-step gradient direction of ac- tual training images, but tend to falter in accuracy due to their short-term focus. In contrast, MTT (Cazenavette et al.) 2022) and DATM (Guo et al.] 2024). trajectory matching methods, en- hances performance by updating the distilled images to minimize the difference in terms of net- works' parameters (i.c. the network trained on distilled images versus the one trained on real data) over several training epochs, addressing the limitations of gradient matching by considering the longer-term training dynamics. The second category shifts focus towards the characteristics of data itself. These methods aim to ensure that the distilled images reflect the distribution of real training samples, with techniques like DM (Zhao & Bilen 2023) and IDM (Zhao et al. 2023) focusing on aligning the means of each class.
II) Another recent advancements have focused on streamlining the complex bilevel optimization as- sociated with DD, Approaches based on kernel methods (Nguyen et al. 2021 Zhou et al. 2022 Loo et al. 2022 2023) aim to derive a closed-form solution for optimization within the inner loop. Notably, FRePo Zhou et al. 2022) concentrates on training only the final layer of neural network to its convergence, maintaining static feature extractor. This methodology allows inner optimiza- tion to be efficiently represented as Kernel Ridge Regression (KRR) (Murphy 2012), significantly reducing computational demands.
III) Image parameterization (Kim et al. 2022. Deng Russakovsky 2022 Liu et al.) 2022 Wei
et al. 2023 Shin et al.) (2023) sceks for oineeiiensional space, aiming not only to compress the size of distilled images but also to narrow the optimization search space for these images. RTP (Deng & Russakovsky 2022) explores image factorization into shared bases, then recall these bases to synthesize the distilled image, thereby minimizing redundant information across different classes and enabling more efficient utilization of constrained storage capacities.
Beyond the supervised setting, DD in the self-supervised setting is under exploration. A recent publication, KRR-ST (Lee et al. 2024), introduces the pioneering framework for self-supervised
4

---

Published as a conference paper at ICLR 2025
DD tailored for transfer learning. This framework aims to distill a set of image and representation pairs from an unlabeled dataset, and the distilled pairs have the ability to train a new model that has similar representation ability what is trained the full unlabeled dataset. Moreover, they prove that the bilevel optimization procedure widely used in DD is incompatible with the random data augmentations, which play critical role in self-supervised learning. In detail, gradient of synthetic samples with respect to SSL objective in naive bilevel optimization is biased due to the randomness originating from data augmentations or masking. To eliminate gradient bias, the data augmentation technique cannot be involved in the bilevel optimization procedure. Hence, in the inner loop they train model (i.e. inner model) by minimizing the mean squared error (MSE) between predicted representations of the synthetic examples and their corresponding target feature representations without using any augmentation. To ensure that the model attains a similar rep- resentational capacity to the teacher model which self-suuperviselly pretrains on the full unlabeled dataset, they also introduce the MSE between representations of the inner model and the teacher model on the original full dataset for the optimization of outer loop. Finally, when other layers are fixed, optimization is restricted to the last layer of the inner model, which can be efficiently resolved by KRR (Murphy 2012), offering closed-form solution for the linear head. In this work, we focus on how to apply both image and representation parameterization in self-supervised DD, as well as releasing the constraints upon augmentations.
3 METHODOLOGY
3.1 PRELIMINARY
The problem of self-supervised dataset distillation for image dataset is the process to create the synthetic dataset X, [Ti,.... _mm= E Rmxxd which consists of m images with size d2 C X h x w into a vector with length dz, and vice versa) and Y, = [uI..... . Vm E xd, which have m target representations (corresponding to X2) with dimension dy. Such synthetic dataset preserves most of the information from the unlabeled dataset X, [æ1,.... Fpp Raxd, while keeping m is significantly less than the size of the unlabeled dataset n. The aim here is to expedite the pretraining phase of neural network with any architecture by employing the distilled dataset (X., Y.) as a substitute for the unlabeled dataset X, to perform model training.
Problem Definition.
x h x (for simplicity, we omit the process that reshape the image with size
To achieve this, teacher model go RRd, + Rdy with parameters & is self-supervised pretrained on the unlabeled dataset X,, where the teacher model de maps input samples to their dy dy-dim repre- sentations. Subsequently, a new model 9e Rdd + Rd with parameters 0 is trained on the distilled dataset (X,, Y.), and the output of ğe encouraged to mimic the output of teacher model Jo by adjusting (Xa,Ys). Yo). This process can be formulated as bilevel optimization:
min Louter xx, Φ), where 0* 9' arg min Limner (0: X,, X..Ya
where Cinner the objective used to train the model ğe. while Couter is the objective which encourages that the output distribution of gø- and the teacher model Js are close on all real data X,, The effectiveness of the distilled dataset (X,, Y,) is assessed by applying it to train random initialized neural network and evaluating this network to a range of downstream tasks through linear evaluation.
Kernel Ridge Regression on Self-Supervised Target (KRR-ST) (eee et al. 2024) KRR-ST repre- sents an self-supervised dataset distillation technique that employs bilevel optimizttion to distill the unlabeled dataset X,. In the context of the bilevel optimization framework, it is theoretically proven that random data augmentation biases the outer loss with respect to the distilled dataset (X., Y,). To avoid randomness, this approach suggests the exclusion of all data augmentations, leveraging the mean square error (MSE) within the bilevel optimization.
In the inner loop, the feature extractor ğo is trained to adapt to the distilled data (X,, Y.) by minimis- ing MSE between the predicted representation of distilled data go [õa (01) ) "o( and its corresponding target Y::
Linner (0; X,, Y.) = (X.) Y. = m i=1
(2:) - Uell 12 (ā0)
(2)
5

---

Published as a conference paper at ICLR 2025
In the outer loop, the goal is to optimize the distilled data (Xg, Y2), ensuring that the representation
upon real data extracted by the inner feature extractor could closely align with the one obtained by the teacher model Jø. This optimization problem is formulated the MSE between Go (X,) and
(x,) where Go (X,) (m)) .......... (=,)) (=.)))))))
Raxdy and go (Xx) (21), ..... 9o (x. )
Ruxdy, However, minimizing the MSE requires backpropagation through the entire inner loop, introducing significant computational and memory overhead. FRePo (Zhou et al., (022) suggests that the model Go can be seen as the composition of feature extractor fa and a linear head lww that is 9o = liw o Ju- and training only the linear head hw to convergence while keeping other layers (i.e. fw) fixed. This optimization can be efficiently solved with a closed form solution, known as kernel ridge regression (KRR) Murphy 2012). in which the outer loop objective becomes:
1 Couter(X, Yo: fw) (X)) ff (X) (Kx,,x.
+ (X.)),,.. A
(3)
where is a hyperparameter for regularization. Im E Rmx= is an identity matrix, Kx,X, (X,) Rmxm is the Gram matrix of all distilled samples, and fu (Xa)
[f far ("m))
Moreover, KRR-ST employs multiple models in inner optimization, enhancing robustness against overfitting compared to the use of single model (Cazenavette et al.) 2022 Zhou et al.) [002]] Zhao
et al. 2023). Building on this concept, KRR-ST introduces a model pool M, comprising feature extractors. To initialize each feature extractor ğo in the pool M, they first sample training step E
(1...., 3). where Z represents the maximum step count. Subsequently, @ optimized to minimize the MSE on X, and Ys, using gradient descent for steps. Afterward, the trained feature extractors and the their trained steps are stored in the model pool, denoted by M = {(āe, (()e,,))).. 21),..., (00z) } and feature extractor is random sampled from M at the start of each inner loop.
3.2 IMAGE AND REPRESENTATION PARAMETERIZATION
Dataset distillation seeks to encapsulate maximal information within a single distilled image. How- ever, image data frequently contains redundant elements or has regularity between adjacent pixels. For decades, the pursuit within the computer vision research community has been to identify effec- tive methods for dimensional reduction of images. A notable method, Eigenface (Turk & Pentland, 1991}. utilizes principal component analysis (PCA) to decompose human face images into set of basis vectors, known as eigenfaces. These eigenfaces have proven to valuable features for facial recognition and classification tasks. Furthermore, linear combination of eigenfaces can accurately represent a wide variety of human faces.
Inspired by this concept, our aim is to parameterize the distilled image X, into a set of bases 1 and a set of coefficients C", where the latter is used to linearly combine the bases to generate distilled images. The initialization of the bases comes from the first U largest principal components obtained by applying PCA upon the unlabeled images X,. Noting that in order to further reduce the dimension of the bases, we scale down the image size from dz scaling factor (following the practice in Kim et al. (2022), ). For the initialization of coefficients, we randomly select m images from the unlabeled dataset X denoted as X = [aq....., ""."" X,.
x h x w to d x & x where s is the
The initial coefficients are determined by projecting the images in X onto the U bases.
In detail, the parameterization of m distilled images X, Rmxd= is formulated as X, D(C" s5) where 8= = w-th largest principal component of the dataset X, for u = I,...,U, ..., C ====b =I [c.....,co. ] E R are coefficients with of being the weight of image x! which is projected onto the bases B= for i i= 1, m, and (.,s) 8) is the upsampling function that upsamples the image from de to dz.
E gu'd" are the image bases with by representing the
Additionally, the representation obtained via self-supervised learning methods suffers from dimen- sional collapse (Ting et al. 2022) Hua et al.) (202T), which means that the representation ends up spanning a low-dimensional subspace instead of the entire available embedding space. Following this idea, our goal is to build up the bases 8 of the feature representations for the distilled dataset, where we apply PCA upon real data representation go (Xt) and take the resultant first V largest principal component as the initialization for B" Regarding the initialization for corresponding co- efficients C' of Y, with respect to B", we also adopt the m images X (which are used to initialize C as described in the previous paragraph) followed by taking the weights of projecting go (X()
6

---

Published as a conference paper at ICLR 2025
onto the bases 8 as the initial CV. That is, Y, C" where C [c]..... T RRxV the coefficients and B "y....,b]] E Vxdy contains V bases with dimension dy-
3.3 PREDEFINED DATA AUGMENTATION AND FEATURE APPROXIMATION
Although KRR-ST Lee et al. 2024) points out that the "random" data augmentation would lead to gradient bias in bilelvel optimization thus abandoning the usage of augmentation, here we advance to relief such limitation upon data augmentation Basically, in order to circumvent the randomness typical augmentation operations, we predeter- mine all augmentations used in the distillation and record all the corresponding representations of the augmented images for future usage (i.e. the distilled dataset now supports the self-supervised learning approaches which adopt the objectives related to augmented images for training new model). Although such idea indeed improves the performance, storing all augmented results occu- pies lot of memory. To better reduce the memory usage. we propose approximation nefworks which are lightweight and learn to predict the representation shift from an unaugmented distilled image its augmented views, where in results we only need store the approximation networks and the representations of unaugmented distilled images.
its important role in self-supervised learning.
In detail, given A predetermined augmentations (Ti..... Tah the unaugmented distilled images X, are extended to be X, = (X., TI (X.). .)....... ..... TA(X)) E Rem(A+Ixdd,, and their corresponding representations Y, [Rm(A+1)xd,, are initialized via BCC (CH
(C* B²)} (following the similar idea as described in Section 3.2) where Cу E R = I,.... ,...,,,, A, the coeffi cient obtained by projecting the representation of distilled images under the a-th augmentation, i.c. (T(Ta(X)) onto the bases B" Furthermore, there are A approximation networks {Qi..... QA} in which the a-th network Qa RV + IR learns to estimate the shift in terms of representation caused by the a-th augmentation Ta- For instance, given unaugmented distilled image à; and its augmented view Ta (āt), the difference/shift terms of their corresponding representations is pre- dicted by Qa(c)) (c""), where of stands for the parameterized coefficient of ÿ; (i.e. the representation of T.))
3.4 OPTIMIZATION AND EVALUATION
The optimization of our proposed self-supervised dataset distillation framework basically follows the bilevel optimization scheme suggested by KRR-ST (Lee al. (024)) with several modifications to incorporate our techniques of image and representation parameterizations, data augmentation, and feature approximation networks:
1. Inner loop of bilevel optimization: The inner model Åo sampled from the model pool M is trained with minimizing the inner loss Limee (8; X., Y,) whose formulation is identical Equation 2 but takes (X,, Y.), i.e. distilled dataset gone through augmentations, as its training dataset.
2. Outer loop of bilevel optimization: The outer loop similarly minimizes the outer loss Lowter Xs, Ya; fw) as defined in Equation 3 while the gradients would be further back- propagated to the bases (B", B*) and the coefficients (C", CV,,Cy, Cy a 1...., A) in the image and representation spaces, due to our proposed parameterizations (e.g. X. D B" s) and Y. C² B"; with noting that the upsampling function is differen- tiable). Please note that our predetermined augmentations (TI,... .1.....TAl .A.} are all differen- tiable thus do not block the backpropagation through X, = =X.,, (X.,,..(......7x(()))
3. After bilevel optimization: Approximation networks (Qa a I,...,A) A) are optimized for minimizing the MSE between (C)) and Cy² C.
In results, our proposed method stores the image bases B", image coefficients C representation bases Bd, representation coefficients C and the approximation networks (Qi,..., Qa). Par- ticularly, our distilled dataset is constructed by X. {X., ..., (X.).... D B", s) and their corresponding representations y. (Y,, Y, + (C ....... Y, + Qx(C")B") with Y, = C" C", The pseudo-code summary as well as the implementation details of self-supervised dataset distillation are provided in Supplementary.
with X,
7

---

Published as a conference paper at ICLR 2025
As the goal of our distilled dataset (X,, (X)) is for further use of training a new model (basically feature extractor, trained to regress from X, Y.9) mimie the characteristics of the self- supervisedly pretrained teacher model Js. its evaluation follows the typical linear evaluation scheme of self- supervised learning works: the new model (i.e. feature extractor) learnt from (X,, Y:?) is frozen and coupled with a linear classifier, where the linear classifier is trained upon the supervised dataset of a downstream task. Higher performance on such downstream task indicates the supe- rior quality of the new feature extractor and consequently the better distilled quality of our dataset (X,, )
EXPERIMENTS
Datasets. CIFARI00 Krizhevsky 2009), TinylmageNer (Le & Yang 2015), and ImageNet (Deng al. 2009) are taken as our source datasets for performing self-supe DD, while the distilled dataset is evaluated upon the target datasets (which include the source datasets themselves, CI- FARIO (Krizhevsky) 2009), CUB2011 (Wah et al. 201T), and Stanford Dogs (Khosla et al. 1 for the classification). Noting that we align the image resolution in target datasets with the one in the source dataset (c.g. CIFARI00 uses 32> x 32, while both TinyImageNet and ImageNet use 64 x 64).
Baselines. Several baselines are adopted for making comparison with method: 1) Random ran- domly draws samples from the source dataset which are couple with their representations extracted by the teacher model to build the distilled dataset; 2) KMeans firstly performs kmeans clustering in the feature space of teacher model, where the centroids and their corresponding image samples con- struct the distilled dataset; 3) four representative approaches of supervised DD, including DSA (Zhao) & Bilen 20) DM (Zhao& & Bilen| 2023), IDM (Zhao et al. (023), and DATM (Guo et al. 2024); and 4) KRR-ST the state-of-the-art method (Lee al.] (024) of self-supervised dataset distillation.
Please note that for the baselines of supervised dataset distillation, the feature extractor used in the linear evaluation is trained by their supervised distilled dataset via cross-entropy loss.
Storage Budget. We follow the common practice of dataset distillation (Deng & Russakovsky 2022 Lee et al.] 2024) to adopt the entire memory consumption equivalent to storing N images (where the pixels are stored in floats) as the reference of storage budget. Noting that as our proposed method stores the image/representation bases and coefficients as well as approximation networks, we ensure that the size summation (in floats) of our tensors (for bases and coefficients) and network weights closely approximates the storage budget. Moreover, for supervised distillation baselines, N is the image per class (IPC) times the number of classes in source dataset.
4.1 EXPERIMENTAL RESULTS
Transfer Learning and Cross-Architecture Generalization. Our experiments investigate the adaptability of our method across datasets and architectures. We highlight the significance of cross- architecture evaluation in determining the quality of distilled datasets. In Table D we present complete comparison of the performance of our method against various baseline approaches across multiple target datasets and network architectures. Results report average accuracy and standard deviation (calculated over three runs). For each experiment, we select or distill samples from CI- FAR100, which serve the source dataset. The storage budget is N=100 images (or equivalent tensor/weight size), which is equivalent to IPC=1 in the conventional supervised dataset distillation setting. For distillation methods, we use 3-layer CNN to distill samples. These samples used to pretrain a feature extractor, including 3-layer CNN, VGG11 (Simonyan & Zisserman 2015).
ResNet18 (He et al., 2016). AlexNet (Krizhevsky et al. 2012). MobileNet (Howard et al. 2018
Vision Transformer (ViT) (Dosovitskiy et al.] (021). Then the feature extractor undergoes a lin- ear evaluation on target datasets like COFARTOO (Krizhevsky 2209). CIFARIO (Krizhessky) 2009).
CUB2011 (Wah et al. 2011), and Stanford Dogs (Khosla et 2011). The result shows that our method effectively distills CIFARI00 into a compact set, which allows to train a generalized feature extractor being compatible for many downstream tasks, where it outperforms all the baselines across various feature extractor architectures and target datasets. The results with adopting Tiny ImageNet and ImageNet as the source dataset are provided in Supplementary, in which sistent observation on our proposed method for providing superior cross- cross-architectur generalization and transfer learning performance (i.e. linear evaluation).
draw the con-
8

---

Published as a conference paper at ICLR 2025
The results on various target datasets and feature extractor architectures. We use 3-layer CNN to perform distillation from the CIFAR100 dataset with storage budge 100 images, then feature extractor are trained on the distilled dataset and linear evaluation is performed on target dataset. We report the average and standard deviation over three runs. The best results are bolded.
Table 1:
Target
Method
ConvNet
VGGII
ResNet18
AlexNet
MobileNet
ViT
Random
43.664055
23.76=0088
19.26*044
28.82
11.99
20.70+aa5
Kmcans
43394
25.13
19.051026
29.82
11.43
20.7710.2
DSA
39.38
19.97
20. +
31.57=4025
9.58++036
20.03+a19
DM
31.93+1.00
10.36=u774
16.24
20.49 =1.27
8.06+1100
NaN
CIFARI00
IDM
38.71
14.24
19.05 1000
33.71
8.18 811.46
17.4 17.411064
DATM
38.73
26.04+1.11
21.20++.44
29.31 +0.3
10.17 1..5
20. 1 +0.44
KRR-ST
47.00
27.78
18,92
31.27=0557
10.1 1 11:00
20.821:02
Ours
52.41
35.35
20.90
36.88
24.14+1.46
23.33+a06
Random
68.561013
50.231247
42.71
52.462004
34.95
45.021003
Kmeans
67.71
50,592108
42.34
53.75
33.53
45.05:000
DSA
65.67 +085
39.58+5.00
44.47 +0.2
54.21 1064
30.93+2.32
42,49+0.4
DM
56.27
28.37=2.m
42.71
12
32.57
NaN
CIFAR1O
IDM
64.45
34.43++29
44.77 4-026
57.71 +1225
31.01 +:158
39.34+1111
DATM
17.
46.952000
45,28
53.6920.5
29.05.0.54
43
KRR-ST
72.14+00.0
51.96+11.2
43.37 +0071
56.24
34.44+1..1
44.90+034
Ours
16
59.60
12
62.45=008
48.73
47.061036
Random
8.88
5.40
20.10
7.46-023
1.82
2.69
Kmeans
+.01
5.92+a05
2.76+0.16
7.61
2.04
2.73+
DSA
2.89
3.70=008
2.56+025
6.222028
1.57 10.00
2.59 1O
DM
4.84
1.74
2.09
2.93
1.25 +0.1
NaN
CUB2011
IDM
7.09
3.05
2.562011
7.07
1.39
2. 2.122025
DATM
6.73
5,88
2.64
6.70
1.32
2.47 +0.53
KRR-ST
10.432028
6.58
2.91
8.09
1.88
2.74
Ours 12.24:0..5
8.92
4.59
8.27=aax
4.37 10.19
3.99 10.07
Random
12.18
6.48
4.42+009
8.42.
2.82
3.75
Kmeans
12.362023
6.48=005
4.29
8.03
2.52
3.97
DSA
9.97
6.46
4.81
7.93
2.17
3 3.66+4..0
Stanford Dogs
DM
7.20+a.1
2.84
3.34
4.26=am1
2.17
NaN
IDM
9.5610.13
4.40
4.22
8.69 a.69
1.61 10.00
3.78
DATM
9.70+t21
6.32=aa5
5,00
7.65
1.90=a.0
3.86
KRR ST
13.42
7.84
10.2
8.42.
2.37
3.88 10.16
Ours
15.34102s
9.36
5.01
1.09
5,48
5.061024
Storage Budget Size. We conduct an experiment to check the impact under variant storage bud- gets. In this experiment, CIFAR100 is used as source dataset and target dataset simutaneously. For Random and KMeans baselines, we get the same numbers coreset images as the given budget. For distillation, we use 3-layer CNN find the distilled dataset from the given dataset within the storage budget. These coresets or distilled datasets are used to pretrain a 3-layer CNN, then linear evaluations for the pretrained CNN are conducted on the given dataset. As the result, Table shows the linear evaluation performance. The proposed method outperforms other baselines in various memory budgets. while two supervised methods, i.e. DSA and IDM, do not scale up as the budget size increases. For comparison, we pretrained the 3-layer CNN using the full CIFAR-100 training dataset as the upperbound for linear evaluation, achieving a result of 59.54%. Notably, supervised methods require memory budget of at least IPC.
Ablation Study for Our Proposed Methods. We conduct a study to evaluate the impact of key components, i.e. image and representation parameterization (cf. Section 3.2) well predefined augmentation and approximation networks (cf. Section 3.3). introduced in our methodology, where the results are provided in Table 3 Utilizing CIFAR 100 for both source and target datasets, our experiments are conducted under a constrained storage budget of N 100 in this study. The base- line for analysis is established by the KRR-ST (Lee et al. 2024) method, whose accuracy is
9

---

Published as a conference paper at ICLR 2025
Table Linear evaluation results on CIFARI00 with various memory budget N. In this experiment, the given dataset is serving as source and target dataset at the same time. We report the average and standard deviation on three runs.
Memory Budget N
25
50
100
1000
5000
Random
41.61
41.80
43.66=0.57
49.82:1022
52.7610..0
KMeans
39.964008
41.68
43.94
49.77
52.80
DSA DM
39.38
36.16+008
36.25 +-0.61
31.93=11066
34.96=292
38.76
IDM
38.71 =0070
37.21 +0,89
42. +
DATM
38.73
44.98
46. 46.231:00
KRR-ST
44.06:1041
45.79
=7.00
51.89
52.49
Ours
51.41
52.08+a.c
52.41 10.10
53.54
55.53+a04
Table 3: Ablation study on CIFAR100 with memory budget N=100. All our proposed tech- niques contribute to the improvement.
Table 4: Performance variation caused by ini- tialization for the parameterization (experiments conducted on CIFAR 100 with N=100).
Accuracy
Basis init. Coeff. init. Accuracy
Baseline (KRR-ST) 47.00+a51
I) Random Random 22.05
Parameterization 48.57+t.1
II) Real
Random 30.99+02s
Aug, & Approx. 52.41
III) PC Projection 52.41
47.00%. Upon integrating the "Image and Representation Parameterization" technique (+Parame- terization), we observe an improvement to 48.57% in accuracy. Further enhancements are achieved by incorporating "Augmentation and Feature Approximation" (+Aug. & Approx.), leading table improvement to 52.41 %. These results reveal that both our proposed techniques significantly contribute the performance of self-supervised dataset distillation.
Different Initialization Methods for Bases and Coefficients. In Section 3.2 and Section 3.3 we
have mentioned the way of initializing the bases and coefficients used in our proposed method. Here we conduct experiments to study the impact of different initializations upon the distillation perfor- mance. Basically, we test three methods, denoted as I). II), and III) respectively, to initialize the bases and coefficients for images and representations, where the experiments are carried out on CIFAR 000 which serves as both source and target datasets with a storage budget N 100. I) Both the bases and the coefficients are initialized randomly by standard Gaussian distribution. In this setting, the final distillation dataset only trains a feature extractor with linear evaluation precision 22.05 22.05%: II) The bases are simply initialized by random images drawn from the source dataset, and the coeffi- cients are initialized randomly by standard Gaussian distribution. This method obtains 30.09% accuracy; IIII Our proposed method utilizes principal components (denoted PC) of the source dataset to initialize the image bases, while the representation bases are initialized by the principal components of the source data representations extracted by the teacher model. It can get 52.41% accuracy. As in Table 4 this experiment shows that the initialization of bases and coefficients is critical, and different initialization methods would cause up to 30% loss on the performance.
5 CONCLUSION
Our work introduces an novel self-supervised dataset distillation method that reduces the size of training datasets while preserving model performance across various architectures. Our method leverages the parameterization of distilled images and representations into bases and coefficients, along with predefined augmentations to prevent gradient bias caused by the randomness in the bilevel optimization. Additionally, we employ approximation networks to capture the relationships between different augmentations, further reducing storage costs. Experiments demonstrate the superior linear evaluation results of the feature extractor pretrained on our distilled datasets across various target datasets and architectures, emphasizing the compactness and generalizability of our approach.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variime-iimariinccccovvrain regulariza- tion for sol--supervised learning, 2021.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML). 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Confer- on Computer Vision and Pattern Recognition (CVPR), 2021.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations (ICLR), 2020.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe. and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research (JMLR), pp. 1-6, 2022.
Deng, Dong, R. Socher, L.JJ Li, and L.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable mem- ories for neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR), 2021.
Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How Well Do Self-Supervised Models Transfer? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 21271-21288 2020.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via dufficulty-aligned trajectory matching. In International Conference on Learning Representations (ICLR), 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Andrew Howard. Andrey Zhmoginov, Liang- Chich Chen, Mark Sandler, and Menglong Zhu. In- verted residuals and linear bottlenecks: Mobile networks for classification, detection and segmen- tation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. On feature decorrelation in self-supervised learning. IEEE International Conference on Computer Vision (ICCV), 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.

---

Published as a conference paper at ICLR 2025
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. International Conference on Learning Representations (ICLR), 2022.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning (ICML), pp. 11102-11111 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances Neural Information Processing Systems (NeurIPS), 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. Technical report, CS231n, 2015.
Dong Bok Lee, Seanie Lee, Joonho Ko, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Self- supervised dataset distillation for transfer learning. In International Conference on Learning Representations (ICLR), 2024.
Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using ran- dom feature approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In International Conference Machine Learning (ICML), pp. 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations (ICLR), 2019.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012. ISBN 0262018020.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5186-1988 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
DongHyeok Shin, Seungjae Shin, and Il-chul Moon. Frequency Nomain--ased dataset distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Matthew A Turk and Alex P Pentland. Face recognition using eigenfaces. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1991.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The caltech-uesd birds-200 200-2000-200 200-2011 dataset, 2011.
12

---

Published as a conference paper at ICLR 2025
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. arXiv 2018.
preprint
Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma. Sparse parameterization for epitomic dataset distillation. In Advances in Neural Information Prucessing Systems (NeurIPS), 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (ICML), 2021.
International Conference on Machine Learning (ICML), pp. 12674-12685, 2021a. Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In
Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching. In International Confer- ence on Learning Representations (ICLR), 2021b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE Winter Con- ference on Applications of Computer Vision ( (WACV), pp. 5514-6523, 2023.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7856-7865, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
In adherence to the methodology described in KRR-ST (Lee et al., 2024), the inner model adopted in our approach utilizes convolutional layers that include batch normalization (loffe & Szegedy) 2015), ReLU activation, and average pooling. The number of layers of inner model is determined by the image resolution, adopting 3 layers for images sized 32 x 32 and 4 layers for 64 x 64 images. The model pool for inner models (cf. the last paragraph of Section 3.1 denoted as M, consists of 10 models, which are initialized and updated via full-batch gradient descent, with learning rate and momentum set to 0.1, 0.9, respectively. The update steps are 1, 1,000 To optimize our distilled dataset, we employ the Adam optimizer (Loshchilov & Hutter 2019). starting with a learning rate of 0.001 that linearly decayed. This distillation process involves 30, 000 outer iterations for CIFARI 100 and 20, 000 for TinylmageNet and ImageNet. The ResNet18 model (He et al.) (016)) serving as self-supervised teacher Io and is trained with the Barlow Twins objective (Zbontar et al., ) (where the training is based on the solo-learn library (da Costa al.) 2022)). Our settings of the number of image bases and the number of representation bases V are listed in the Table 5 Notably, the number of bases is generally set to be twice the memory budget, N, except in cases where this exceeds the dimensional limits of the images or representations. We set the size of image basis 3 16x 16 and the size of representation basis 512. We adopt bilinear upsampling as our upsampling function D(-) and and the upsampling scale is set to 2 for CIFAR 100 and 4 for TinylmageNet and ImageNet. We use image rotation as our augmentation function, which rotates the image by 90", 180 and 270". The approximation networks are designed as 2-layer perceptron, with the hidden layer sizes being for CIFAR100 and 16 for TinylmageNet and ImageNet.
Upon completion of the distillation process and stepping forward to evaluation, we pretrain a model (which acting as feature extractor later for linear evaluation) on the distilled dataset for 1, 000 epochs. This pretraining employs stochastic gradient descent (SGD) optimizer with a mini-batch size of 256, where the learning rate and momentum are maintained at 0.1 and 0.9, respectively. The weight decay parameters during pretraining the feature extractor are listed in Table 5 we set the weight decay parameters which depends on the size of distilled dataset. For training the linear classifier to conduct linear evaluation, we standardize the experimental settings to utilize the SGD optimizer with a momentum of 0.9, excluding weight decay, and initiate the learning rate of task- specific head to 0.2 with cosine scheduling.
13

---

Published as a conference paper at ICLR 2025
In Table 14 provide detailed list of all the notations/variiales used in our description and their corresponding meanings.
Table 5: Hyper-parameter configurations for our experiments. The storage budget is allocated to equivalentyy store N images, while the numbers of image bases and representation bases are set to U' and V, respectively.
Dataset
N
U
Weight Decay
CIFAR 100
25
50
50
0.001
CIFARTOM
100
100
0.001
CIFARI00
100
200
200
0.001
CIFARIO6
1000
500
500
0.0001
CIFAR 100
5000
700
500
0.0001
Tiny Imageeet
50
100
100
0.005
TinyImageNet
100
200
200
0.005
TinyImageNe
200
400
400
0,005
ImageNet
250
500
500
0.005
A.2 PSEUDO CODE
Here provide the pseudo code (i.e. Algorithm 1 together with the detailed but compact expla- nation to emphasize the systematic approach of our proposed method for self-supervised dataset distillation, which begins with initializing the framework and proceeds through bilevel optimiza- tion process, ending with the training of approximation networks to capture representation shifts due to the augmentations (e.g. rotations). Such pseudo code offers clear understanding of our method's structure and objectives.
The initialization phase, from Line to Line 6, starts with training teacher model using self- supervised learning (SSL) objective on an unlabeled dataset (i.e. the source dataset that we are about to distill). It involves setting up the bases and coefficients for the image and representation, following the guidelines detailed in Section 3.2 This phase also includes the initialization pool of feature extractor models, each associated with specific training iteration drawn from predefined range, and the models are trained on the initial distilled images and their corresponding targets through the process defined in Algorithm
The core of the Algorithm I spanning from Lines 7 to 20, engages in bilevel optimization. It involves iteratively refining the distilled images and representations by adjusting their bases and coefficients. First, it randomly selects model from the pool to update the bases and coefficients in order to minimize the outer loss, as shown in Equation 3 This step also includes an inner loop where the selected model is further trained using Equation if its assigned steps are below the maximum limit; otherwise, it is reinitialized.
In the final stage (from Lines 21 to 23), the algorithm addresses the shifts in representations sulting from our predefined augmentations. This is achieved by training approximation networks to minimize the mean squared error (MSE) between the predicted coefficient shifts and the differ- ence coefficients caused by augmentations. The distilled dataset is thus represented by the bases, coefficients, and trained approximation networks.
A.3 TRANSFER LEARNING AND CROSS- ARCHITECTURE GENERALIZATION ON TINTTMAGENET
Our experiments explore the versatility of our method across different datasets and architectures. We emphasize the importance of evaluating across architectures to assess the efficacy of distilled datasets. In Table 6 we offer comprehensive comparison of performance against several base- line approaches across a variety of target datasets and network architectures. The results include select or
average accuracy and standard deviation, based three runs. For each experiment, distill samples from TinylmageNG which serve as the source dataset. The storage budget is 2=200 images (or equivalent tensor/weight size), which corresponds to Image Per Class (IPC) of 1 as
14

---

Published as a conference paper at ICLR 2025
Algorithm The proposed self-supervised dataset distillation
Input: Unlabeled dataset X,, the number of image bases U, the number of representation bases
V. upsampling function D, scaling factor s, predefined augmentations (Ta 14 total steps Z
1: Optimize teacher model Io with the SSL objective on Xt-
2: Initialize bases and coefficients B², 13, C*, C²2 {CX) J0= (c.f. Section 3.2
3: Get X,, Y, from Algorithm 2
4: Randomly initialize models ge, and integers {l...., 2} for I I.....L
5: Train model Gø, for z steps on X, and Y, for I...,.. L.
6: Initialize model pool M +
(00l z1)).
7: for each distillation step do
8: Get X,, Y, from Algorithm
2
9: Sample a model Ôe, = hw, o fu,, and its trained steps 2j from M.
Compute the outer objective Lomer Xx., Ys; fay) using Equation 3 A. A.
11: Get the gradient V Louwer w.r.t. B", By C C² and C for Update bases and coefficients, D= By, C C and C for
13:
if 2j Z then
Set 2j f +1
15:
Evaluate inner loss Cimne((f: X,, using Equation 2 Update 0, according to V Cinner (0y; X., Y.).
16:
else
18:
Reset 2j +- O and randomly initialize 9
end
20:
end for
21: Randomly initialize approximation networks {Q) (Q1..... Qa}
22: Get the representation shift, C C for 1, ................................ A
23: Train Qa by minimizing MSE between (C and C - C for 1....,A
Output: distilled data 8", B C C² , (Q..........................._.. QA
Algorithm 2 Generate distilled images and target representations
Input: image bases 8 image coefficients C", representation bases B", representation coeffi-
cients C* upsampling function D, scaling factor predefined augmentations )4
1: Generate distilled images X, D(C B s).
2:
Generate augmented images X, (X,, ..., TA(aa))
Generate target representations Y, ((C*B") (C" (",,B,,,,,,,,,))
Output: X., Y..
15

---

Published as a conference paper at ICLR 2025
typically defined in the supervised dataset distillation setting. We employ 4-layer Convolutional Neural Network for the distillation process. These distilled samples are then utilized to pretrain a feature extractor, which includes architectures such as 4-layer CNN, VGG11 (Simonyan & Zisser-
man 2013), ResNer18 (He et al. 2016). AlexNet (Krizhevsky et al. 2012), and MobileNet (Howard perform linear evaluation assess performance of the feature tractor on target datasets like TinyImageNet (Le & Yang 2015). CUB2011 I (Wah et al. (011)) and Stanford Dogs (Khosla et al. (011). The findings demonstrate that our method efficiently condenses TinyImageNet into a compact distilled dataset. This enables the training of a versatile feature extrac- tor that performs better than baselines in various downstream tasks across various feature extractor architectures and target datasets.
al. 2018). Subsequently,
Table 6: The results various target datasets and feature extractor architectures. We 4-layer CNN to perform distillation from the TinylmageNe dataset with storage budget 200 images, then feature extractor are trained on the distilled dataset and linear evaluation performed target dataset. We report the average and standard deviation over three runs. The best results are bolded.
Target
Method
ConvNet
VGGII
ResNet18
AlexNet
MobileNet
Random
24.44
11.71
14.11 1071
9.84
6,51 6.51200 103
TinylmageNet
KRR-ST
2044 29.54
14.66+1.17
14.78
10.28
6.194
Ours
25.12
16.75
23.28
14.02=0029
Random
9
7.95
3.25
5.02+1.22
2.1 2.17
CUB2011
KRR-ST
11.59
8.27
3.70=a.17
5.24
2.17
Ours
11.11:0026
10.58
5.17:1.119
9.78
5.78
Random
12.38
7.41
4.79
5.99
2.97
Stanford
KRR-ST
14.52
8.79+1.0
4.61 =0.29
5.49
2.48
Dogs
Ours
14.62
12.29
6.60=0.06
11.30
7.21
A.4 VARY THE STORAGE BUDGET ON TINYIMAGENET
We examine the impact of different storage budgets. this experiment, TinyImageNet serves as both the source and target dataset. For the Random and KMeans baselines, we select the same number of coreset images as allowed by the given budget. For the distillation process, 4-layer CNN is employed extract distilled dataset from the original data within the specified storage budget. These coresets or distilled datasets are then used to pretrain a 4-layer CNN, followed by linear evaluations the dataset. As shown in Table 7 the lincar evaluation results indicate that our proposed method surpasses other baselines across various memory budgets.
Table 7: Linear evaluation results TinylmageNet (Le & Yang 2015 with various memory budget N. this experiment, the given dataset serving as source and target dataset at the same time. We report the average and standard deviation three runs.
Memory Budget N
50
100
200
Random
22.431054
22.73
23.95
KMeans
23.17
23.96
25.03
KRR-ST
25.294a30
25.64
27.23
Ours
28.03
29.35
31.25
A.5 TRANSFER LEARNING AND CROSS-ARCHITECCURR GENERALIZATION ON IMAGENET
In this experiment, ImageNet is used the source dataset, with storage budget of N = 250 images (or an equivalent tensor/weight size). For the Random baseline, we select the same number of images permitted by the budget. Following the settings of KRR-ST (Lee et al.] 2024 a 4-yer CNN is used during the distillation process to generate distilled dataset with a resolution 64 64 within the specified storage constraints. These coresets distilled datasets are then used pretrain
16

---

Published as a conference paper at ICLR 2025
feature extractor with architectures such as 4-layer CNN, VGGI1 (Simonyan & Zisserman 8 (He et al. 2016), AlexNet (Krizhevsky et al. 2012), and MobileNet (Howard et al. 2018 Linear evaluations subsequently performed target datasets like TinyImageNet (Le & Yang 2013), CUB2011 (Wah et al. 2011), and Stanford Dogs (Khosla et al. 201T). J). As shown in Table the linear evaluation results demonstrate that proposed method consistently outperforms other baselines in various downstream tasks across different backbone architectures.
2015
Table 8: The linear evaluation results on various target datasets and feature extractor architectures. We use 4-layer CNN to perform distillation from the ImageNet (Deng al., 2009) dataset with storage budge - images, then feature extractors are trained on the distilled dataset and linear evaluation performed on various target dataset. The best results bolded.
ConvNet
VGGII
ResNet18
AlexNet
Mobilenet
Random KRR-ST
13.65
6.59
5.79
5.87
2.45
ImageNet
16.75
10.22
6.19
6.81
3.18
Ours
21.17
14.8
8.04
11.31
10.72
Random
25.44
10.51
13.84
9.12
7.28
TinylmageNe
KRR-ST
29.17
19.03
13.74
8.71
7.84
Ours
33.04
27.23
18.89
18.83
22.35
Random
10.65
7.56
3.61
4.90
2.66
CUB2011
KRR-ST
12.32
10.44
3.78
4.76
2.38
Ours
12.08
11.70
6.04
10.87
9.10
Random
13.16
7.04
4.97
6.27
2.93
Stanford Dogs
KRR-ST
15.90
10.62
4.86
5.87
3.67
Ours
17.2
12.80
7.83
11.74
10.84
A.6 RESULTS ON IMAGENET WITH LARGER IMAGE RESOLUTIONS
ImageNette (128 x 128). We evaluated our method on hegleroresolution data using ImageNette, 10-class subset of ImageNet, as both the source and target dataset. We the resolution 128 x 128 and used storage budget of 10 images. For the distillation process, we employed a 5-layer CNN to generate distilled samples. These samples were then used to pretrain a randomly initialized 5-layer CNN feature extractor, and we report the linear evaluation accuracy this feature extractor. As shown in Table our method scales effectively to larger resolutions.
resolution of 224x224 with storage budget of 100 images. For distillation process, we use a 6-layer CNN to create distilled samples. These samples
ImageNet-1K (224 x 224). We further tested our method ImageNet-IK
used to pretrain random initialized 6-layer CNN feature extractor, and we report the linear evaluation accuracy which is conducted on this feature extractor. The results, showed in Table 10 demonstrate that our method surpass baseline methods on larger scale datasets.
Table 9: Linear evaluation accuracy on a layer ConvNet pretrained with 10 distilled Im- ageNette images (128x 128).
Table 10: Linear evaluation accuracy on
6-layer ConvNet pretrained with 100 distilled ImageNet- images (224x224).
Method Accuracy
Method
Accuracy
Random KRR-ST
46.93
Random 8.75
50.14 14
KRR-ST
9.22
Ours 59.31
Ours 9.60
A.7 MODELING THE REPRESENTATION SHIFT CAUSED BY AUGMENTATION
In this study, we explore the impact of different methods in predicting the representation shift through experiment. Using CIFAR 100 (Krizhevsky 2009) both the source and target dataset
17

---

Published as a conference paper at ICLR 2025
with storage budget N = 100, examine three distinct scenarios to model this phenomenon. The first scenario, termed "Same." involves treating the representation of all augmented images as iden- tical, technique commonly employed in self-supervised learning. This approach typically regards augmented views generated from single image as a positive pair, aiming to align their represen- tations closely. The second scenario, "Bias," presupposes that the augmented view diverges from the original view by a specific bias vector, with each predefined augmentation associated with its unique bias vector. The third scenario, "Ours," " is described in our main paper. It adopts approxi- mation networks to predict the representation shifts, and the subindex 2, 4, and 8 indicate the width of the hidden layer in the approximation networks. Last, "Ideal", which serves as an upperbound in this experiment, represents storing all of the representation without considering the storage budget. According to the results presented Table [1] including linear evaluation accuracy of the feature extractor trained on distilled data and prediction mean square error (MSE), the proposed lightweight approximation networks can achieve better accuracy than "Same" and "Bias" baselines. Notably, while larger networks achieve lower MSE, they do not always improve accuracy due to the storage budget constraint. These findings indicate that the proposed design can effectively predict represen- tation shifts caused by augmentations, achieving reasonable trade-off between MSE and accuracy.
Table 11: Ablation study on modeling the target of augmented images. The result shows the linear evaluation accuracy which is conducted on a 3-layer ConvNet. It is pretrained on the distilled CI- FARI00 with N=100.
Method
Same
Bias
Ours2
Ours
Oursg
Ideal
Accuracy MSE
50.10
50.32
52.30
52.41
51.19
53.51
0.31
0.30
0.07
0.06
0.04
A.8 AUGMENTATION.
Our investigation delves into the effects of various predefined augmentations applied within our proposed method, utilizing the CIFAR 100 both source and target dataset. We specifically explore three distinct augmentations: rotation, jigsaw, and crop, each of which is differentiable and yields predetermined (i.e., non-random) result. For the rotation augmentation, we subject an image to rotations of 90º, 180%, and 270 The jigsaw augmentation involves dividing an image into four patches. We then perform augmentation by swapping the patches in three ways: left with right, and top with bottom, as well combination of both swaps. In the crop augmentation process, an image is cropped into its four corners and central portion, with each cropped area being 20 x 20 pixels. These cropped sections are subsequently resized to the original resolution of the images. The result of the experiment shown Table 12, all them are better than no augmentation, indicating that our "Predefined Data Augmentation and Feature Approximation" is not sensitive to the choice of augmentation, while adopting rotation augmentations achieves the best.
Table 12: Study on choices of predefined augmentations (dataset: CIFARI00; -=100).
No augmentation
Roatation
Jigsaw
Crop
48.57
52.41 +0.0 50.934o21
51.03+0.00
A.9 VISUALIZATION
We conduct a qualitative analysis of our outcomes by distilling CIFAR100 and TinylmageNet within a storage budget = 100 and 200, respectively. First, we show the bases which are initialized by the first 64 largest principal components in Figure 2a To illustrate the distilled images, we combine the image bases with their coefficients into distilled images and present a subset of these images, as depicted in Figure 2b Additionally, we conduct comparative analysis between the CIFAR100 representations extracted by the teacher model and those obtained from our distillation process, which are parameterized by the combination of representation bases and coefficients. For
18

---

Published as a conference paper at ICLR 2025
this purpose, employ the UMAP method, detailed in (McInnes et al. 2018 project both the CIFARIOU 100 representations and the distilled target representations into din-dimensional vectors. The resultant visualization is shown in Figure 2c, where similar distribution as real dataset, indicating that the distilled data captures the characteristics of the real dataset. Finally, we perform the same visualization TinylmageNet, and Figures Ba 3b and show similar visualization results.
that the distilled data have the
Roal Data
Distiled Data
(a) CIFARI00 bases
(b) CIFAR images
(c) CIFARIO0 UAMP
Figure 2: Visualization result of CIFARIO0 (N 100)
Real Data Ditilled Data
(a) TinyImageNet hases
(b) TinylmageNet images
(c) TinyImageNet UAMP
Figure 3: Visualization result TinylmageNet (N -200)
A.10 COMPUTATIONAL COST
Based on CIFAR-106 100 with storage buffer N 100, we evaluated GPU memory usage and execu- tion time. The results demonstrate that while our method requires approximately 1.5x more GPU memory than KRR-ST. ST. it remains feasible within modern hardware constraints (c.g.. NVIDIA RTX 4090 with 24GB memory). Furthermore, the execution time of our method is comparable to other baselines, as shown in Table B
Table 13: Computational cost of distilling CIFAR-100 with storage buffer N 100 using a single Nvidia RTX 4090 GPU card.
Target
DSA
DM
IDM
DATM
KRR-ST
Ours
GPU memory (MB)
3561
2571
2407
4351
4483
6917
Time (mins)
81
78
313
121
189
205
19

---

Meaning Published as a conference paper at ICLR 2025
Variable
the parameters (w, W) of model Óe hw o fw the a-th predetermined augmentation the index of augmentation the number of augmentations
To
A
B
[bf....,00 by T E Uxx image bases
B
[b.....,bby by IR Vxd, representation bases
number of channels
O
RmxU coefficient for generating distilled images
C
RTXV coefficient for generating distilled representation
CY
RmxV for a 1, , A corresponding the target representation of Ta (X₂)
d
c h x w the dimension of the sample e x A - the dimension of image basis the dimension of representation y/ feature extractor with parameter w teacher model
d
f
9
the model used mimic the teacher model (usually smaller than g)
ga-
the model used mimic the teacher model with optimized parameters height of image
h
hw
linear head with parameter W
K X,,x,
Gram matrix of distilled samples X,
the index of model pool
L
the size of model pool
M
( 21),... (ğæz =r)} model pool the number of diltilled data
the number of real data total storage budget
N
Qa
approximation network corresponding to a-th augmentation the scaling factor of the image bases the number of image bases
U
the number of representation bases width of image
Ta
real image
iz
distilled image
X.
E mmxd the set of distilled images [ā1,... F'm
X,
[r1.-.., mx E RRndx the set of real images
X
random sample images from X;
X.
augmented distilled images target representation for distilled image ä; [ū....., 3m.] E IR xd,, the set of target representations
Y.
the target representation for X,
the target representation which approximates by network (Qa for X.
21
the trained steps of model ğo
Z
the maximum steps to update the feature extractor
Table 14: The List of Mathematical Notations
20


Reviews:
[{"review_id": "oWYK9EK4k6", "reviewer": "Reviewer_TeEU", "summary": "This work targets the cross architecture generalizability challenge in dataset distillation. When performing distillation, the data is often biased to the model used in the distillation process -- in this work the proposed self-supervised approach parameterizes the representations of images while studying/leveraging the effects of augmentations. This approach features a 5 stage method involving pertaining a network on the source dataset, followed by image parameterization (encoding the images and augmentations via low-dimensional bases vectors), bi-level optimization on the images, approximation to handle the distribution/representation shift, and reconstruction of the images using the bases and learned features. The method reports strong performance improvement on a variety of datasets against most of the current SOTA methods.", "questions": "I've highlighted a few of the issues/suggestions for the Authors to consider in the rebuttal phase above in the Weaknesses Section. These are crucial in determining the significance of the work and wide scale adoption.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The key strengths of this paper include:\n\n1. More diverse datasets: Not many dataset distillation papers venture beyond the CIFAR/ImageNet datasets, however these authors included results on CUB2011 and StanfordDogs. Additionally, the ViT performance has been reported, and overall it appears that the authors performance improvement is maintained on Transformer architectures, albeit smaller.\n\n2. The basis and coefficient initialization ablation provides interesting insight into the sensitivity of the proposed framework.\n\n3. Personally, I found the use of the approximation networks to be a clever solution to reducing memory usage while preserving the essence of image augmentation. By learning a mapping between and subsequently the shift in distribution of the unaugmented distilled representation into it's augmented views, one can efficiently store simply the network rather than all the augmented views.\n\n4. Strong baselines: This work accurately surveyed some of the most seminal and current SOTA in the field of dataset distillation (with the exception of a few missing citations that should be added). I find the included competitive methods to be comprehensive enough to support the statements however, further comments on the benchmarking are included in the Weaknesses section.", "weaknesses": "Despite the interesting approach taken in this work, I find a few crucial weaknesses:\n\n1. I find that the experimental support is a bit lacking. As is common in Dataset Distillation works, it is generally good practice to show the scaling over different memory budges (N) on various datasets, rather than just a single dataset, in order to show generalizability.\n2. I noticed that the resolutions on ImageNet scale to 64 x 64 -- however recently, the field has shifted to higher resolutions such as 128x128 or even 512 x 512 -- I think it would be important to see if the method can scale well to larger resolutions.\n3. I think another important criteria that should be included is Applications -- as alluded to in the paper tasks like continual learning or neural architecture search (line 43) are important in the field, however none of these results were included in the main paper -- I think it is important to test the applicability of the method in order to determine significance and impact.\n4. Given that this approach involves multi-level optimization, I think efficiency metrics should be compared as well (time per step, GPU memory etc). -- This will demonstrate wether the gain in performance is justified over other methods when comparing the relative compute demands.\n\n[Minor] Some missing citations including DataDAM (ICCV'23), CAFE (CVPR'22)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DfcqKFCOdl", "reviewer": "Reviewer_MMPZ", "summary": "This paper proposes a novel approach to self-supervised dataset distillation aimed at reducing training costs by creating compact datasets that maintain model performance. This method, intended to address challenges in self-supervised learning (SSL) for dataset distillation, introduces three key contributions: 1. Parameterization 2. Predefined Augmentation  and feature approximation 3. Optimizations with approximation Networks. Generally they have shown a very contributing method.  \n\nThe paper introduces a solid contribution to self-supervised dataset distillation, with innovative approaches to parameterization, augmentation handling, and memory efficiency with upgraded existing method named as KRR-ST. While the approach is complex, it provides a promising direction for reducing training costs in SSL, particularly in resource-limited settings. With further optimization and extension to diverse tasks, this method has the potential to make dataset distillation more accessible and applicable in real-world scenarios.", "questions": "1. The datasets in the experiments are CIFAR 100 and datasets with similar image attributes. I can understand it is possible to get a distilled dataset in a lab environment and the datasets are very feature-controllable. Do you have space to show that your experiment can be successful in other different scenarios? For example, some randomly taken images. \n\n2. Though this is a memory saving method, a very large portion of the whole method is still computing intensive. Do you have any benchmark to show that the whole method could be executed in an efficient way?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper demonstrated a very strategic parameterization.\nThe use of bases for image and representation parameterization is a sophisticated approach to compress dataset information without sacrificing accuracy. This addresses both storage efficiency and computational cost.\n\n 2.Effective Augmentation Handling:\nBy predefining augmentations, the method successfully mitigates the bias introduced by random augmentations, a notable challenge in SSL distillation methods.\n\n3. Improved Memory Efficiency:\nThe inclusion of approximation networks to predict representation shifts from unaugmented to augmented views significantly reduces memory usage by eliminating the need to store augmented representations. This makes the approach more scalable.\n\n4. Transfer Learning Potential:\n\nThe method shows strong transferability to downstream tasks, making it particularly appealing for real-world applications where labeled data is scarce, and transfer learning is critical.\n\n5. Ablation Studies and Hyperparameter Analysis:\n\nThe paper includes ablation studies that isolate the contributions of parameterization, augmentation, and approximation networks, offering clear insights into each component's impact on performance.", "weaknesses": "1. Complexity and accessibility\nCritique: The method involves several sophisticated techniques, including low-dimensional basis parameterization, predefined augmentations, and approximation networks. This complexity may make it difficult for practitioners to implement and tune the method without extensive expertise in self-supervised learning and dataset distillation.\n\n2.  Computational and memory trade-Offs\nCritique: While the method claims to be memory-efficient due to approximation networks, the additional computational overhead introduced by these networks might reduce the method’s overall efficiency, especially in resource-constrained environments.\n\n3. Dependence on Synthetic Data for Evaluation:\nThe experiments rely heavily on benchmark datasets like CIFAR100. However, these datasets have well-structured labels and relatively consistent image quality, which may not fully represent real-world data variability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wdc9c9BCgT", "reviewer": "Reviewer_u46N", "summary": "This paper proposes a self-supervised data distillation method based on image decomposition. By initializing with principal components and learning the impact of data augmentation, the performance of the distilled dataset is enhanced. The experiments provide a comprehensive analysis of the method’s effectiveness.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The topic is both valuable and practical, especially in the era of large datasets. While most current research on data distillation focuses primarily on classification tasks, which may be too narrow, this work seeks to improve self-supervised tasks. This approach is more general and can better support feature learning for downstream applications.\n\n2. The paper is well-written and easy to follow, with a straightforward method that is simple to understand. For each component, the authors clearly explain the rationale behind its inclusion.\n\n3. The experiments demonstrate the method’s effectiveness, as it consistently outperforms baseline methods in both transfer learning and linear probing tasks.", "weaknesses": "I did not find any major weaknesses in this paper. However, there are some concerns regarding its novelty. The techniques employed are largely derived from previous work on data distillation for classification tasks. It would be helpful if the authors could clarify what unique challenges exist for self-supervised data distillation and how their method specifically addresses those challenges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QH87LgvoOp", "reviewer": "Reviewer_Q6us", "summary": "In this paper, the authors propose a method for dataset distillation based on KRR-ST. Two techniques are introduced: (1) PCA-based dimensionality reduction, which transforms images and their representations into lower-dimensional bases and coefficients; and (2) Data Augmentation, which employs predefined data augmentations and approximation networks to address the limitation of KRR-ST in utilizing data augmentation during dataset distillation. The authors conduct an extensive experimental evaluation and demonstrate significant improvements over previous baselines.", "questions": "1. Could the authors provide more details about the approximation networks, such as the number of networks used, structure, and layers?\n2. Could the authors show a comparison of the distilled data sizes?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Reducing data size is a critical direction in self-supervised learning research.\n2. Fixing the issue of incorporating data augmentation into data distillation is important, as it significantly improves performance.\n3. The authors conduct a wide range of experiments, evaluating model performance with various network architectures and different numbers of training examples.", "weaknesses": "1. The proposed techniques in the paper are not new, such as PCA and augmentation approximation networks.\n2. The proposed technique leverages data augmentation while minimizing bias, and similar ideas have been explored in self-supervised learning. It is important to cmopare it with other analogous methods [1][2][3].\n\n[1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision. NeurIPS 2021\n[2] Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration. NeurIPS 2021\n[3] RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning. NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)