Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks
Authors: Siddharth Joshi
Jiayi Ni
Baharan Mirzasoleiman
Venue: ICLR 2025 Poster (2025)
Keywords: dataset distillation
self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We present the first effective method for dataset distillation (i.e. creating a small synthetic dataset to summarize a large real dataset) for self-supervised learning.

Abstract: Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. Despite the success of DD methods for supervised learning, DD for self-supervised pre-training of deep models has remained unaddressed. Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. In this work, we propose the first effective DD method for SSL pre-training. First, we show, theoretically and empirically, that naiive application of supervised DD methods to SSL fails, due to the high variance of the SSL gradient. Then, we address this issue by relying on insights from knowledge distillation (KD) literature. Specifically, we train a small student model to match the representations of a larger teacher model trained with SSL. Then, we generate a small synthetic dataset by matching the training trajectories of the student models. As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data. Code at https://github.com/BigML-CS-UCLA/MKDT.

Published as a conference paper at ICLR 2025
DATASET DISTILLATION VIA KNOWLEDGE DISTILLA- TION: TOWARDS EFFICIENT SELF-SUPERVISEE PRE- TRAINING OF DEEP NETWORKS
Siddharth Joshi, Jiayi Ni, Baharan Mirzasoleiman Department of Computer Science, University of California, Los Angeles s sjosh18048cs www..cllae......o.... edu, ni mijjayy11196225g ucl. edu, baharan cs ucla.
ABSTRACT
Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. Despite the success of DD methods for supervised learning, DD for self-supervise pre-training of deep models has remained unaddressed. Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. In this work, we propose the first effective DD method for SSL pre-training. First, we show, theoretically and empirically, that naïve application of supervised DD methods SSL fails, due to the high variance of the SSL gradient. Then, we address this issue by relying insights from knowledge distillation (KD) literature. Specifically, we train small student model to match the representations of a larger teacher model trained with SSL. Then, we generate small synthetic dataset by matching the training trajectories of the student models. As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. Through extensive experiments, we show that our distilled sets lead up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data. Code at httpps://ghht thupp://///////....o//pppppp thub. com/Rooo///ooooo gML- CS--------- CS-UCLA/MKD"
INTRODUCTION
Dataset distillation (DD) aims generate a very small set of synthetic images that can simulate train- ing on a large image dataset, with extremely limited memory and compute (Wang et al. (018)) This facilitates training models on the edge, speeds up continual learning, and provides strong privacy guar- antees (Kim et al. 2022 Cazenavette et al., 2022, Sajedi et al. 2023 Dong et al. 2022 As result, there has been a surge of interest in developing better DD methods for training neural networks in su- pervised manner. However, in many applications, very few labeled example are available. In this case, supervised models often fail to generalize well. Instead, models are pre-trained, using self-supervised learning (SSL), on a large amount of unlabeled data and then adapted to the downstream task using the limited labeled data by training a linear classifier using the labeled examples of each downstream task (linear probe). Remarkably, Chen al. (2020) showed that SSL pre- pre-training. followed by linear probe, can outperform Supervised Learning (SL) by nearly 30% on ImageNet (Deng et al. 2009) when only 1 % of labels available. More impressively, by only training the linear layer (linear probe), SSL pre- training is able to generalize to variety of downstream tasks nearly as effectively as full fine-tuning on the downstream task, for fraction of the cost. Thus, SSL pre-training's 's benefits are invaluable in today's modern MI. ecosystem, where unlabeled data is plentiful and it is essential to generalize to a plethora of downstream tasks, effectively and efficiently. With the datasets for SSL. being far larger than those for SL, the computational and privacy benefits of dataset distillation for SSL are important than they are for SL. Nevertheless, DD for SSL pre- mee-triinng has remained an open problem.
more
DD for SSL is, however, very challenging. One needs to ensure that pre-training on the synthetic dataset, distilled from unlabeled data, results in a encoder that yields high-quality representations for variety of downstream tasks. Existing DD methods for SL generate synthetic data by matching
gradients (Zhao et al. 2021: Zhao & Bilen 2021 Lee et al., (022) or trajectory (Cazenavette et al.
| Cui et al. 2022) of training distribution of real data (Zhao & Bilen 2023
2022 Du al. 2023

---

Published as a conference paper at ICLR 2025
Sajedi al. 2023: Wang et al., 2022a), meta-model matching by generating synthetic data such that training on it achieves low loss on real data (Wang et al.) 2018 Nguyen et al. 2021 Loo et al.) 2022 Zhou al. 2022). Among them, gradient and distribution matching methods heavily rely on labels and will suffer from representation collapse otherwise. Hence, they are not applicable to SSL DD. Very recently, [Lee al. (2023) applied meta-model matching generate synthetic examples for SSL pre-training, and evaluated its performance by fine-tuning the entire pre-trained model on the large labeled downstream datasets. However, we show that SSL pre-training on these distilled sets does not provide any advantage over SSL pre-training on random real examples.
In this work, we address distilling small synthetic datasets for SSL pae--rrreeing via trajectory matching. First, we show, theoretically and empirically, that naive application of trajectory matching to SSL fails, due to the high variance of the gradient of the SSL. loss. Then, we rely on insights from knowledge distillation (KD) to considerably reduce the variance of SSL trajectories. KD trains smaller student network to match the predictions of a larger teacher network trained with supervised learning (Hinton) al. 2015 In doing so, the student network can match the performance of the larger teacher model.
Here, we apply KD for SSL by training a student encoder to match the representations teacher encoder trained with SSL. Then, we propose generating synthetic data for SSL by Match- ing Knowledge Distillation Trajectories (MKDT). Crucially, as the KD objective for training the student model has considerably lower variance, it enables generating higher-quality synthetic data by matching the lower-variance trajectories of the student model. As a result, the encoder can learn high-quality representations from the synthetic data. We also provide theoretical and empirical evidence showing that KD trajectories are indeed lower variance than SSL trajectories and that this lower variance enables effective dataset distillation for SSL.
a larger
Finally, we conduct extensive experiments to validate the effectiveness of our proposal MKDT for SSL pre-training. In particular, we distill both low resolution (CIFARIO, CIFAR100) and larger, high resolution datasets (Tiny ImageNe)) down to 2% and 5% of original dataset size and show that, across various downstream tasks, MKDT distilled sets outperform all baselines by up to 13% in the presence of limited labeled data. Moreover, we confirm that the datasets distilled with smaller ConvNets can transfer to architectures as large as ResNet- Finally, we demonstrate that MKDT effective across SSL algorithms (BarlowTwe Twins (Zbontar al. 202 and SimCLR (Chen et al. 2020
RELATED WORK
2.1 DATASET DISTILLATION
There has been a large body of recent work dataset distillation for supervised learning. These techniques can be broadly characterized into meta-model matching, gradient matching, distribution matching and trajectory matching (Sachdeva & McAuley (2023).
Meta-model Matching Meta-model matching generates synthetic data such that a model trained on the synthetic dataset achieves low training loss the real dataset (Wang et al.) 2018). The traditional meta-model matching approach is computation and memory inefficient as it requires solving bi-level
optimization problem. Thus, several methods (Nguyen et al., 202T; Loo et al. 2022, Zhou al.)
2022) have been proposed to solve the inner-optimization problem in closed form with kernel ridge regression.
Gradient Matching Gradient matching generates synthetic data by matching the gradient of network trained on the original dataset with the gradient of the network trained on the synthetic dataset (Zhao et al. 2021 Zhao & Bilen 202T; [ee et al. (022)). Gradient- Gradient-matching is done for each class separately, otherwise optimizing the synthetic images to match gradients is not possible (Zhao et al., 22221). As result, these methods require labels to be applicable.
Matching Training Trajectories (MTT) MTT, first proposed by Cazenavette et al. 2022)) generates synthetic data by matching the training trajectories of models trained on the real dataset with that of the synthetic dataset. (Cui et al.) (2022) reduced the memory footprint of MTT and Du et al. (2023) minimized the accumulated error in matching trajectories by distilling flatter trajectories.
Distribution Matching Distribution matching generates synthetic data by directly matching the distribution of synthetic dataset and original dataset. One line of work does so by minimizing the
2

---

Published as a conference paper at ICLR 2025
maximum mean discrepancy (MMD) between the representations of the synthetic and real data using a large pool of feature extractors (Zhao & Bilen, 2023 Sajedi et al., 2023; Wang et al.] 2022b). For these methods, distilling examples per class is essential, without labels, the models trained on the synthetic data suffer from representation collapse and cannot learn any discriminative features (Zhao
& Bilen 2023). More recently, (Yin et al.] 2024 Zhou al. 2024 Shao et al.) 2024) apply ideas from data-free knowledge distillation
et al. (2017) to match the distributions of synthetic and
real images using the batch norm statisties of models trained on the full data. While these methods
do not distill per class, the distillation loss relies on the labels of the data and is essential to distill data preserving dlas--dicrrmminatiee features.
Dataset Distillation for SSL Very recently, KRR-ST (Lee et al. 2023 applied the kernel-based meta-model matching to distillation for SSI.. However, kernel ridge regression, with relatively unchanging encoder as the kernel function, prevents distilling synthetic data that is useful for training the encoder effectively. We empirically confirm that the encoder learnt by pre-training on these generated examples cannot outperform encoder learnt directly using SSL pre-training on random real images. While KRR-ST also uses a MSE loss to representations instead of directly performing SSL, they claim to do so mitigate the bias of bi-level optimization in meta-model based matching for SSL. In MKDT, we instead, use the knowledge distillation loss of MSE to representations of a larger teacher model to reduce the high variance of SSL gradients, and thus lower variance trajectories enable trajectory matching.
MTT is another dataset distillation method that is agnostic the labels, and hence can be potentially applied to SSL. Nevertheless, application of MTT to SSL has not been explored before. In our work, we show that that naïve application of MTT to SSL yields poor performance. Then, we propose method that leverages knowledge distillation to enable effective dataset distillation for SSL.
2.2 DATA EFFICIEFI LEARNING VIA SUBSET SELECTION
Another line of work that enables data-efficient learning is selecting subsets of training data that gen- eralize par with the full data. This has been extensively studied for supervised learning (Coleman ett al.] 2020 Toneva et al. 2019 Paul et al. [022 Mirzasoleiman et al.) [2020] Yang et al.] 2023). At a high level, these works show that difficult-to-een
examples with higher loss or gradient norm un- certainty benefit SL. the most. More recently, SAS (Joshi & Mirzasoleiman 2023) has been proposed
for selecting subsets of data for self-supervised contrastive learning (CL.). Interestingly, the most bene- ficial subsets for SL are shown to be least beneficial for self-supervised CL. We use SAS as a baseline and show that the synthetic data distilled by our method can outperform training on these subsets.
2.3 KNOWLEDGE DISTILLATION
Knowledge distillation (KD) is technique used transfer knowledge from large teacher model to a smaller student model, with the aim of retaining high performance with reduced complexity (Hinton et al. 2015). For supervised learning, some techniques align the student's outputs with
those of the teacher (Hinton et al.) (201)). while others concentrate on matching intermediate features (Romero al. 2015). attention maps ZZgorrykoo & Komodakis
(017) or pairwise distances
between examples (Park et al.) 2019). Recent works have adapted KD for SSL models (Passalis
Tefas 2018 Chen al. 2017) Koohpayegani al., [2200] Yu al. 2019). DarkRank (Chen et al.
2017) approaches KD for SSL as rank matching problem between teacher and student encoders. PKT Passallis & Tefas 2018) and Compress (Koohpayegani
al. 2020) model the similarities in data samples within the representation space probability distribution, aiming to align these distributions between the teacher and student encoders. Yu et al. (2019) introduced the concept of minimizing the Mean Squared Error (MSE) between the representations of student and teacher encoders. In this work, we rely on KD to enable effective dataset distillation for SSL,
3 PROBLEM FORMULATION
Consider a dataset Dreal = ( heen of n unlabeled training examples drawn i.i.d. from an unknown distribution. Contrastive SSL methods (Zbontar et al. 2022 Chen et al., 2020 learn an encoder f that produces semantically meaningful representations by training on Preat- Barlow Twins, in particular, learss these representations using the cross-correlation matrix of the outputs of different augmented
3

---

Published as a conference paper at ICLR 2025
views of given batch of training data:
CBT E F.)
Σ (=1 j=.j=
Σ Fi
(1)
where is the cross-correlation of outpuls within in a batch B s.t. Fij with A(x) being the set of augmented views of H. d is the di- mension of the encoder ff and À is a hyperparameter. After pre-training, linear classifier is trained on representations and labels of downstream task(s).
Ezen =,,,,,))(s))ll (()
Our goal is to generate a synthetic dataset Dayn from Dreat, such that [Daynl < DDreal| and the representations of Dreal using the encoder fög,. trained on the the synthetic data Dayn= are similar to those obtained from encoder fe... trained on the real data Dreat- Formally,
syn = arg min EgwDies D(So (2), So (x)),
Dyn
where -): is a distance function.
Evaluation To evaluate the encoder trained on the synthetic data, for every downstream task Dd, we train linear classifier 9D, on the representations of fo... and evaluate the generalization error of the linear classifier Da:
Err (Da)l == (( [fem + y]
(3)
An effective encoder achieves small Err(So (Da)) across all downstream tasks.
4 MATCHING TRAINING TRAJECTORIES FOR SSL
As discussed in Sec. 2 distribution matching and gradient matching methods cannot work without labels, and meta-model matching cannot effectively update the encoder. Therefore, in our work, we fo- on application of MTT to SSL distillation. First, we discuss the challenges of applying MTT in the SSL setting and show that its naive application does not work. Then, we present our method, MKDT, that relies recent results in knowledge distillation (KD) to enable trajectory matching for SSL.
4.1 CHALLENGES OF MATCHING SSL TRAINING TRAJECTORIES
In this section, we first introduce trajectory matching (MTT) for supervised learning (SL). Then, we discuss why naively applying MTT to the SSL setting does not work.
Matching Training Trajectories for SL MTT Cazenavette et al. (2022) generates a synthetic dataset by matching the trajectory of parameters 0 of model trained on the synthetic data with trajectory of parameters 0* of the model trained on real data (expert trajectory). This matching is guided by the following loss function:
110++N 0
CDD(D =
(4)
00 0 2|
In equation 4 0g denotes the model parameters after training on real data up to step t. The term 0 represents the model parameters after training on the synthetic dataset for N steps, starting from 0ž- Similarly, f refers to the model parameters after M steps of training on the real dataset. The primary goal MTT is ensure that the encoder's weights after training on the synthetic dataset for N steps closely match the encoder's weights after training on real data for significantly larger number of steps M, usually with N & M. MTT agnostic to the training algorithm and doesn't 't rely on labels; thus, can be applied to dataset distillation for SSL. However, naïve application of MTT cannot effectively distill synthetic data for SSL pre-training. as we will discuss next.
High Variance Gradients Prevent Effective Trajectory Matching for SSL SSL losses rely on interaction between all examples in a batch and consequently have high variance over choices of random batches (c.f., the Barlow-Twin loss in equation I.. As result, the contribution of examples to the loss and hence their gradients varies significantly based on the rest of examples in the batch Robinson et al. (02T)) unlike SL where each example's contribution to the loss is independent of
4

---

Published as a conference paper at ICLR 2025
1.00
www.mmmma
Mean Change in Synthetic Image
0.00
Varancca of Fira Trajectory
000077
0373
0.05
MTT SSL (4x batch size)
MTT-SL
0.30133
Dist oottonttoon Locs
0.32
0,00223
0.04
0.03
0.00373
0.00533
SSL
0.75
MTT SSL
0.03
MTT SSL (4x batch dooo
MTT SL
0832
SSL SIOOI
0.00
0J3232
20
300
0
20
40
60
80
100
Distillatior Berssion
Distilation Reretion
(b) Distillation Loss (Error
(c) Average Absolute Change in Pixel w/s Distillation Iterations
(a) Variance in Weights at End of Trajectory ws Trajectory Length
Matching Trajectories) w/s
Distillation Iterations
Figure I: Challenges MTT for SSL (Dataset: CIFARIO0 (1%); Arch: 3-layer ConvNet)
other examples. The high variance in gradient over mini-batches in each iteration results in high variance of the trajectories of SSL.
Theoretical Evidence for Higher Variance Gradients in SSL. We now present, in Theorem 4.
theoretical evidence, in simplified setting, demonstrating that the variance of the gradient of SSL over mini-batches indeed greater than that of SI., i.e., Var ( Lssı(B) Var(Vn Lsz (B)) Proof appears in Appendix C Appendix presents a more general version of this analysis, when optimizing with synchronous parallel SGD.
Theorem 4.1. Let D {(ai, Pi be a dataset with examples, where x; the i-th input and E {0,1) 1 is its corresponding class label. Assume the data I; are generated using the sparse coding model Xue er al. (2023); Joshi et al. (2024): for class I; Co + G, and for class G1 C, where and are basis vectors and c; ~ N oN is noise. Note that using mean class vectors en, er w.l.o.g. models the setling of arbitrury mean class vectors that are orthogonal to each other. Each class hus 3 examples.
Consider a linear model fo(x) Wx, with W initialized as I (the identity matrix). The supervised mean squared error (MSE) loss is given by:
(B) = B Σ
[[o()))) (a:)))))) Eyu Eg.
where Ey: is the one-hot encoded vector for class ys, and is a mini-batch.
The SSL Loss (spectral contrastive loss used here for simplicity of analysis) is given by:
LSSL = 22E
aim [fo(x))² Fo(x2]] + +En,zEeB [folz:]* fo(x,))
B
where A(xi) = + Cauge with Caug (0, I) representing augmentation noise.
Under stochastic gradient descent (SGD) with mini-batch size B 2:
Var( L... B)) Var( w L.sst. (B))-
where for a matrix M, Var(M) == E[|[M- E[M]] 2
as in Gower et al. 2020
Empirical Evidence for Challenges of Matching SSL Trajectories Due the high variance in the gradient of SSL objectives, the naive application of MTT to SSL does not succeed. Firstly, the slower convergence caused by high variance gradients necessitates much longer trajectories for both training on real and synthetic data, Secondly, the higher variance of gradients results in greater variance the weights at the end of trajectories starting from the same initialization (henceforth referred to variance of trajectories), as illustrated theoretically in the simplified setting above. Attempting to match SSL's longer, higher variance trajectories extremely challenging, as matching such trajectories results in chaotic updates to the synthetic images. Thus, the synthetic images cannot move away from their initialization meaningfully. Fig. [aa shows empirically that the variance of SSL trajectories larger than that of SL. trajectories, across different trajectory lengths. Additionally, the
5

---

Published as a conference paper at ICLR 2025
variance of trajectories grows faster, with length of trajectory, for SSL than for SL, exacerbating the problem for longer trajectory matching. Fig. [b compares a simplified distillation using MTT with a single expert trajectory for SSL and SL. Despite extensive hyper--arameter tuning, matching even single expert trajectory is challenging for SSI., confirmed by the slow decrease of distillation loss. This indicates that the training trajectory on the distilled set is unable match the training trajectory on the real data for SSL. Fig. Ic] shows that the difficulty in aligning trajectories is due to the chaotic updates of the synthetic image, as evidenced by the synthetic images being unable move away from their initialization. To further confirm that the inability to distill effectively is indeed due to the variance of trajectories, we also include a comparison to MTT SSL with 4x larger batch size, which leads to slightly lower variance. Fig. confirms that indeed the larger batch size reduces the variance of the trajectories slightly. However, Fig. Tb and IC show that reducing the variance of SSL trajectories via larger batch size is insufficient to help distillation since an infeasibly large batch size will likely be required to achieve the necessary low variance trajectories.
Next, we will present our method, MKDT, designed to address the above challenges.
4.2 MATCHING KNOWLEDGE DISTILLATION TRAJECTORIES
To reduce the length and variance of SSL trajectories, our key idea is to leverage the recent results in knowledge distillation (KD) Kim et al. (2021). We first introduce KD, and then discuss our method, MKDT, Matching Knowledge Distillation Trajectories, that leverages KD to reduce the length and variance of SSL trajectories.
Knowledge Distillation (KD) KD refers to distilling the knowledge of a larger model (teacher) into smaller model (student) to achieve similar generalization as the larger model, but with reduced complexity and faster inference times. Here, we rely on the knowledge distillation objective for SSL models, introduced in Yu et al. (2019)
(5) where Is and Jr represent the student and teacher encoders respectively. (Yu al. 2019) trains student models with the aforementioned KD objective and the original SSL Loss. However, we only minimize the MSE between student and teacher representations to avoid the issues with matching SSL training trajectories (discussed in Sec.
LCD De (IS(fs(II,frr())
Converting SSL to SL trajectories via KD We use the objective presented in equation D j.c. min- imizing the MSE between the representations of student and a teacher model trained with SSL. In doing so, we train the student model to match the performance of the teacher trained with SSL. Note that training the student model by minimizing the MSE loss in equation 5 is a supervised objective. Therefore, while the trained student model will produce similar representations to that of the teacher, training with MSE loss is much faster than SSL, as its gradients have much smaller variance (c.f. Fig. Lab. Thus, we can get shorter and lower variance expert trajectories from the student models trained with KD using the MSE loss, instead of the teacher model trained with SSL. Then, we can generate syn- thetic examples by matching these shorter and lower variance trajectories, without relying on labels.
Matching KD Trajectories (MKDT) We now describe our method MKDT: Matching Knowledge Distillation Trajectories. MKDT has the following three steps: (1) training a teacher model with SSL, (2) getting expert trajectories by training student models to match representations of the teacher using KD objective, (3) generating synthetic examples by matching the expert trajectories. Below, we discuss each of these steps in more details.
(1) Training the Teacher Model with SSL First, we train the teacher encoder fe, with CSSl on Dreat:
Or arg min [SsL(So, Preat)]
(6)
In experiments, LSsl is the Barlow loss function shown in equation I but method is agnostic to the choice of SSL algorithm. Since KD with larger models leads to better downstream performance (Huang al. 2022), encoder used for creating the expert trajectories for distillation. For example, in our experiments we use ResNet-18 as the teacher encoder and 3 or 4-layer ConvNet as the student encoder.
use teacher model that is much larger than the student
(2) Getting Expert Trajectories with KD For training the expert trajectories, we encode the full real data with the teacher model and train the student model minimize the MSE between its

---

Published as a conference paper at ICLR 2025
Algorithm 1 MKDT: Matching Knowledge Distillation Trajectories
Require: K: Number of expert trajectories
Require: S: Number of distillation steps
Require: M: of updates between starting and target expert params.
Require: # updates to student network per distillation step.
Require: 1 < ':: Maximum start epoch.
I: Train model fo, using LSsL on Dreal using augmentations A
2: {r;"} +-rain K expert trajectories to minimize (xc(ffe: for)
3: Initialize distilled data Dayn: Zsyn Dreal: Zreal
Initialize trainable learning rate Csyn = cxp Dayn
5: for S steps do
6: & Sample expert trajectory: T [ with r" {0])0
7: P Choose random start epoch, t S T
8: D Initialize student network with expert params:
9:
8, = 87
10: for 0 1
I Sample mini-batch distilled images:
12:
betn ~ Dayn
13:
b Update student network w.r.t. MSE loss to reference representations:
14:
0cta+1 = 0.+0 Csyn V LMSE (bi+ns Zuyn)
15:
end
16:
b Compute loss between ending student and expert params:
17:
(DD((Du))) = ||014N f 100 0 & Update Dayn and Ckayn with respect to LDD
18:
19:
end for
Ensure: distilled data Dayn Zayn and learning rate ctsyn
representations and that of the teacher model. We refer the representations of the real data from the
teacher model as teacher representations denoted by 2T- = [.. Se, (T:) Va, E D reaat- Formally,
minE
(c.).[Zr))
(7)
We train K' such student encoders and save the weights after each epoch of training to generate the expert trajectories that we will match in the distillation phase.
(3) Data Distillation by Matching KD Trajectories We now optimize the synthetic data such that training on it results in trajectories that match the aforementioned expert trajectories. First, we initialize Day as a subset of Dreal and Z as the corresponding teacher representations from Zy. Then, every distillation iteration, we sample an expert trajectory starting at epoch I, where S T+, of length M. We then train on the synthetic data for N steps by minimizing the MSE between representations of synthetic data from feom and Zsyn fer. Formally, Vn [N].
0 0,pn Okay V LLME(J (Dayn) Zayn)
(8)
Now, we compute our distillation loss (DD (shown in equation 4 using the parameters of the encoder trained on the synthetic data and the encoder trained on the full data, and update the synthetic data and learning rate, Dayn and O'gyn. respectively, to minimize this. Note that Zayn remains unchanged. We repeat this distillation for S iterations. Pseudo-code of MKDT provided in Alg.
Initializing Synthetic Data Empirically, we find that initializing Deyn from the subset of examples from Dreal that have high loss across the expert trajectories, leads better downstream performance than initializing with random subsets. In particular, for all expert trajectories. we use the encoders after epoch of training and use to compute the MSE loss for all examples x; E Dreal i.e. LMSE (fo; (x.,), (2),)). We then average the loss for examples across encoders from all expert trajectories and choose the examples with highest loss to initialize our synthetic data. Sec. compares initializing MKDT with random subsets and such high loss subsets.
Evaluating the Distilled Dataset For evaluation, we first pre-train the encoder Je.. on the distilled data by minimizing the MSE between the representations of the synthetic data Dsyn and Zsyn using the distilled learning rate asyy-
0sy = arg minlÉ ED,, CMSE (fe((,)( [Zyyn,) with l.r. asyn
(9)
7

---

Published as a conference paper at ICLR 2025
Table 1: Pre-training on CIFARIO (2% of Full Data)
Downstreem
Method
Training CIFARIO
Downsttram
Labeled Data (58)
Tiny ImageNet
CIFAR100
Aircraft
CUB2011
Dogs
Flowers
No Pre-Training
35 84
2. 2.52=0.09
8.01
2-43a0.17
1.27740.10
1.92a0.18
2. 2.02+0.70
Random Subset
36.35
2.40s
7.42
2.41
1.1620.0
1.90 22
19
Subset
36. .1.02
2.63
7.77
2:29.0.25
1.78.0.32
KRR-ST
37. 17.19:0,40
2 2.8/2
8.67=9441
2.53e0.05
1.25
1.8840.32
2. 2.02+0.22
MKIYT
44.36_1.01
courr
10.58
2.58
1.37.0..0
10.2
Full Data
58. :0.22
2.94
14.53.0.4
2.92
1.69
2.40
3..33
No Pre-Training
46. 16.23=00.0
5.37
16. 13
5.61
1.97
2.90
5.22:0000
Random Subset
46. 46.02.11.02
5.491.0.12
15.2810.56
5.35
1.51
2.52.0.22
3.64.10.6
Subset
46.52:u0.1
5.41:00.2
15.90:.0.a
5.63
1.48
2.69
3.75:1..0
KRR-ST
46. 75=0.44
6,83
16.65=0.51
5.41=0.45
1.88+0.10
2.76e0.40
1.52+0.14
MKITT
53.08..011
2.25
19.37
5.97
2.06
3.06.0.44
4.97.0.84
Full Data
67. 67.16:0.44
10.85
26.382n.52
6.92
2.51
3.8820.25
6.37
We then evaluate the encoder foy. using Err fegm (Da), defined equation i.e. the generalization error of linear classifier gD, trained on the representations obtained from encoder Se... and corresponding labels of downstream task Dz.
EXPERIMENTS
Table 2: Pre-training on CIFARI00 (2% of Full Data)
Size of Downstream Labeled Data (%)
Method
Pre-Training
Downstream
CIFAR100
Tiny ImageNer
CIFAR 10
Aircraft
CUB2011
Dogs
Flowers
Pre-Tmining
8.01
2.52g0.00
35.84g1.5 .39
2.4310.17
1.27
1992
2.021. 76
Random Subset
9.2011015
3.16.0.21
38.03.1.2
2.41
1.431912
1.901010
2.8110.43
SAS Subset
9.3990.18
3.23.0.19
98.7321.44
2.53ea.04
1.426004
2,07a014
2.95e
1%
High Loss Subset
10.03_..32
3.33
30.78
1.47
1.58
2.131o21
2.631
KRR-ST
8.31=0.00
2.73a0.0
37.19s0.83
2.56
1.20
1.92
2.58e0.14
MKDT (Rad Sub)
11.4=0030
3.90=0.20
13.35=1.04
2.53+0.22
1.6620.13
2.22+0:20
2.63+110
MKDT
12.36.0.87
4.13
44.90
2.74
1.61 20.1
2.15
3.24
Full
21.44:00.0
6,80g0.37
58.21a00.
3.20
1.79aa0s
2.50g
3.09
No Pre- Training
16. 02.03
5.37 20.7
46, 16.23
5.61
1.97aa11
2.90aaas
5. 6.2250.2
Random Subsei
17.75=0.42
6.79e0.00
18.59e0.30
5.66ee077
2.12:3.25
3.02=0.11
5.41+0
SAS Subset
17.94::0.54
6.71
48. 13.00.a..0
5.95an.8s
2.15:0.07
3.22
5 6.5620 48
5%
High Loss Subset
18.72_n.m
6.94.00.3
40.5010.31
5.6310.52
2.5810.3
3.1810
6.1410si
KRR-ST
16.40=0.44 6a
6.1620.45
77.0622..3 s2
5.54aa0.rr
2.00aa.0s
2.95+025
1.60
MKDT (Rnd Suh)
21.71 me
8.01 20.00
63.08
6.2410.75
2.53
1.38
6. 1.2610
MKDT
22.64_0.42
$4.1220.28
2.501aae
1.25
6.3710-45
Full Data
35.78.0.54
14.11
67.25..0.4
7.46
3.00
g.23
8.4110m
Table Pre-training on TinyImageNet (2% of Full Data)
Size Downstream
Method
Pro-Training
Downstream
Labeled Data (%)
ImugeNet
CIFARIO
CIFARI00
Aizeralt
CUB2011
Dogs
Flowers
No Pre- Trainrainng
2.03g2 29
30.40a1.00
7.1420.3
2.25-0.11
1.28
1.76a0 13
2.84
Random Subset
3.03
34.4611.13
7.661o.77
2.27
1,2410c6
1.92
2.16 2.
1%
KRR-ST
3.32.0 29
34.2420.a
7.8420.s
2.1420.30
1.30gace
1.95 22
2.23
MKDT
3.87.1o
37.2510.47
8.9510.31
2.30__.19
1,361022
1.9__0.2
2..6
Full Data
9.12e0.30 36
50.52e0.57
15. 18
2.57=0.20
1.50e0.a1
2 2.20
3.29=0.35
Pre Training
5.609
39.91
13.324000
4.46-0081
1.71i000
2.51is.18
4.83.0a3
Random Suhset
6.76.0. n6
43.74
10.8316.15
4.40
1.6810.4
2.6710:11
4.23.10.00
KRR-ST
42.44.1.50
18.8550072
3.99-0.57
1.774007
2.47
4.14
MKDT
7.99.0.33
45.97.00.7
16.5010.35
4.66
1.07
2..1
5.49
18.93_...4
58.9010.s3
26.47.1.77
5.07__0.71
2.47.1006
3.8511.18
7.09
In this section, we evaluate the downstream generalization of models trained using the synthetic sets distilled by MKDT that are 2% and 5% of the original dataset's 's size for CIFAR1O, CIFAR 100 (Krizhevsky & Hinton| 2009), TinyImageNet (Le & Yang 2015). We also conduct ablation stud- ies over initialization of the distilled set and the SSL algorithm. Finally, we also consider the generalization of the distilled sets to larger architectures.
8

---

Published as a conference paper at ICLR 2025
Table 4: Pre-training with Larger Distilled Set Size (5% of Full Data)
Pre-Training Dotaset
Size Downsrram Labeled Dam (%)
Method
Downstream Task Accurney
CIFARIO
CIFAR100
Thy ImageNet
Aireraft
CUB2011
Degs
Flowers
Random Subset
37 27.a3
7.63e0.15
2. 2.G3esaa
2:28
1.13=s.11
1.81
2.05
1%
KRR-ST
36 69..
8 3.09
3. 8.20en.s
2,26
1.38=a
1.91
2.39
CIPARIO
MKDT
50.23
12.33
4.27
1.59
2.3610.20
oz
Randon KRR-ST
16 16
6 a.2a
5.08aess
1.
3.30 20.0
5%
16.05 47.40
68
7:10
5.66:20 77
1.
2.78 30
4.381001
MKDT
58.37
23.15
8.8
6.79
2.30
3.34.0.35
5.94
Random Sulwet
42.45
4 4.17
2.475017
2.12
2.6
1%
KRR-ST
86.334
9.02
2.94.0.33
2.42
1.50...07
1.99
1.04
CIFARI00
MKDT
47.77s1.32
13.40
4.45
2.93an442
161:s.0
2.22.
2.50
5%
Random Subset
KRR ST
47 17.311
51. 31.92
20 34.0.0
17 17.24
7.83
8 6.00en.32
5.37
6.06s0.7
2.27
2,31
2.87
3.22
5.7810.4
5.23
MKDT
56.6159.s
28.18
9.1240.00
6.66
2.66:0.25
3.6610.44
6,93
Randon Suboot
10.55-1.36
9. 4/410.29
2.23
144
2.06
2.77
KRR-ST
34.2721.30
7.5
3. 8.199
2:11.023
1.30
1.68...0
2.652081
TinyImageNet
MKDT
10.29 41.4
5.09
2,16
1.20:a..s
2 2.02e0.25
2.
5%
Random Sulwer
KRR-ST
42.82.6.46
48 26.4625.0 t
13.71:0.30
15 15.63390.0 68
6.50
8 4:00
4.36.0.4
4.55
1.97..006
1.98 18
2.7516.37
2.9
3.97.0.11
81
MKDT
50.79 10.49
19.25
10.63.0.33
1.88
2.08
2.89
1.58
Table 5: Transfer to Larger Architectures Sire-Trannin on CIFAR100 5%, 5% Downsteam Labels)
Pre-Training
Downstream
Method
CIFAR100
CIFARIO
TinylmageNet
Aircraft
CUB2011
Dogs
Flowers
No Pre- Traininnn
1.36
13.1841774
1.03
1.00a0.01
0.43
0.60=a.01
0.75
ResNet-10
Random
18. 80
41.24+0.85
10.33=0.16
2.15
18
2.52+a.29
5.41+0.01
KRR-ST
13.84207s
39.21
8.0420.32
2.1249.15
1.162005
1.7749.14
4.56220-2
MKDT
23.23
49.13:0.60
13.35
1.68
L.7700.00
2.64
6.15+0.47
No Pre- Training
1.01
10.0020.00
0.91 to
1.01
0.56...07
0.67
0.9320as
ResNet-18
Random
16.82:0.60
40.11:1.18
8.9520:23
1.84+0.25
L.220.06
2.40+0.2
5.1620.99
KRR-ST
12.30=0.83
35.73
7.21
2.32
1.18
1.81+0.14
2.45#0.12
MKDT
21.51
46.10=0.00
11.57=2017
2.0520.43
1.86
2.36.0.29
5.17
Distillation Setup We use ResNet- 18 trained with Barlow Twins (Zbontar al. 2021 the teacher
encoder and train K = 100 student encoders (using ConvNets) to generate the expert trajectories. As
in previous work (Cazenavette et al.) 2022 Zhao & Bilen 202T| Zhao al.]
2021 Chen et al. 2023
Du et al. [023] Cui et al.) 2022), we use a 3-layer ConvNet for the lower resolution CIFAR datasets and 4-layer ConvNet for the higher resolution TinylmageNet. Exact hyperparamters in Appendix A
Evaluation Setup To test generalization in presence of limited labeled data, we evaluate encoders pre-trained on the distilled data using linear probe on 1% and 5% of downstream labeled data.
Baselines We compare pre-training with MKDT distilled sets to pre-training with random subsets, SAS subsets (Joshi & Mirzasoleiman 2023), KRR-ST distilled sets (Lee et al.) (023) as well as pre-training the full data. For KRR-ST, we use the provided distilled sets for CIFAR100 and Tiny ImageNet, and distill using the provided code for CIFARIO. We omit SAS for TinyImageNet as this subset was not provided. For distilled sets of size 5%, we consider only Random and MKDT, since other baselines did not provide distilled sets.
Downstream Generalization Performance Table I demonstrates that pre-training on CIFAR10 using MKDT with a 2% distilled set improves performance by 8% on CIFARIO 10 and 5% on down- stream tasks over the KRR-ST baseline. Gains CIFARIO are consistent across 1% and 5% labeled data, but improvements on downstream tasks are more pronounced with 5%, indicating MKDT scales well with more labeled data. On CIFAR100, MKDT 2% distilled set improves performance by 6% and 8% on CIFAR 100 and downstream tasks, respectively. Additionally. MKDT shows up to 3% improvement on downstream tasks for larger, higher-resolution datasets like TinyImageNe (200K examples, 64x64 resolution). highlighting MKDT's scalability. KRR-ST consistently fails to outperform SSL pre-training on random subsets across all settings. In Appendix ( this holds for fine-tuning experiments from KRR-ST (Lee et al. 2023). affirming MKDT as the only effective DD method for SSL pre- trainini. Table 4 shows that pre- pre-training with larger distilled sets (5% of full data) further enhances performance by up to 13% confirming MKDT scales effectively with distilled set size as well. Table ( shows that MKDT outperforms the strongest baseline (random subsets) by 5% on pre-training and 7% downstream tasks when using 10% and 50% labeled data.
we verify that
9

---

Published as a conference paper at ICLR 2025
Table 6: Ablation over SSL Algorithm (SimCLR), Distilled Set Size 2%
Pre-Training
Size Downstrrea Laheled Data (%)
Method
Downstream Task Accuncy
Dotaset
CIFARIO
CIFARI00
Tiny ImageNett
Aircraft
CUB2011
Degs
Flowers
Random Subset
35,20e1.12
T.35an.2
2.20
2.21=
1.19:s.06
1.8320.15
1.83a9.2
1%
KRR-ST
36 36.5021. asc
8 8.38
2.96 12
2. d.
1:10
1.77+015
2.35
CIFARIO 10
MKDT
40.77.1.05
9.17an13
3.
s.30
3.
5%
Random KRR-ST
46.87.49
15.6949..
15 15.00
16. 16.29a0.3 37
1.
1252-
4.36.0
3.36
MKDT
$1.77 24
18.07
6,55
2. m
1899
2.98
1.09
1%
Random Subset
96.57
34.67.0.5 as
5.35
8.384002
2.29
2.23.
2:41:00
1.10
1:28:
1.71+0.0
1.7810.30
13
1.984924
CIFARI00
MKDT
39.59.1
9.44
2.00
1.33
1.931037
2.49
Random Subset
45.0730.00
15.11.0.44
5.21an.2
5.21
1.51:8.14
2.54
3.37
5%
KRR-ST
46, 76.7611022
15.76 46
6.17
5.43.000
1.9319.00
2.6
MKDT
49.87
18
6.68
1..6
1.93.0.13
1.98
Table 7: Larger Labeled Data Fractions (10% 50%), Distilled Set Size 2%
Pre Training
Datasel
Size of Downstream Labeled Data (9)
Method
CIFAR1O
CIFARI00
Downstream Tisk Accuracy
Tiny ImtogeNet
Aircrant
CUB2201
Dogs
Flowers
10%
Rundem Subart
51.21 120024
10.75 tho
5.04 tr..
7.42 tt.45
2.15 ht.15
3.10 to..
5.22 m0
KRR-ST
61.02 -m.50
21.01 1
8.95 +1.56
7.01 +o
2.44 hh.12
3.54 +h
6.82 mn
CIAARIO
MKDT
56,88
24.61 1o.42
9,76 Lo.z
8.96 10.71
2.78
4.37 1uor
7,38 11:00
50%
Random Subset
KRR-ST
08.09
57.18
25. 86 Looe
29.01
15.54
15.16 Locs
16.11 10:00
4.18 Jan
6,81
0.17 1oo
15:32
12.98
MKDT
63.63
34.06 $.030
17.57 45.77
10.03
2:0.22
20.17 7.63
16.89 2:037
10%
Randous Sabset
KRR-ST
52.23
62.40 2m:73
21, 39 20.36
20.60
8:21 20.07
8.57 20.43
7:64 3:3.66
#3.03
2.73 #0.17
2.34
do.11
3.04
3.76
7.50
6.52 1:
CIFARIO0
MKITT
$7.67 m0
27.28 +0.18
11,05 +0.50
9.25 +3.70
3.42
tho 10.10
4.43
th0.0
9.35 + 1..
50%
Randen Subaet
60.39 m0
31 02 tr
16,02 +1.14
18. 10 +a.19
5.65
th.18
7.37 th
16.18 moos
KRR-ST
08.57
29. A6 #1
15.70 +044
15.80
80 +h.54
4.82
+h.41
7.00 #h..1
15.01 -n
MKDT
65.85
38.09 10.35
17.46 10:24
20.73
1:0.34
6.67 11g.
8.48 10:3
21.15
10%
Random Subsel
KRR-ST
16.08
16. Lo.32
10
8.84 Loas
5.57 1a.3s
2.12 La.::
3:21
3.07 1ost
6.51
TisyImageNer
MKDT
49.33
20.19 1:.45
10.43
d:3.3
2.47
22.18
3,86 2.0.0
7,33
50%
Randoms Sabset
KRR-ST
47.35 22000
48.16 21.16
20.01 10.40
19.
13.59
14.01 20.35
8.00 13.29
3.48
do.3:
5:18 #50.2
4.08 da.m
13:00
15.12 2n.
MKDIT
$1.98
m0
24.41
+0.30
16.99 +0.05
10.84 +a.as
4.44 m0.20
6.22 m0.15
16.47 t0..0
Ablations We perform ablations over two factors: 1) initialization and 2) SSL. algorithm. Table 2 presents results for pre-training with MKDT using random subset initialization, well results for pre-training directly on the high-loss subset initialization used by MKDT. Interestingly, the KD objective for SSL pre-training benefits slightly more from the high-loss subset than from random subsets. Consequently, MKDT initialized from the high-loss subset performs better than when initialized from random subset. Ta shows results for MKDT using a teacher model trained with SimCLR (Chen et al.) (020) instead of Barlow Twins. Specifically, we train a ResNet- 18 for 400 epochs using SimCLR. Here too, MKDT achieves approximately 6% higher performance compared to random subsets across downstream datasets. This confirms that MKDT generalizes across different SSL training algorithms.
Generalization to Larger Architectures Table 5 compares CIFARI00 5% size MKDT distilled set to 5% size random subsets, using the larger ResNet- and ResNet-18 architectures. Across all down- stream tasks, confirm that MKDT distilled sets outperform baselines even when using larger archi- tectures. Surprisingly, the larger ResNet- 18 slightly merdor-e the smaller ResNet- 10. This trend is observed across all baselines, including no pre-training. We conjecture this is due to larger models requiring a lot more data to be able to use their extra capacity to surpass their smaller counterparts.
6 CONCLUSION
To conclude, we propose MKDT, the first effective approach for dataset distillation for SSL pre- training. We demonstrated the challenges of naively adapting previous supervised distillation method and showed how knowledge distillation with trajectory matching can remedy these problems. Empiri- cally, we showed up to 13% improvement in downstream accuracy when pre-training with MKDT distilled sets over the next best baseline. Thus, we enable highly data-efficient SSL pre-training
Acknowledgments This research was partially supported by the National Science Foundation CA- REER Award 2 2146492, and an Amazon PhD fellowship.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Xuxi Chen, Yu Yang. Zhangyang Wang, and Baharan Mirzasoleiman. Data distillation can be like vodka: Distilling more times for better quality. International Conference Representation Learning (ICLR), 2023.
Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Darkrank: Accelerating deep metric learning via cross sample similarities transfer, 2017.
Coleman, C Yeh, Mussmann, B Mirzasoleiman, P Bailis, P Liang. Leskovec, and M Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations (ICLR), 2020.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. arXiv preprint arXiv:2211. 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-2552 2009. doi: 10. 109////P...... 5006.420
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference Machine Learning, pp. 5378-5396. PMLR, 2022.
Jiawei Du Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation, 2023.
Wele Gedara Chaminda Bandara, Celso M De Melo, and Vishal M Patel. Guarding barlow twins against overfitting with mixed samples. arXiv e-prints, pp. arXiv-2312, 2023.
Robert M. Gower, Mark Schmidt, Francis Bach, and Peter Richtarik. reduuceee-reueed methods for machine learning, 2020. URL htttt:t//s//xi...rtt absp//bs//20000 00892
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher, 2022.
Siddharth Joshi and Baharan Mirzasoleiman. Data-efficient contrastive self-supervised learning: Most beneficial examples for supervised learning contribute the least. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 15356-15370. PMLR, 23-29 Jul 2023. URL
httpp::/ http::s//prcceeiingg mlr www.sppesssseie..ee.ee j00002/ooshiibbbhhm po.
Siddharth Joshi, Arnav Jain, Ali Payani, and Baharan Mirzasoleiman. Data-efficient contrastive language-image pretraining: Prioritizing data quality over quantity. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artifi- cial Intelligence and Statisties, volume 238 of Proceedings of Machine Learning Research, pp. 1000-1008, PMLR, 02-04 May 2024. URL Joshi24a. html
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Tachyeon Kim, Jachoon Oh, Nak Yil Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback- leibler divergence and mean squared emror loss in knowledge distillation, 2021.

---

Published as a conference paper at ICLR 2025
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self learning by compressing representations, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report University of Toronto, Toronto, Ontario, 2009.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https: htt.aaatttiaaacooaaaooooo/orptpa
Dong Bok Lee, Seanie Lee, Joonho Ko, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Self- supervised dataset distillation for transfer learning, 2023.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensa- tion with contrastive signals. In Proceedings of the 39th International Conference on Machine Learning, pp. 12352 12364, 2022.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Advances in Neural Information Processing Systems, 2022.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks, 2017. htl.:atteee//raaa.....o///////12
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-leaminn from kernel ridge- regression. In International Conference on Learning Representations, 2021. URL https
//openreview net. htt:://oummmiiiie wtt.:/ettoooooie111--rr----
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early in training. Advances Neural Information Processing Systems, 34: 10596-20601 2021.
Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
Noveen Sachdeva and Julian McAuley, Data distillation: A survey, 2023.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 17097-17107, 2023.
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-
scale data condensation via various backbone and statistical matching, 2024. URL https
17950
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Ben- gio, and Geoffrey Gordon. An Empirical Study of Example Forgetting During Deep Neu- ral Network Learning, November 2019. URL fttp://arrivv org/
arXiv:1812.051599 [cs, stat].
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang. Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12196-12209 2022a.
12

---

Published as a conference paper at ICLR 2025
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang. Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features, 2022b.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
Yihao Xue, Siddharth Joshi, Eric Gan, Pin- Yu Chen, and Baharan Mirzasoleiman, Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression, 2023.
Yu Yang, Hao Kang, and Baharan Mirzasoleiman. Towards sustainable learning: Coresets for data-efficient deep learning. International Conference Machine Learning (ICML), 2023.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation imagenct scale from new perspective. Advances in Neural Information Processing Systems, 36, 2024.
Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, and Arnau Ramisa. Learning metrics from teachers: Compact networks for image embedding, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny, Barlow twins: Self-supervised learning via redundancy reduction, 2021.
Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pp- 12674-12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2021.
Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: good compression is all you need, 2024.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
A.1 ADDITIONAL DETAILS FOR EXPERIMENTS IN TABLES 1 3
5 6
7
Training the Teacher Model Using SSL We trained the teacher model using Barlow/Wins (Zbontar eet al. (022T) using the training setup Re specified in (Gedara Chaminda Bandara et al. 2023)) We used the Adam optimizer with batch size 256, learning rate 0.01, cosine annealing learning rate schedule, and weight decay 10-6. The feature dimension is 1024, Finally, we use the pre- rre-projection
head representation of the trained model for teacher representation, and its dimension is of size 512.
Training Expert Trajectories Using KD We trained 100 expert trajectories for each dataset with random initialization of the network for 20 epochs, using Stochastic Gradient Descent with learning rate 0.1, momentum 0.9, and weight decay le-4. Similar to other DD works (Cazenavette et al., 2022 Lee et al. 2023), we used depth ConvNet for Tiny ImageNet and depth ConvNet for both CIFAR 10 and CIFAR 100. We did not apply any augmentation except normalization, and did not apply the ZCA- CCC-whitening.
Distillation Hyperparameters We distilled 2% of CIFAR 10, CIFAR 100, and Tiny ImageNet. We used SGD for optimizing the synthetic images with batch size 256, momentum 0.5. We distilled CIFAR 10 and CIFAR 100 with depth 3 ConvNet and Tiny ImageNet with depth 4 ConvNet. We initialize the synthetic learning rate as 0.1 and used SGD with learning rate 10 and momentum 0.5 to update it. We distilled the datasets for 5000 iterations and evaluated their performance for all the experiments except those in Table where we use the distilled dataset after 1000 iterations. The other
are recorded in Table
Table 8: MKDT Hyperparameters on 2% Distilled Set
CIFARIO
CIFAR100
IinyInugeNet
Percentage Distilled Model
2%
2%
2%
ComNNeD33
ComNNeDD3
ConvNeeiD4
Synthetic Steps (N)
40
40
Expert Epochs (MF
2
2
2
Max Start Epoch (T*)
5
Leaming Rate (Pixels)
10
10a
10a
Pre-training on Synthetic Data For the synthetic data, we pre-train them using the MSE loss with their learned representation for 20 epochs using SGD with batch size 256, their distilled learning rate, momentum 0.9, and weight decay 10-4, We use depth 3 ConvNet for CIFAR 10 and CIFAR 100, and depth 4 ConvNet for Tiny ImageNet. For distilled datasets, we use the synthetic learning rate asyn- For other datasets (c.g., random subset), we use the same setting except a learning rate of 0.1.
Evaluation We use the models' penultimate layer's 's representations (as is standard in contrastive learning (boottar et al., 2021 Chen et al., )020 of the downstream task's 's training set and train linear classifier, using LBFGS with 100 iterations and regularization weight 10- 3 We then use the pre-projection head representations of the test set of the downstream task and evaluate using the aforementioned linear classifer report the downstream test accuracy.
Using SimCLR for Obtaining Teacher Representation. We conducted an ablation study using teacher model trained with SimCLR (Chen et al.. 2020) instead of Barlow Twins (Zbontar et al.) 2021) for CIFAR 10 and CIFAR 100. The experiment steps are similar to the ones in A. During the "Training the Teacher Model Using SSL" step, we used the Adam optimizer with batch size 512, learning rate 0.001, and weight decay 10-6 to train a ResNet18 along with 2-layer linear projection head for 400 epochs. The projection head included Batch Normalization and ReLU after the first layer, and Batch Normalization after the second layer, projecting to 128 dimensions. Then, we used the pre-projection head representation of the trained model for getting the teacher representation of size 512. The other steps are the same the one in I. Table 6 shows that MKDT consistently outperforms the random subset across all downstream datasets and various sizes of labeled data, highlighting the method's 's generalizability to other contrastive learning methods.
Scaling the Method to Larger Distillation Sets. In addition to distilling 2% subsets, we also conducted experiments distilling 5% subsets of CIFAR1O, CIFAR 100, and TinylmageNet to evaluate the generalizability of the method to larger distillation sets. Table o shows the scalability of the method to larger distilled set sizes. The experiments procedure the same as the ones in A. except
14

---

Published as a conference paper at ICLR 2025
Table MKDT Hyperparameters on 5% Distilled Set
CIFARIO
CIFAR
TinyImageNet
Percentage Distilled
5%
5%
5%
ConvNetD3 ComNNerD3
ConNNeDD4
Synthttic Steps (N)
40
40
10
Expert Epochs (M)
Max Start Epoch (T*)
5
Learning Rate (Pixels)
10
107
that we use different distillation hyperparameters for the 5% distilled set. The hyperparameters are summarized in T
Scaling the Method Larger Downstream Labeled Dataset Sizes. We evaluated the performance for CIFAR 10, CIFAR 100, and TinylmageNet on larger downstream labeled data sizes, specifically 10% and 50% labeled data sizes, using the 2% distilled set obtained with the method illustrated in A.1 As shown in Table MKDT continues to outperform random subset across all downstream datasets and data sizes, demonstrating its scalability with larger labeled data sizes.
Details for KRR-ST et al. (2023) We use the code and ssper-parameters provided in (Lee et al. 2023). As the original paper did not provide the results and the hyperparameters for CIFAR 10, we use the same hyperparameters as CIFAR 100 to distill CIFAR 10. In particular, this invovles using
BarlowTwins ResNet- 18 as the teacher model as well.
A.2 ADDITIONAL DETAILS ON EXPERIMENTS IN FIG.
For the experiment in Figure we train 5 trajectories of each of MTT SSL and MTT SL for CIFAR 100 using the same random initialization of the network, respectively. For MTT SSL, we train the models with the Adam optimizer with batch size 1024. learning rate 0.001, and weight decay 10 10-6 For MTT SL, we train the model with SGD with batch size 256, learning rate 0.01, momentum O, and no weight decay.
For both of the experiments in Figure [b] and [c learning rate 1000, max start epoch synthetic steps 20, and expert epochs We distill using MTT SLL with image learning rate 1000, max start epoch O, synthetic steps 10, and expert epochs 2. We distilled them for 100 iterations and record the change the loss function and the average absolute change in pixels.
we distill the dataset using MTT SL with image
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL COMPARISON OF RANDOM SUBSET SSL PRE-TRAINING TO KRR-ST
This experiment is conducted in the setting Le et al. (203) for pre-training CIFARIOM This experiment pre-trains on a distilled set / subset of 2% the size of CIFAR 100 and then evaluates on downstream tasks by finetuning the entire network with the entire labeled dataset for the downstream task. KRR-ST compares with the random subset baseline by pre-training using SL. However, we use SSL on the random subset as baseline instead. Here, we show that, fact, even in the finetuning setting reported by Lee et al. )2023), KRR-ST cannot outperform SSL pre- wre-training on random real images. The baseline reported as 'Random' Lee et al. (2023) refers to SL pre-training as opposed to SSL pre-training.
Table 10: Pre-training CIFAR100
Method
Pre Pee-Trining
Downstreim
CIFARIOU
CIFARIO
Aircratt
Cars
CUB2011
Dogs
Flowers
No trai-trainnn (frum Lee Lev al. 2021)
64.05ana:
87.5 37.50a
34 00
10. 10.13an0
18. 18.16.0
22.31
58 75s8-4
Random (Supovised
65.23
Lee Random Pre-franngg
66.441m34
38.7416.20
42.02amas
28.751m21
15.12...
26.57
68.21:6.4
16

---

Published as a conference paper at ICLR 2025
C PROOF FOR THEOREM 4.1
Proof. To analyze the variance of the gradient for both SL and SSL, we will conduct a case analysis. There 3 possible unique cases for constructing the mini-batch. Assuming n is extremely large s.t. n/2 n/2-1 we have:
1. Both examples are from class 0 with probability
2. Both examples are from class with probability
3. example from each class with probability
Let 31. refer to example and its corresponding label vector; similarly, let 22, y/2 refer to example 2 and its corresponding label vector.
To show the effect of each term, we refer to the class mean vectors for class and class as μo and yn, respectively; and use en and E1, the basis vectors to represent the labels for class 0 and class 1.
Analyzing cases for SL.
1 2
(LsL((B)) = Σ 2(Wx; x4 - y1)
(10)
2 i=1
= =( x1 - (x1) + (Wx2
(x))
(11)
Case 1: Both examples from class
(w((sr(B)) = (W., - (o) + (Wx2
co2
(12)
(LsL(LB B((Wx; ω1 - (z1) + (Wa, - (o)
(13)
E[(Was x1 ( (22)
(14)
E[(W μO + WEN, - (e0 + =NN,) " BB( WEN, eo) + EN.)
T
(15)
E[(uo tEN, - E00 + eN. I El(uo + tENs - euu + ENN) Jbby substituting W
(16)
= =Elμo/ + ENN T - coo4 + toet, + + cN - 1
+ EEHHOH + +NNHH -aa4 + moEt, + tENSEN, eoe
(17)
2(H0N4 col'o
(18)
(w(LsL(B)) = pnno + + Mo eapo + MoET, + +EN, EN, - EDEN T
+ Holot CNaHu coju HOCN,, CNN.T. CDCN2
= =2u0ro (EN, + +N2)HD 2cuuu + HoleTh (1T)) T T + ENGENN =o(eer + +eTr)
+eNGEN,
(19)
Case 2: Both examples from class By symmetry, we have:
V (LsL((BB)
E
2(P14 -1HTT
(20)
17

---

Published as a conference paper at ICLR 2025
and
VWLSI(B)
-2ii1 (EN, +ENs)ui 22114 + +Mi(EN, + +o
EEN + ENN - -2 + +
(21)
Case 3: I example from each class
(bs(( polld eollD + +11n eij1
(22)
and
LsLLL(B) polod + EN, PD copo poe tEN - EDEN,
+ Mint + CNaHt c1fl + uicNo + tNN. - C1
(23)
Pulting together
E[ (L.s
2wolo eoHl)
(24)
+ 2( -(HHT)
(25)
+ 2 (HoN) - -copo + MiH1 c1
(26)
mo copo + +HIH 6144
(27)
Finally,
Var(Vw(LsL
(28)
| BLW case 2]
(L)) (B) case 3] (29)
Simplifying term 1: [I( V (LsL (LV 1]
vw(Lst(B)) V (W
2pano + (eN, + EN2)NO 2200g + -o(eT, + +e
+ EEN,EN + EN, - co(ET, + +eX,)
- (MOHo eoHoo + HH1 -E1
=Mono CN,HO +N,HH cajl + por T + MoC
T +EN,EN + +ENgEND - EDEN, -606N2
-MIMT + +eiu
(30)
By symmetry, term 2 i.e. follows:
E(V (LsL (B))]II) case 2] can be simplified as
(w(Lsr(B)) E[V Ind + eN. of + +eNNu enn Maee nn
18

---

Published as a conference paper at ICLR 2025
+ ENiCN, To +CN2CN2 cich, -IEN2
Hoold + topp
(31)
Simplifying term 3:
(Dsr(B) E[V (Ls 3]
(32)
(w((LL(B)) E[Vw(LsLL(B))
MoHl CN collo + MOCN, T +en eoCN:
+HIi + CNaH1 T EIM + MICNo + CNoo CICN2
- -HOPp -e0lo + +Mi4 - -E14T)
EN,MO MOCN: T CN, EDEN:
+EN,Hi + HIEN, +EN,EN CIENS
(33)
Analyzing cases for SSL
We will now analyze the same for SSL and show by comparing each of the 3 terms, element-wise, above to their counterparts that Var(aaasa (B)) > Var(Lst( (B))
From
Xue al. (2023). have that LSSL can be re-written as VwLssL(W) -T((2WWTT) + Tr(MW WMW W)
where
1 2m
M
Ty
2m
with being the number of augmentations, and M represents the covariance of the training data. The matrix M is defined as:
M
Σ
Σ
T
2 i=1
zEA(>:)
sEA(z;)
where A(x;) denotes the set of augmentations for the sample x4.
As m - xc, M = M = ( + x2222)
Hence, Vw (LssL (B)) = 4W + WW WMM
Substituting W T, get V (LssL (B)) -4M 4.2
Case 1: Both examples from class 0
M
[ + x242)
(34)
((vo + eN₂ (uo + TT, (wo + ENN + +N2)
(35)
(HaRo -ot +eNuHo teN T
+ Lolo + poct, + CNaHu + (eN, =oon- 1 (MoEk, teMuu + +eN + Hoc + CNsHD + (N,
(36)
(37)
M (lnM + (MocT, + CN + teN
(38)
19

---

Published as a conference paper at ICLR 2025
poeN, + +eNalo +eNz
(39)
= -oolo HoPo
1
+ PoHlo (HOEN, + ENNiM tENIEN)
(HOE + cNald I + +NN, + +Pott 2
+ (JocT, + +NN Ho + +N, Ni) Holo
2
+ (MOrTs + tNalo + N,, pollo
(MOCT, + CN.lo T tc, Mi)
+ (HOET, + teNaHD + =N2
+ (HOCT, + +CN. HO + tN, (HUcks + eeald + cN,
(HOcTa + cNolo + (Na (Hoct, + tcn (N)
(40)
E E + 4MM] 4poro + 4poHo mopo
(41)
Case 2: Both examples from class 1
By symmetry,
- (ur + +eNu + + pnln + 2
1
+ +1I tNNHH + ENN
(42)
E[-4M + 4MM Apii + (enuTann
(43)
Case 3: I excample from each class
M
(Mas) T x222)
(44)
((H + eNi MHO + teNi) + (μi + ENN + (
(45)
2 (HUPP + roct + +eNind + tcN,ch
+ minn + -IET, + EeNJH + +eN
(46)
B[-4M + 4MM
-2pon1 - 2n1 + 2nano mhno + 2pr pual
(47)
Putting together
Hence.
1
BM-4M + 4M
Ahonr T + ahanr HoHo)
4
( -411t + A paph)
4
20

---

Published as a conference paper at ICLR 2025
1 2HaMo 2n1 2pamo
HoNg + 201 piph))
(48)
2
2pond 201n4 20o00
+ nonld 2prpi pip
(49)
Comparing each term, element-wise
Comparing Term I
Vw((ss(B)) ([V((Ls((B)) = mopo + CN, + +eNald colld +Mo Moc
+CN, T +NN coCis cocN2 -MIn +iip
(50)
(w((sss(BB) B[V = -po04 T -100 T N0N - 2e Ne
N.
2uDEN; 2EN 2ENGEN2
2oaur HaM4
+ HONo 2 (HDET, + +ENuHO + ENN ENI
+ Hopo 2 (MOCTT ENaHo + CN,CNT)
+ 2 2HOET, + tNiMD ENNIENn) HOHO
+ 2 (HOCT + +eNal0 + + 1000
+ (HOCT, + cNil0 + +NNi (T))
+ (HOCT, + EN2100 + +eNgCNg) NT)²
+ (HOC + CNiPo +CNi (HOCTZ T + #NHHO + (NN
+ (MOEN ENaMU +eN,EN,) (HOEN, T
+EN +EN ENi. (51)
Recall that Ao eg.
Use and to denote the value of an abtrirary element of cN, and CN2, respectively,
Comparing the expectation of the square of element (0, 0) of w (L(B)) E[V
- For SL: E[(1 + 62 - + +
- 1 + 1) = E[((1 + + + =
From Eq. 51 is easy to see that the expectation of the square of element (0, 0) is far larger due to far more f1 E2, ef and éø terms.
Similar argument holds for diagonal terms that (0, 0). due the greater number of terms that only have ef and es (all terms having only and are only present in element (0, 0) since they are always multiplied with en).
An analogous argument holds for off diagona elements. Here, the contribution is only due to terms containing only CN, and/or 6N,-
Comparing Term 2
By symmetry, it must be that the same holds for term 2.
Comparing Term 3
(B) E(Vw = CNiPo + Moct, + +CN T CoC
+eNgus + tuieT + -NN2NN E1ENN
T
(52)
(w((sss(BB)
-2(moro poet, ENN ENN T
21

---

Published as a conference paper at ICLR 2025
+ Hill + &ueet + ENssi cN2
A(oold + noen, + eN.Ho + +NN, padl picky tcnuu (NN
(53)
Again, from analogous arugments from Term 1, we can see that the expression for Lsst. has larger expected square element wise than that of L.L.
Conclusion
Since, for each term (corresponding to the cases of mini-batch), have that
||| (Lsssr EIVW(Lssr(B)]I1}
- -BV (54)
Thus, we can conclude
(B)(V < Var(VwLssz(BB),
22

---

Published as a conference paper at ICLR 2025
D PROPOSITION D.I. GENERALIZED ANALYSIS OF VARIANCE OF TRAJECTORY UNDER SYNCHRONOUS PARALLEL SGD
Proposition D.1. Let D ((),, be dataset with N examples, where xi is the i-th input and yi E (1,2,....,F K} is its corresponding class label from K classes. We assume the data Iı is generated from distribution where each example from class & can be modeled as d-dimensional (k < d) 1-hot vector ex plus some noise e, i.e.
ey, + Eg where py, is the vector corresponding to class yi and E ~ (0, I). Moreover, we consider a linear model fa(x) = Ox where 0 E Ka
The supervised loss function Lst, is defined as a loss function that depends solely on each individual example:
1 Σ (su(fo(z.).
(B)
(55)
[B]
iEB
where fst. is an arbitrary supervised loss function (e.g., MSE, Cross-Entropy, etc.) and B is mini-batch of examples.
For SSL, we consider the spectral contrastive loss:
LSSL aag [f<21) + (Ln f(x))) 2r
(56)
where A denotes the augmentation that are modeled as: A(x,) ~ I; + Caug. where Caug (0,
We analyze this setting under synchronous parallel SGD, where the number of mini-batches equals the number of parallel threads. Let represent a partition of the dataset D into a set of disjoins mini-batcher each of size [B). The synchronous parallel SGD update for an arbitrary loss function LBB) on mini-batch B at time step t p.+1 := -- a Eer Vo, L(B).
Let be the initial parameters and Ost. Osst. be the model parameters after a single epoch of training with synchronous parallel using the supervised loss Lsg and the self-supervised loss LSSL. respectively, with a learning rate a. Then, for sufficiently large batch size [B).
Varp(Ost) < (arp(Osst).
(57)
where Varp represents the variance over different partitions P, and for vector random variable X, we define the scalar variance as Var(X)= = |||| X [[[]]|]]
Proof. For synchronous parallel SGD, have:
0.+1 = 02 - a Σ LBB),
BEP
where L is an arbitrary loss function.
Variance of Trajectory in Supervised Learning
Consider two arbitrary partitions P1 and Pz. Let represent the initial parameters, and let 0SL, and 69L, represent the parameters after a single epoch of synchronous parallel SGD for supervised learning using L.s, on the partitions P1 and Pz, respectively.
We can express the updates as:
sSL, = =0+a
Σ
VoLsı
(58)
BEP,
= = Σ
Σ
1
V VoLsi(1 (due to independence of loss for each example)
B
BEP, (Zu.)eB
(59)
=0+a Σ
V VoLsI Lo
(60)
(R.Wi)DD
=0+a Σ
Σ
V VoLsE((2:
(61)
B
BEP₂ (Z..:))B
23

---

Published as a conference paper at ICLR 2025
=8LL2-
(62)
Since OSL, = Ost, for arbitrary partitions P1 and P2, we can conclude that:
(0s)( =
Variance of Trajectory in Self-Supervised Learning
Next, we need to show that Varp(Ossr.) > 0. Let 0 represent the initial parameters, and let Osst. and 8SSL, represent the parameters after single epoch of synchronous parallel SGD using the self-super loss LSSL with partitions P1 and P2, respectively.
To show (0rp(( > it suffices show that there exist partitions Pi and Pa such that 0SsL, * OSSL2
Consider the following two partitions:
Let P1 be a partition where each mini-batch is constructed by iterating over classes, until examples are selected, and picking single example from the remaining examples in each class
Let P2 be partition where each mini-batch is constructed by picking examples from the remaining examples each class until [ examples
It is easy to see, from prior work in hard negative mining for contrastive SSL such as Robinson al.] (202T), that P2 will have different gradient across all batches from P1 P2 only contrasts examples within the same class, whereas Pi contrasts examples across different classes. Hence, Osst. * Osst.
We will now prove this for our setting, considering two random partitions Pi and P2.
Xue al. 2023 shows the LSSL can be rewritten as LSSL -Tr(2Me0T) Tr(M
where M
Em .. M' B ( EoEA(..)) 2) ( ExeA(.)) x) where m is
the number of augmentations sampled.
From Xue et al. (023), we also have vo(LssL) = 40M 40 OM
With i.e. number of augmentations large enough, M M'. We now refer to for P1 as M1 and M for P2 as M2.
From the construction of Pl. have M1 and from the construction of P2, we have [Male 5r, where ō,, = 1 = j] for batch where examples from class are selected. From this,
that, the ve(Lsst) is not equal for P1 and P2-
Hence, we can conclude that OSsL, + Os512
24

---

Published as a conference paper at ICLR 2025
E EXAMPLES FROM MKDTAND KRR-ST (LEE ET AL. 2023
Here, we show examples from CIFARIO, CIFARIO0 and TinyImageNet for the data distilled using MKDTand KRR-ST (Lee al. 2023).
(a) MKDT
(b) KRR-ST
Figure 2: Examples of Distilled Images for CIFARIO
(a) MKDT
(b) KRR-ST
Figure 3: Examples of Distilled Images for CIFARI00
(a) MKDT
(b) KRR-ST
Figure 4: Examples of Distilled Images for TinylmageNet
F DM AND MTT-SSL EXPERIMENTS
We also compared our methods with two DD methods adapted to SSL. Specifically, we focused on distilling 5% of CIFAR 100. The results all show that those methods do not outperform MKDT.
To adapt DM for SSL, we used the same codebase and hyperparameters as DM (Zhao & Bilen 2023). However, during distillation, instead of DM's original approach of sampling random images from specific class, we modified the process sample random images from the entire dataset. This adjustment allows us to approximate the behavior of running DM a label-free setting. The results are shown in
To adapt MTT to SSL, we modified the supervised learning algorithm in getting the trajectory and training the network in the inner loop of the distillation to be the SimCLR SSL algorithm. Due the
25

---

Published as a conference paper at ICLR 2025
high variance of SSL analyzed in [.
distillation fails to converge, and the poor results further
shows that MTT with SSL does not work. The results are shown [2
Dataset
CIFAR100
CIFAR10
TinyImageNet
Aircraft
CUB2011
Dogs
Flowers
CIFARTO (1% Labels)
1.
35.98
2.4
2.1520.21
1.1940.13
1.71
1.60
CIFARIO (5% Labels)
14.77=0.10
46.41=0.35
5..1
0.94
1.57
0.13
3.10+0.33
CIFARI00 (1% Labels)
8.74+0.35
37.75=0.
2. 2.722.222
2.51
1.20#0.111
2.07
2.53+0.27
CIFARI00 (5% Labels)
17.01
47.32-0.38
6.40+0.21
5.08+0.82
1.9940.13
2.77+0.28
4.62+0.28
Table 11: Results for Distilling with DM with SSL.
Dataset
CIFARI00
CIFARIO
TinyImageNet
Aireraft
CUB2011
Dogs
Flowers
CIFARIO IO (1% Labels)
3.63
22.06+1.80 1.86
1.11
1.92
0.77=+0.1
1.38 28
1.26=0.37
CIFARIO (5% Labels)
7.10+0..4
29.04 40.58
1.90=0.17
2.81
0.88+0.08
1.82+0.19
1.42=0.53
CIFAR 100 (1% Labels)
4.26:10.34
26.064 1.58
1.20#0.14
1.49
0.87+0.26
1.24
1.41
CIFARI00 (5% Labels)
9.66#t0.24
38.16+0.57
2.11+0.19
2.83#t0411
1.06+0.04
1.74+0.15
2.38=0.19
Table 12: Results for Distilling with MTT using High Variance SSL Trajectories.
26

---

Published as a conference paper at ICLR 2025
G SOCIETAL IMPACT
Our work democratizes the development of models trained with SSL pre-training, by reducing the volume of data needed to train such models, and thus the costs of training these models. However, the risk with dataset distillation is augmenting the biases of the original dataset.
27


Reviews:
[{"review_id": "lXNv0rG7Qm", "reviewer": "Reviewer_YkVh", "summary": "This paper explores dataset distillation methods for self-supervised learning (SSL). The authors demonstrate that the MTT method cannot be directly applied to SSL due to high trajectory variance, both theoretically and empirically. To address this, they propose a solution leveraging knowledge distillation (KD) to reduce the length and variance of SSL trajectories.", "questions": "- Line 295: Incorrect format in “[41] trains student ….”\n- The MKDT method uses ResNet-18 as the teacher model and ConvNet as the student model. What is the backbone network used for KRR-ST in Tables 2 and 3?\n- Presenting some distilled images would help demonstrate the proposed method’s effectiveness and its advantage over the baseline KRR-ST.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Self-supervised dataset distillation is an important yet underdeveloped area with wide-ranging applications.\n- The empirical and theoretical evidence provided (Theorem 4.1 and Figure 1) for high gradient and trajectory variance of MTT in SSL settings is convincing and insightful.", "weaknesses": "- The performance improvements over random subset and high-loss subset are marginal in many cases, especially on the larger TinyImageNet dataset (Table 3). This raises concerns about the method’s practical value, as it requires significant computational resources to distill synthetic data while yielding minimal improvement.\n- The absence of KRR-ST results in Tables 4, 5, 6, and 7 limits the ability to assess the proposed method's effectiveness, given that KRR-ST is a closely related baseline.\n- The MKDT method appears to add only a knowledge distillation process to MTT, where a student model mimics the SSL teacher model’s representations. The synthetic data is then learned by matching the student model’s trajectory, rather than the teacher model’s trajectory, as in MTT. It is unclear why introducing a student model as an intermediary reduces trajectory variance, or what specific role the student network plays throughout the data distillation process. The authors are encouraged to provide further discussion or insights into this mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SXFTEbMzDN", "reviewer": "Reviewer_1SyT", "summary": "- This paper propose a data distiallation method for self-supervised pretraining. This methods MKDT is a based on knowledge distiallation method, which is mainly motivated by the study on the gradient's variance of different loss function.", "questions": "It seems like that the logic of the paper is: the gradient variance of SSL is larger than the one of SL, thus the knowledge distillation is needed to lower the variance due to the relationship between gradient and trajectory matching. The questions are:\n- It's due to the high variance of the SSL gradient, why not show and print the empirical gradient variance directly?\n- How is the straightforward relationship between model performance and gradient variance, if albation study on trajectory sampling is considered?\n\nI first give 6 here, actually, I tend to give 5.5. therefore, some important concerns should be discussed during the rebuttal period.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The first, as this paper claims, effective data distillation method for SSL pretraining is proposed.\n- Outstanding experimental performance on provided datasets, \\textit{e.g.}, CIFAR-10, 100/Tiny ImageNet.", "weaknesses": "- Not matched theory and implementation. The toy case in theoretical analyses is too simple and the gap between analyses and implementation is too large.\n    1. The linear model is provided in the main paper only. However, even in linear probe experiments, the tested model is non-linear. Moreover, in the literature of the related ones (\\textit{e.g.}, sharp-aware generalization, bias-variance trade-off, Out-of-Distribution Generalization), analyses about gradient's variance on different loss design are usually conducted on non-linear case (\\textit{e.g.}, 2-layer network with non-linear Activation function, such as ReLU in early years).[1,2,3]\n    2. The derivative and presentation of its proof is too complicated however the logic is actually trival (i.e., simply unfold the deviation of the gradient of SL and SSL, and then compare each term by \\textit{easy to see is larger}).\n- Pretraining is important nowadays. However, the model architecture is too small and simple, for which pretraining is not as important as the large models (\\textit{e.g.}, LLMs), which is not discussed in the main paper.\n    1. In the assumption of the theorem, citation [4] about sparse code is on multi-modal pretraining, but similar architecture is not mentioned in the rest of the paper.\n- Experiments about generalization is confusing.\n    1. In the distillation setup, it claims that the teacher model is ResNet-18 and the student model is ConvNet. However, the models in the experiments about generalization to Larger architecture are ResNet-18 and ResNet-10. The models are not larger at all, and ResNet-L (L>18) are available in the provided code.\n- Typos and inconsistent prensentation, \\textit{e.g.}, synhronous in Line 215, pre-training and pretraining.\n\n[1] Estimating Example Difficulty using Variance of Gradients. CVPR 2022\n[2] Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization. ICML 2022\n[3] Rethinking Bias-Variance Trade-off for Generalization of Neural Networks. ICML 2022\n[4] Data-efficient contrastive language-image pretraining: Prioritizing data quality over quantity. AISTATS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Sp8bCtGC2", "reviewer": "Reviewer_SPPk", "summary": "This paper introduces one of the first approaches to dataset distillation for self-supervised learning (SSL) pre-training, addressing the challenge of generating compact, synthetic datasets that can efficiently pre-train deep networks without labeled data. By leveraging a teacher-student framework, the authors demonstrate that using knowledge distillation (KD) significantly reduces the variance in training trajectories, a common issue in SSL due to high variance in gradient updates. This approach, termed \"Matching Knowledge Distillation Trajectories\" (MKDT), trains a smaller student model to match the embeddings of a larger teacher model, resulting in a low-variance objective that allows effective dataset distillation for SSL. Experimental results indicate that MKDT outperforms prior methods by up to 13% across various downstream tasks with limited labeled data, underscoring its potential for memory- and compute-efficient SSL pre-training​", "questions": "1. In Figure 1, what are the primary insights that should be drawn from each sub-figure? Specifically, could the authors clarify how each metric in Figure 1 (e.g., variance in weights, distillation loss) directly impacts the effectiveness of the KD-based trajectory matching approach?\n\n2. The paper argues for the superiority of trajectory matching in the SSL setting. Could the authors elaborate on any specific challenges or limitations of gradient matching and distribution matching that make them unsuitable for SSL dataset distillation, as compared to trajectory matching? (this question ties to one of the points in the weakness section)\n\n3. Regarding the high-variance problem in SSL gradients, has there been an investigation into alternative approaches, such as sharpness-aware minimization, to reduce gradient variance? What led to the decision to focus solely on KD rather than considering variance-reducing techniques as a complementary approach? (also raised in the weakness section)\n\n4. In Table 2, could the authors clarify the relationship between initialization choice (e.g., high-loss vs. random subsets) and performance? Is there a grounded rationale behind the preference for high-loss subset initialization?\n\n5. Could the authors provide more context on how the choice of SSL algorithm (e.g., SimCLR vs. Barlow Twins) affects the efficacy of the MKDT method? Is there a fundamental reason one algorithm would be preferred over another in the KD-based trajectory matching framework? Which method among SimCLR and BarlowTwins as faster convergence while learning the distilled samples? Can this method also be extended to other SSL methods such as masked reconstruction?\n\n6. Lastly, in the methodology, could the authors clarify how they determined the optimal values for hyperparameters like the number of distillation steps, K expert trajectories, and distillation loss thresholds?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This work is among the first to address dataset distillation for self-supervised pre-training, providing an innovative approach to generate compact, synthetic datasets that enable efficient pre-training without labeled data.\n2. The theoretical motivation for introducing a teacher-student learning model is compelling. By using knowledge distillation, the method effectively reduces the high variance in gradients commonly seen in self-supervised learning objectives, improving upon naive trajectory matching approaches. While I have not throughly checked the the mathematical proofs, but nonetheless, the proofs intuitively seem convincing.\n3. The related work section is particularly well-constructed, offering a thorough exploration of the dataset distillation literature and providing valuable insights into the challenges and developments in both supervised and self-supervised settings, positioning this work within the broader context of current research.", "weaknesses": "1. **Limited Discussion of Alternative Distillation Paradigms**: The paper presents knowledge distillation (KD) as a novel proxy for enhancing trajectory matching in SSL dataset distillation. However, the authors omit a comparison with other established dataset distillation methods, such as gradient matching or distribution matching. The Related Work section briefly acknowledges these methods but dismisses them based on label dependency. This lack of comparative analysis leaves a gap: could these paradigms outperform trajectory matching if adapted to SSL? Addressing this gap would strengthen the paper by either justifying the choice of trajectory matching for SSL or empirically showing why KD-based approaches are superior.\n\n2. **High Variance in SSL Gradients and Potential Regularization Techniques**: A core motivation for the KD approach is the high variance in SSL gradients, which makes naive trajectory matching ineffective. Yet, the authors do not explore whether regularization techniques, such as Sharpness-Aware Minimization (SAM), could mitigate this variance issue. SAM, which reduces oscillations in model updates by flattening the loss landscape, could be a promising candidate to stabilize SSL gradients. An empirical analysis incorporating such techniques would clarify whether the KD setup is necessary or if regularization alone could suffice.\n\n3. **Clarification and Takeaways in Figure 1**: Figure 1 illustrates the challenges of applying MTT to SSL, emphasizing issues like high gradient variance and chaotic updates to synthetic images. However, the figure captions lack concise takeaways summarizing the implications of each plot. Clearer captions explaining the significance of variance trends and how they justify the proposed KD approach would improve the paper's readability and impact.\n\n**Minor Weakness (Formatting and Page Limit)**: The submission uses a numerical citation format, which does not adhere to the ICLR 2025 requirements of (Author, Year) format. Given the 10-page length, updating the citation style within the limit should be manageable.\n\n**Reference**:  \n[1] Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKq95dHHOj", "reviewer": "Reviewer_Juoa", "summary": "The authors of the paper deal with applying dataset distillation on unlabeled dataset as part of a self-supervised pre-training. A fundamental observation presented in the paper is that optimizing for dataset distillation while using SSL objective is hard to convergence because of a high gradient variance in the batch. This observation was demonstrated empirically and theoretically under some assumptions. To mitigate the high gradient variance, the authors proposed to use Matching Training Trajectories following knowledge distillation, which leads to a much smooth gradients, allowing a stable dataset distillation optimization and accuracy improvements on several downstream tasks.", "questions": "- My main question is more general, concerning the concept of dataset distillation (DD). Despite over four years of research on DD, progress appears to be limited, with most work focused on small datasets and controlled settings. I would appreciate the authors' insights on the future potential and development of dataset distillation.\n\n- I think the authors may consider presenting some connections to the field of data-free knowledge distillation. What do the authors think?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- In my view, the proposed approach effectively transforms the problem of data distillation in an unlabeled setting (SSL objective) into a supervised approach using knowledge distillation loss.\n\n- The concept of enhancing self-supervised approaches by incorporating dataset distillation is interesting.\n\n- The approach presented in the paper is simple and clearly explained.\n\n- The theoretical and experimental motivations regarding high variance gradients appear reasonable and valid.", "weaknesses": "- Line 75: “… datasets distilled with ConvNets can transfer to other larger architectures such as ResNets.” Note that ResNets are also a type of Convolutional network (ConvNet).\n\n- What about considering larger (and more modern) architectures? For example, using larger ResNet architectures instead of just 3-4 convolutional layers.\n\n- Do the authors consider connections to data-free knowledge distillation (DFKD)?\n\n- The authors' theoretical derivation is based on very strict assumptions. Statements such as \"as confirmed theoretically above\" (line 242) may be overstated.\n\n- The paper uses the Barlow Twins SSL objective. Did the authors try other SSL approaches? It would be useful to know if the observations are generalizable and if the accuracy improvements hold across different SSL algorithms.\n\n- Experiments: the authors only compare linear probes on 1% and 5% of the downstream labeled data. Why not present results for additional subset sizes? Especially when comparing to SAS, which is a data pruning method where the main focus is on higher subset sizes (e.g., SAS provides results for 20% subset size and above). This raises the question of whether the comparison to SAS is fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)