Dataset Distillation for Pre-Trained Self-Supervised Vision Models
Authors: George Cazenavette
Antonio Torralba
Vincent Sitzmann
Venue: NeurIPS 2025 poster ()
Keywords: dataset distillation
self-supervised learning
few-shot learning
Primary area: deep_learning
TL;DR: We distill large datasets to just one image per class and use them to train linear probes on top of self-supervised vision models.

Abstract: The task of dataset distillation aims to find a small set of synthetic images such that training a model on them reproduces the performance of the same model trained on a much larger dataset of real samples. Existing distillation methods focus on synthesizing datasets that enable training randomly initialized models. In contrast, state-of-the-art vision approaches are increasingly building on large, pre-trained self-supervised models rather than training from scratch. In this paper, we investigate the problem of distilling datasets that enable us to optimally train linear probes on top of such large, pre-trained vision models. We introduce a method of dataset distillation for this task called Linear Gradient Matching that optimizes the synthetic images such that, when passed through a pre-trained feature extractor, they induce gradients in the linear classifier similar to those produced by the real data. Our method yields synthetic data that out-perform all real-image baselines and, remarkably, generalize across pre-trained vision models, enabling us, for instance, to train a linear CLIP probe that performs competitively using a dataset distilled via a DINO backbone. Further, we show that distilled datasets provide a valuable tool for model interpretability, predicting, among other things, how similar two model's representations spaces are under the platonic representation hypothesis or whether a model is sensitive to spurious correlations in adversarial datasets.

Dataset Distillation for Pre-Trained Self-Supervised Vision Models
George Cazenavette
Antonio Torralba
Vincent Sitzmann
Massachusetts Institute of Technology
Abstract
The task of dataset distillation aims find small set of synthetic images such that training model on them reproduces the performance of the same model trained on a much larger dataset of real samples. Existing distillation methods focus synthesizing datasets that enable training randomly initialized models. In contrast, state-of-the--e vision approaches are increasingly building on large, pre- pre-trained self self-super models rather than training from scratch. In this paper, we investigate the problem of distilling datasets that enable us to optimally train linear probes top of such large, pre-trained vision models. We introduce a method of dataset distillation for this task called Linear Gradient Matching that optimizes the synthetic images such that, when passed through a pre-trained feature extractor, they induce gradients in the linear classifier similar to those produced by the real data. Our method yields synthetic data that outperform all real-image baselines and, remarkably, generalize across pre-trained vision models, enabling us, for instance, train linear CLIP probe that performs competitively using a dataset distilled via a DINO backbone. Further, we show that our distilled datasets are exceptionally effective for fine- mne-grained classification and provide a valuable tool for model interpretability, predicting, among other things, how similar two models' embedding spaces are under the platonic representation hypothesis or whether model is sensitive to spurious correlations in adversarial datasets.
Introduction
The task Dataset Distillation involves the synthesis small set of synthetic samples such that a model trained from scratch on this synthetic set will achieve test-time performance comparable to that model trained on the full real dataset. Since this problem's 's first introduction and proposed solution in the self-titled paper [47]. many methods [6, 27, 51, 54-56] and extensions thereof [7, 11, 16, 25, 28, 40, 46, 53] have made strides towards the lofty goal of learning a high--qulity model from just handful of synthetic images.
Meanwhile, computer vision has increasingly adopted paradigm of using the representations of large, pre-trained self-supervised vision models for downstream tasks, either via fine-tuning or by using these models as feature extraction backbones. Given this trend, in this work, we explore dataset distillation in the regime of training models top of features extracted by pre-trained vision foundation models. Specifically, we study linear classification on top of a pre-trained feature representation.
In our new method, Linear Gradient Matching, we distill synthetic datasets by optimizing such that their representations extracted by pre-trained feature extractors induce gradients in a linear classifier similar to those obtained from real images. We find that a single synthetic image per class suffices train linear classifiers to competitive performance across wide variety of large vision
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CLIP
Pizza
Volcano
Lorikect
Monarch Jack-O'-Lanle
Carbonara
Jay
Jellyfish
DINO-v2
Scotch Torrier
Beaver
Tiger
Ladybug
Cello
Cheeseburger
Banana
Timber Wolf
EVA-02
Frog
King Penguin
Cab
Flamingo
Balloon
Bookcase
Strawberry
MoCo-v3
Castle
French Horn
Church
Blue Jeuns
Kimono
Espresso
School Bus
Cardoon
Figure I: ImageNet-Ik Distilled for Self-Supervised Models: Using method of Linear Gradient Matching, we distill vision datasets just synthetic image per class using different pre-trained self-supervised backbone models. These learned images can then be used to linear probes that achieve high accuracy on unseen test data, outperforming all real-image baselines. Furthermore, each backbone model seems to yield its own "style" of distilled image, giving insights into the aspects on which these models tend to focus (structure, texture, color, etc.).
model backbones, outperforming real-image baselines. Figure shows samples distilled from ImageNet- 1k [12] with our method using various self-supervised feature extractors.
Motivated by recent hypotheses that different large models converge similar representations even when trained on different modalities [20]. we investigate whether distilled datasets transfer across architectures. We find that gradient matching objective alone leads to images that are overfit to particular model architecture and do not yield competitive performance across founda- tion models. However, we overcome this issue through differentiable augmentations and a simple re parameterizztion of images via a multi-scale pyramid. Compared to those retrieved via naïve pixel optimization, the resulting distilled images not only look remarkably realistic but also readily transfer across foundation models, such that dataset distilled using, for example, a DINO backbone yields competitive performance when used to train linear classifier on top of different model's representation, such CLIP's.
We also observe that our distilled datasets offer several interesting interpretability results, including predicting alignment between different models, explaining susceptibility (or robustness) to spurious correlations adversarial datasets, and highlighting distii-iittiitttion capabilities.
Extensive experiments and ablations validate our Linear Gradient Matching method's 's effectiveness on this dataset distillation task and highlight its potential interpretability tool.
Related Work
Dataset Distillation. As dataset and model sizes continue grow, has the interest in more efficient forms of learning. To this end, researchers have worked towards methods of learning optimal training data such that one could train effective model from scratch using as few samples as possible. One such solution to this problem was the initial proposal of Dataser Distillation [47] in which the model's final performance was expressed as a function of the synthetic training data that was optimized end-to-end by back-propagating through many inner training iterations. Follow-up works introduced proxy losses and learned synthetic images that matched gradients [55], feature distributions [54], training trajectories [6] and more [27, 51, 56]. Some works extend dataset distillation large models [50, 51], but these methods do not excel in the ultra-small data regime, i.c., one image per class. For such settings, Trajectory Matching [6] and

---

Real Images
Meta-Gradient
Gradient
Hl real aw
W
Creal
Forward
Cosine Distance
Synthetic Images
alayn aw
W
Lsyn
Figure 2: Linear Gradient Matching for Pre-Train Pre-Trained Vision Models: Given pre-trained self-supervised vision model (Ø), we perform a distillation step by first passing a batch of real and synthetic data through and a randomly-inttialied linear classifier (W) to get the real and synthetic classification losses (feas and Eyn)). Our meta loss (mmen) is then defined the cosine distance between the gradients of these classification losses (feet and Eygn) with respect to random linear probe (W). This meta is then nack-propagated through ini- tial synthetic gradient calculation and used to update our synthetic images. This technique allows us to distill large datasets to just single image per class while still achieving high performance when training new linear probes.
its modifications [7, 11, 16, 32, 40] still reign supreme. However, this method fails to scale up to large models due to memory constraints and instability in the bi-level optimization.
This work introduces new problem in the space of dataset distillation: for the purpose of training linear probes on top of pre-trained self- self-superiiee feature extractors instead of training randomly initialized models from scratch. Our proposed solution, Linear Gradient Matching, takes inspiration from prior work on gradient matching [55] and trajectory matching [6] but only considers gradients of the linear classifier opposed to the entire model.
learning synthetic images
Supeviiee Learning. Given that overwhelming majority of available visual data lacks any useful labels, Self-Supervised Learning has become the defacto method of pre-training neural networks to be later used for down-stream tasks. In recent years, several different paradigms of self-supervised training have emerged, including contrastive Iearners [5, 8-10, 18, 31]. masked auto-encoders [17, 37], vision-language models [34, 41, 44, 52], and hybrid approaches [13, 14]. Despite the various training methods, researchers have noticed that these different models tend to learn similar representations, even across different modalities, and dubbed this observation the "Platonic Representation Hypothesis" [20].
In this work, we focus on distilling datasets using four pre-trained self-supervised models in particular: CLIP [34], DINO-v2 [31], EVA-02 [13], and MoCo-v3 [10]. Since our feature extractors were pre-trained purely self-supervised manner, our linear probes still only see a single labeled sample per class while achieving competitive performance.
3 Method
Dataset Distillation aims to synthesize tiny synthetic dataset of (typically) images that are optimal for training. Unlike previous works designed to produce synthetic samples for training models from scratch, our new method, Linear Gradient Matching, aims to distill datasets for the purpose of training linear classifiers in the embedding space of pre-trained feature extractors.
3.1 Linear Gradient Matching
Formally, given pre-trained self-supervised feature extractor, d. along with real dataset Dreat of images Xreat and labels Yral,, we wish to distill a small synthetic set Day (X.ym. Ygn) such that Day can be used train linear classifier in feature extractor é's embedding space that will have similar test-time performance to one trained D

---

Inspired by previous works that aimed to match single-step gradients [55] or multi-step trajectories [6], our solution is designed ensure that training on our synthetic dataset results in similar updates as training on real data. In other words, the gradients of the classification loss using synthetic images (with respect to the linear classifier) should match those using real images.
To achieve this, we sample random linear classifier matrix W ~ (O 1)cx/ at each distillation step where e is the number of classes and f is the feature extractor o's output dimension. After passing the real and synthetic images through the feature extractor & and linear classifier W, we find the real and synthetic classification losses where CE is the multiclass cross-entropy loss:
Leal = CB W (((ral); Yea))
E Eyn CB(W of (xay)); Ysya)
We then take the gradients of these classification losses with respect to the linear classifier W, and our meta loss is then the cosine distance between them:
Of aw
Olsyn
Leeea 1 - cos
vec
vec
aw
This meta loss is then back propagatdd through the inner gradient computation, linear classifier, and feature extractor to find @Cmea/ 10X syn and update our synthetic images. This process is repeated until distillation is complete. An outline of this Linear Gradient Matching method can be found in Figure 2.
3.2 Implicit Regularization
As noted in prior works [6, 55), Dataset Distillation tends to yield synthetic images that are overfit to the model used to perform the distillation. This issue manifests in what appear like adversarial patterns and inhibit the images' usefulness when used to train other models.
One recent work [15] proposed using a pyramid representation rather than naïve pixels for image opti- mization problems. While the authors used this technique for CLIP inversion, we find it works remark- ably well for our Dataset Distillation task. Rather than simply optimizing pixels, each synthetic sample instead stored as set of images of different resolutions p = [1 x 1,2 2 2, x 4,..., 256 256).
Before each optimization step, the composite images X are "rendered" by bilinearly upsampling each level of the pyramids P {Pr|r p} to the max resolution (256) and adding them together before smoothly clamping the pixel values with a sigmoid function:
X sigmoid
Σ resize250 (P.)
Furthermore, we progressively optimize our pyramid, starting with just the lowest-resolution com- ponent and periodically adding more tiers during distillation. The effects of using the pyramid representation are quite dramatic, as seen in Figure 5.
As an additional step to combat overfitting, we also learn distilled images in decorrelated color space, as described in prior feature visualization work [30]. In short, we apply a fixed linear transform to the channels of our images after the pyramid reconstruction that brings them back into the standard correlated color space. This helps ward off any potential color-based biases induced by the model used during distillation.
3.3 Differentiable Augmentations
As first noted in the work on Differentiable Siamese Augmentation [53], applying differentiable augmentations to the synthetic images during the distillation process greatly improves the quality of the distilled data. As such, we apply horizontal flipping, random resized cropping, and Gaussian noising to our distilled images at each step. In practice, we actually apply multiple rounds of this augmentation to different copies of the synthetic data at each iteration and concatenate the results. We find this improves distillation since the optimization now encourages all these augmented copies together to the ideal training set rather than attempting to fit all pertinent information into a single augmented version of the images.

---

Train Set
ImageNet-100
ImageNet-Ik 1k
Img/Cls)
CLIP
DINO-v2
EVA-02
MoCo-v3 Average
CLIP
DINO-v2 EVA-02
MoCo-v3
Average
Distilled (Ours)
84,9
91.5 0.1
89.0
13.4
87.2
63.0
70.3 75.0
63.2
67.9
Neighbors
67.8.10.3
86.0
78.8:02
77.1:0.1
77.4:002
38.8
67.7
49.910011
56.4100.0
53.2
Centroids
77.1=00.1
86,9
80.9
77.7+0.1
80.6=0.2
53,9+000
69.5+0.1
58.1
$7.40000
59.7=001
Random
16.6
74.8
64.5+227
61.4+2..
64.3+224
31,7++05
50,3+0.5
37.7=+0.4
38.8++0.6
39.6=0.5
Full Dataset
92.5.10.0
95.21.01 94.1.10.1 89.4.10.3 92.8.0.. 78.7.10.0
83.0
81.7.10.11 76.5
80.0....0
Table 1: Linear Probes with One Image-per-Class We compare our method (Distilled) several real-image baselines ImageNet-109 (left) and ImageNet- (right). Images are distilled (or selected) using the given model in each column. "Neighbors" are the real images with embeddings closest to those of our distilled images. "Centroids" are the real images with embedding closest to the mean of each class. "Random" is random selection of real images. Our method outperforms each baseline across all models and both datasets.
Distill Model
ImageNet-100
ImageNet-Il 1k
CLIP
DINO-v2
EVA-02
MoCo-v3
Average
CLIP
DINO-v2
EVA-02
MoCo-v3
Averuge
CLIP
84.91-0.1
80.8
83.8
61.6::0.2
77.8
63.0
56.4+10.1
59.7:10.1
39,5+:0.0
54.7:10.1
DINO-v2
77.0=0011
91.5+001
86.8+0.1
78.8+0.1
83.5+0.1
54.1
75.0+001
65.4
60.0=000
63.7+00.1
EVA-02
75.5+002
86.4+:0.1
89.0+0.0
67.7400.1
79.7+0.1
55.54-0.1
65.940.1
70.3
51.8:000
60.9
MoCo-v3
65.6++0.1
86.6+=0.1
82.3+002
83.4::0.1
79.5++0.1
41.40:00
66.9+..11
57.2:10.1
63.2:10.0
57.2+00.
Full Dataset 92.5.10.D 95.2.10.1 94.1110.1
89.4.10.3
92.8....1
78.7.1000
83.0.10.
81.7.10.1
76.5.10.0
80.0.10.0
Table Cross-Model Performance of Distilled Datasets: Here, we see ImageNet-100 (left) and ImageNet-1k 1k (right) distilled using a given model and then evaluated across all models. We see that images distilled from DINO have the best average cross-model performance for both datasets. The distilled datasets generalize well, aside from an outlier pair of CLIP and MoCo. Columns are colored based on percentage of the "Full Dataset" benchmark.
4 Experiments
We evaluate our method various datasets, including ImageNet-1k [12] and ImageNet-100 100 [42] for our primary results, Spawrious [26] and Waterbirds [38] for a study on adversarial datasets, Stanford Dogs [21] and CUB-200-2011 [45] for fine-grained visual classification, and ArtBench [24] to test the method's duiti--ditt capabilities. The majority of our experiments use four pre-trained
self-supervised feature extractors as backbones: CLIP [34], DINO-v2 [31], EVA-02 [14], and MoCo- v3 [10]. In all our experiments, we distill the given dataset for 5000 iterations before training linear probes to convergence on the resulting synthetic images. All experiments are conducted at 224 x 224 resolution and use the "VIT-B" version of the given model. All distilled datasets by default use 10 sets of augmentations per batch except for ImageNet- Ik, for which only 3 sets of augmentations are used due to compute constraints. For further implementation details, please see the Appendix.
Evaluation and Baselines. To measure our method's performance on a given feature extractor, we randomly initialize linear classifier and optimize it to convergence using the distilled images before evaluating on the test set. The same procedure is used to evaluate real-image baselines. In prior works self-supervised learning [3 the evaluation consists of grid scarch across number of hyper-parameters including from which layer(s) the features used to train the linear probe should be taken from. For simplicity. we instead use only the features from the backbone's 's last layer and keep the training hyper-parannn parameters consistent across the training of all linear probes.
We compare our method against three real- real-image baselines. For the Neighbors baseline, we choose the real image for each class whose distill-model embedding is closest to that of the corresponding synthetic image produced by our method. Similarly, for Centroids, we take the real training image with embedding closest to the mean embedding for each class. I Lastly, for Random, we simply select random image for each class and average the performance over 10 different seeds.
5

---

ImageNet- 100 Distilled with DINO-v2
ImageNet-Fruits: DINO-v2 Embeddings
90
30-
Dishiled
Cenwood
20-
Test Accuracy
Eval Model
CLPP
DING=2
PC2
EVA-22
80-
0-
75-
-10
-20
2
4
6
10
30
20
10
0
10
20
30
Augmentations per Batch
PC1
Figure 3: Performing more rounds of differentiable augmentation on the synthetic data during each distil- lation step improves both the single-model and cross- model performance of the distilled images.
Figure 4: We distill ImageNet- Fruits and observe the PCA of the training image embeddings. Each color represents class. Note that the distilled images lypi- cally lie the edge outside of their class's 's cluster.
4.1 Linear Gradient Matching Out-Performs Real- Leal-Image Baselines
First, we evaluate our method on the original dataset distillation task: using a backbone model to distill dataset and then training a model of the same architecture on the resulting synthetic images. Note that this setting differs from previous dataset distillation works in that we use pre-trained backbone distillation and reuse that backbone during evaluation by training a new randomly-mittalizzed linear classifier on top of it.
In Table 1, see results for the single-model setting when distilling ImageNet- 100 and ImageNet- 1k. Across both datasets and all four models, our method convincingly out-performs all real-image baselines. In particular, our method enables linear probe trained on top of DINO-v2 [31] reach 75 % test accuracy on ImageNet-Ik while only ever having seen a single labeled image per class. In comparison, training on the full dataset of 1.3 million real images reaches just points higher 83%. Similar results are seen for the other models and datasets as well.
In Figure 4 visualize the embeddings of the distilled images relative to the real training data by plotting the 2D principal component analysis (PCA). For the sake of visual clarity, we use Fraiitst-Fruits [6, 7], a toy dataset of 10 fruit classes from ImageNet- The embeddings in this figure are from the same model used to distill the dataset (DINO-v2). We observe that the embeddings of the distilled images tend to lie far away from their respective class's 's centroid, often falling on the outside edge of the class's distribution. We hypothesize that this is due in-part to the distillation embedding highly-diseriminative features within the synthetic images.
4.2 Distilled Images Generalize Across Models
Next, we investigate the cross-model generalization capabilities of our distilled datasets. That is, we first distill dataset using one backbone model (e.g.. CLIP) and then evaluate the synthetic images' per- formance on the other models (DINO-v2, EVA-02, MoCo-v3) that were not seen during distillation.
We visualize these results in Table 2. The diagonal elements are equivalent to the first row of Table 1 and represent single-model performance while the off-diagonals show cross-model capabilities. We see that the distilled datasets tend to generalize well to unseen models save for an outlier between CLIP and Moco, possibly due to poor model alignment (Section 4.6). We also observe that the highest- accuracy model on the full dataset (DINO) also has the distilled dataset with the best cross-mode per- formance, suggesting that a model's quality affects the generalization capabilities of its distilled data.
4.3 Regularization and Augmentation Enable High Performance
In this section, we quantitatively (Figure 3) and qualitatively (Table 5) analyze the effects of the various regularization and augmentation techniques included in our distillation process. Table 3 shows the effects of ablating various components when evaluating using the same model used to distill (top)

---

Train Set
ImageNet- 100
Img/Cls)
CLIP
DINO-v2
EVA-02 MoCo-v3 Average
CLIP
Avg. Cross Same Eval
Full (Ours)
84,9:1011
91.5 :1.1
83,4 89.0
87.2:10.1
-Decorrelate
82.6:1011
91.
09.
83.2:10.1
86.5
-Pyramid
83,5=002
91.0+003
87.9=0.1
80,5+-0.1
85.7++0.1
McCo-v3 EVA-02 DINO-12
-Augment 58,4=002
82.6+00.
74,0=0.3
59.6:
68.6++0.4
Full (Ours) 75.4 1.0.3
1..1
76.61.0.1
78.2 1.01
77.8 1.0.2
-Decorrelate
69.4
79.5 1..1
76,8
10.1
79.9..02
76.4..0.2
-Pyramid
57.3=+0.2
74.5+:0.2
68.5+00.
68.1
67.1
-Augment 35.3:+0.4
31.6::0.2
34.3+00.5
32.2:+0.5
33.3++0.4
Full Dataset 92.5 95.2.10.1 94.1:10.1 89.4.003
92.8
Table 3: Evaluating Ablations:
While all three components provide improvements, the Augmentation has the most dra- matic effect, especially in the cross- model setting. Likewise, the Pyramid optimization seems to matter more in the cross- model setting than the same-model setting by mitigating over- fitting to the model used during distillation.
Ours
-Decorcelate -Pyramid -Augment
Figure Visualizing Ablations: Removing various components of our pipeline causes visual degradation in the distilled images.
and averaging performances across the other three models (bottom). Figure visualizes the distilled Pineapple class from ImageNet-100 100 under the same ablations.
Color Decorrelation Inspired by prior work in feature visualization [30] and motivated by a desire to spare our synthetic images from any color-related biases imposed by the model used to distill them, we apply a color decorrelation technique to our distillation process. In Figure 5, we see that the images distilled without the color decorrelation (column 2) look less realistic than those using our full method (column 1); they are over saturated and contain high levels of incorrect colors (blue in this example). Quantitatively, however, this component has the lowest effect of the three analyzed in this section. While generally slightly helpful, it only offers a large improvement in the cross-model setting when distilling using CLIP.
Pyramid Representation It has been shown in prior work [7] that pixel- optimization for dataset distillation does not scale well to higher resolutions; the synthetic images tend to become riddled with
patterns that inhibit their usefulness as training data. As discussed in Section 3.2, we instead adopt pyramid representation for the distillation process. In Figure 5, we see samples distilled from ImageNet- 100 optimized using either the pyramid (column 1) or naive pixel representation (column 3). We encourage the reader zoom in and observe the high-frequency patterns and overall lack of coherence in the pixel-basee based images. In Table 3, we see that the images distilled without the pyramid representation do indeed make far worse training data in the cross-model setting, causing the model to overfit to the high-frequency patterns.
Differentiable Augmentations Since its first proposal [53], the incorporation of differentiable augmentations during the distillation process has proven critical the efficacy of the synthesized images as training data. As such, we also perform differentiable augmentations in this work, as described in Section 3.3. In Figure 5, we see samples from ImageNet- 100 distilled both with (col- umn 1) and without (column 4) augmentations. Visually, we observe that the images distilled without augmentations lack much mcaningful structure and appear to simply be blobs of color with just a hint of geometry. In Table 3, we see that the lack of augmentations severely limits the synthetic images' usefulness as training data, even when evaluation on the same model used to distill.
As earlier discussed, we also apply multiple rounds of augmentations at each distillation step and concatenate the multiple augmented versions together to form the synthetic batch. Figure illustrates how the number of angmentations per batch affects both the single-model and cross moosl perfor- mance of the distilled data. Specifically, we see that when distilling ImageNet- with DINO-v2, raising the number of augmentations per batch from to 5 and eventually 10 steadily increases the distilled images' effectiveness as training data both on the backbone model used to distill (DINO-v2) and the other unseen models (CLIP, EVA-02, MoCo-v3).
7

---

McCo-v3 DINO-v2
Train
Test
Bulldog
Corgi
Dachshund
Labrador
Buildog
Corgi
Dachshund
Labrador
Real Images
Distilled Images
Figure 6: Distilling Datasets with Spurious Correlations: The 4-class "'ppawrious" dataset contains spurious
background correlations in the training set that are then subverted in the test set (left). A DINO linear probe trained the full training performs well, reaching 78% test accuracy. while MoCo probe fails catastrophically, only reaching 36%. The distilled images (right) hint as to why: those distilled with DINO-v2 still contain mostly decipherable dog breeds while the MoCo-v3 counterparts focus almost entirely on the background environments. These images likely reflect the same biases held by the models used to distill them.
Train Set Img/Cll)
Spawrious
WaterBirds
CLIP
DINO-v2
EVA-02
McCo-v3
Average
CLIP
DINO-v2
EVA-02
McCo-v3 Average
Distilled (Ours)
43.1:1644
80.8 13.1
36.5
32.7
48.7
17.9
82.1
78.0
77.8
1..0
Neighbors
41.7:14.6
76.9
40.7:127
33.0
43.1
74.6
67.3
14.3
45.2
65.3
Centroids
:5.8:1500
80.2
38.1
30.5
48.4
69.8
65.7:334
64.6
62.0::5.2
65.5 #6.
Random
46.2
68.148.1
33.3 5:3
31.8
1.25
44.9
71.9445
57.1
59.8
67.4
64.0
Full Dataset
50.4 78. 1:0.1 50.1
35.8:10.0 53.6:0.2 86.0+...1 95.5+0.1 90.4
74.3:E0.1
86.5.2..1
Table 4: Performance on Datasets with Spurious Background Correlations: The Spawrious (left) and Water- birds (right) training sets contain intentionally adversarial background correlations that are then subverted in the test sets. On Spawrious, our method no longer out out-performs the real-image baselines as with the standard datasets (Table 1). This is perhaps due to the synthetic data adopting the models' biases and overfitting to the backgrounds on the training sets. We also see interesting interpretabillity results in the images themselves (Figure 6).
4.4 Distilling Adversarial Datasets Reveals Interpretable Model Weaknesses
The "Spawrious"" dataset [26] consists of four classes of dog breeds and is designed to evaluate model's ability to focus on the relevant content of an image rather than spurious correlations. In the training data, each breed is paired with its unique environment: (Bulldog, Forest), (Corgi, Desert), (Dachshund, Mountain), and (Labrador, Snow). However, in the test set, the environments are inten- tionally different: (Bulldog, Mountain), (Corgi, Forest), (Dachshund, Snow), and (Labrador, Desert). We also experiment with a similarly constructed data, "Waterbirds" [38] consisting of just two classes (land birds and water birds) with similarly spurious background correlations.
Quantitative results on these datasets are shown in Table 4. Our method generally outperforms the real-image baselines, but to a significantly smaller extent than the standard datasets (Table 1). We also observe that when training on the full Spawrious dataset, DINO-v2 [31] far out-performs the rest of the models (81 % test accuracy) with MoCo-v3 [10] doing dramatically worse (37%), both to degrees far greater than seen when training on standard datasets (as in Table 1).
By distilling Spawrious with method, we gain potential insights as to why we might such dramatic results. As seen in Figure 6, the images distilled with DINO-v2 contain clear (albeit abstract) depictions of the correct dog breeds. On the other hand, those yielded by MoCo-v3 are almost entirely indecipherable, with the only clear portions being the spurious backgrounds from the training set. This interpretability result gives clue towards explaining MoCo-v3's poor performance on this dataset; the model clearly seems to focus on the spurious background correlations present in the training set rather than the relevant subjects, causing catastrophic overfitting.
4.5 Distillation Excels in Fine-Grained Visual Classification
While computer vision methods are typically benchmarked against datasets containing broad cate- gories of objects, such as ImageNet [12], much research also focuses on the task of Grained Visual Classification (FGVC) wherein datasets consist of many closely related classes that can be
8

---

Border Toriir
Irick Temrir
Norlolk Tearier
Nerwich Temner
Yarkehire Terrer Lakelend
Tenier
Whestinn Tetrier Auttralinn Terrier
Silky Tarker
Saalyhum Terricr
Bard Spartow Beewer Spariow House Spurrow
Fleld Sparrow
Fox Sparro
Hams Sparrow Lincodn Sparrow
Song Sparrow
Tiee Spanow
Vesper Spanow
Figure 7: Distilling Fine-Grained Datasets:
Our distillation method captures the details necessary to teach a
classifier to distinguish between highly similar classes. Pictured above are just 10 of the 120 classes distilled from the Stanford Dogs [21] dataset (top) and 10 of the 200 classes distilled from Caltech-UCSD Birds [45] using DINO-v2. For full distilled datasets, please see the Appendix or our project page.
Train Set
Stanford Dogs
CUB-2011
(1 Img/Cls)
CLIP
DINO-v2
EVA-02 MoCo-v3 Average
CLIP
DINO-v2
EVA-02
MoCo-v3 Average
Distilled (Ours)
5..1
83.0
74.8 10.1 69.6
102 62.2.102
86.0
74,1 10.2
42.5
66.2.102
Neighbors
33.4110.1
71,
58.5
56.3
54,9
39.410.1
76.9
52.6
103
28.1
49.2.00.1
Centroids
43.3
73.0
60.9
55.2
58.1
54.3.002
78.5
59.9103
30.2..01
55.7
Random
23.3:115
11
38.3
36.61114
1..
37.5
64.4
44.3
19.1100
1..
Full Dataset
76.9+t0.1
88.6
82.6#0.1 72.3 80.1 77.5
90.2
84.0
43.74-08
73.8
Table 5: Performance on Fine-Grained Datasets: Our distillation method captures the most discriminative aspects of each class, therehy enabling down-stream classifier to correctly identify samples from datasets where all classes are closely related. In particular, the performance gap between our method and the real-image baselines even higher these fine-graine mraiggained datasets (Stanford Dogs [21] and CUB-200-200 [45]) than on the standard ImageNet benchmarks.
challenging for even human experts to distinguish. Such datasets require models to learn fine- Ene-grained features in order to identify the subtle differences between each class.
We apply our linear method to two common FGVC datasets: Stanford Dogs [21] and Caltech-UCSD 2irds-200-2011 [45]. Samples these datasets distilled using DINO-v2 can seen Figure 7. As seen in Table 5, in this more challenging fine-grained setting, our method outperforms the real-image basclines to significantly higher degree than on the standard ImageNet benchmarks (Table 1). It seems as though the distillation process's ability to store didactic information in single image per class is even more relevant in this setting since any one real image is less likely to contain the information necessary to distinguish its entire respective class from the other highly similar classes.
4.6 Cross-Model Performance Predicts Model Alignment
Recent research has suggested an "alignment" of various self-supervised models despite large differences in their training methods even data modalities. The notion that all these models are indeed converging to a unified embedding space has been dubbed the "Platonic Representation Hypothesis" [20]. While the representations of today's 's models not yet perfectly synchronized, it is still of interest to measure the degree to which they are aligned.
In the hypothesis's titular work, the authors introduce method of measuring the alignment of two models called "mutual nearest neighbors." " In short, this method finds each sample's nearest neighbors in the embedding spaces of each model and then computes the fraction of neighbors that are shared between the two.
When we distill images with given model A and measure the I nearest neighbor performance (since there is just one image per class) on a differrnnt model B, we find that the normalized test accuracy is strongly correlated with the mutual k-nn alignment between models A and B, as seen in Table 6. This is an interesting interpretability result showing that the cross- moosll performance of a distilled dataset highly depends on the alignment between the two models. Given this dependency. our distillation method then acts as a method of visualizing how well these models align by backing the discrepancies in the embedding space out to image space. For example, given the two least aligned models (CLIP and MoCo), we can see in Figure that even for different sets of classes, the two models clearly distill very different styles of images, alluding towards their misalignment.

---

Distill Model
Normalized k-NN Accuracy
Source Model
Model Alignment
CLIP
DINO-v2 EVA-02 MoCo-v3
CLIP
DINO-v2
EVA-02
MoCo-v3
CLIP
69.5
72.3
46.6
CLIP
0.21
0.26
0.18
DINO-v2
77.2
95.5
91.7
DINO-v2
0.21
0.39
0.31
EVA-02
66.1
70.2
EVA-02
0.26
0.39
0.30
MoCo-v3 57.2
87.4
76.8
MoCo-v3
0.18
0.31
0.30
Table 6: Distilled datasets predict model alignment. We distill ImageNet- Ik and find the synthetic data's cruss-model performance (left) by evaluating on model other than one used during distillation. We find that this cross- performance correlates well with the alignment between models' embedding spaces (right). Note the similarity of the per-row trends between the two tables. Rows are colored from highest to lowest.
Neighhors Distilled
Nouveou
Baroque
Expressionism Impressionism Post-Impres.
Realium
Rommiticism
Suateal.sm
Likyo-e
Figure 8: Distilling Distrr-Distritttion Datasets: Despite only ever being trained on the worldw images in ImageNet-IK [12], our method can still use DINO-v [5] to distill out-of-distribution datasets such as ArtBench [24] that have no overlapping content. We that the distillation is not simply "copying" samples from the dataset by comparing to the nearest real neighbor the model's embedding space and observing the stark diffcrences. This interpretability insight granted by our method highlights DINO's remarkable ability to generalize beyond its training distribution.
4.7 Self-Supervised Backbones can Distill Out-of-Distribution Datasets
Note that while we have thus far been discussing DINO-v2, this section in particular will reference DINO-v1 since this version's 's training set is open-sourced. Since DINO-v1 [5] was only trained on ImageNet [12], a dataset of nearly all "real-world" images, we can easily test its ability to distill an out-of-distribution dataset such as ArtBench [24] that consists 10 classes of art styles. In Figure we visualize the distilled images (top) as well as the nearest neighbor in DINO-vI's embedding space. Surprisingly, despite having previously only ever seen woollw images from ImageNet, DINO-v1 is still able to effectively distill ArtBench into single image per class. Furthermore, confirm that the distilling is not simply "copying" real images from the training set by observing the stark differences between the synthetic images and their nearest neighbors. This is especially apparent in the cases of Realism, Renaissance, and Romanticism where the nearest neighbors are black and white while the distilled images colorful.
5 Conclusion
In this work, we introduce new task in the area of Dataset Distillation: learning tiny set of synthetic images designed train linear probes on top of pre-trained self-supervised vision models. Our proposed solution, Linear Gradient Matching, optimizes a meta gradient objective to ensure that our synthetic images induce training updates similar to those obtained from the real data. Quantitatively, our method outperforms all baselines, enabling, for example, DINO-v2 linear probe to reach 75% test accuracy on ImageNet- 1k while only having ever seen one labeled image per class.
We also showcase the importance of secondary aspects of our Linear Gradient Matching method, such the pyramid representation and differentiable augmentations, and highlight their importance in learning highly efficacious distilled datasets. Furthermore, we evaluate our method on challenging fine- grained datasets and show that out-performs the real-image baselines by an even larger margin than on the standard ImageNet benchmarks. Lastly, our method yields several interesting interpretability results, such as giving insights into how these pre-trained models "see." predicting how well different models align, or elucidating a model's 's ability to generalize beyond its training distribution.
We hope work brings to light the interesting of problems posed by this task and inspires others to continue this area. Code and distilled datasets can be found on our project page.
10

---

References
[1] Adam McLean. The surrealism website. attps://surrealiiee website/. 15
[2] J. Ansel, E. Yang. H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski,
G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. DeVito, E. Ellison, W. Feng, J. Gong. M. Gschwind, B. Hirsh, S. Huang. K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, Liang. Lu, C. K. Luk. B. Maher, Y. Pan, C. Puhrsch, M. Reso, M. Saroufim, M. Y. Siraichi, H. Suk, S. Zhang. M. Suo, P. Tillet, X. 7hao, E. Wang, K. Zhou, R. Zou, X. Wang. A. Mathews, W. Wen, G. Chanan, P. Wu, and S. Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPL.OS '24, New York, NY, USA, 2024. Association for Computing Machinery. 14, 15
[3] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014. 15, 16,
[4] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary. D. Maclaurin, G. Necula. A. Paszke, VanderPlas. S. Wanderman- Wanderman-Milne, Milne, and Q Zhang. JAX: composable transformations of Python+NumPy
programs, 2018. 14
[5] M. Caron, H. Touvron, Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in sulf-supervised vision transformers. In ICCV, 2021. 3, 10, 15
[6] G. Cazenavette,, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu, Dataset distillation by matching training trajectories. In CVPR, 2022. 2, 3, 4,
[7] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu. Generalizing dataset distillation via deep generative prior. In CVPR, 2023. 6, 6,7
[8] T. Chen, Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. JCML, 2020. 3
[9] X. Chen, H. Fan, R. Gishick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
[10] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In ICCV, 2021. 3, 5. 8, 16, 20
[11] J. Cui, R. Wang. S. Si, and C.-J. Hsieh. Scaling up dataset distillation to imagenet- Ik with constant memory. In ICML, 2023. 1,3
[12] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li. and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 5, 8, 10, 15, 16, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30
[13] Y. Fang, Q Sun, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva-02: A visual representation for ncon genesis. Image and Vision Computing, 2024. 3, 15, 19
[14] Y. Fang, W. Wang, B. Xie, Q. Sun, I.. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023. 3,5, 16
[15] S. Fort and J. Whitaker. Direct ascent synthesis: Revealing hidden generative capabilities in discriminative models. arXiv preprint
2025. 4
[16] Z. Guo, K. Wang, G. Cazenavette, H. I.I, K. Zhang, and Y. You. Towards lossless dataset distillation via aifficulty-aligned trajectory matching. In ICLR, 2024. 1,3
[17] K. He, X. Chen, S. Xie, Y. Li, Dollár, and R. Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022. 3
[18] K. He, Fan. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3
[19] Hugging Face. Hugging Face: The Al community building the future. https:////// http://hggggggggggeccooo 2016. 15
[20] M. Huh, B. Cheung, T. Wang, and P. Isola. Position: The platonic representation hypothesis. In ICML, 2024. 2, 3.9

---

[21] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel dataset for fine-grained image categorization.
In CVPR. 2011. 5,9, 15, 16,31
[22] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICIR, 2015. 14
[23] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, T. Lee, E. David, I. Stavness, W. Guo, B. A. Earnshaw, I. S. Haque, S. Beery, J. Leskovec.
A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang- WILDS: A benchmark of wn-lhe-wild distribution shifts. In ICML, 2021.
[24] P. Liao, X. Li. X. Liu, and K. Keutzer. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint arXiv:2206.11 1 1104, 2022. 5, 10, 15
[25] D. Liu, J. Gu, H. Cao, C. Trinitis, and M. Schulz. Dataset distillation by automatic training trajectories. In ECCV, 2024.
[26] A. Lynch, G. J.-.. Dovonon, Kaddour, and Silva. Spawrious: A benchmark for fine control of spurious correlation biases, 2023. 5, 8,
[27] T. Nguyen, Z. Chen, and J. Lee. Dataset meta-learning from kemel ridge-regression. In ICI.R, 2020. 1,2
[28] T. Nguyen, R. Novak, L. Xiao, and J. Lec. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, 2021.
[29] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), Kolkata, India, 2008. 15, 16,
[30] C. Olah, A. Mordvintsev, and Schubert. 4,7
Feature visualization.
Distill, 2017.
[31] M. Oquab, Darcet, T. Moutakanni, H. Vo, M. Szafranice, Khalidov, P. Fernandez, D. HAZIZA,
F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang. S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnacve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024. 3, 5, 6, 8, 15, 16, 18, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35
[32] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Vetioo, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation pytorch. In ICLR Workshop, 2017.
[33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein,
I. Antiga, A. Desmaison, A. Kopf, F. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, rer--performance deep learning library. In NewrIPS. 2019. 14, 15
[34] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 3,
5, 15, 16, 17
[35] J. Resig, Ukiyo-e scarch.
2012. 15
[36] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski, Kornia: an open source differentiable computer vision library for pytorch. In WACV, 2020. 15
[37] C. Ryali, Y.TT. Hu, D. Bolya, C. Wei, H. Fan, P.-Y. Huang, V. Aggarwal, A. Chowdhury, O. Poursaced, In /CML, 2023.
J. Hoffman, et al. Hiera: A hicrarchical vision transformer without the
[38] S. Sagawa, P. W. Koh*, T. B. Hashimoto, and P. Liang. Distrihutionally robust neural networks. In ICLR, 2020. 5, 8, 15
[39] S. Sagawa, P. W. Koh, T. Lee, I. Gao, S. M. Xie, K. Shen, A. Kumar, W. Hu, M. Yasunaga, H. Marklund,
S. Beery, E. David, I. Stavness, W. Guo, J. Leskovec, K. Saenko, T. Hashimoto, S. Levine, C. Finn, and
P. Liang. Extending the wilds benchmark for unsupervised adaptation. In ICLR, 2022. 15
[40] B. Son, Y. Oh, D. Back, and B. Ham. Fyi: Flip your images for dataset distillation. In ECCV, 2024. 1,3
[41] Q Sun, Y. Fang, L. Wu, X. Wang. and Y. Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 2xXiv22303,15388 20231,
12

---

[42] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In ECCV, 2020. 5, 15, 16, 17. 18, 19,
[43] TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https: wwt.ub.......ttteeeeeeeeeee
2016. 15
[44] M. Tschannen, A. Gritsenko, X. Wang, M. E. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans,
L. Beyer, Y. Xia. B. Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502. 14786, 2025. 3
[45] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-UCSD birds 200. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 5, 9, 16, 33
[46] K. Wang, B. Zhao, X. Peng, Z. Zhu, S. Yang. S. Wang. G. Huang, H. Bilen, X. Wang, and Y. You. Cafe: Learning condense dataset by aligning features. CVPR, 2022. 1
[47] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint arXiv: 1811.0999. 2018. 1,2
[48] R. Wightman. Pytorch image models. httpa://hogginggtn co/t mm, 2019. 15
[49] WikiArt.org. WikiArt Visual Art Encyclopedia. https: /www. wikiart. org /, 2024. 15
[50] Z. Yin and Z. Shen. Dataset distillation via curriculum data synthesis in large data era. 2024.
[51] Z. Yin, Xing. and Z. Shen. Squeeze, recover relabel: Dataset condensation imagenet scale from new perspective. NeurIPS, 2023. 1,2
[52] Zhai, Mustafa, Kolesnikov. and Beyer. Sigmoid loss for language image pre-training, ICCV, 2023. 3
[53] B. Zhao and H. Bilen. Dataset condensation with differentiable siamese augmentation. ICML, 2021. 1,
4, 7
[54] B. Zhao and Bilen. Dataset condensation with distribution matching. WACV, 2023. 1,2
[55] B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. In ICLR, 2020. 2,3,4
[56] Y. Zhou, E. Nezhadarya, and J. Ba. Dataset distillation using ncural feature regression. Newr/PS, 2022. 1,2
13

---

Appendix
To begin the appendix, we first speak on our method's limitations, broader impact, and compute budget. In the following sections, we include more information that was omitted from the main paper due to space constraints. Section A, we provide more detailed description of our method's implementation details. In Section C, we include visualizations of complete distilled datasets.
Limitations Our method is largely limited by memory and data loading. For example, we were lim- ited in our ImageNet-1K lk experiments to only using 3 rounds of augmentations per batch due to data loading bottleneck. We hope eventually develop method that does not require the loading of thou- sands of real images per step. Additionally, due to the bi-level optimization, we are limited to using Py- torch's [2, 33] nn. siistributee ara 1 el, rather than nn. Distr www.rruueeddD. aatataPrra Paralla lell which causes significant slowdowns due to our very large batch sizes. Using a different automatic differentiation framework, such as Jax [4], might alleviate this issue, but this would also require porting all our sulf-super backbones to the new framework.
While our method quantitatively out-performs the real image baselines, the broader impacts likely lie in the introduction of this new task well as the resulting images acting as an interpretability tool. As shown the paper, the distilled images offer insights into things such as model alignment and how different networks see the world in varying ways.
Broader Impact
Furthermore, the ultimate goal of dataset distillation is the reduction of training time, which equates to less energy spent. A linear probe trained on our distilled datasets can be trained to convergence in just few minutes while training on the full dataset can take up to an entire day.
Compute Budget We used a variety of GPUs for this work depending on what available on the shared cluster. Specifically, we used a combination of H200, A 100, L40s, Ada6000, and 4090 GPUs. Distilling ImageNet-100 with the default settings takes about hours using H200 GPU, and ImageNet- Ik takes about 12 hours using 4 H200 GPUs.
A Implementation Details
In this section, we provide more detailed explanation of our methodological and experimental implementation details.
A.1 Distillation
We implement our method in Pytorch [2, 33], which has unique license which we comply.
We optimize our pyramid representations using Adam [22] with a learning rate of 0.002.
We distill for 5000 iterations and add a new level to the pyramids every 200 iterations until we reach the maximum resolution (256).
Each level of the pyramid is initialized with (O, 1) normalized by the current number of levels in the pyramid. When new level is added, the existing levels are appropriately re-normalized.
At each distillation iteration, we first sample a new linear classifier consisting of a weight matrix W and bias vector b using the default initialization for Pytorch's 'ss nn. Linear. For simplicity, we omit the bias the main paper.
Then, we reconstruct the synthetic images from their pyramid representations using the following equation (copied from the main paper for convenience):
1 X
tanh
Σ TEP
resize250 (P))
2
The synthetic images are then augmented several (10 by default) different (not sequential) times and concatenated together to form the full synthetic batch. We obtain our synthetic loss (fayn) by passing this batch through the feature extractor Φ and linear classifier (W, b) and then compute the gradient of the synthetic loss with respect to the linear classifier (and bias). These gradients are then vectorized.
14

---

For our real batch, we sample a number of real images equal to the length of the full synthetic batch (i.e.. after the different augmentations and concatenation) and augment them once (since the batch size already matches that of the full synthetic batch). We then get the real loss and gradients via the same procedure. The meta loss is then calculated as the cosine distance between the real and synthetic vectorized gradients.
A.2
Augmentation
For our differentiable augmentations. we use a custom re-implementation torchvision trans- forms that run on GPU and support different random transforms for each element in batch. We ini- tially used the Kornia [36] implementations, but found that they cause CPU bottlenecks with very large batch sizes. We use RandomHor zontalFlis with default parameters, RandomRes i zedCrop with size= (224, 224), , and RandomGauss Na se with std=0.2.
A.3 Evaluation
Linear Probes. To evaluate given distilled dataset (or selected coreset for the real baselines) on given target model, we first randomly initialize new linear classifier. We then train the linear classifier for 1000 epochs with a batch size of 100. We use an Adam optimizer with learning rate of 0.001 0.001/256 (taken from the DINO-v1 [5] evaluation protocol) along with a cosine decay learning rate schedule. We stop training early if the test accuracy has not improved over the last 50 epochs. For the training set, we perform the same set augmentations as during distillation (horizontal flip, random resized crop, and Gaussian noise). The output of the random resized crop is of size 224 224. For the test set, we resize the shortest side to 256 and then do a center crop of 224x2244
Nearest Neighbors. For our nearest neighbor evaluation, by resizing the shortest side of each test image to 256, taking a 224x224 center crop, and passing through the feature extractor. We then do the same for the test set and find the training embedding closest to each test embedding (by cosine distance) and report the class of said neighbor. We only do one nearest neighbor since distill down one image per class.
first find the train set embeddings
A.4 Datasets
For ImageNet [12] and ImageNet [42], we build on the Torchvision [43] implementation as part the
Pytorch [2, 33] ecosystem. ImageNet has unique non-commercial license, to which we have agreed.
Spawrious [26] is taken from the paper's official GitHub repository and uses CCO-1.0 license.
Waterbirds [38] is taken from the WILDS [23, 39] benchmark. Both the Waterbirds and WILDS repositories an MIT license.
Stanford Dogs [21] is taken from the project website. It presumably uses the same license as ImageNet [12].
Caltech-UCSD-200
is taken from the project website, No license is given.
ArtBench [24] taken from the paper's GitHub repository. The repository itself uses an MIT license.
The data is sourced from WikiArt [49], Ukiyo-e Search [35], and The Surrealism Website [1], all of which have Fair Use licenses.
Flowers- 102 [29] is taken from Torchvision. No license is given.
Food-101 [3] is taken from Torchvision. No license given.
A.5 Models
For CLIP [34], we use the official GitHub repository which has an MIT license.
For DINO-v1 [5], we use the official GitHub repository which has an Apache-2.0 license.
For DINO-v2 [31], we the official GitHub repository which has Apache-2.0 license.
For EVA-02 [13], we use the Pytorch Image Models [48] implementation hosted on Hugging Face [19]
which has an MIT license.
15

---

Train Set
Flowers- 102
Food-101
Img/Cls)
CLIP
DINO-v2
EVA-02 MoCo-v3
Average
CLIP
DINO-v2
EVA-02
MoCo-v3
Average
Distilled (Ours)
79,5
9..
97.8 0.1
13.4
83,3
78.7
83.7
83,9 10.2
31,61:32
69.5
Neighbors
69.4::0.8
:9.4
9088:02
48.6
77.0
58.8
73.8:0.4
71.5
25.2
57.3
Centroids
77.5
99.4+0.1
95.6403
48.24226
80.2=0.9
74.1
75.6=0.1
76,44:4
22.74:20
62.2.0.8
Random
67.9
98.8=044
91.0=088
36.8+2.2
73.6=1.1
48.5++10
52,8=19
49.9::06
13.5
41.22
Dataset
93.4
99.7.10.0 98.91000 82.2
93.5 1.111
91.9.000
92.7
92.0.10.0
78.3..0.0 88.7 1.000
Table B.1: Additional Datasets: Distilling Flowers- 102 [29] and Food-101 [3] show similar results; distilled dataset consistently out-perform the real-image baselines.
For MoCo-v3 [10], the official GitHub repository which has Creative Commons Attribution- NonCommercial 4.0 International Public License.
B Additional Results
In Table B.1, we include results for Flowers- -0verss-120// [29] and Food-100 1ood-1021g [3]. We see results following the same trends as the rest of the paper, the distilled datasets out-perform the real cel--image baselines across the board.
C Visualizing Full Datasets
In Figures C.I. C.2, C.3, and C.4 present the entirety of ImageNet-1000 [42] distilled with CLIP [34], DINO-v2 [31], EVA-02 [14], and MoCo-v3 [10] respectively. Figures C.5-C.14, we show ImageNet- 1k [12] distilled with DINO-v2 and omit the other models for the sake of brevity (and ink).
We also include Stanford Dogs [21] (Figure C.15), CUB-200-2011 [45] (Figures C.16 and C.17),
Flowers- 102 [29] (Figure C.) and Food-101 [3] (Figure C.19) distilled with DINO-v2 and once again omit the other models. All distilled datasets, including those not inculded here, can be viewed on our project page.
16

---

Figure C.I: ImageNet- 100 [42]
distilled using CLIP [34]
17

---

Figure C.2: ImageNet- 100 [42] distilled using DINO-v2 [31]
18

---

Figure C.3: ImageNet- 100 [42] distilled using EVA-02 [13]
19

---

Figure C.4: ImmaeeNet 100 [42] distilled using MoCo-v3 [10]
20

---

Figure C.S: ImageNet-Ik [12] distilled with DINO-v2 [31] classes [0-99]
21

---

Figure C.6: ImageNet- Ik [12] distilled with DINO-v2 [31] classes [100-199]
22

---

Figure C.7: ImageNet- 1k [12] distilled with DINO-v2 [31] classes [200-299]
23

---

Figure C.8: ImageNet- 1k [12] distilled with DINO-v2 [31] classes [300-399]
24

---

Figure C.9: ImageNet- 1k [12] distilled with DINO-v2 [31] classes [400-499]
25

---

Figure 10 ImageNet-1k 1k [12] distilled with DINO-v2 [31] classes [500-599]
26

---

Figure C.II: ImageNet-Ik 1k [12] distilled with DINO-v2 [31] classes [600-699]
27

---

Figure 12 ImageNet-Il 1k [12] distilled with DINO-v2 [31] classes [700-799]
28

---

Figure C.13: ImageNet-Il 1k [12] distilled with DINO-v2 [31] classes [800-899]
29

---

Figure C.14: ImageNet-Ik [12] distilled with DINO-v2 [31] classes [900-999]
30

---

Figure 15: Stanford Dogs [21] distilled using DINO-v2 [31]
31

---

Figure C. LUB-200-2011 [45] distilled using DINO-v2 [31] classes [0-99]
32

---

Figure C.17: CUB-200-2011 [45] distilled using DINO-v2 [31] classes [100-200]
33

---

Figure C.18: Flowers- 102 [29]
distilled using DINO-v2 [31]
34

---

Figure C.19: 19: Food-101 [3] distilled using DINO-v2 [31]
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our work introduces the new method, Linear Gradient Matching, and presents the results and insights described the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
36

---

Justification:
We do not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss our experimental implementation high level in the main paper and in-depth in the appendix. We will also release the source code after having time to make it more user-friendly.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
public/ guides/ Souides/CodeSuuaeie Cowe/ seroomssssinns ssionpoicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cc/ cc/publlccccccccccccccccccccccc c/gu i des /CodeSubunn ss onPol ecy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We discuss the most relevant experimental settings in the main paper and provide an in-depth explanation in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Ye]]
Justification: Results are reported over 5 runs as mean std.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
38

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma erro bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We include section on compute resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp::///neuuue /neurips....p// cc hwtpppblccc/ttiieeuuueeeeeeeeeee
Answer: [Yes]
Justification:
We have read the code of conduct, and our research conforms it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include a broader impacts section in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
39

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks, as we do not release any pre-trained models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all code, data, and models used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw's thoode com/ dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
40

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor reserach with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor reserach with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
41

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work does not use LLMs at all.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: / /////reerppppp//epe/eee cc/Conferences / 2025/ LLM) for what should or should not be described.
42


Reviews:
[{"review_id": "gGAnNnz3h8", "reviewer": "Reviewer_Kyf1", "summary": "The paper tackles the problem of dataset distillation for linear classifiers built on top of fixed, pre-trained vision encoders. It proposes a method that optimizes synthetic images such that, when passed through the frozen feature extractor, they induce gradients that closely match those produced by real training data. The approach is evaluated across a range of popular vision encoders, including MoCo v3, DINO v2, EVA v2 and CLIP. Experimental results show that the proposed technique outperforms baselines of nearest neighboors, random images and centroids on several benchmarks. The paper also investigates its application to model interpretability, suggesting that the method offers insights into the relationship between input images and classifier behavior.", "questions": "I found the experiments in Table 6 particularly interesting. While the models achieve similar performance on ImageNet, there is a significant gap between DINO v2 and MoCo v3 on the spurious background dataset. If I understand correctly, all \"DINO\" models in the paper refer to DINO-v2. (A minor suggestion is to consistently use “DINOv2” throughout the paper to avoid confusion.) Given that both DINOv2 and MoCo v3 are based on contrastive learning, this performance gap raises intriguing questions. One possible explanation is the use of a patch-level loss in DINOv2, which might make it more robust to spurious correlations. Alternatively, it could be related to the use of smaller crops in DINOv1. Including experiments on DINOv1 could help disentangle these factors and provide further insight into the source of robustness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and clearly presented.\n\n- It conducts thorough ablations of key components in the proposed technique, including Pyramid Representations, and Differentiable Augmentations, showing that both are crucial for achieving strong performance.\n\n- The method is evaluated across a variety of pre-trained vision encoders and includes insightful analyses such as:\n\n  - Cross-model performance comparisons,\n\n  - Evidence supporting the platonic representation hypothesis,\n\n  - Identifying spurious correlations and analyzing out-of-distribution behavior.\n\nWeaknesses:\n\n- The motivation for focusing specifically on dataset distillation for linear classifiers is underdeveloped:\n\n  - Why is this particular task important?\n\n  - Why prioritize linear probing over more general settings like full model fine-tuning?\n\n- While the dataset distillation technique is well-executed, the paper might have been even stronger if it had been framed primarily as a study in model interpretability, using dataset distillation as a tool rather than the main focus. In this framing, the cross-model comparisons and analyses could have been pushed further to explore deeper insights into the behavior and differences among pre-trained models.", "was_revised": true, "final_justification": "My questions are properly addressed by the rebuttal. I have raised my confidence score accordingly.", "rebuttal": "Thank you for your constructive comments. \nThe insights provided have already improved our paper and inspired us to continue pushing along this line of research.\n\nWe hope to resolve your concerns by first addressing your broader points followed by the specific questions.\n\n## Addressing \"Weaknesses\"\n>The motivation for focusing specifically on dataset distillation for linear classifiers is underdeveloped:\n>Why is this particular task important?\n\nThe traditional field of dataset distillation has somewhat stagnated; \nwhile promising results have been seen on toy datasets like CIFAR, achieving even moderate success on more realistic \ndatasets like ImageNet requires so many synthetic images per class that it almost goes against the spirit of the task. \nFor ImageNet-1k, from-scratch dataset distillation methods have not come close to 5% accuracy at 1 image per class, \nseverely limiting the distilled data's usefulness.\n\nGiven that most vision tasks are solved today by starting from a pre-trained feature extractor rather than training a \nmodel from scratch, we feel it is time for dataset distillation to explore this area as well. \n\nWhile prior distillation methods worked well in small-scale settings, none of them scaled up to large models or \nrealistic datasets (without distilling a very large number of synthetic images per class), calling into question the field's practical usefulness. \nUsing our method, it is possible to train a linear classifier to nearly **75%** test accuracy on ImageNet-1k with just a \nsingle image per class in under a minute. Since most models used today start from a pre-trained backbone, the data \ndistilled with our method show potential for practical use cases rather than just toy problems.\n\n> Why prioritize linear probing over more general settings like full model fine-tuning?\n\nThank you for raising this question, as the answer is not immediately intuitive.\n\nWhen distilling a dataset down to just one (or a handful of) image(s) per class, fine-tuning an entire large vision \nmodel on such a small set of images tends to lead to catastrophic overfitting. \n\nOverfitting to the distilled dataset is a known issue of dataset distillation, but training a model of such high \ncapacity on the distilled dataset dramatically exacerbates the issue. \n\nBelow, we include a table of results obtained by fine-tuning the models on the distilled data from ImageNet-100. \nWe observe the expected trend: fine-tuning the models on the tiny distilled datasets leads to catastrophic overfitting.\n\nWe will touch on this briefly in the main paper and include the quantitative results in the appendix.\n\n#### ImageNet-100, Fine-Tuning on Distilled Images\n| **Train Set (1 Img/Cls)** | **CLIP**      | **DINO**      | **EVA**        | **MoCo**        | **Average**    |\n|---------------------------|---------------|---------------|----------------|-----------------|----------------|\n| **Distilled (Ours)**      | 2.7 ± 0.3     | 3.7 ± 0.2     | 12.7 ± 6.7     | **46.6 ± 12.9** | **16.4 ± 5.0** |\n| **Neighbors**             | 3.1 ± 0.2     | 4.2 ± 0.4     | 14.4 ± 9.8     | 33.5 ± 13.8     | 13.8 ± 6.0     |\n| **Centroids**             | **4.4 ± 0.3** | **4.8 ± 0.4** | 17.8 ± 11.9    | 38.0 ± 13.0     | 16.2 ± 6.4     |\n| **Random**                | 2.5 ± 0.3     | 2.6 ± 0.5     | **32.9 ± 2.5** | 21.1 ± 1.9      | 14.8 ± 1.3     |\n| **Full Dataset**          | 92.5 ± 0.0    | 95.2 ± 0.1    | 94.1 ± 0.1     | 89.4 ± 0.3      | 92.8 ± 0.1     |\n\n> ... the paper might have been even stronger if it had been framed primarily as a study in model interpretability, using dataset distillation as a tool rather than the main focus.\n\nThank you for this suggestion! \n\nIndeed, we also find the interpretability results to be some of the most exciting in the paper. \nWe have reworded the results section to further highlight these results and emphasize the distillation method's usefulness \nas an interpretability tool. \n\nWe will more aggressively pursue this angle in future works now that the problem (Dataset Distillation for Pre-Trained \nSelf-Supervised Vision Models) has been formally introduced, opening up space for ancillary endeavors.\n\n## Addressing \"Questions\"\n\n> If I understand correctly, all \"DINO\" models in the paper refer to DINO-v2. (A minor suggestion is to consistently use “DINOv2” throughout the paper to avoid confusion.)\n\nYes, this is true throughout the paper *except* for the section on ArtBench. \nWe used DINO-v1 for ArtBench because we wanted a model trained only on ImageNet such that the artworks would be out of distribution. \n\nWe initially dropped the \"v2\" in the tables to save space, but we see how this could cause confusion and have since labeled them more specifically as you suggested.\n\n> One possible explanation [for DINO-v2's much better performance than MoCo-v3 on the spurious background dataset] is the use of a patch-level loss in DINOv2, which might make it more robust to spurious correlations. Alternatively, it could be related to the use of smaller crops in DINOv1. Including experiments on DINOv1 could help disentangle these factors and provide further insight into the source of robustness.\n\nThis is an interesting insight; thank you for the suggestion!\n\nBelow, we include an extended version of the Spawrious table from the paper that includes DINO-v1 as well.\n\n#### Spawrious, DINO Comparison \n\n| Train Set (1 Img/Cls) | DINO-v1 ViT-B/16 | DINO-v2 ViT-B/14 | MoCo-v3 ViT-B/16 | Average        |\n|-----------------------|------------------|------------------|------------------|----------------|\n| Distilled (Ours)      | 48.6 ± 1.8       | **83.3 ± 3.8**   | **35.6 ± 2.3**   | **55.8 ± 2.6** |\n| Neighbors             | **50.4 ± 1.8**   | 81.7 ± 3.1       | 33.9 ± 2.5       | 55.3 ± 2.5     |\n| Centroids             | 50.0 ± 2.4       | 80.7 ± 2.8       | 31.3 ± 1.9       | 54.0 ± 2.4     |\n| Random                | 46.7 ± 3.4       | 64.2 ± 3.5       | 33.5 ± 8.4       | 48.1 ± 5.1     |\n\nThis DINO-v1 model has a similar patch size as the DINO-v2 model (16 versus 14).\n\nWe see that this DINO-v1 model, like MoCo-v3, also performs significantly worse than DINO-v2 on this spurious background dataset.\n\nQualitatively, we observe that the images distilled using DINO-v1 also seem to focus too much on the background environments (although we have no way of sharing these images in the rebuttal).\n\nHowever, we still cannot be sure of the precise reason for this performance gap. \nIt could be due to DINO-v2's patch-based losses, but it could also simply be due to the training data. \nDINO-v1 and MoCo-v3 were both trained on ImageNet, while DINO-v2 was trained on an undisclosed highly-curated dataset.\n\nRegardless, it is quite clear that this distillation method can indeed act as a useful tool with which to study model interpretability. \nWe plan on further exploring this direction in future work.\n\n## Overall\n\nThank you again for your thoughtful review. \n\nWe appreciate your enthusiasm for the interpretability results, and we aim to explore this direction more deeply in future work.\n\nWe hope to answer any lingering questions during the upcoming discussion period."}, {"review_id": "XyjKnIPOab", "reviewer": "Reviewer_pNCJ", "summary": "The authors propose a new method for creating synthetic training data.  The method uses *Linear Gradient Matching* to create a synthetic image that has the same gradients in a linear classifier as data in the original target class.  The synthetic images successfully train different model architectures, suggesting that a similar feature representation is shared between disparate model architectures.  The method is tested using several datasets and models.", "questions": "1. The presented work is only valid for linear classifiers.  What assumptions for non-linear classifiers would cause the proposed method to fail? \n\n2. Since the feature extraction backbones are pre-trained, is it necessary to use the full dataset for distillation? Would it be possible to use the linear gradients as a pre-screening method to identify a smaller subset of data which is capable of achieving the same results without the extensive compute resources?\n\n3. Please provide pseudo code for the main training algorithms until the main code repository is suitable for release\n\n4. Would it be possible to provide a toy example that allows users to apply the technique to smaller datasets?", "limitations": "Because of the extensive compute and data required for this paper, it will be challenging for the community to apply this to smaller datasets or verify the results independently.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths** \n\nI think this paper does a good job providing a novel experimental method that touches on SOTA models, interesting theoretical underpinnings, and a practical problem.   The presented experiments are convincing, extensive, and provide a good direction for future work.  \n\n**Weaknesses**\n\nThe code is currently unavailable, although the authors state their intention to release code upon acceptance.  The issue with this is that the authors report that the proposed method's success largely relies on the pyramid upsampling method, but details of the pyramid construction (\"we progressively optimize our pyramid, starting with just the lowest-resolution component and periodically adding more tiers during distillation...\") are not presented in enough detail to re-create the code from the paper alone, or to understand how this optimization affects the data distillation method proposed.  Although this is discussed further in the supplementary info, a complete algorithm or pseudo code is omitted from the paper. It also isn't clear why the pyramid plays such a large role if the linear gradient matching is the primary theoretical explanation for why the method works.", "was_revised": true, "final_justification": "The authors clarified my concerns regarding compute and the generalizability of this work. Including the pseudocode makes the work accessible to a broader audience, and I hope that the authors will release the full code repository on acceptance. If this work is broadly implemented across different fields, it could provide good experimental confirmation (or refutation) for different theories in foundations of AI, including universal feature approximations and desirable properties for model hessian spectra.", "rebuttal": "Thank you for your valuable feedback; \nyou have helped us identify several areas of improvement which have already further improved the quality of our paper. \n\nWe will now address the concerns and questions raised in your review. \n\n## Addressing \"Questions\"\n\n> The presented work is only valid for linear classifiers. What assumptions for non-linear classifiers would cause the proposed method to fail?\n\nWe chose to study linear classifiers in this work simply because they (along with fine-tuning) are the standard for measuring the quality of a pre-trained model. \nOur method may indeed transfer to non-linear classifiers, but we did not have time to adequately explore this direction during the rebuttal week. \nWhile single-step gradient matching is adequate for a linear model, a non-linear classifier might require adapting trajectory-matching techniques from previous dataset distillation methods. \nThis would introduce memory issues due to the large footprint of the pre-trained feature extractor. \nWe intend to investigate this further in future work. \n\nOn the other hand, a full-scale fine-tuning would *certainly* prove problematic. \nWhen distilling a dataset down to a single image per class, fine-tuning the entire large pre-trained feature extractor \ntends to lead to catastrophic overfitting. \nWe present a table here for ImageNet-100 when fine-tuning on the distilled data. \n\n| Train Set (1 Img/Cls) | CLIP          | DINO          | EVA            | MoCo            | Average        |\n|-----------------------|---------------|---------------|----------------|-----------------|----------------|\n| **Distilled (Ours)**  | 2.7 ± 0.3     | 3.7 ± 0.2     | 12.7 ± 6.7     | **46.6 ± 12.9** | **16.4 ± 5.0** |\n| Neighbors             | 3.1 ± 0.2     | 4.2 ± 0.4     | 14.4 ± 9.8     | 33.5 ± 13.8     | 13.8 ± 6.0     |\n| Centroids             | **4.4 ± 0.3** | **4.8 ± 0.4** | 17.8 ± 11.9    | 38.0 ± 13.0     | 16.2 ± 6.4     |\n| Random                | 2.5 ± 0.3     | 2.6 ± 0.5     | **32.9 ± 2.5** | 21.1 ± 1.9      | 14.8 ± 1.3     |\n| Full Dataset          | 92.5 ± 0.0    | 95.2 ± 0.1    | 94.1 ± 0.1     | 89.4 ± 0.3      | 92.8 ± 0.1     |\n\n> Since the feature extraction backbones are pre-trained, is it necessary to use the full dataset for distillation? Would it be possible to use the linear gradients as a pre-screening method to identify a smaller subset of data which is capable of achieving the same results without the extensive compute resources?\n\nThe compute required for our method scales with the *number of classes* in the real dataset, **not** the number of samples. \nCurrently, the batch size is equal to the number of images per class (typically 1) times the \naugmentations per batch (typically 10) times the number of classes. \n\nSo in short, pruning the real dataset before distillation will not reduce the method's cost, but future work may lead to \nmethods that can get away with smaller batch sizes.\n\n> Please provide pseudo code for the main training algorithms until the main code repository is suitable for release\n\nOf course, please see below.\n\n```python\nimport torch\nfrom typing import List\ndef reconstruct_pyramid(pyramid: List[torch.Tensor]) -> torch.Tensor:\n    # The pyramid is stored as a list of Tensors of different shapes\n    # (3, 1, 1), (3, 2, 2), (3, 4, 4), (3, 8, 8), (3, 256, 256)\n    sum = torch.zeros(3, 256, 256)\n    for layer in pyramid:\n        # each layer is bilinearly up-sampled to the final resolution and added together\n        sum += F.interpolate(layer, (3, 256, 256), \"bilinear\")\n    # we normalize by the current number of layers to account for progressively adding more layers\n    image = sum / len(pyramid)\n    return torch.tanh(image)\n```\n\nAbove is the code for reconstructing the distilled image from its pyramid representation. \nIn practice, this is equivalent to reconstructing an image from its Laplacian pyramid.\nAt initialization, the pyramid only consists of the 1x1 layer. \nAn additional layer is added every 200 distillation steps until the final layer (256x256) has been added.\n\n```python\nimport torch\ndef distillation_step(\n        pyramids: List[List[Tensor]]# our set of distilled image pyramids\n        labels: Tensor              # labels for distilled images/pyramids\n        optim: torch.optim.Adam,    # optimizer for distilled pyramids\n        backbone: nn.Module,        # the pre-trained feature extractor \n        num_features: int,          # number of features in the backbone's output\n        num_classes: int,           # number of classes in the dataset\n        augs_per_batch: int,        # how many sets of augmentations per batch (typically 10)\n        augment: Callable,          # augmentation function\n):\n    # initialize a random linear classifier\n    W = torch.randn(num_features, num_classes)\n    \n    # get synthetic images by reconstructing from their pyramids\n    x_syn = [reconstruct_pyramid(p) for p in pyramids]\n    y_syn = labels\n    # we augment the synthetic images multiple times to get better gradients\n    x_syn = torch.cat([augment(x_syn) for i in range(augs_per_batch)])\n    \n    z_syn = backbone(x_syn)\n    out_syn = torch.matmul(z_syn, W)\n    loss_syn = cross_entropy(out_syn, y_syn)\n    \n    # we have to retain the graph here because we will backprop through this backprop later\n    grad_syn = torch.autograd.grad(loss_syn, W, retain_graph=True, create_graph=True).flatten()\n    \n    # load from disc, this is our biggest bottleneck because we have to load over a network filesystem\n    x_real, y_real = get_real_data()\n    x_real = augment(x_real)\n    \n    z_real = backbone(x_real)\n    out_real = torch.matmul(z_real, W)\n    loss_real = cross_entropy(out_real, y_real)\n    \n    grad_real = torch.autograd.grad(loss_real, W).flatten()\n    \n    # our overall loss is the distance between the real and synthetic gradients w.r.t. W\n    meta_loss = 1 - cosine_similarity(grad_syn, grad_real)\n    \n    optim.zero_grad()\n    # the meta loss is back-propagated through the earlier synthetic gradient calculation\n    # it is a \"bi-level\" optimization\n    # the loss is back-propagated past the synthetic images all the way to their pyramid representations\n    meta_loss.backward()\n    optim.step()\n```\n\nAbove is the pseudo code for a single distillation step. \nAt each step, we:\n1. Sample a random linear classifier\n2. Reconstruct the synthetic images from their pyramids and augment them.\n3. Get the gradient of the classification loss w.r.t. W using the *synthetic* images.\n4. Load real images from disc.\n5. Get the gradient of the classification loss w.r.t. W using the *real* images.\n6. Calculate the meta loss as the distance between these two gradients.\n7. Back-propagate the meta loss through the initial gradient calculation all the way to the distilled pyramids.\n\n> Would it be possible to provide a toy example that allows users to apply the technique to smaller datasets?\n\nYes! In fact, we already included some results several such datasets. \n\nAs discussed earlier, the compute requirements scale with the number of *classes* in the dataset, so any dataset with \nfewer classes will distill much faster. \n\nThe 4-class (Spawrious) and 2-class (Waterbirds) datasets discussed in Section 4.6 were distilled in under 20 minutes on a single GPU. \n\nThe 10-class ArtBench dataset from section 4.7 was distilled in 30 minutes on a single GPU.\n\n## Addressing \"Weaknesses\"\n> The code is currently unavailable ... \n\nThank you for bringing to our attention how this can be challenging for reviewers; \nwe will strive to include clean code bases in future submissions.\n\nWe hope that the pseudo-code included in the previous responses adequately answer all your questions about our method.\n\n> It also isn't clear why the pyramid plays such a large role if the linear gradient matching is the primary theoretical explanation for why the method works.\n\nThe pyramid representation solely acts as an implicit regularizer on the distilled images while still allowing for the \nsame potential set of solutions. \n\nAn example of another such implicit regularizer would be optimizing the fourier coefficients of the images instead of the pixels directly. \nWhile still allowing for the same set of result images, this alternate representation affects the direction of steepest \ndescent [1], thereby affecting the optimization dynamics and resulting in fewer high-frequency artifacts.\n\nWe simply found that optimizing the pyramid representation as an implicit regularizer yielded better results than other\nmethods, such as optimizing the fourier coefficients. \nFurthermore, we found that progressively adding more layers to the pyramid over the course of distillation performed \nbetter than initializing it with all layers already in-tact. \nQualitatively, we observed that this progressive method led to more global patterns across the images.\n\n[1] Distill.Pub, Feature Visualization ([link](https://distill.pub/2017/feature-visualization/))\n\n## Addressing \"Limitations\"\n> Because of the extensive compute and data required for this paper, it will be challenging for the community to apply this to smaller datasets or verify the results independently.\n\nAs discussed in our earlier responses, the compute requirements for our method scale solely with the number of classes \nin the dataset. \nAs such, only the full ImageNet-1k was prohibitively expensive since the batch size was so large.\n\nIn fact, as noted in the \"Compute Budget\" section of our included supplementary material, \nit takes 6 hours to distill ImageNet-100 using 4 A100 GPUs or 4.5 hours using 4 H200 GPUs.\n\nDepending on a user's compute budget, this may still be considered expensive. \nHowever, distilling a 10-class dataset like ArtBench or ImageWoof takes only 40 minutes on a single RTX4090 GPU. \nThis should fall comfortably within most users' compute budgets if ImageNet-100 distillation is too expensive."}, {"review_id": "AGSEs7ggdy", "reviewer": "Reviewer_Cicx", "summary": "The paper describes a method, Linear Gradient Matching, that optimizes a synthetic image generation model to produce gradients similar to real data. Prototypical training images generated with this approach suffice to train capable classifiers. Several image encoders are used on top of the synthetic and real images, and they passed through randomly initialized layers to", "questions": "* What do you mean on line 96 with sampling a classifier at each distillation step, what is a distillation step?\n* How do you generate the synthetic images? Do you use GANs, diffusion or flow based models? Do you finetune, train from scratch? How many parameters? This could be a part of figure 2\n* Is the flow like this: generative model -> encoder (e.g. dino) -> learned linear projection W -> classify -> get loss -> propagate to the generative model and W\n* What is the batch size used for training?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n* I find the approach intuitive, matching the learning signal, instead of surface similarity makes a lot of sense\n* The paper is mostly easy to follow and well written \n* Clear empirical quantitative and qualitative evaluations, e.g. the pyramid approach\n* Four vision encoders are used to get loss\n* Both high level iNet and spurious datasets are used\n\nWeaknesses\n* Unfortunately I do not find the presentation of the core method clear. What parameters are being updated, and what exactly is being optimized (recommend you clarify this pedagogically, this is the main issue for me to raise my score) – see questions\n* Missing training details (even if the model being trained was clear to me)\n* Would have been good to see an ablation with multiple images per class\n* Would have been good to see how the approach works on a few fine grained datasets\n* No information about how expensive the method is\n* No error bars, not even bootstrapped\n\n\nNotes\n* I would summarize the pyramid findings when you mention figure 3 in 3.2, referencing table 2 and 3.\n* The mentions of the platonic representation hypothesis seem a bit tangential and perhaps not necessary", "was_revised": true, "final_justification": "During my initial reading of the paper, a core part of the methodology flew over my head. After clarification from the authors, I now understand what they are doing and where my confusion came from. The added fine-grained results, error bars etc. cover the rest of my worries. I am thus happy to raise my score.", "rebuttal": "Thank you for your thoughtful review. \nWe appreciate the constructive feedback and have already used it to improve the paper.\n\nWe hope to resolve your concerns by first addressing your specific questions and then the broader concerns.\n\n## Response to \"Questions\"\n\n> What do you mean on line 96 with sampling a classifier at each distillation step, what is a distillation step?\n\nWe have made this clearer in the revised version of the paper, but our method can be broken up into two phases: \nthe Distillation Phase and the Evaluation Phase.\n\nThe \"Distillation Phase\" is the core of the method. This is the part where we perform the gradient matching and update the synthetic images.\n\nIn the \"Evaluation Phase,\" we evaluate the synthetic images by *training* a new linear classifier from scratch on the \ndistilled data and then *evaluating* the linear classifier on the real test set.\n\nSo to answer your question, a \"distillation step\" involves \n1. Passing a real/synthetic batch of images through the encoder *ϕ* and a randomly initialized linear classifier *W*, \n2. Getting the real/synthetic classification loss with respect to the real/synthetic images, \n3. Getting the *gradients* of the real/synthetic classification losses with respect to *W*\n4. Getting the meta loss by measuring the distance between the real and synthetic gradients\n5. Updating the synthetic images by calculating the gradient of this meta loss with respect to the synthetic images (a bi-level optimization)\n\n> How do you generate the synthetic images? Do you use GANs, diffusion or flow based models? Do you finetune, train from scratch? How many parameters? This could be a part of figure 2\n\nWe apologize for the confusion; we hope to clear up our optimization process here. \n\nTo be clear, the images are ***not*** the output of a generative model.\n\nRather, the images themselves are directly optimized using the bi-level optimization described in the previous answer. \nAs such, there is no fine-tuning or training from scratch since there is no generative model.\n\nThe naive approach to image optimization is to directly optimize the *pixels* of the images, but we find that this performs quite poorly due to overfitting (Table 3).\n\nInstead, we parameterize each image as a pyramid of different resolutions (Section 3.2) and reconstruct the image before each distillation step. \n\nIn practice, this equates to optimizing the *Laplacian pyramid* of an image rather than its pixels directly. \nWhile having the same solution set as directly optimizing the images, optimizing the pyramid representation instead acts as an implicit regularizer to combat over-fitting. \nThe visual difference of the naive pixel versus pyramid optimizations can be seen in Figure 3. \nAnother example of such an implicit regularizer would be optimizing the images' fourier coefficients instead of the pixels directly [1], but we found the pyramid approach to work better.\n\n[1] Distill.Pub, Feature Visualization ([link](https://distill.pub/2017/feature-visualization/))\n\n> Is the flow like this: generative model -> encoder (e.g. dino) -> learned linear projection W -> classify -> get loss -> propagate to the generative model and W\n\nThere are a few misconceptions here to clear up. \n1. As discussed in the previous response, there is no generative model; the images themselves are directly optimized. \n2. *W* is **not** learned at distillation time (only at *evaluation* time to measure the quality of our distilled images).\n\n> What is the batch size used for training?\n\nAt *distillation time* (where we optimize the synthetic images), the \"batch size\" is equal to the number of synthetic images multiplied by the number of augmentations per batch. \nWe sample an equal number of real images at each step to keep the real and synthetic batch sizes equal.\n\nAt *evaluation time* (where we train a new linear classifier on the synthetic images to measure their quality), the batch size is 100.\n\n## Response to \"Weaknesses\"\n\nNow we will attempt to address the broader weaknesses you mention.\n\n> Unfortunately I do not find the presentation of the core method clear. What parameters are being updated, and what exactly is being optimized\n> \n> Missing training details\n\nWe hope the detailed responses in the previous section have alleviated these concerns. \n\nTo briefly recap:\n* There is no generative model; we directly optimize a small set synthetic images themselves\n* The linear classifier is randomly sampled at each \"distillation step\" (i.e., every time we do gradient matching). It is only ever *trained* at evaluation time when we are measuring the performance of the distilled images.\n\n> Would have been good to see an ablation with multiple images per class\n\nBelow is a table comparing our method to various baselines for ImageNet-100 at 10 images per class. \nWe see the same trends as with 1 Image per Class overall, but the absolute gap between our method and the baselines is now smaller since we are closer to the upper bound (Full Dataset).\n\n### ImageNet-100, 10 Img/Cls, Baseline Comparisons\n\n| Train Set (10 Img/Cls) | CLIP           | DINO           | EVA            | MoCo           | Average        |\n|------------------------|----------------|----------------|----------------|----------------|----------------|\n| **Distilled (Ours)**   | **86.1 ± 0.1** | **92.6 ± 0.1** | **90.6 ± 0.1** | **84.2 ± 0.1** | **88.4 ± 0.1** |\n| Neighbors              | 82.6 ± 0.1     | 91.0 ± 0.1     | 88.2 ± 0.2     | 83.0 ± 0.1     | 86.2 ± 0.1     |\n| Centroids              | 77.4 ± 0.2     | 87.3 ± 0.2     | 83.1 ± 0.2     | 77.8 ± 0.2     | 81.4 ± 0.2     |\n| Random                 | 85.0 ± 0.4     | 91.0 ± 0.2     | 88.0 ± 0.4     | 82.9 ± 0.4     | 86.7 ± 0.3     |\n| Full Dataset           | 92.5 ± 0.0     | 95.2 ± 0.1     | 94.1 ± 0.1     | 89.4 ± 0.3     | 92.8 ± 0.1     |\n\n\nBelow is a table measuring our method's cross-model performance for ImageNet-100 at 10 images per class. \nAgain, we see the same overall trends as with 1 Image per Class: DINO with the best overall generalization and a large negative outlier between MoCo and CLIP. \n\n### ImageNet-100, 10 Img/Cls, Cross-Model Performance\n\n| Distill Model    | DINO       | EVA        | CLIP       | MoCo       | Average    |\n|------------------|------------|------------|------------|------------|------------|\n| **DINO**         | 92.6 ± 0.1 | 88.4 ± 0.1 | 85.4 ± 0.2 | 79.8 ± 0.4 | 86.6 ± 0.2 |\n| **EVA**          | 90.1 ± 0.1 | 90.6 ± 0.1 | 86.7 ± 0.1 | 76.2 ± 0.3 | 85.9 ± 0.1 |\n| **CLIP**         | 86.8 ± 0.3 | 86.2 ± 0.2 | 86.1 ± 0.1 | 66.8 ± 0.1 | 81.5 ± 0.2 |\n| **MoCo**         | 88.5 ± 0.1 | 83.7 ± 0.1 | 78.3 ± 0.1 | 84.2 ± 0.1 | 83.7 ± 0.1 |\n| **Full Dataset** | 95.2 ± 0.1 | 94.1 ± 0.1 | 92.5 ± 0.0 | 89.4 ± 0.3 | 92.8 ± 0.1 |\n\n> Would have been good to see how the approach works on a few fine grained datasets\n\nThank you for this suggestion!\n\nWe conducted experiments on CUB-200-2011 (a dataset of 200 bird species) and Stanford Dogs (a dataset of 120 dog breeds).\n\n### CUB-200-2011\n\n| Train Set (1 Img/Cls) | CLIP           | DINO           | EVA            | MoCo           | Average        |\n|-----------------------|----------------|----------------|----------------|----------------|----------------|\n| Distilled (Ours)      | **63.2 ± 0.6** | **85.7 ± 0.2** | **75.4 ± 0.4** | **41.6 ± 0.3** | **66.5 ± 0.4** |\n| Neighbors             | 33.9 ± 0.5     | 74.9 ± 0.1     | 52.4 ± 0.4     | 21.2 ± 0.4     | 45.6 ± 0.3     |\n| Centroids             | 39.6 ± 0.3     | 76.4 ± 0.3     | 55.2 ± 0.4     | 23.8 ± 0.4     | 48.7 ± 0.4     |\n| Random                | 37.2 ± 1.4     | 62.8 ± 1.6     | 44.1 ± 1.6     | 15.2 ± 1.0     | 39.8 ± 1.4     |\n| **Full Dataset**      | 77.5 ± 0.7     | 90.2 ± 0.2     | 84.0 ± 0.3     | 43.7 ± 0.8     | 73.8 ± 0.5     |\n\n### Stanford Dogs\n\n| Train Set (1 Img/Cls) | CLIP           | DINO           | EVA            | MoCo           | Average        |\n|-----------------------|----------------|----------------|----------------|----------------|----------------|\n| Distilled (Ours)      | **52.9 ± 0.6** | **82.4 ± 0.2** | **74.7 ± 0.4** | **68.8 ± 1.2** | **69.7 ± 0.6** |\n| Neighbors             | 21.9 ± 1.4     | 69.5 ± 0.3     | 53.2 ± 0.4     | 50.8 ± 0.8     | 48.9 ± 0.7     |\n| Centroids             | 32.4 ± 1.7     | 71.3 ± 0.4     | 51.3 ± 0.7     | 47.5 ± 1.0     | 50.7 ± 0.9     |\n| Random                | 21.0 ± 1.9     | 50.6 ± 2.1     | 36.3 ± 1.3     | 27.6 ± 2.1     | 33.8 ± 1.9     |\n| **Full Dataset**      | 76.9 ± 0.1     | 88.6 ± 0.1     | 82.6 ± 0.1     | 72.3 ± 0.5     | 80.1 ± 0.2     |\n\nThe performance gap between our method and the baselines is *significantly* larger on these fine-grained datasets \ncompared to ImageNet. \n\nIt seems that the distilled images focus heavily on the fine-grained details necessary to distinguish between the highly similar classes.\n\n> No information about how expensive the method is\n\nIn the **Compute Budget** section of the Appendix (in the included supplementary material), we note that \"Distilling ImageNet-100 with the default settings takes about 4.5 hours using 4 H200 GPUs\nand around 6 hours using 4 A100 GPUs.\" It was difficult to get an accurate measure of how long it took to distill ImageNet-1k since we were largely bottle-necked by our network speed, but it typically took around 12 hours.\n\n> No error bars, not even bootstrapped\n\nWe have since had time to repeat our experiments and now report all results as mean ± stdev over 5 runs. \n\nFor the sake of space, we will not re-include all the updated tables from the paper here, but you can see examples of  \nthe \"error bars\" as ± stdev in the previous responses.\n\n## Overall\n\nThank you again for your constructive review. \n\nYour feedback has helped us make our paper even stronger by increasing the clarity and including more compelling results.\n\nWe hope to answer any lingering questions during the upcoming discussion period."}, {"review_id": "DiFUA8iznG", "reviewer": "Reviewer_gLBa", "summary": "Summary\n\nThis paper addresses the performance of dataset distillation when the models to be trained are continued on pretrained models, e.g., linear probe, instead of from scratch. They propose to match: the gradient of loss wrt linear classifier when trained on [real, synthetic] data. The real/syn images are stored in different resolutions to avoid overfitting.\n\nExperiments\n\nAcross 4 SSL methods, including CLIP and DINO, on ImageNet, linear probe on distilled dataset is better than other dd methods, and comparable to full data.", "questions": "How is your approach different from directly applying GM to linear probing? What about other trajectory matching methods?", "limitations": "The methodology seems to be the same as gradient matching. Are there any challenges when applying GM, or any other gradient based DD approaches for linear probing? Otherwise the technical contribution might be limited.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The paper is clearly written and experiment results are well presented. Finetuning might be a more prevalent task than training ilnear classification, imo.\n\nWeaknesses: The technical contribution is limited as it directly applies gradient matching https://arxiv.org/abs/2006.05929 to another scenario, linear probing.", "was_revised": true, "final_justification": "Thank you for the clarification in the discussion.", "rebuttal": "Thank you for taking the time to review our paper.\n\nIt seems your main concern is a potential lack of novelty, citing the \"Dataset Condensation\" paper by Bo Zhao et al. \nAs discussed in our paper, this work was indeed a major inspiration for our Linear Gradient Matching method. \nHowever, it should be noted that this method and those that came after it (trajectory matching, etc.) generally fail to scale up to practical models and realistic datasets.  \n\nIt is true that our method, like that of Zhao et al., primarily functions by matching the gradients induced by real and synthetic gradients. \nHowever, our method makes several core improvements that allow it to scale, allowing us to distill practically useful datasets at just a single image per class. \nFurthermore, our method also serves as an interpretability tool, allowing researchers to better understand why different vision models behave in certain ways. \n\nScaling up gradient matching in and of itself would be a novel contribution, but we additionally offer a number of new improvements unique to our work.\n\nTo list some specific points of novelty:\n### 1. Task-Specific Gradients.\nThe original method of Zhao et al. optimizes each class of distilled images independently.\nThis is particularly problematic in the 1 image-per-class setting (where we focus) since they do not even update the teacher network between iterations.\nBy optimizing each class independently, the learning process misses out on the crucial information necessary to distinguish each class from *each other*.\nWe jointly optimize across all classes such that the distilled images learn to induce the optimal N-way decision boundary between the classes.\n\n### 2. Multiple Simultaneous Augmentations.\nTo the best of our knowledge no existing dataset distillation method has employed our method of multiple augmentations. \nBy creating a \"mega batch\" by combining several independently augmented versions of the distilled data, the synthetic images receive significantly better gradients.\nRather than the optimizer forcing a single augmentation of the distilled image to be the \"ideal\" exemplar for that class, \nmultiple simultaneous, disjoint augmentations allow for the joint set of augmented versions to serve as the informative class representatives together .\n\n### 3. Pyramid Representation.\nPrevious works have explored alternative representations for distilled data, but they typically either add a tremendous \namount of overhead or reduce quality in the name of compression. \nTo our knowledge, ours is the first work to represent the distilled data as a Laplacian pyramid of different resolutions. \nThis unique parameterization implicitly regularizes the distilled data for only a minimal amount of extra computation. \nWithout the pyramid representation, we would not be able to distill ImageNet or other high-resolution datasets to this level of quality. \n\n### 4. Model Interpretability.\nBy distilling datasets in the space of a pre-trained model, we gain valuable insights into that model's behavior. \nDespite coming from largely the same transformer architecture, the images distilled from our 4 different feature \nextractors differ quite significantly in appearance. \nThis offers us insights as to how these models actually \"see\" and how they behave differently from each other. \nWe further explore this in the section on spurious backgrounds and identify possible reasons as to why, for instance, MoCo gets fooled far more often than DINO.\n\n### 5. The Task Itself.\nDataset distillation (especially at 1 image per class) has largely been seen as a toy problem with limited practical uses. \nWhile achieving decent results on CIFAR and toy networks, existing methods failed to scale up to realistic settings. \nGiven that most models today start from a pre-trained backbone, extending dataset distillation to this regime was a natural step. \nIn doing so, and through the development of our new method, we can now train a linear classifier to nearly 75% test \naccuracy on ImageNet-1k in a matter of minutes using just a single image per class, as opposed to the 12+ hours and over \n1 million images it would take to train using the full, real dataset. \n\n### To answer some specific points you mentioned:\n\n> Finetuning might be a more prevalent task than training ilnear classification, imo.\n\nOne of dataset distillation's points of failure is overfitting to the synthetic data. \nIn the low image count regime (i.e., 1 image per class) networks easily overfit, leading to a large performance gap versus training on the full dataset.\n\nThis overfitting is already a known issue with the toy ConvNets traditionally used in dataset distillation works, \nbut training on a large pre-trained vision transformer heavily amplifies the problem. \n\nWe present a table of results obtained by fine-tuning the models on the distilled data from ImageNet-100. \nIndeed, we observe overfitting to an extreme degree, with CLIP and DINO reaching a performance level barely above random guessing. \n\n#### ImageNet-100, Fine-Tuning on Distilled Images\n| **Train Set (1 Img/Cls)** | **CLIP**      | **DINO**      | **EVA**        | **MoCo**        | **Average**    |\n|---------------------------|---------------|---------------|----------------|-----------------|----------------|\n| **Distilled (Ours)**      | 2.7 ± 0.3     | 3.7 ± 0.2     | 12.7 ± 6.7     | **46.6 ± 12.9** | **16.4 ± 5.0** |\n| **Neighbors**             | 3.1 ± 0.2     | 4.2 ± 0.4     | 14.4 ± 9.8     | 33.5 ± 13.8     | 13.8 ± 6.0     |\n| **Centroids**             | **4.4 ± 0.3** | **4.8 ± 0.4** | 17.8 ± 11.9    | 38.0 ± 13.0     | 16.2 ± 6.4     |\n| **Random**                | 2.5 ± 0.3     | 2.6 ± 0.5     | **32.9 ± 2.5** | 21.1 ± 1.9      | 14.8 ± 1.3     |\n| **Full Dataset**          | 92.5 ± 0.0    | 95.2 ± 0.1    | 94.1 ± 0.1     | 89.4 ± 0.3      | 92.8 ± 0.1     |\n\n> What about other trajectory matching methods?\n\nTrajectory matching methods suffer from having to track gradients through an expensive inner loop. \nThis issue is further exacerbated when introducing a large vision transformer as a feature extractor. \nAt this point, it does not seem feasible to apply existing trajectory matching methods to the space of a pre-trained feature extractor.\n\nFortunately, single-step gradient matching should be adequate for a linear classifier since the points along an optimal \ntrajectory should be approximately co-linear. \n\n### Overall\n\nThank you again for taking the time to review our paper. \n\nWe hope we have properly conveyed the significance of our work.\n\nPlease let us know during the discussion period if there are any other points we can clarify."}]

Author rebuttal:

Meta-review: This paper on dataset distillation tackles the case of pre-trained self-supervised models that are often used in computer vision problems. The core of this work is an approach for "generating" synthetic data through linear gradient matching that ensures same gradients in a linear classifier for synthetic and real data. The proposed approach is widely evaluated with several popular encoders (e.g., DINOv2, CLIP, MoCo v3) and compared against several relevant methods and baselines.

During the review process several strengths identified for this work include: a diverse set of convincing evaluations and ablation studies, presenting a relevant theoretical basis, the use of relevant datasets showing the practicality of the approach and its potential use in the research community. The initial reviews also noted questions on: the importance of this task, the motivation for using linear probing (e.g., compared to fine-tuning), reproducibility of this work, use beyond linear classifiers, the novelty of the work. The rebuttal and the discussion that ensued clarified all the questions with convincing evidence.

Thus, all the reviewers and the AC recommend the paper for acceptance. The final version of the manuscript should include the clarifications provided in the rebuttal and discussion that ensued.
Decision: Accept (poster)