On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models
Authors: Juliette Marrie
Michael Arbel
Julien Mairal
Diane Larlus
Venue: Accepted by TMLR ()

Abstract: Large pretrained visual models exhibit remarkable generalization across diverse recognition tasks. Yet, real-world applications often demand compact models tailored to specific problems. Variants of knowledge distillation have been devised for such a purpose, enabling task-specific compact models (the students) to learn from a generic large pretrained one (the teacher). In this paper, we show that the excellent robustness and versatility of recent pretrained models challenge common practices established in the literature, calling for a new set of optimal guidelines  for task-specific distillation. To address the lack of samples in downstream tasks, we also show that a variant of Mixup based on stable diffusion complements standard data augmentation. This strategy eliminates the need for engineered text prompts and improves distillation of generic models into streamlined specialized networks.

Published in Transactions on Machine Learning Research (05/2024)
On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models
Juliette Marrie
juliette. siliettemmarree@nneetddb..o.m
NAVER LABS Europe
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble
Michael Arbel
michael.arfell@irrii...
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble
Julien Mairal
qulien..miinl@iinri.....
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble
Diane Larlus
diane. wnaverlabs.cco
NAVER LABS Europe
Reviewed on OpenReview: https: openreview. net/ net/ forum? id= oyISaaeHuu
Abstract
Large pretrained visual models exhibit remarkable generalization across diverse recognition tasks. Yet, real-world applications often demand compact models tailored to specific prob- lems. Variants of knowledge distillation have been devised for such a purpose, enabling task-specifie compact models (the students) to learn from a generic large pretrained one (the teacher). In this paper, show that the excellent robustness and versatility of recent pretrained models challenge common practices established in the literature, calling for new set of optimal guidelines for task-specific distillation. To address the lack of samples in downstream tasks, we also show that a variant of Mixup based on stable diffusion comple- ments standard data augmentation. This strategy eliminates the need for engineered texl prompts and improves distillation of generic models into streamlined specialized networks
1 Introduction
Recent large pretrained visual models demonstrate robust generalization across diverse computer vision tasks. Developed by leveraging substantial computational resources, these models trained enormous (often internal) sets of visual data, enabling them to learn rich visual representations. Such models exhibit remarkable transfer performance on downstream tasks with frozen features, achieving competitive results through simple linear probing (see, e.g. Li et al. 2022 He et al. 2022 Fang et al. 2023 Oquab et al. 2024). However, the size of the best performing models often poscs limitations for various real-world applications, both lerms of inference time and memory usage, especially in scenarios with constrained resources.
An essential question thus emerges: How can we most effectively transfer the rich visual representations from these large models to smaller architecture? While smaller models distilled from the larger oncs sizeable generic dataset are sometimes available (Oquab et al. 2024 is simply finetuning them to specific tasks optimal? As visual pretrained models are becoming larger, the cost of finetuning them is often out of reach for many users. It is therefore natural to ask whether a teacher trained with simple probing (linear or with a small multilayer perceptron) sufficiently competent to guide the training of a smaller model, specialized for given computer vision task. Finally, as distillation often benefits from data augmentation (Beyer et al. 2022), and given the offectiveness of data augmentation methods based on Stable Diffusion (Rombach al.
Project page: https://europe.
http::///errooennnellbbbbb wwverlabs..c.... cot..abbb..cmm/ddddddd....

---

Published in Transactions on Machine Learning Research (05/2024)
2022
Saharia et al. 2022b) in supervised learning (Trabucco et al., 2023 Azizi et al. 2023 Zhou et al,
2022 leveraging generative models for distillation seems promising. However, unlike in superv ieservsedddoo learning where the generative model is usually conditioned by text prompts, e.g. with class labels, the dependence on class information becomes questionable when used for distillation, as labels are not technically roquired. This raises the question of how to best leverage these models in the context of knowledge distillation.
In this paper, we study these fundamental questions. (i) We delineate optimal practices for leveraging large pretrained visual models in real-world applications constrained by limited resources, supported by an extensive experimental analysis. Our work shows that simple, cost-efficient approach to supervised distillation from large pretrained models consistently achieves superior results. (ii) We investigate various data augmentation strategies based on Stable Diffusson and demonstrate that a variation of Mixup is notably efficient for distillation. Originally proposed by Pinkney (2022) in a different context to generate visually appealing combinations of images, it proves particularly effective when employed as a data augmentation tochnique for distillation. It operates solely on unlabeled images, eliminating the necessity for text prompt engineering, and remains agnostic to the rownstream lask.
Concretely, our work reaches a series of experimental conclusions that ground our guidelines. Our experi- ments are conducted using DINOv2 teachers (Oquab et al. 2024), recognized for providing strong basclines (see Section 4.2) and extended to EVA-02 MIM- and CLIP-pret -l.e--retrained models (Fang et al., 2023 Sun et al. 2023 (see Appendix B.3 g. Our findings, summarized below, are validated across different architectures and
various tasks: classification on specific image modalities, fine-grained classification, and semantic segmenta- tion.
Probing can yield better teachers then fineturing. The remarkable adaptability of recent large-scale pretrained models, such DINOv2, challenges the need for finctuning the teacher, which is standard
practice in prior research on Lask-specific distillation (Jiao et al. 2020 Sun el. al. 2019 Touvron
et al.
2021 Beyer et al. 2022 Huang et al. 2023
2. Task-specific distillation complements task-agnostic distillation. Task-spocific distillation allows transferring task-specific knowledge, leading to better representations compared to simply finetuning the student after task-agnostic distillation, as illustrated in Figure We show that tpsk-specific dis- tillation consistently outperforms simple finetuning, which aligns with conclusions from prior works (Jiao et al., 2020: Huang et al. 2023 ; drawn for teachers finetuned on the target task. Our study extends their results to teachers that are only probed for the task, thus reducing the cost of the distillation procedure.
3. Teachers do not need to be as accurate as their students. This observation generalizes conclusions from early works ( (Yuan et al. 2020 Furlanello et al. 2018), conducted with teacher/student CNN models trained from scratch in a supervised manner. We show that even when DINOv2's pretrained VIT-S outperforms its teacher with simple finetuning, distillation can still be beneficial.
4. Small models can directly learn from much larger ones. Prior works suggest that large capacity gap between teacher and student hinders distillation, and employ middle-sized 'teacher assistant' to learn from the large model and teach the small one (Jiao et al.) 2020, Mirzadeh et al. 2020: Wang et al. 2020). However, DINOv2's ViT-S was directly distilled from their ViT-g and yet demonstrates excellent generalization capabilities. Similarly, we show that task-specific distillation works equally well when using DINOv2's ViT-g or their middle-sized ViT-L teach ViT-S.
5. Diffassio models can be effectively leveraged dala augmentation for distillation without relying class information, making them applicable to tasks where text-conditioned image generation is non-trivial (such as semantic segmentation). To bypass the need for class information in Stable Diffusion, we leverage diffusion model that generates mized images conditionally on multiple images provided as input, taking inspiration from the classical Mixup augmentation. We show that, while being ineffective in the context of supervised learning, this mixing strategy consistently helps task-specific distillation.
2

---

Published in Transactions on Machine Learning Research (05/2024)
Teacher
Student distilled PCA projection of student's features
PCA projection of the tencher's features
Image
Student init
Student finetuned
Mixer
PCA projection of the student's features
PCA projection of the stadent's features
This paper advocates for distilling large pretrained teacher (top, left.) to train small task-specific student model (top, right). This distillation process results in better clustering of the representations compared to simply finetuning the student on the task (bottom, right). Distillation is improved by class- agnostic data augmentation based on Stable Diffusion that consists in mixing real images to create synthetic ones, producing features shown in gray in the teacher plot. Each plot shows image features for 30 classes of the CUB Bird dataset, after PCA (one color per class).
Figure 1:
2 Related work
In this section, we first discuss relevant prior work on knowledge distillation (Section 2.1 works that leverage Stable Diffusion for data augmentation, and discuss data augmentation in the context of distillation (Section 2.2h.
We then cover
2.1 Knowledge distillation
Task-specific vs. generic distillation. Following the pioneering work of Hinton et al. (015)), distillation has become a standard approach to transfer knowledge from one model into another (see Gou et al. 2021 Wang & Yoon 2021 for detailed surveys). Initially, knowledge distillation was conceived as a method to transfer ledge from a large teacher network trained on a specific task to a small student network (Hinton et al.
2015 Ba & Caruana 2014). With the rise of self-supervised learning, the approach was extended
transfer general representations produced by large generic model into small ones (Abbasi Koohpaycgani
et al.
2020
Fang et al. 2021 Xu et al. 2022 Gao et al.) 2022 Navancet et al., 2021 Wu et al.) 2022 Duval
et al.
2023 There, distillation is used as a knowledge compression mechanism, which is motivated by the
observation that directly pretraining small models on large amounts of data leads to underwhelming results
compared to learning them by distillation from large pretrained models (Abbasi Koohpayegani et al. 2020
Fang et al. 2021 Xu et al. 2022 Wu et al. 2022 Oquab et al. 2024h.
In the context of self-supervised learning, it is then common finetune distilled models on various down- stream tasks, without further exploiting the teacher's knowledge (Sun et al. 2019 Touvron et al.) 2021; Beyer et al.] 2022). Surprisingly, only few studies have explored task-specific distillation procedure that leverages both the teacher and the downstream task. An example is the two-stage distillation introduced in natural language processing by Jiao el al. (2020) and recently applied to vision tasks by Huang et al. (2023). Specifically, their approach involves a conventional generic distillation, followed by finetuning the teacher on downstream task and applying a second task-specific distillation involving the finetuned teacher. In con- trast, our findings indicate that finetuning the teacher is not always the optimal strategy, and we advocate for less computationally demanding approach.
3

---

Published in Transactions on Machine Learning Research (05/2024)
Architecture-eppedenni distillation. Some variants of knowledge distillation directly exploit the specific architecture of both the teacher and the student. These include feature-based knowledge distillation often tailored to CNNs (Romero et al. 2015 Zagoruyko & Komodakis 2017 Chen et al. 2021a b), where knowledge is distilled by matching representations from any intermodiate layor(s), or aligning mutual relations in the Teature space (Yim et al. 2017 Tung & Mori 2019 Approaches specific to transformers have also emerged, consisting, for instance, of adding a separate distillation token (Touvron et al.] (021). Simultancously, other works have proposed architeclure-agnosii distillation approaches relying on particular loss functions (Tian) et al. 2020 Zhao et al., 2022). For example, Tian et al. (2020) propose contrastive objective inspired by se -ssperiissd learning approaches. In our work, we adopt task- and architecture-agnosiic distillation framework, therefore bypassing the need for adjusting to the model's architecture.
2.2 Data augmentation
Traditionally, data augmentation has been used to improve the generalization capabilities of deep neural networks (Wang & Yoon, 2021). Recently, Stable Diffusion models have emerged as another compelling tool for data augmentation, and have been broadly studied in the context. of supervised learning. In the context of knowledge distillation, data augmentation is not constrained by the need of class labels or segmentation masks, which suggests that optimal augmentation approaches may differ from those delincated in the context of supervised learning. Below we discuss prior works on the use of Stable Diffusion for data augmentation and prior studies on data angmentation for knowledge distillation.
Data augmentation with Stable Diffusion. Recent generalive models such as latent diffusion models Rombach et al. 2 have emerged as a compelling way to artificially augment training data (Tabucco
et al. 2023 Azizi et al. 2023 Dunlap et al. 2024) or even replace it (Sarıylldzz et al., 2023), usually using
class names as textual prompts. Yet designing prompts can be difficult for tasks such as segmentation, as
it requires featuring the multiple classes found in image. Prior works often resort to prompt engineering
Fang et al. 2024 or to language models to generate prompts from class names (Nguyen et al. 2023 Zhou
et al. 2022
An alternative to text-to-image generation is to leverage imago-to-image diffusion models to directly provide training images as prompts. Imago-to-image diffusion models have proven successful at various tasks such as restoration (Saharia et al.] (202a)) or image editing (Brooks el al. 2023). However, using them as a tool for data augmentation raises significant challenges. These models can struggle with producing meaningful variations such as viewpoint changes object shape variations,
pointed out by Brooks et al.. (2023) Properties such as object shape, location, and appearance can be extracted and controlled from the internal representations of diffusion models (Epstein et al. 2023) but this requires manual interventions and cannot be universally applied to any task. For dense segmentation tasks, Yang et al. (2023) propose to generate /mask pairs without resorting to prompt engineering, but is restricted to supervised tasks with access to segmenta- tion masks, and synthetic images are bound to be generated with these fixed masks. In contrast, we advocate an approach that can be universally applied to any task and that produces substantial image variations by interpolating between multiple training images (Pinkney 2022).
synthetic data based on the segmentation mask of real images. This approach allows generating
Data augmentation in the context of distillation. In the context of knowledge distillation, Beyer al. ) recommend to apply the same augmentations to the inputs of both teacher and student networks to ensure they are provided with consistent views. [Wang et al. (2022) suggest that a good data augmentation scheme should reduce the covariance of the teacher-st.udent cross-entropy, and propose an enhanced CutMix augmentation. Alternatively, Stanton et al. (2021) show the positive impact of Mixup on knowledge distil- lation. In this work, we show that distillation works better when performing data angmentation that goes beyond simple photometric and geometric transformations such as vanilla Mixup or CutMix, by exploiting the richness of generative models such as Stable Diffusion.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Pretrained Model
Lrask
label
Trad
Original dataset
Step 1: Teacher probing
Diffusion -assed data augmentation
x
fe(x")
Teacher
Cdietill
fs((e)
ft
Extended set images Original images Synthetic images
Ss(x)
x
Student
fz
Lrssk
label
Original dataset
Step 2: Distillation
Figure 2: Overview of the task-specific distillation pipeline. The pretrained model is probed to build teacher (top). Then its knowledge is distilled (Hinton et al. 2015) by minimizing the distillation loss Caistill jointly with the task loss Lrask (bottom). Laistall is optimized with both i) original images x and ii) synthetic images obtained with Stable Diffusion x', while Lrask is only optimized on the original dataset (2r y). Note that and x' are also transformed using standard data augmentation (not shown here).
3 Method
Our study focuses on task-specific distillation, which consists training small model for a specific su- pervised task while transferring knowledge from large pretrained encoder. In Section 3.1 we detail the standard approach for task-specific distillation of pretrained models. It usually divides in two steps: first training a teacher model on the target task, then transferring the knowledge from the trained teacher to a student. Unlike prior work, our distillation is performed without teacher finetuning: we only train task head on the pretrained encoder. Finetuning can be computationally expensive, especially when dealing with large teachers such as ViT-g, but it may also compromise the quality of the visual representation acquired during pretraining (e.g., from self=supervision). In Section 3.2 we present a mixing data augmentation based on Stable Diffusion that leverages the teacher's knowledge more effectively to enhance the distillation process. The overall method is illustrated in Figure 2
3.1 Task specifc distillation
We consider a task, such as classification or segmentation, where the goal is to predict a label (e.g. a class or segmentation map) given an input image x. Typically, one could learn model f to perform such a task using a training set Dirain of image/labels pairs (a, y) by simply optimizing the following training loss (which is possibly regularized):
Crast((f) (..aa))Daaam brank (f())
(1)
One directly leverage pretrained model by using it to initialize the model and then performing either finetuning or probing using objective (f). However, these direct approaches require using the same archilecture as the original pretrained model, which can be limiting for applications where inference speed and memory are critical factors. Instead, we are interested in learning a lightweight model fa that can still leverage knowledge from a much larger pretrained encoder model e, perform the task. To this end, first construct teacher model f, from the pretrained encoder e, and then use it distill knowledge relevant to the task on the light model fs-
5

---

Published in Transactions on Machine Learning Research (05/2024)
Step 1: Teacher probing. We augment the encoder model et with a task-specific prediction head Pa creating the teacher model St (i.e., fa(x) = p=(e)()))) The teacher is then probed for the supervised task by training the prediction head Pt to minimize the training loss (fts Notably, the parameters of the encoder e₁ remain frozen. This not only significantly reduces the training cost compared to finctuning but it also helps preserving information acquired during (sell-supervised) pretraining. Our experiments (Table 2 indeed show that this probed teacher leads to better distillation results than its finetuned version in general.
Step 2: Distillation. After probing the teacher ft, we use to guide the training of smaller student model fs on the downstream task. Specifically, we supplement the task loss Leask with distillation loss Laistin that encourages the student's predictions to match the teachers', resulting in an overall objective of the form:
C(f.) = (1 - alase + a fz,
(2)
where is a weighting parameter controlling the strength of the distillation loss. We define Edistill as an average of some dissimilarity measure la between the student's and the teacher's predictions over a set D of well-chasen images:
Laistill (Js, = BL,, (La (J.(t).
(3)
The loss eq. (3) ensures our distillation protocol is agnostic to the architecture since the dissimilarity measure la depends solely on the student's and the teacher's outputs and not on their internal structure. We set the dissimilarity measure la to be the KT-divergence rescaled by temperature parameter T proposed by Hinton et al. (2015):
[1(x)) T
f((x)) T
((({(()) DKL
(4)
The choice of images DD in the distillation loss (eq. (3) is crucial as it defines the nature of images for which the student is required to match the teacher's predictions. While is only natural define D as the set of training images Dtrain, this choice is not necessarily the most effective for extracting relevant knowledge from the teacher Dirain could offer view that is too narrow. Instead, we propose to build D by extending Dtrain using an augmentation protocol based on Stable Diffusion, described in the next subsection.
3.2 Distilling with synthetic data
The distillation process outlined in Section 3.1 aims to aligu the teacher's and student's outputs for set of images D that is sufficiently large and diverse to extract relevant knowledge. While it is possible to augment a dataset with standard data augmentation, our experiments indicate that this may not introduce enough diversity. When aiming for increased diversity, generating images relevant to the task-with suitable semantics and originating from the correct domain is crucial. However, this step should be task-agnostic avoid the need for manual tailoring to each downstream task, or avoid providing class names or any other ground truth.
We propose to use variant of Stable Dilfusion, originally introduced by Pinkney 2022 for hesth purposes, named ImageMixer. It is finetuned version of Rombach et al. (0022) that enables the mixing of CLIP image representations from two or more input images to generate a new one. More precisely, CLIP embeddings are concatenated along the sequence dimension and serve as a conditional input. We use this method as a variant of Mixup for data augmentation, which involves mixing random pairs of images, regardless of their classes. This enables to create an augmented dataset Daa containing both the original images from Drrain and the synthetic ones. During training, use Dea by randomly sampling synthetic images and original ones with equal frequency.
Example images generated for the CUB (Wah et al. 2011), Pascal VOC (Everingham et al. 2010 and NomsinNet's 's Painting (Peng et al. 2019) datasets can be found in Figure Additional examples can be found in the appendix.
It is crucial to note that the corresponding augmented set is exclusively used for the distillation loss Edistill We have experimentally observed that. introducing synthetic data in the optimization of Lrast degrades
6

---

Published in Transactions on Machine Learning Research (05/2024)
Synthetic image
Real images
Synthetic image
Real images
Synthetic image
Real imagea
Figure 3:
Diffusion- based data augmentation. Examples of synthetic images generated using Im-
ageMixer Pinnkeey 2022) as described in Section 3.2 mixing two training images from CUB (Wah et al.
2011
(left), Pascal VOC (Everingham al. 2010 (middle) and Painting from DomainNet (Pegg et al. (right). Those populate the exlended dataset Daa for distillation.
2019
performance, even for a variant that only mixes images of the same class (see Section 4.3) This supports the intuition that the generated images are diverse enough to potentially extend beyond the scope of each class, while remaining close enough to the overall training domain to still be useful for distillation.
4 Experiments
We evaluate our distillation protocol across three families of tasks: classification on various domains, fine- grained classification, and semantic segmentation. For classification, we consider the painting, sketch and clipart datasets from DomainNet. (Peng et al.) 2019), each composed of the same 345 classes, for which we isolate 20% of the training set for testing. Fine- sise-grained classification conducted on the CUB (Wah et al. 2011), FGVC Aircraft (Maji et al.) (2033) and DTD (Cimpoi ct al.) 2014) datasets respoctively consisting of 200 bird species, 100 aircraft models, and 47 textures. Finally, we use three benchmarks for segmentation: ADE20K (Zhou et al. 2017). Cityscapes (Cordts et al. 20166), and the augmented Pascal VOC (Everingham) et al. 2010). After overview of our experimental setup (Section 4.1 present our main distillation
results (Section 4.2
followed by additional ablation studies (Section 4.3
4.1
Experimental setting
We present the design choices for our student and teacher models and detail the data augmentation applicd, the training hyperparameters and our evaluation protocol.
Backbone models. For the teacher, we start from one of the pretrained models provided by DI- NOv2 (Oquab et al. 2024). either ViT-S, ViT-I. or ViT-g, three architectures of increasing capacity. Note that the ViT-I. and VIT-S models provided by DINOv2 are distilled from their ViT-g. The teacher is then one of these pretrained models probed for the target downstream task (see top part of Figure 2 We also consider finetuned VIT-L teacher in our study to investigate the impact of finetuning versus probing strategies for the teacher. We do not explore finetuning the ViT-g model, in line with the paper's focus on maximizing the utility of pretrained models within constraints of limited computational resources.
For the student, we explore two lightweight architectures. The majority of our experiments use a ViT-S model initialized with DINOv2's pretrained weights. We also show that our observations generalize to randomly initialized models: we report experiments with a ResNet-50 model for classification and a DeepLabv3 model CCen et al. 2017) with Net-5t backbone for segmentation.
7

---

Published in Transactions on Machine Learning Research (05/2024)
Prediction head. We use MLP head for classification (unlike DINOv2 which evaluates with linear head) and DINOv2's linear head for segmentation, for students and for teachers. Note that there is no prediction head for the ResNet-50 and DeepLabv3 models as those are trained from scratch.
When available, the input for the prediction head is defined as follows. In classification tasks, we adhere to DINOv2's process: i) we concatenate the CLS tokens from up to the last four blocks (choosing 4 for DomainNet and 3 for fine-grained tasks), ii) optionally, we concatenate the average pooling of the patch embeddings from the last block (which we only do for DomainNet). For segmentation tasks, we adopt DINOv2's linear evaluation protocol, directly evaluating from the patch embeddings of the last block.
Synthetic image generation with Stable Diffusion. We generate synthetic datasets with times more images than the original training set, setting n to 5 for DomainNet and segmentation tasks, and 10 for the relatively smaller fine-grained classification datasets. As noted carlier, this data augmentation strategy may be viewed as a variant of Mixup (Zhang et al. 2018). akin to interpolating between random pairs of images using the ImageMixer method proposed by Pinkney 2022
Standard data augmentation. In all experiments, we apply classical data augmentation to both the original training images and the synthetic images. For classification tasks involving transformers, we use RandomResizzdCCop, ColorJitterr and Mixup, while for ResNet-50, we use TrivialAngment (Müller & Hutter 2021). Note that Mixup is excluded for synthetic images obtained from ImageMixer, which is already variant of Mixup based Stable Diffusion. For segmentation tasks, adopt the same augmentations DINOv2 (Oquab et al. 2024) see details in the appendix. Following the recommendation of Beyer et al. (0022), the student and teacher models receive exactly the same batch of images, transformed with the same data augmentation.
Training hyperparameters. Probing runs for 20 epochs for ViT-L/g and 30 epochs for VIT-S, while finetuning lasts for 50 epochs for ViT-L and 80 epochs for ViT-S. We use the AdamW optimizer for training ViTs and SGD with momentum for ResNet-50, and a cosine scheduler in both cases. The selection of weight decay and learning rate is determined through a grid search on the validation set, with specific details available in the appendíx. In instances where no predefined validation set exists, we allocate 10% of the training set for this purpose. We use a fixed distillation temperature of T = 2 and constant weighting between trsk and Laistil set to 0.5 for all experiments.
Evaluation. We report results averaged over three independent runs with different random seeds. For distillation evaluations, we consider three different teachers, each from independent runs, and conduct 2 runs per teacher for Net and 3 runs for fine-grained and segmentation tasks.
About our probing results. We remind that for classification, we use a MLP head while DINOv2 (Oquab) ct al., 2024) uses a lincar head. Please also note that for segmentation, we use an image size of 560x560 pixels while DINOv2 uses 512 x 512. This explains why our probing results are slightly higher than those reported by Oquab et al. (2024) (comparison the appendix).
4.2 Experimental results
We explore distillation with two different students: i) DINOv2's VIT-S pretrained with task-agnostic distil- lation, and ii) randomly initialized models: ResNet-50 for classification and DeepLabv3 with ResNet-50 backbone for segmentation.
Our results are presented in Tables 1 to 3 Table 1 reports probing and finctuning results for DINOv2 ViT-S, VIT-L and ViT-g pretrained models. Table 2 reports distillation results using ViT-S as the student, and using a probed ViT-S, a probed and a finetuned ViT-L or a probed VIT-g as the teacher. Table g reports distillation results using a probod ViT-g as the teacher, and randomly initialized ResNet-50 as the student. Distillation results reported in Tables 2 and 3 both with and without augmenting the training set with synthetic images for distillation, are compared to those obtained with a simple probing or finetuning of the
8

---

Published in Transactions on Machine Learning Research (05/2024)
Classification on Nomain.et (acc)
Fine-grained classification (acc)
Semantic segmentation (mloU)
Model
Painting
Sketch
Clipart
CUB
Aireraft
DTDD>
ADE20K
Cityscapes
VOC
VIT-S (1a) Probing
77.3
71.9
79.3
88.2
77.1
82.1
45.1
67.0
81.8
(1b) Finctuning
79.4
76.0
81.8
87.3
87.8
81.6
49.8
75.8
84.6
VIT-L (2a) Probing
80.4
85.3
91.3
87.8
85.5
47.8
70.4
82.7
(2b) Pinctuning
83.9
81.4
85.9
91.5
94.0
85.8
57.4
78.6
88.0
ViT-g (3a) Probing
81.2
85.7
91.6
88.1
85.8
48.8
71.2
83.5
Table 1: /fnettinnn of DINOv2 pretrained models for classification on DomainNet, fine- grained classification and semantic segmentation. We report accuracy for classification and mIoU for sog- mentation. Relative distillation gains in Table are with respect to underlined results in this table
Classifleation DomainNet (nee]
Fine-grained classification (acc)
Semantic segmestattin (mloU)
Student
Teacher
SD
Painting
Slutch
Clipart
CUB
Aiseraft
I/ED
ADE20K
Cityseapes
VOC
Vir-S
(4u)
80.0 (+0.6)
76.9 (40.9)
82.2 (-6.4)
89.4
86.5 (-1.8)
82.9 (+0.8)
49.6 (-0.2)
71.2
84.6 (+6.0)
probed
(48)
80.2 (.008)
77.1 (10.2)
82.4 (4066
89.7 (41.6)
86.6 (-1.2)
83.4 (+1.))
50.3 (40.5)
72.3 (-8.6)
84. (10.))
VIT-L
(5a)
x
80.5 (111.)
17.8 (11.8)
83.4 (11.8)
89.7 (11.5)
89.2 (11.4)
63.4 (1
50.7 (1
74.0 (-1.8)
60.5 (10.9)
probed
(58)
80.8 (11.4)
78,0 (12.0)
83.2 (11.4
90.0 (1.88)
89.8 (12.0)
84.0 (11.9)
51.7 (11.0)
74.7 (-L.)
86.1 (11.5)
V-T-L
(6a)
x
79.7 (+0.3)
77.0 (+1.0)
82.5 (90.7)
88.6 (10.4)
88.0 (+1.3)
61.5 (10.6)
50.7 (+0.0)
76.3 (10.5)
8188 (10.2
flrecaned
(68)
80.3 (4-0.))
77.2 (41.2)
82.9 (41.1)
88.6 (4-0.4
89.1 (4115)
82.5 (4004)
51.0 (4-1.8)
76.4 (40.6)
80.7 (+1.1)
ViT-g
(7a)
80.5 (+1.1)
77.7 (41.7)
83.4 (+1-6)
80.1 (+0.9)
89.8 (41.8)
63.1 (+1.0)
51.6 (+1.8)
74.4 (+1-4)
85.7 (+1.1)
probed
(78)
/ 80.8 (11.4)
78.0 (12.0)
83.3 (11.5)
89.8 (11.))
90.1 (12.3)
83.6 (12.5)
52.1 (12.3)
75.0 (-0.8)
86.3 (1177
Table 2: Distillation on VIT-S initialized with DINOv2 for classification on DomainNet, fine-grained classification and semantic segmentation. We report accuracy for classification and mIoU for segmentation. We report results with and without data augmentation based on Stable Diffusion (SD), for various choices
of teachers. Relative gains with respect to simple probing or finetuning (best underlined in Table I are in
parentheses. Bold numbers: within 95% confidence interval of the best score for each task.
Sturlent
Teacher
8D
Classification on DomainnNe (acc)
Fine grained clastiication (acc)
Semansic segmentation (mlott)
Painting
Skrtch
Clipart
CUB
Aireratt
DTD
ADE20K
Cityscape
(8a)
68.0
68.1 1
72.5
73.3
85.0
63.5
37.8
67.9
67.5
R50
ViT-g
(9a)
67.7
20.5 (+2.4)
74.9 (+2.4)
76.0 (427)
65.7 (-0.7)
(43.3)
(-0.4)
67.7 (402)
67.7 (40.2)
probed
(96)
69.1 (42.7)
71.0 (-
75.2 (+2.))
79.1 (45.8)
87.8 (+26)
60.4 (+5.9)
42.1 (+1.5)
69.3 4)
73.9 (46.9)
Table 3: Distillation from ViT-g to ResNet-50 (resp. DeepLaby3-RosNetta for sogmentation) trained from scratch for classification on DomainNet, finc- grained classification and semantic segmenta- tion. We report accuracy for classification and mIoU for segmentation. We report results with and without data augmentation based on Stable Diffusion (SD). Relative gains with respect to simple training (underlined) are in parenthescs. Bold mumbers: within 95% confidence interval of the best score for each task.
ViT-S (best underlined in Table 1 and with simple training of ResNet-50 (underlined in Table 3 with relative gains indicated in parentheses.
We discuss the results of Tables [ to according to four separate axes supporting the main claims of this study: i) the relative gains of distillation over finetuning when teaching a small pretrained model, ii) the impact of finetuning the teacher, iii) the impact of using a teacher that is less accurate than the student, and iv) the generalization of our observations to students trained from scratch.
In what follows, observations are discussed by comparing lines of Tables to 3 The lines from these three tables are denoted by unique alphanumerical reference such that e.g. the mention (2a us 2b) refers to comparing lines 2a and 2b in the Table 1
Task-specific distillation complements task-agnostic distillation. The key observation from Table 2 is that task- specifcc distillation generally outperforms probing and finetuning. This can be observed by comparing any line from Table 2 4 to 7, a or b, with the corresponding number in line from Table []] referred to as (4-7 us 1) following the notation introduced in the previous paragraph. Figure this observation ADF20K: the PCA of patch embedding representations exhibits better clustering structure
9

---

Published in Transactions on Machine Learning Research (05/2024)
Teacher
Student init.
Student finetuned
Student distilled
Figure 4: PCA of patch embedding representations for 20 classes of ADE20K for the ViT-g g teacher (a) and for the ViT-S student in its imitial state (b), after finetuning (c) and after distillation (d), colored by their main class (details in the appendix). Classes are better clustered after distillation than after finetuning.
after distillation than after finetuning (see also Figure D. Cityscapes is the only exception where distillation from a probed ViT-L or a probed ViT-g does not improve over finetuning. Interestingly, on Cityscapes, the finetuned VIT-S student already outperforms the probed ViT-g and ViT-L teachers by a large margin (+44.6 and +5.4 mIoU) (1b us 3a, which may explain why distilling from those is not beneficial.
Our dataset augmentation based on Stable Diffusion further enhances distillation results (4a 4b, 5a 5b, etc.), except on Clipart, where it performs par with distillation on the original training images alone.
While ViT-g exhibits slightly higher accuracy than ViT-L when probed on downstream tasks (Sa us 2a), both models serve as almost equally effective teachers for distillation (7 vs 5). In exploring experiments with smaller teachers, we evaluate distillation from a probed ViT-S (1a), placing ourselves in the context of self-distillationn We observe that when the performance gap between probing and finetuming is not too large, self-distillation (4) improves over finetuning (1b), evident across all baselines except for Aircraft and Cityscapes, where the probed VIT-S has accuracy accuracy/mloU /mIoU approximately 10% lower than with finctuning (Ia 1b). However, it is important to note that ViT-g and ViT-L remain superior teachers compared to VíT-S. This implies that, even if ViT-S was pretrained with generic distillation from ViT-g, it is more effective to directly leverage the largest teachers for dounstream tasks.
Finetuning yields a poorer teacher than probing. Next, we study the impact of finetuning our teacher prior to distillation, comparing distillation results using either a probed or finctuned pretrained ViT- L model from DINOv2 (Oquab et al. 2024). Finetuning significantly enhances ViT-L's accuracy compared to probing (2a us 2b). However, employing the finetuned ViT-L model as a teacher generally results in a poorer performance for the student (5 us 6). For example, finetuning brings about 6% increase in accuracy for Aircraft and Pascal VOC compared to probing (2a 2b), yet the distillation results with the probed teacher are better (5 vs 6). This suggests that preserving the rich represenlations learned during pretraining is crucial, even if it leads to a teacher with lower accuracy for the specific task. Cityscapes is the only exception where distillation of a finctuned teacher significantly improves results, while using a probed teacher degrades them (5 us 6). This may be attributed to the substantial performance gap between probing and finetuning on this dataset, with +8 to +9 mIoU (1a us 1b, 2a vs 2b).
In summarr, our experiments indicate that finetuning the leacher for lask-specific distillation is oflen un- necessary and sometimes even detrimentol. Given the relatively fast training of the MLP head, the primary computational cost in knowledge distillation with teacher probing lies in training the student, with the additional overhead of performing forward passes through the teacher.
Teachers are not required to be as accurate as students. Sometimes, simply finctuning our VIT-S model (1b) gives better results than probing ViT-g (3a). This is the case for all three segmentation tasks. Still, distilling from ViT-g proves beneficial for ADE20K and Pascal VOC (7b), as it gives around 2% mloU gain compared
finctuning (1b), even though the finctuned ViT-S model is already about 1% higher mIoU than the teacher (3a). This supports the more general observation that a student can surpass teacher and still benefit from distillation.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Student: VIT-S
Student: ResNet-50
Text prompt-free
CUB
Aircraft
DTD
CUB
Aircraft
DTD
Baseline (distillation only from Ptrain)
89.1
89.6
83.1
76.0
85.7
66.7
Ded Teet-to-image (parent class)
89.4
90.0
83.6
77.8
87.9
68.1
Ded Teet-to--mmaee (class name)
89.5
90.2
83.5
79.8
87.6
68.7
Dad composed of Random ImageNet images
89.2
89.4
83.3
77.3
86.5
65.8
Dat uses ImageVariations Dat uses ImageMixer
89,5
90.4
83.4
78.8
87.7
68.7
80.8
90.1
83.6
79.1
87.8
69.4
Table 4: Building Dxd from Dtrain- We compare distillation results, using ViT-g as a teacher and ViT-S or ResNet-50 as a student, for our mixing approach based on stable diffusion (ImageMixer, by Pinkney 2022) with i) model producing image variations from single images (ImageVariations, by Pinkkecy 2022). ii) simply adding subset of ImageNet, and iif) text-to-image diffusion using the parent class only, e.c. bird, aircraft or texture (*A photo of {parent class)"), or using class information as well (*A photo of a (class name} {parent class}")). We observe that the ImageMixer variant we advocate for is surprisingly competitive despite not requiring a text prompt.
Data augmentation based Stable Diffusion substantially helps students trained from scratch. Here, we replace DINOv2's pretrained ViT-S student with ResNet-50 (resp. DeepLabv3 with ResNet-50 backbone for segmentation) trained from scratch, while retaining DINOv2's pretrained ViT-g as the teacher. Results are reported in Table 3 and consistently demonstrate that distillation is beneficial, leading to 2% accuracy gain on average. Notably, data augmentation based on Stable Diffusion significantly enhances results, yielding a further 2-3% accuracy gain on fine-grained lasks and a 4-6% mIoU gain for segmentation compared to standard distillation (9a as 9b). Surprisingly, the ResNet-50 model benefits even more from distillation than the pretrained ViT-S model. These findings indicate that the observations made for pretrained ViT-S student generalize to students that i) did not undergo generic distillation or any form of pretraining, and ii) whose architecture is not based on transformers like the teacher.
4.3 Ablation studies
Data augmentation with Stable Diffusion. We now compare various strategies for creating an aug- mented dataset D&a used for distillation. Our evaluation focuses on fine-grained classification tasks, involving both a pretrained ViT-S and a ResNet-50 trained from scratch as students.
We conduct a comparative analysis of our data augmentation strategy based on ImageMixer (Pinkney 2022) We compare
i) another model by Pinkney (2022) creating image variations from single images; ii) an augmentation approach incorporating an ImageNet subset; and fii) the text-to-image diffusion model used by (Saryyldız et al. (2023). For the latter, we explore textual prompts with the parent class, and with and without class names. Specifically, prompts with class names take the form "A photo of a {class name] {parent class)" while prompts without class names follow the pattern "A photo of {parent class)", where {parent class) represents either bird, aircraft, or texture.
Table shows that our prompt-frce approach performs cqually well, not better, than prompt-basod data augmentation that leverages class information. Additionally, prompt engineering poses challenges for certain tasks, especially segmentation, and necessitates the model used for parsing prompts (e.g., CLIP) to be trained with semantic information about the data. This may be impossible for some modalities such as medical or microscopy images, which are not easily described with text and might fall outside the semantic scope expected by CLIP-like models.
Our chosen strategy based on synthetic images produced using the ImageMixer model outperforms the ImageVariations model on out of 6 settings, validating the benefits of a mixing mased approach conditioning image generation with multiple images.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Data used in
CUB
Aircraft
DTD
Loss
SD
CUB
Aircraft
DTD
Caistill
Liosk
Finetuning
Cieain
$7.3
87.8
81.6
Finetuning
Dad-ianea
83.8
85.4
80.3
Darain
87.3
81.6
Catamn
88.8
86,2
82.7
89.6
86.9
82.9
-idtintra
Ded-intea
89.6
Distillation
Distillation
Dud-intra
Derain
89.6
90.1
83.9
Duain + Catatill
x
89.1
89.6
83.1
Dad
Dirain
89.8
90.1
83.6
89.8
90.1
83.6
Table 5: Impact of synthetic data on each loss. Impact on fine-grained classification tasks for finetuning and distillation with ViT-$ as student. and ViT-g as teacher, using dataset Dad-intra augmentod with synthotic images by mix- ing original images inside each class separately.
Table 6: Role of the different losses. We com- pare optimizing Luain only (i.e. finetuning), Ldistill only, or both losses with equal weighting (standard distillation followed in our experiments). Results are with ViT-S as student and ViT-g as teacher.
Using augmented data for supervision. In our study, we use synthetic data only for optimizing the distillation loss Edistill; while the task-specific loss Ltask is trained solcly on real data. In this section, we explore the outcomes when incorporating synthelic images as additional labeled data for optimizing Lirinn for both finetuning and distillation. For this purpose, we compare two different ways of leveraging the difu- sion model of P (2022) described in Section 8 mixing images regardless of their labels (inter-class), or mixing images from each class separately (intra-class). These approaches result in two augmented datasets, Dad and sad-intra; containing both the original images and the synthetic ones, as explained in Section 3 Ta- ble presents distillation results for the fine-grained datasets. We observe comparable performance between inter-class and intra-class approaches, but incorporating synthetic data for supervision is not beneficial. In particular, including synthetic images for fineluning considerably degrades results. This aligns with the inlu- ition that the diffusion model may not be faithful enough to each fine-grained class, and even when provided with two images of the same class, it may generate a new image beyond the scope of this class.
Relative weighting between task and distillation losses. Here, we investigate the influence of com- pletely excluding the loss Lrask during distillation. Table 6 presents the results for fine-grained classification when solely optimizing Cuain (i.e., finetuning), solely optimizing Edisull, and optimizing both with equal weighs, as implemented in our study. The outcomes reveal that training without label information (i.e. op- timizing Ldistill only) yields competitive results for CUB but significantly lower results for Aircraft. Overall, the student achieves the best results when exposed to both hard labels and soft teacher labels.
5 Discussion and concluding remarks
Since the seminal work of Sharif Razavian et al. (0014), it has been known that generic pretrained models could be reused directly or adapted for many target tasks instead of learning new models from scratch. Yet, the rapid development and public release of even larger, rich and generic models, pretrained on up to billions of images (Oquab al., 2024 Fang et al., 2023), raisos pressing question with heavy practical implications: How to best leverage the knowledge of large and generic visual models when training smaller model for specific task? Our work aims at addressing this question by reexamining current good practices for knowledge distillation in the light of these new large models, and draws series of experimental conclusions. Below we summarize the main messages of our study, and relate them to previous discussions on neighboring topics.
Task-specific distillation and sclf-supervised lcarning. In the context of self-supervised lcarning, knowledge distillation has emerged as a compelling way to compress large pretrained models into smaller ones, yielding significant improvements compared to directly pretraining these small models (Abbasi Koohpayegani ct al. 2020 Fang et al.) 2022 Xu et al.) 2022 Wu al. 2022 Oquab et al. 2024). Yet, our study shows that simply finetuning or probing these small pretrained models yields sub-optimal results compared to leveraging
the knowledge of the larger models for specific downstream task.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Accurate teachers may not be the best for distillation. In prior works, Cho & Hariharan (0019)
Mirzadeh et al. (2020) have observed that the most accurate teachers are not always the best for distilationn, attributing this observation to the model capacity gap between the student and the teacher. To make the most of the largest models, Mirzadch et al. (2020) propose a multi-stage approach where knowledge is distilled from a large model to successively smaller ones, thus reducing the capacity gap between two successive distillation steps. Pointing to the inefficiency of such approach, Cho & Hariharan (2019) show
that instead, this capacity gap can be mitigated by stopping the teacher's training early. This early-stopping approach may be related to our observation: by freezing the teacher's pretrained backbone, i.e., probing the model instead of finetuning it, we prevent it from specializing too much to the given task. This results in better distillation despite a lower teacher accuracy. Nevertheless, we did not find any evidence that a large capacity gap may be detrimental to distillation, since our best results were obtained by distilling directly from DINOv2's (Oquab et al. 2024g largest ViT-g model to much smaller models, such as ViT-S or ResNet- 50. Instead, the fact that a probed model can serve as a better teacher than a finctuned one suggests that some aspects of the representation that are still relevant to the lask are lost when training for that (ask. This phenomenon is further discussed in the next paragraph.
Probing can yield better teachers than finetuning. One of the key observations of our study is that, comparable performance (experimentally, difference in accuracies smaller than 6% for the three families of models considered here, DINOv2, EVA-02 and EVA-02-CLIP), a probed model makes a better teacher than a finetuned one. This observation could be due to the catastrophic forgetting which happens (Kirkpatrick ltt al. 2017) when performing finetuning. Our experiments suggest that, when specializing for a given task, the finetuned teacher has forgotten some features that are not immediately relevant for optimizing the task loss, but that still help generalization. For instance, when finetuned on the CUB bird classification task, the pretrained model could end up only relying on spurious correlations (such as the background, standard source of spurious correlations in CUB, as studied in Sagawa ct al. 2019) that provide shorteuts to the optimization. These shortcuts help improving the teacher accuracy but are detrimental to the distillation process.
A teacher need not be more accurate than its student. Prior works have showed that student models can learn from poorly trained teachers (Yuan et al., 2020), or teachers with the same architecture (Furlanello et al. 2018), somctimes even outperforming them. Our results are consistent with these findings, since we also observed that a small model can benefit from distillation even when that same model already outperforms its larger teacher after simple finetuning. Additionally, distillation also resulted in improvements when using a teacher of the same size as the student (f.e., self-distillation). However, our results highlight that distilling from the largest models works considerably better than self-distillation, thus supporting the idea that knowledge from larger models can further guide the training of a smaller student.
Stable Diffusion as a source of additional information. Our study shows that our data augmentation strategy producing synthetic images with Stable Diffusion can be leveraged to extend the set of images used to optimize the distillation loss Ldistill- However, as illustrated in Table 5 including these synthetic images in the optimization of the task loss Lrssk can degrade results, particularly for finetuning. This shows that it may be difficult to efficiently leverage this augmentation strategy outside the context of knowledge distillation. Using synthetic imago-label pairs for supervised lcarning requires i) the generation process to be good enough for images to pertain to their class, which may be challenging for fine-grained tasks, and ii) the generation process to create the right label for each new image, which is particularly challenging for dense tasks such as semantic segmentation. Knowledge distillation alleviates the need for generating class-specific images and for labeling generated images, and hence can more easily leverage Stable Diffusion.
Acknowledgments
This project was supported by ANR 3IA MIAT@Grenoble Alpes (ANR-19-P3IA-0003) and by ERC grant number 101087696 (APHELEIA project). This work was granted access to the HPC resources of IDRIS under the allocation AAD001003343R1 made by GENCI.
13

---

Published in Transactions on Machine Learning Research (05/2024)
References
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self-supervised learning by compressing representations. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Shekoofch Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Flect. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research (TMLR), 2023.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Newral Information Processing Systems (NIPS), 2014.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IERE/CVF Conference on Computer Vision and Pallern Recognation (CVPR), 2023.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. In Proceedings of the AAAI Conference Artificial Intelligence, 2021a.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint ar 1.X.vv1706.0000 775605587, 2017.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
M. Cimpoi, S. Maji, 1. Kokkinos, Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), 2014.
Marius Cordts, Mohamed Omran, Scbastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-based augmentation. In Advances in Newral Information Processing Systems (NeurIPS), 2024.
Quentin Duval, Ishan Misra, and Nicolas Ballas. A simple recipe for competitive low-compute self supervised vision models. Xiv preprint ar Xii:2331.00454 2023.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In Advances in Neural Information Processing Systems (NewrIPS), 2023.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Wimn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV), 88(2):303-38) 2010.
Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmentation for object detection via controllable diffusson models. In Proceedings of the IEEE Winter Conference Applications of Computer Vision (WACV), 2024.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. arXio preprint arXi::2303.110 11331, 2023.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self- supervised distillation for visual representation. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Yuting Gao, Xin Zhuang, Shaohui Lin, Hao Cheng, Xing Sun, Ke Li, and Chunhua Shen. Disco: Remedy self-supervised learning on lightweight models with distilled contrastive learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey.
International Journal of Computer Vision (IJCV), 129:1789-1819, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint AAXiv:1503.02330 2015.
Wei Huang, Zhiliang Peng, Li Dong, Furu Wei, Jianbin Jiao, and Qixiang Ye. Generic-to-speeifc distillation of masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2023.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tiny- BERT: Distilling BERT for natural language understanding. In Findings of the Association for Compu- tational Linguistics: EMNLP, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsse et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114((3))3521-33) 2017.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In Proceedings of the International Conference Learning Representations (ICLR), 2022.
S. Maji, J. Kannala, Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Aug Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2020.
Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
K I. Navancet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Simrog: Regression as a simple yet effective tool for self-supervised kuowledge distillation. In Proceedings of the Brilish Machine Vision Conference (BMVC), 2021.
Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic dataset generation for lixel-level semantic segmentation. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syn- naeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.
15

---

Published in Transactions on Machine Learning Research (05/2024)
DINOv2:
learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the International Conference Computer Vision (ICCV), 2019.
Justin Pinkney. Image mixer, 2022. Lambda Labs.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition (CVPR), 2022.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnels: Hints for thin deep nets. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, Distributionally robust neural networks for group shifts: shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.0879) 2019.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In Proceedings of the ACM SIGGRAPH Conference on Compuler Graphics and Interactive Techniques, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems (NewrIPS), 2022b.
Mert Bülent Sanyyllız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cun features off-the-shelf. astounding baseline for recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPRW). 2014.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? In Advances in Newral Information Processing Systems (NeurIPS), 2021.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXio preprint ar 2rv::2300 1 15899, 2023.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Proceedings of the Conference on Empirical Methods Natural Language Processing (EMNLP), 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Je- gou. Training data-cfficient image transformers & distillation through attention. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. Xixi preprinl ar 4rXin:2302.07944 2023.
Frederick Tung and Greg Mori. Similarity-preservvin knowledge distillation. In Proceedings of the Interna- lional Conference Computer Vision (ICCV), 2019.
16

---

Published in Transactions on Machine Learning Research (05/2024)
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes "good"' data augmentation in knowledge distillation statistical perspective. In Advonces in Neural Information Processing Systems (NeurIPS), 2022.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 44 (6):3048 1103048-3068,,0 2021.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep sell-attention distillation for aask-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tiny Fast pretraining distillation for small vision transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Haohang Xu, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Bag of instances aggregation boosts self-supervised distillation, In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast opti- mization, network minimization and transfer learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance convolutional neural works via attention transfer. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the BEEE/CVF Conference Computer Vision and Pattern Recogration (CVPR), 2022.
Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adcla Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. International Journal of Computer Vision (IJCV, (33)))233772344) 2022.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Appendix
In this appendix, first introduce additional experimental details regarding the choice of training hyperpa- rameters and data augmentation, the prediction heads, PCA visualizations and training time (Appendix A Next, we provide additional experimental results, with a comparison of linear and MLP heads for classifi cation, an extended version of Tables 1 to with confidence intervals, additional distillation results with EVA-02 MIM- and CLIP-pretrained models (Fang et al. 2023 Sun et al., 2023) instead of DINOv2, and comparison of our chosen distillation loss (Hinton et al. 2015) to alternatives from the literature (Ap- pendix Bj. Finally, include additional visualizations of synthetic images produced by our mixing based Stable Diffusion (Appendix Ch.
A Additional experimental details
A.1 Datasets
Table A reports the number of classes and the number of images in the training set of the datasets used for our study.
Classes
Size (train)
Painting
60617
DomainNet
Peng et al.
2019
Sketch
345
56304
Clipart
39064
Fine-grained
CUB (Wah et al.
2011
200
5994
classification
Aircratt Maji al.
2013
100
6667
DTD Cimpol
2014
3760
Semantic segmentation
ADE20K (Zhou et al.
2017
150
20210
Cityscapes (Cordts
2016
2975
Pascal VOC (aug.. |Everigham et al.
2010
10582
Table A: Number of classes and size of training set of each dataset.
A.2 Training hyperparameters
Table details the weight decay and lcarning rate used for each task (classification on DomainNet, finc- grained classification, semantic segmentation), each archilecture, and each training procedure (with/withe withhut freezing the pretrained backbone). Values are chosen based on grid search on the validation set. More preciscly, coarse grid scarch is first performed finer one that is reported in Table B When the grid search leads to values that are nearly identical for all tasks, we fix the value and report it in the table. Note that distillation with synthetic images based on Stable Diffusion is run with the best hyperparameters found for distillation without synthetic images. Also note that for finetuning experiments on ViT-L, we use a smaller batch size (8 for segmentation, 32 for classification) and reduce the learning rate in the grid search accordingly.
logarithmic scale using powers of 10, before defining
A.3 Generic data augmentation
In all experiments, we consistently apply classical data augmentation both training and synthetic images (except for Mixup which is only applied to original images). The list of augmentations with their parameters is detailed below for each task (classification or segmentation) and architecture.
Classification. When training ViTs, apply:
18

---

Published in Transactions on Machine Learning Research (05/2024)
Arch
Learning rate
Weight decay
Probing distilaaion
00002, {1, 43 x 10-5
VIT
DomainNet
{.025, 05, 113
ResNet-50
Training/distillation
.0005
Fine-grained classification
Probing
000,,,002,.000 {1,2,4} x 10-6
(,5, 2,4,, 4, 8) {.025, .05, .1]
ViT
Finetuning/
Mnnetuning/distilaationn distillation
ResNet-50
Training/distilattton
{.01,-02,.00
{.005, .013
Semantic
Probing
.008
ViT
segmentation
Finetuning/Cisstllaani
[1,2,4] x 10-5
{.001,
DeepLabx3(R50)
Thaining/distillation
.08, (00001, 01
Table B: Training hyperparameters for batch size of 128 for DominNet and fine-grained classification tasks and 16 for segmentation tasks (32 for probing). Hyperparameters in O are chosen based on grid search on the validation set.
RandomPesizzedCrog with scale 0.08
ColorJitter with range (0, 0.4)
RandomFlip with probability 0.5
Mixup with parameter 0.2.
An exception is for probing on fino-grained classification tasks, whore we simply apply Resize and CenterCrop instead RaandomResizedCrrre and do not apply Mixup. We found that these transformations were too strong for Aine-grained classification with a frozen backbone.
When training the ResNet-50, use TrivialAugment's Müäller & Hutter 2021 strategy (ImageNet version). that consists of
RandomResizedCCrpp with scale 0.08
ColorJitter with range (0, 0.4)
RandomPlip with probability 0.5
A fourth transformation randomly sampled among pool.
However for fine-grained classification, we use a scale parameter of 0.4 for RandomResizeeCrrop. as 0.08 proved too strong for training from scratch on fine-grained tasks.
For validation and testing, we follow the standard procedure of applying Resize and CenterCrop.
Semantic segmentation. We train with images of size (s, with = 560. We apply the following data augmentations for all experiments, which correspond to the mmsegmentation augmentations also used by DINOv2:
Resize to s.) with ratio range (0.5, 2.0)
RandomCrop s), with cat max ratio 0.75
RandomFlip with probability 0.5
PhotoMetricDistortin..
For validation and testing, sliding windows of size (8 and stride 3.
19

---

Published in Transactions on Machine Learning Research (05/2024)
A.4 Details prediction heads
We remind the reader that for segmentation, use the samne linear evaluation head as DINOv2's (Qquab et al. 2024 while for classification, we use a MLP head, unlike DINOv2 which uses a linear head.
More preciscly, let Pin: Rhidden: Rout respectively denote the number of input, hidden, and output ncurons in the MLP head. Pout the number of classes, and Rin the number of input features extracted from the pretrained backbone, meaning that Tin n (nCLS luse avgpool), where
f is the embedding dimension (ny = 1536, 1024,38, 384 for ViT-g, ViT-L and ViT-S respectively)
nCLS is the number of blocks from which the CLS tokens are concatenated (NCLS 4 for Demain 3 for fine-grained tasks)
Luse avgpool indicates whether we also concatenate the average pooling of the patch embeddings of the last block (true for DomainNet).
As for the number of hidden neurons Thidden: we set Thidden = nin for ViT-S and Whidden = Rin X Yout for ViT-L/ViT-g, as we experimentally found that this choice gave the best results. Intuitively, using such intermediate size for ViT-L./VIT-,, whose embedding sizes are larger (1024 and 1536), allow for more progressive decrease toward Nout-
A.5 Details on the PCA
The main paper provides PCA-based visualizations of the learned representations for CUB and ADE20K datasets, respectively in Figure [ and Detailed step-by-step descriptions of how these visualizations were constructed are provided below.
For the PCA visualization of teacher predictions from Fig. 1 on the CUB fine-grained classification task, the steps are the following:
1. Feature computation for both original and synthetic CUB training images, giving class token predictions of shapes (N, D) and (Nayathetie, D) with D the embedding dimension (D = 1536 for ViT-g and 384 for ViT-S)
2. Subsampling: we only keep the first 20 classes. We keep synthetic images that result from mix of images belonging to this set of 20 classes. This leaves M < N and Maynthetic Neynthotic images.
PCA over the (M, D) predictions on original images
4. Visualization of the (M + Maynthetie. D) data points projected onto the two main principal com- ponents, colored by class label for the M images, and in gray for the Msynthetic images.
For the visualization of student predictions on Figure 1, the steps are the same but without the synthetic images.
For the PCA visualization from Figure 4 on the ADE20K segmentation task, we visualize patch embedding representations as follows:
1. Feature computation on N = 500 test images, giving patch embedding predictions of shape (N, D. H, D, W) with D the embedding dimension; and H W = 40 (=
image size
560
patch size
14
Resizing of the corresponding 500 segmentation maps to shape (N, C, H, W) where C' is the number of classes. We use mmssegmentation's resize method on onc-hot encoded labels.
3. Flattening of predictions and labels to (N x H x W, D), (N x H W. C) respectively.
4. Filtering: keep patches whose labels are well defined, with a probability over 0.9.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Model
Head
Clussification on DomainNet
Fino-grained classification
Semantic segmentation
Painting
Sketch
Clipart
CUB
Airerafi
DTD
ADE20K
Cityscapes
VOC
Linoar Oquah at al.
2024
91.6
87.2
84.5
49.0
71.3
83.0
ViT-g
Probing
Linear
91.7
87.8
71.2
83.5
MLP - outs
83.0
81.2
85.7
91.6
88.1
85.8
Linear Oquab et al.
2024
90.5
81.5
84.0
47.7
70.3
82.1
VIT-L
Probing
Linear ours
82.2
91.8
86.5
MLP ours
82.9
80.4
85.3
91.3
87.8
85.5
Linear Oquab et al.
2024
88.1
74.0
80.6
41.3
66.6
81.1
Probing
Linear ours
75.6
78.7
89.0
75.8
81.8
VIT-8
MLP ours
77.3
71.9
79.3
88.2
77.1
82.1
Finetuning
Linear ours
78.5
75.6
81.3
87.0
87.3
81.2
19.8
73.8
81.6
79.4
81.8
87.3
87.8
Table C: Comparison of our linear probing results with DINOv2's (Oquab et al., 2024) and comparison of lincar and MLP heads for classification. We report accuracy for classification and mloU for segmentation. We report results from probing ViT-g, VIT-L, ViT-S, and from finetuning VIT-S. The underlined figures correspond to those in Table !
5. Subsampling: we only keep 20 classes. We select those whose size (number of patches of this class) is closest to the median size.
6. Filtering and subsampling yield a number M of data points, M < H
PCA on the DM predictions.
8. Visualization of the two main principal components, colored by class label.
A.6 Training time
All our experiments were performed on a single GPU (either V100 or A100). We detail the training time with a pretrained ViT-g as teacher and a pretrained ViT-S as student. When using a ResNet-50 from scratch as student, the training time per epoch similar to that of the VIT-S, but we train for longer (200 epochs instead of 80). Experimentally, we observed that probing the teacher (ViT-g) either takes less time about the same amount of time as finetuning the student (ViT-S). Distillation with the probed ViT-g as teacher takes approximately twice as long as finetuning. Lastly, adding data augmentation based on Stable Diffusion further increases the training time by 1.5 times in average. For example, finetuning the ViT-S for ADE20K takes 16 hours on a A100 GPU, while distillation with data augmentation based on Stable Diffusion takes hours, and probing the ViT-g takes 14 hours.
As for the generation of synthetic data, our image mixing procedure (Pinkney 2022) roughly takes hours for 1000 images (on V100 GPU). We remind that generated synthetic datasets with times more images than in the training set, with n = 5 for DomainNet and semantic segmentation, and = 10 for the relatively smaller fine-grained tasks. The size of the training set of each task is reported in Table A
B Additional experimental results
B.1 Additional results with linear prediction head
In Table C we compare classification results using linear and MLP heads when probing VIT-S, ViT-L and ViT-g, and when finetuning VIT-S. We also compare our linear evaluation results with DINOv2's (Qquab ct al. 2024). For lincar probing, our performance is similar to DINOv2's. Relative differences can be altributed to varying choices of data augmentation for classification, and to differences in image size for segmentation (we train with size 560 while DINOv2 uses 512). For classification, we observe that using a MLP head consistently improves results, both for probing and finetuning. In other words, using a MLP head for both the teacher and student models independently boosts their accuracy, and we experimentally observed that it also makes better teacher, as it yields the best distillation results for the student.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Classification on NomainNet (acc)
Fine-grained classification (acc)
Semantlc segmentation (mloU)
Model
Painting
Sketch
Cliport
CUB
Aireraft
DTD
ADE20K
Cityacapes
VOC
(3a) Probing
77.3 +.2
71.9 #.3
79.3 +.2
88.2 +.1
77.1 +.4
82.1 #.6
45.1 #.1
67.0 4.2
81.8 #.2
VII-S
(10) Finetuning
79.4 4.3
76.0 #.2
81.8 2.2
87.3 4.8
87.8 4.9
81.6 1:11
49.8 d..4
75.8 #.3
81.6 4.7
VIT-L (8e) Probing
82.9 4.2
80.4 #.2
85.3 1.1
91.3 1.5
87.8 113
86.5 4.3
47.8 #.2
70.4 #.1
82.7 4.3
(2h) Finetuming
83.9 +.2
81.4 #
85.9 1.1
91.5 1.1
94.0 4.5
85.8 #.6
57.4 1.
78.6 #:2
88.0 #:4
VIT-g (8c) Probing
83.0 2.4
81.2 #.2
86.7 1.1
91.6 2.2
88.1 #.6
86.8 1.3
48.8 2.2
71.2 #.2
83.5 #.1
Table D: Probing/ Trobing/finetinng finntuningg of DINOv2 pretrained models for classification on DomainNet, fine- grained classification and semantic segmentation. We report accuracy for classification and mloU for seg- mentation. We report the 95% confidence estimation (1.96o) averaged to the upper decimal.
Classifiation on DomainNet (acc)
Fine-grained classification (acc)
Semantic segmentation (mloU)
Student
Teacher
SD
Painting
Slotch
Clipart
CUR
Aireraft
DTD
ADF20K
Citysrapes
VOC
VIT-S
(4n)
x
80.0 +.2
76.9 #.4
82.2 #33
+.3
86.5 =.7
82.9 +.4
49.6 #.2
71.2 +.3
84.6 +.2
probed
(46)
80.2 +:3
77.1 #.2
82.4 #.5
89.7 4:3
86.6 =.6
83.4 +.5
50.3 #.2
72.3 4:3
84.9 4:2
VIT-L
(5a)
x
80.5 +.3
77.8 +.3
83.4 #.2
89.7 4.4
69.2 =.7
83.4 +.4
50.7 #.3
74.0 +.2
85.5 +.3
VIT-S
probed
(0b)
80.8 5:3
78.0 4:2
83.2 4
90.0 =.3
89.8 4:4
84.0 4:4
51.7 4:2
74.7 1.2
86.1 1:3
ViT-L
[6u]
x
79.7
77.0
82.5
88.6
81.5
50.7
81.8 1.4
finetuned
(66)
/
80.3 +:2
77.2 #.2
82.9 +.3
88.6 +.3
89.1 m.4
82.5 #.0
51.6 #:7
76.4 #.3
85.7 #.4
VIT-g
(Taj
x
80.5 +:1
77.7 #.3
83.4
2.2
89.1 +
89.6 1.5
83.1 #.8
51.6 #.4
74.4 #.2
85.7 2:3
probed
(76)
/
80.8 1.2
78.0
1.2
83.3
89.8
90.1
83.6 1.6
52.1 1.4
75.0
86.3
Table E: Distillation on VIT-S initialized with DINOv2 for classification on DomainNet, fine-grained classification and semantic segmentation. We report accuracy for classification and mloU for segmentation. We report results with and without data augmentation based on Stable Diffusion (SD), for various choices of teachers. We report the 95% confidence estimation (1.96g) averaged to the upper decimal. Bold numbers: within 95% confidence interval of the best score for each task.
Clessiftsttion on DomainNet (soc)
Fine-grained chassifccation (acc)
Semantic segmentation (mloU)
Student
Teacher
SD
Painting
Sketch
Clipart
CUB
Aireraft
DTD
ADE20K
Cityscapes
VOC
(8a)
66.0 1.4
68.1 4.3
72.5 4.9
73.3 4.2
85.0 #.9
63.5 41.2
37.8 4.7
67.9 #.7
67.5 41.0
R50
ViT-g
(9u)
67.7 1.7
70.5 L1.0
74.9 1.4
76.0 1.8
85.7 L.4
06.7 1.6
38.2 1.6
67.7 11.8
67.7 1.5
probed
(96)
69.1 4.8
71.0
75.2 4.6
79.1
#.9
87.8
4.5
69.4
#1.
42.1
69.3 #1.9
73.9 #.7
Table F: Distillation from ViT-g to ResNet-50 (resp. DoepLabv3-Resseeee for segmentation) trained from scratch for classification on DomainNet, fine-grained classification and semantic segmenta- tion. We report accuracy for classification and mIoU for segmentation. We report results with and without data augmentation based on Stable Diffusion (SD). We report the 95% confidence estimation (1.96o) aver- aged to the upper decimal. Bold numbers: within 95% confidence interval of the best score for each task.
B.2 Main results with standard deviations
In Tables D to F we provide uncertainty estimations to the results reported in Tables | to 3 of the main paper, respectively. We use 1.96a as 95% confidence estimator, and report its value averaged to the upper decimal, where a is the standard deviation of the results obtained through different runs. We remind that we use runs for probing, finetuning and training, 3 2 = 6 runs for distillation on DomainNet and 3x3 = 9 runs for distillation on fine-grained and semantic segmentation tasks.
B.3 Distillation with EVA-02 pretrained models
In this section, we assess whether the good practices we have drawn as experimental conclusions of our study using DINOv2's pretrained models as teachers also transfer to EVA-02 (Fang et al., 2023) and EVA-02-CLIP (Sun et al.) 2023) models, that were pretrained with masked image modeling (MIM) and CLIP training, respectively. More precisely, we use i) either EVA-02 or EVA-02-CLIP ViT-I. models as teachers, pretrained
22

---

Published in Transactions on Machine Learning Research (05/2024)
Classification on DomainNet
Fine-grained classification
Model
Painting
Sketch
Clipart
CUB
Aircraft
DTD
(1a) Probing
51.9
31.7
56.7
45.4
31.4
53.7
EVA-02 VIT-$
(1b) Finetuning
80.1
76.0
82.3
88.3
84.8
81.1
(8a) Probing
84.1
82.4
87.0
85.1
63.5
83.9
EVA-02 VIT-L
(2b) Finetuning
85.2
83.2
86.7
91.8
90.7
86.8
(2a) Probing
83.9
82.4
86.9
85.2
63.9
83.4
CVAI VIT-L
(2b) Finetuning
85.3
83.4
86.6
91.5
93.5
86.9
Table G: Probind/ /fibiinninneuning of EVA-02 Fang et al., 2023) and EVA-02-CLIP (Sun et al. 2023 pretrained models for classification on DomainNet and fine-grained classification. We report the probing/
probingffinctuingg /finctuning accuracics of EVA-02 ViT-I. EVA-02-CLIP ViT-L used as teachers, and EVA-02 ViT-S used student. Relative distillation gains in Table H are with respect to underlined results in this table.
Classification on DomainNet
Fine-grained classification
Student
Teacher
SD
Painting
Sketch
Clipart
CUB
Aircraft
DTD
EVA-02 VIT-L
(Sa) (56)
81.0 (-0.9)
78.0 (+2.0)
83.7 (++14)
87.3 (-1.0)
78.9 (-6.9)
83.6 (+2.5)
probed
81.4 (+1.3)
78.1 (+2.1)
83.8 (+1.5
87.6 (-0.7)
81.3 (-3.5)
(+2.8)
EVA-02 VIT-L
(Gn)
80.2 (-0.1)
76.6 (40.6)
82.5 (+0.2)
88.4 (-0.1)
85.6 (40.8)
81.6 (40.5)
finctuned
(66)
/
80.5 (-D.4)
77.2 (4-1.2
83.1 (4-0.)
88.6 (-0.3)
86.2 (4-1.4)
82.8 (4117)
EVA-02 VIT-S
EVA-02-CLIP VIT-L
(Ga)
81.0 (-0.9)
78.0 (+2.00
83.9 (+1166
87.4 (-0.9)
79.0 (-5.8)
82.9 (+1.8)
probed
(56)
81.2 (-1.1)
78.2 (42.2)
83.7 (1-1.4
87.4 (-0.9)
81.7 (-3.1)
83.5 (42.4)
EVA-02-CLIP VIT-L
(6a)
79.9 (-0.2)
76.8 (+0.8)
82.6 (+003)
88.5 (-0.2)
85.7 (+0.9)
81.3 (+0.2)
finetuned
(6b)
80.5 (-0.4)
77.0 (+1.0)
82.9 (+0.6)
88.8 (-0.5)
86.4 (+1.6)
82.7 (+1.6)
Table H: Distillation with EVA-02 (Fang et al.; 2023) and EVA-02-CLIP (Sun et al.) 2023
pretrained models for classification on DomainNet and fine-graing grained classification. We report results with and without data augmentation based Stable Diffusion (SD), using EVA-02 VIT-S as student and EVA- 02 VIT-L, EVA-02-CLIP ViT-L as teachers. Relative gains w.r.t. to underlined result from Table G are in parentheses. Bold numbers: within 95% confidence interval of the best score for each task.
datasets composed of 38 million images and 2 billion images respectively, and ii) EVA-02 VIT-S model as student, pretrained on ImageNet-21k.
Results are reported in Tables G and H The line numbering is kept consistent with that of DINOv2 ex- periments. Compared to DINOv2, probing results for ViT-L are stronger on DomainNet but weaker on fine-grained tasks, especially on Aircraft (2a). Probing results for VIT-S are overall very low compared DINOv2's ViT-S (1a), which is certainly due to the fact that EVA-02's VIT-S was pretrained from scratch while DINOv2's ViT-S was distilled from their ViT-g. Distillation with probed ViT-L is detrimental for CUB and Aircraft (5). On the four other tasks, it boosts results and outperforms distillation from finetuned VIT-L (5 us 6). Similarly to the case of Cityscapes with DINOv2 models (Tables [ and 2], poor distillation results with the probed ViT-L for CUB and Aircraft can be explained by fact that the finetuned ViT-S student outperforms the probed ViT-L teacher by a large margin (resp. 3.2% and 21.3% accuracy).
B.4 Ablation for the distillation loss Ldistill
We compare the distillation loss originally proposed by Hinton et al. (0015) (and that we used in all exper- iments in the main paper) to other more recent alternatives from the literature: i) Hard-label distillation, that consists in a cross-entropy loss with the hard label prediction produced by the teacher and is employed by Touvron et al. (2021) for learning their distillation token (here, to be agnostic to the architecture, we simply reuse their loss, not the full distillation protocol), ii) CRD (Tian et al. 2020). that aligns teacher and student representations through a contrastive learning objective, and iii) DKD (Zhao et al., 2022). that decomposes the classical knowledge distillation loss into target-class and non-target-class losses.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Castilll
Mixup
CUB
Aircraft
DTD
Finetuning
87.3
87.8
81.6
Hard-label KD (Touvron et al,
2021
x
88.1
88.3
81.2
CRD
Tian et a..
2020
89.0
88.4
83.3
Distillation
DKD
Zhao et al.
2022
x
88.5
85.7
82.8
89.0
82.6
KD Hinton al.
2015
89.6
Table I: Choice of distillation loss Ldistill- Comparison of the classical distillation loss KD introduced by Hinton et al. (2015), chosen for our study, with i) hard-label distillation, used by [Touvron et al. (2021), it) CRD (Tian et al., 2020), and iii) DKD (Zhao et al.) 2022). We evaluate the version of CRD that uses KD and a temperature T = 0.5, as it gave the best results. Note that the Mixup augmentation is not applied for hard-label distillation and DKD, as these losses are not compatible with Mixup. Results are with ViT-g leacher and ViT-S as student, withhout data augmentation based Stable Diffusion.
Synthetic
Real Images
Synthetic
Real
Synthetic Image
Real
Figure A: Diffusion-based data augmentation. Examples of synthetic images generated using Im-
ageMixer [Pinkney| 2022) as described Section 3.2 mixing two training images from ADE20K Zhou
et al. 2017 (left), Sketch from DomainNet (Peng et al. 2019 (middle) and DTD (Cimpoi et al. 2014
(right).
Those populate the extended dataset Ded used for distillation.
Results for fino-grained classification reported in Table Mnd show that, for the task-specific distillation of DINOv2 pretrained models, the classical knowledge distillation loss introduced by Hinton et al. (2015) performs best.
C Additional visualizations
Visualizations of the synthetic images produced by our augmentation protocol based on stable-diffusion for
ADE2OK (Zhou et al. 2017). DomainNet's Sketch (Peng et al. 2019 and DTD Cimpoi et al. (2014) can be found in Figure A
24


Reviews:
[{"review_id": "EK2VlSQ8vi", "reviewer": "Reviewer_FHF3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. the paper evaluates a big topic regarding how to learn smaller models given a pretrained model, and it covers several topics regarding knowledge distillation, data augmenation, and fine-tuning. \n\n2. the paper offers several practical guidelines that might benefit the community of similiar interests. \n\nWeakness\n\n1. the claim of the paper seems a bit way too big. While the paper indeed evaluates a couple of datasets, from image classification to semantic segmentation, the community have studied knowledge distillation way bigger than this scope. Although the paper can achieve consistent findings in these topics, it's not convincing enough the paper's conclusion can live upto its claim. \n     - tons of backbone choices, tasks choices, and other minor changes of the method might affect the effectiveness of the method. In addition, several parts of the paper is written in a fairly generic way, while the contents are mostly for computer vision studies. \n\n2. the data augmentation with stable diffusion is fairly strong. It is reasonable to believe that data augmentation with stable diffusion itself can benefit strongly to the improvements of the model, for the reason that stable diffusion model has also seen tons of data, probably even more than the teacher model per se.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper offers a systematic overview of techniques that can used to learn a smaller models from a large pretrained models (mostly knowledge distillation, also using fine-tuning and data augmentation). Across their tasks, the paper shows several findings that achieve universal performances for the datasets and settings they have tried. The paper conclusion regarding how to train smaller models in general seems not supported.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I would recommend the author to rephrase this into more like a method paper, focusing particular applications and phrasing the study into a new method, which might be OK as a novel contribution, instead of a systematic study, so that the paper will be free of the issues of overclaiming. \n   - if the author prefer the current wording, the work might need substantially more experiments to live to its claims. \n\n2. the paper will need thorough studies of how much the stable diffussion contributes to the learning, with or without distillation. In particular, answering the question that stable diffusion model likely has seen more data than teacher model, why do we still need the teacher model.", "broader_impact_concerns": "none noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XXkTGvOC6B", "reviewer": "Reviewer_QYsY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Well written, easy to read\n2. Thorough experiments, evaluation on many datasets, appropriate comparisons, and ablation studies\n3. Reasonable gains from the proposed techniques - the data augmentation strategy and distilling a “probe-tuned” teacher instead of a fully fine-tuned one \n\nCons: \n1. The gains compared to a fine-tuning baseline are not huge (for pre-trained ViT-S, they are somewhat larger for the from-scratch models)  \n2. On a related topic, I am really missing some baselines. While the study in the paper can be seen as self-contained and interesting of itself, and it does not necessarily need to beat state of the art, I find that its relevance is difficult to judge without knowing how the reported results compare to some relevant baselines in the field (including SOTA) - and I urge the authors to include some of those. Not proposing specific papers because I am not familiar enough with this sub-field, but I am sure the authors can find relevant baselines to report. \n3. It would be really helpful to report results with multiple types of teachers, not only DINOv2. Are the proposed techniques still relevant to e.g. CLIP-style models? \n\n\nMore minor cons: \n\n4. Some findings are formulated in an overly general way, not justified by the evidence. For instance, “Linear probing generally yields better teachers than finetuning” is only shown on DINOv2 and should not be stated like this (it is somewhat clarified in the text, but still, shouldn’t be stated so generally in the first place) \n5. A few times the term “stable diffusion” is used in the context where, it seems, just any diffusion model is meant - it’s unclear why “stable” has to be there. If referring to just some diffusion model, please just write “diffusion”, and if referring to specifically Stable Diffusion, please write so, with capitalization. \n6. In Figure 2, seems both the augmented and the non-augmented image go into the student, what does that mean? The text seems to suggest that it’s the augmented image in both cases “Note that we feed the same augmented images to both the teacher and the student models” \n7. “Linear probing runs for 20 epochs” → it’s not actually linear, but MLP? \n8. How were the durations (#epochs) of probing and fine-tuning selected? They can play a huge role for the final results", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies “task-specific distillation” - distilling a (typically large) teacher model to a student model on a specific task. The teachers are pre-trained on DINOv2 and the downstream tasks are several classification datasets and several semantic segmentation datasets. The paper studies transfer from teachers of different sizes to students of different sizes in different settings and additionally proposes a data augmentation stratedg based on using a diffusion model for mixing real images. The paper shows that the proposed augmentation strategy is helpful and come to several conclusions about task-specific distillation in the considered setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Compare to relevant baselines and/or state of the art (Con 2)\n2. Very desirably, run at least some experiments with CLIP-style models (Con 3) - do not have to be as thorough as the current DINOv2 ones, but some\n3. Address the \"more minor cons\", mostly those are presentation issues", "broader_impact_concerns": "No specific concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "InyAf9ygVQ", "reviewer": "Reviewer_dfB3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think the paper presents an interesting study of the transfer problem. Although I mention some weaknesses here, I still think it should be interesting to the TMLR’s audience and is technically sound.\n\nStrengths:\n\n- The problem of efficient transfer is very important and solving it has become even more urgent in recent years with the rising popularity of very large models.\n- The paper aims to perform a thorough study of the problem and update best practices for researchers and practitioners in the field. They offer several useful takeaways.\n- The experiments checking the impact of stable diffusion-based image augmentation are a valuable addition to the analysis. The experiments in Table 2 are especially interesting in this regard.\n- To the best of my knowledge, the paper faithfully describes related work.\n\nWeaknesses:\n\n- The presentation of the paper could be improved. It took me some time to learn how to read Table 1 which is the main result of the work.\n    - Additionally, I’m not sure how to read Table 4. I assume that the first row ($L_{train}$ only) should correspond to row 3(b) in Table 1, but it clearly doesn’t.\n- The paper shows a surprising result that a probed teacher is more useful than a finetuned teacher, but it does not try to explain this phenomenon. In my opinion, the paper already does enough to warrant acceptance in TMLR but explaining this behavior would definitely strengthen the paper.\n- The novelty is somewhat limited, but obviously, this is not an important factor in TMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the question of distilling the knowledge of large vision models on a specific task to smaller models. Additionally, the paper studies using stable diffusion image generation methods as a data augmentation-like technique. The authors run experiments using the family of VIT models from DINOv2 and a randomly initialized ResNet. They find that linear probing yields a more useful teacher than fine-tuning. Additionally, the paper finds that self-distillation can work remarkably well and that there is no performance gap when distilling directly from very large to much smaller models. Finally, they perform additional analysis and ablations of their results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the authors to consider improving the presentation of the main results. For example, having the first column in Table 1 say \"Student Arch\" rather than just \"Arch\" would help. Possibly moving some of the info to a different table would help as well -- e.g., self-distillation is an interesting but a bit unintuitive case, and maybe moving all \"no-teacher\" setups would make the case clearer. It's hard for me to provide more precise feedback without actually changing the paper, but I'd like to highlight that working on the presentation would make it easier to parse for wider audiences.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As described above, the reviewers were mostly convinced after the discussion phase that the paper provides interesting claims with sufficient evidence.
Decision: Accept (poster)