Self-Distillation for Further Pre-training of Transformers
Authors: Seanie Lee
Minki Kang
Juho Lee
Sung Ju Hwang
Kenji Kawaguchi
Venue: ICLR 2023 poster (2023)
Keywords: self-distillation
adaptation of pre-trained models
regularization

Abstract: Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks. However, direct fine-tuning of the pre-trained model may be suboptimal if there exist large discrepancies across data domains for pre-training and fine-tuning. To tackle this issue, several previous studies have proposed further pre-training strategies, where we continue to pre-train the model on the target unlabeled dataset before fine-tuning. However, all of them solely focus on language models and we empirically find that a Vision Transformer is vulnerable to overfitting as we continue to pretrain the model on target unlabeled data. In order to tackle this limitation, we propose self-distillation as a regularization for a further pre-training stage. Specifically, we first further pre-train the initial pre-trained model on the target unlabeled data and then consider it as a teacher for self-distillation. Then we take the same initial pre-trained model as a student and enforce its hidden representations to be close to those of the teacher while optimizing the student with a masked auto-encoding objective. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks. Experimentally, we show that our proposed method outperforms all the relevant baselines. Theoretically, we analyze the proposed method with a simplified model to understand how self-distillation for further pre-training can potentially help improve the performance of the downstream tasks.

Published as a conference paper at ICLR 2023
SELF-DISTILLATION FOR
FURTHER PRE-TRAINING OF TRANSFORMERS
Seanie Leelt Minki Kang Juho Lee Sung Ju Hwang' Kenji Kawaguchi 3+
KAIST", AITRICS", National University of Singapore" {1snfamily02, zzxc1133, juholee, sjbwangs2)ekaist ac. jinccmccpppp nus. edu.sq
ABSTRACT
The application of pre-training large transformer models on massive amounts of unlabeled data and fine-tuning them on labeled datasets for diverse downstream tasks has demonstrated remarkable success in various vision and natural language processing tasks. However, the direct fine-tuning approach may result in subopti- mal performance if there exists significant discrepancy between the pre-training and fine-tuning domains. To address this issue, some previous studies have pro- posed further pre-trat mre-training strategies to continue pre- pre-training the model on the tar- get unlabeled dataset before fine-tuning. However, these strategies are limited to language models and may result in overfitting when applied to Vision Transform- ers. To overcome this limitation, we present a novel approach of self-distillation as a regularization method for the further pre-training stage. Our method first fur- ther pre-trains the initial pre-trained model on the target unlabeled data, and then uses as teacher for self-distillation. Then we take the same initial pre-trained model as a student, and enforce its hidden representations to be close to those the teacher while optimizing the student with a masked auto-encoding objec- tive. Our experiments demonstrate the superiority of self-distillation over relevant baselines on various benchmark datasets for image and text classification tasks. Furthermore, we provide theoretical analysis of proposed method using a simplified model to shed light on how self-distillation for further pre- vee-training can potentially enhance the performance of downstream tasks.
INTRODUCTION
Pre-trained transformer models (Devlin et al., 2019; Brown et al., 2020; Liu et al., 2019; He et al., 2022) have been effective on various vision and natural language processing tasks. The pre-trained models learn general representation from large volume of unlabeled data so that they generalize well to various downstream tasks when they are fine-tuned on each task with a labeled dataset. However, in many of real-world applications, it requires considerable amount of effort to adapt the pre-trained model to a specific downstream task domain since there exists a significant distributional discrepancy between data for the pre- pre-training and fine-tuning stage. Moreover, it is difficult to collect a large amount of labeled data for such specific domains, which renders adaptation of the pre-trained model downstream tasks more challenging.
Image Chassifiation on CUB
Several works have proposed to tackle the problem of adapting pre-trained models specific domain. A prevalent approach for adaptation of the pre-trained model is further pre-training where continue update the parameters of the pre-trained unlabeled data
Tost: Acmursay
56
56
model on additionally curated domain-specins with self-supervision (Beltagy et al., 2019; Lee et al., 2020), before fine-tuning it on the target labeled data depicted in Figure 2b. Gururangan et al. (2020) also show that further pre- training only with the target unlabeled data is still effective with- out any extra data. However, most of the existing further pre- training approaches have focused language models, and
52
Seld Distillttios
Parther Pre Pre-traming
10k 20X 30k 40K 50K box 50
Further Prostrsining Stops
Figure 1: Acc. with varying the number of further pre-training steps.
find that the further pre-training
"Corresponding Author
The work was done while the author was intern NUS.

---

Published as a conference paper at ICLR 2023
Labeled Datavel
Unlaseled Datseet
Labeled Dataset
Clacoifir Res
Classifior Rad
Dacader Bebu
Clasotfer Aas
Classifier Aug
Transformer
Transformer
Transformer Fome
Transformer faz
y Aes
Transformer
Te
EMaE
(u) Fine-tuning
(b) Further Pre-training
Unlabeled
Dataset
Extac
Decoder flay.
Transformer
Labeled Dataset
Unlabeled
Classifier Pas
Classifier Aap"
Decoder Bônir
Teacher
Dataset
Transformer
Transformer
Transformer
Student
fa+
for
T0an
Lytt
(c) Self-distillation in Further Pre-training
Figure 2: Concepts. Comparison between methods adapting pre-trained transformers to the target domain. (a) Fine-tuning without any further pre-training. (b) Further pre-training and fine-tuning, (c) Self-distillation in further pre-training and fine-tuning.
strategy not effective for Vision Transformer (ViT) (Dosovitskiy et al., 2021). As shown in Fig- ure 1, ViT is vulnerable to overfitting and does not generalize well to downstream tasks as when we continue to pre-train on the target unlabeled data.
Several regularization methods (Chen et al., 2020a; Gouk et al., 2021; Aghajanyan et al., 2021) have proposed to tackle the overfitting issue large pre-trained models, however, they do not consider the adaptation process such as further pre-training. Instead, they enforce the distance between the final fine-tuned weight and the pre- pre-trained weight to be small to promote the transfer of the knowledge acquired from pre- pre-training to downstream tasks for better generalization. However, these regular- izations hinder the adaptation of pre-trained models to downstream tasks especially when there is a significant distributional shift between the pre-trained data and target data. It eventually results in worse generalization than the simple fine-tuning strategy.
To tackle these limitations, we propose self-distillation a target unlabeled dataset so that we can effectively adapt pre-trained models to the downstream task of various domains with a limited amount of labeled data. For self self-supervision, we focus on masked outo-encoding for pre-training since it does not depend on any data augmentations, compared to other superviedd learning methods (Chen al., 2020b; He al., 2020; Grill et al., 2020; Zbontar et al., 2021; Chen & He, 2021; Caron et al., 2021) which require data augmentations to construct positive pairs for self- supervieed learning objective such as contrastive learning. This especially useful when it is hard to define meaningful data augmentations for target domain.
regularization for further pre-training on
Specifically, we take the pre-trained model with an encoder So.... and a decoder ger. which are pre- trained on massive amount of unlabeled data from general domain, and continue to pre-train it with masked auto-encoding (MAE) (Devlin et al., 2019; He et al., 2022) objective on the target unlabeled data to obtain fa, and Jou After that, we set the encoder fe, as a teacher for self-distillation. Then we take the copy of the pre-trained model (for. Idm.:) ) as a student, and match the representations of the student encoder and those of the teacher encoder while optimizing the student with the MAE on the target unlabeled data. Finally, we fine-tune the self-distilled student fa, on the target labeled data for the downstream task. We illustrate the overview of method in Figure 2c.
To verify the efficacy of our method, we empirically show that it significantly improves the general- ization performance of pre-trained VIT and language model RoBERTA (Liuu et., 2019), and out- performs the relevant baselines on various image and text classification datasets. Moreover, we the- oretically analyze the proposed method with a simplified model understand how self-distillation for further pre-training can potentially help improve the generalization performance on the target tasks after fine-tuning.
Our contribution is threefold:
We propose self self-distillation for further pre-training on the target unlabeled dataset, where we enforce representations of the student to be close to those of the further pre-trained teacher while training the student with masked-auto encoding objective.
We theoretically analyze the proposed method with simplified model to understand how delf-distillation for further pre- pre-training can potentially lead to better generalization perfor- mance of downstream tasks.
2

---

Published as a conference paper at ICLR 2023
We extensively validate our method on various image and text classification datasets with pre-trained transformers and show that ours outperforms the relevant baselines.
2 RELATED WORK
Self-Distillation Knowledge distillation is to transfer knowledge of teacher to student by minimiz- ing a divergence between output of teacher and student (Hinton et al., 2014). When the parameteri- zation of student and teacher is identical, we call it self-distillation as a special case of the knowledge distillation. Although there is new information during self-distillation process, Furlanello et al. (2018) have shown that the student from self-distillation achieves better generalization performance than the teacher. A similar phenomenon has been consistently observed in other works (Yang al., 2019; Ahn et al., 2019). Several works propose self-distillation without a pre rreitrained teacher net- work (Sun al., 2019; Zhang al., 2019; 2022). They add auxiliarly classifiers to intermediate layers and train the classifiers to minimize divergence between the output of the classifier of the last layer and that of the auxiliary classifiers. Mobahi et al. (2020) theoretically analyze how self- distillation induces regularization and reduces overfitting in Hilbert space. However, all of them focus on self-distillation for supervised learning. Instcad, we empirically and theoretically show that self-distillation for further pre- pre-training with self-supervision leads to better generalization of downstream tasks after fine-tuning the self-distilled model with target labeled data.
Further Pre-training Lee et al. (2020); Beltagy et al. (2019); Sun al. (2020) have shown the success of continual pre-training language model on a large number of corpora collected from target domain and fine-tuning the model on target labeled dataset. However, it is computationally expen- sive to further pre-train the model on a large amount of unlabeled text data and it may not be feasible to collect such large scale of unlabeled data on certain domains. Instead, Gururangan et al. (2020) devise a task-adaptive pre-training where we use only target unlabeled data for further pre-training language model before fine-tuning the model on the target labeled data. To improve the effective- ness of further pre-training, Kang et al. (2020); Ye et al. (2021) propose learning to mask input for masked auto-encoding with bilevel optimization, which requires a prohibitive computational cost. However, all of them solely focus pre-trained language models and we empirically find that naive further pre- pre-training is not effective for Vision Transformers.
Regularization for Fine-tuning There are several works proposing regularization for fine-tuning a pre-trained model. Chen et al. (2020a) propose to modify Adam (Kingma & Ba, 2015) optimizer, called RecAdam, which enforces the fine-tuned model close to the initial pre-trained model by minimizing Le distance between fine-tuned and initial pre-trained weight. Similarly, Gouk et al. (2021) project the fine-tuned weight for every gradient descent update such that it lies within the sphere centered on the initial pre-trained weights with the distance induced by the norm of maximum absolute row sums (MARS). Instead of explicitly minimizing the distance, motivated by trust region theory, Aghajanyan et al. (2021) propose to minimize symmetric KL-divergence between the model output of an original input and that of the input perturbed by Gaussian noise. However, all of them do not consider adaptation of pre-trained models to a specific target domain, which results in worse generalization performance of downstream tasks than a simple fine-tuning strategy.
3 METHOD
3.1 PRELIMINARIES
Problem Statement We assume that we are given (finit, dinit) parameters of the neural network Jo ferss which is pre-trained on large volume of unlabeled data with masked auto-encoding objective, where fa... is an encoder which extracts hidden representation of an input and Ja.. is an decoder reconstructing a masked input. Our goal is fine-tune the pre-trained model fom. with a randomly initialized task specific head hn labeled dataset D ((x(), ())) of downstream classification task such that the model generalizes well to unseen test dataset Drest A typical approach to achieve this goal is empirical risk minimization as follows:
w; DUFE) via algorithm A as minimize Lee (0,w; Biw
(1)
(0" ,,=) A(Lre) Ditr)
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 Self-Distillation
Algorithm 2 Further Pretrain
Require: Unlabeled dataset D", initialization Oinic Dinits
Require: Unlabeled dataset D", initialization
learning rate a R20, round of self-distillation I" E
fiast; Pisit, learning rate E R20, mask- ing probability 7 (0, 1). and batch size B.
N4. masking probability 7 (0, 1) and batch size B.
1: Og <- Further PriieerrrPetrain(1 (D",, Oisit, Quas,,
2: for all <- 1..... T" do
Initialize 0g t Dinit and @o f- Pinit 2: while not converge do
3: Initialize 8, - Ginit and dz + Qinit
4: while not converge do
3: Sample mini-batch (x(²) ) from D
Sample a batch (x(1))
د from D
6: for all j + L...., B do
for all j + 1, .,..., do
7:
Sample mask z(1) (z)
5: Sample mask z(5) Pz.
8:
2 2k=1= =(()
6:
z Σ 2=== =k ((5)
9:
Get masked input x(1) with 25))
7:
Get a masked input x00 with 2
10:
() - - EK スा log po.. ( 1x)))) (
20)
8:
Pk t Poo. (a* (z)))
(9)
11:
G for (x - Stoporad (x())
9:
e t kk=1 203 log Pk
10:
end for
12:
end for
11:
+ B ΣØ=1 Lj
13:
14:
0g 0. +cc++t2
L1 + B E
2/==
Cy L2 + En
0=0e
12:
13:
0o + 0o
Φ t 60 a 0p 1040
a 30 0=00
ac
2C
15:
di Φi 061 1=
14:
end while
16:
end while
15:
return Pg
17:
Bu + 02
18:
end for
16:
19:
return 0y
17:
where Cce is cross-entropy loss and A denotes stochastic gradient descent algorithm to minimize fCE on the dataset D with the initialization 0init.
Further Pre-training However, the pre-trained model is prone to overfitting when it is fine-tuned on a small amount of doomain-specine labeled data. Gururangan al. (2020) have shown that further pre-training, where we continue to pre-train the model Je.s o fe on the target unlabeled dataset D" = { Fim1 and then fine-tune it D is effective for improving generalization performance when there is not enough domain-speciff labeled data. Note that D" is the exactly same as Dt except that we remove the labels y(²) In this work, we focus on the masked auto-encoding (Devlin et al., 2019; He et al., 2022) pre-training objective function since its generality compared to other supervupe methods (Chen et al., 2020b; He et al., 2020; Grill et al., 2020; He et al., 2020; Chen & He, 2021; Caron al., 2021) which require well-defined data augmentations construct positive pairs for self-supervised learning.
Masked Auto-Encoding We briefly describe the masked auto-encoding objective (Liu et al., 2019; He et al., 2022) for a language model such as RoBERTA (Liu et al., 2019) and Vision Transformer
(ViT) (Dosovitskiy et al., 2021). Let x()) (x11) ...... (n(()) be sequence of patches for image or tokens for a sentence with length K. Then we independently sample a binary mask from Bernoulli
then (f) is replaced with special "mask" token. Otherwise, we use the same Tx for masked input. 4k
distribution with probability y for each 2 denoted ( (21")
()) If 2
(8)
Let (0) = (+(1) (10) be masked input and let fo, 9o be an encoder and decoder, respectively. Then the final objective for masked auto-encoding is defined as follows:
K
K
(0, D²)
E (x) (x)
log Po, (z (x (E(0)) 2 =
(2)
O
1=1
where xy, (z) denotes Binomial distribution with its parameters for probability that = 1 and K for the number of trials. Note that the negative log-likelihood instantiated as cross-entropy loss for language models or mean square error for Vision Transformers. See Appendix C. for more detail.
3.2 SFLF-DISTILLATION FOR FURTHER PRE-TRAINING
Although further pre-training strategy has been effective on text domain (Gururangan et al., 2020; Lee al., 2020; Sun al., 2020), we empirically find that ViT with further pre-training overfits
4

---

Published as a conference paper at ICLR 2023
the target unlabeled data and does not generalize well to downstream image classification tasks. In order to tackle the issue, we propose self-distillation as a regularization for further pre-training. Specifically, given pre-trained model Iø.ni= fo..... we first continue train the model on the target unlaheled data D with the masked auto-encoding objective as descrihed in equation 2 to obtain the encoder fog and decoder Jdo- We discard the decoder and consider the encoder fo, teacher for self-distillation. Then we take the copy of the pre-trained initial network Por.s fe.... as student and further pre-train the student with masked auto-encoding objective but enforce hidden representation of the encoder of the student fo.... to be close to that of the teacher fo, as follows:
(01, E arg min (Craz (0, o; D + Coistal1 (0;00 D*))
8,4
(3)
Σ fo(x(0)) StopGrad (Fo, (x((
Coistil1 (0; 00: D²)
where and & are initialized with the pre-trained parameters fimi= and Oinst. respectively and StopGrad denotes the stop-gradient operation which does not back-propagate through the in- put. As described
Algorithm I, we can repeat this process to perform multiple rounds of self- distillation (T' > 1) where the student of the previous round becomes teacher and new student is initialized with the pre-trained weights finit and dinit for the next round. We empirically find that the first round of self-distillation plays the most significant role in improving the final gener- alization performance of downstream tasks. Furthermore, theoretical analysis shows that the first round of self-distillation has the largest impact regularization. Thus, we perform single round of self-distillation for computational efficiency. After self-distillation, we discard the decoder gø, and fine-tune the encoder of the student Se, along with randomly initialized task-specific head ha by minimizing Lez(0, w.; ("t") with the initialization 0 as described in equation 1.
4 THEORETICAL ANALYSIS
In this section, we analyze how seff-distillation affects the final model after fine-tuning in terms of generalization and regularization. This section proves a generalization bound on the supervised loss for our method and shows that the generalization bound strictly decreases as the number of self- distillation increases. Moreover, between the initial weight before further pre-training and the final weight after fine-tuning. The regularization effect is shown have the largest impact in the first round of self-distilaaiin, which suggests that the first round of self-distillation plays more significant role in the final performance when compared to the other rounds.
show that self-distillation acts regularizer the distance
We consider the dynamics of the weight vector wt, over time r of fine-tuning after rounds of self- distillation, where Wo,0 the result of further pre- pre-training, and W²,0 E minimize Le L((w) ( is the result of the self-distillation of & rounds with Lr(w)= J E [ w) - f(,, w;-1,0) + X||w|] ll2 for some X > 0. After rounds of self-distillation, we consider the dynamics over fine-tuning time 7 via gradient flow (Saxe et al., 2014; Kawaguchi, 2021): dway -V((wrr) with the initialization w1, obtained by the self-distillation where [(w)= 1 - f(w, Nı, 31) with f(w, x, v) [ w) u1l2 and y E IRP Here, the seff-distillation and fine-tuning share same training dataset ((E,, y4) H=) In this section, to obtain theoretical insights, we consider the regime of d > and a simple abstract model, (xr w) W(( IR", with some nonlinear map y and the weight matrix W E RRxd where w = vee [ de and (o(x) E Rdd Here, vee(W" is vectorization of the matrix WT. Let us fix the fine-tuning time length as 1 < 7 S T oo. Since d > n, there are infinitely many solutions to the problem of minimizing L(w). Thus, each of the finite length T and the peeeeearaettiiiiionnnn d > n implies that the initialization Wro0 at the fine-tuning phase via self-distillation plays an important role.
n
dr
Let > and E No. We then define F, = (Ar(s) : E S}. where S is a set of all training datasets of size n such that with probability at least 1 - 8. the training dataset is in S. For each training dataset s E S, (sss) = Wer is the final weight vector of the model after t rounds of self- distillation and T time of fine-tuning. Let us define the matrix & E Rdxn by D, (((;)), We assume that 4 is of full rank; i..., rank( () n since d 2 n. This is typically satisfied because if rank( () < n, there is some redundancy in the rows of the matrix Q. Denote by [I, @ & e dppxxp the Kronecker product of the identity matrix I, XRx and the matrix $. We write its singular
5

---

Published as a conference paper at ICLR 2023
value decomposition by 1 @ Φ] UEVT where U [U1. u2 ..... Udpl d contains the left-singular vectors u, E Re for i {l,...,dj and Σ E Rdxxpp is rectangular diagonal matrix with Σn = oi E R>0 for E {L..... .ll.....p).p.................... and σ1 2 Ø2 2 an upper bound norm of the features as Eg [x I R. We assume that Wo,0 + 0: wo,o = O, then the target function in the self-distillation phase is always zero as f( Wo,n) for all i, which is unlikely the case in practice. We define Winit E dp be the weight before further pre-training and define Y = vec[lyn, Vn ]=] E RRP.
2 Onp 2 0. Define M to be the loss l(w, T. V) M. Define R to be upper bound on the expected
The following theorem shows that the generalization bound the supervised loss E(wer r. 2. y) of the fine-tuning phase strictly decreases as we increase the number of self-distillation rounds in the further pre-training phase:
Theorem 1. There exists a constant e (that only depends on M) such that with probability at least 1 1-6 6, the following holds:
4c RRP
In(2/8) 2n
Ee 8)1 -
(Wr. M.) ) + 5(t))
M
(4)
where the function ((t) is strictly decreasing in E No-
The proofs of all results in this section are presented in Appendix A. Moreover, the following theo- rem shows that the tight upper bound on the distance between the initial weight Winit and the final weight w4,r after T steps of fine-tuning (i.e., || Wr,T (2) strictly decreases as the number of seff-distillation rounds increases:
Theorem 2. There is a function d/ No - Rzo such that (1) || - wrrrll2 = (b()) for some Winit E dRd,, (2) W[iii wo,rllz I (t) for all Winit E Rd,, (3) the function (t)) is strictly decreas-
ing in t E No, (4) the function (b(1) can be decomposed to (b(() /G, + 1/1 (t) + 1 [tt = 0).B G2
with constants G1,Gr G2 2 0 in where (t) is strictly decreasing in E No and
dp
np 1 (u, "0,o)²² 2 O.
Theorem 2 shows that the self-distillation acts regularizer on the distance between the initial weight Wnit and the final weight Wr,r. Since the Rademacher complexity of set of vectors is invariant to shift by constant vector, this distance has been shown to control the generalization bound in previous papers in various models and settings, including deep neural networks (Bartlett & Mendelson, 2002; Bartlett al., 2017; Nagarajan & Kolter, 2019). This suggests that self-distillation helps generalization via a regularization effect on the distance. Moreover, the first round of self- distillation is expected to have the largest impact based on Theorem 2 since Theorem 2 shows that we can completely remove the unnecessary component B of woo0 in the first round of ceff-distillation We have verified these theoretical predictions in the experiments where we show the correlation between the improvement via self-distillation and the distance that appeared in the generalization bound the previous paper (Nagarajan Kolter, 2019).
5 EXPERIMENT
Dataset For image classification problem, we use six datasets - FGVC Aircraft (Aircraft) (Maji et al,,, 2013), Caltech UCSD Birds 200 (CUB) (Wah et al., 2011), Chest X-ray (Kermany et al., 2018), Describable Textures Dataset (DTD) (Cimpoi et al., 2014), Stanford Dogs (Khosla et al., 2011) 1), and Oxford 102 Flower (Nilsback & Zisserman, 2008). For text classification problem, we use four datasets - Chemprot (Kringelum er al., 2016), ACL-ARC (Jurgens et al., 2018), SCIERC (Luan al., 2018), and Twitter-Emotion (Mohammad et al., 2018). Please see Appendix D for more detail.
Implementation Detail For the image classification problem, we use Vision Transformer pre- trained on unlabeled ImageNet dataset with masked auto-encoding (He et al., 2022) and fine-tune it on the downstream task with AdamW optimizer (Loshchilov & Hutter, 2019) for 10,000 steps with batch size 32. Regarding further pre- mre-trannig and self-distillation, we continue to pre-train the model for 20,000 steps with batch size 64. We evaluate the Vision Transformers with accuracy. For text classification. following the experimental setup from Gururangan et al. (2020), we use RoBERTA (Liu et al., 2019) backbone network and fine-tune it on the target labeled dataset

---

Published as a conference paper at ICLR 2023
Table 1: Average and standard deviation of accuracy with runs for image classification datasets.
Method
Aireraft
CUB
Chest X-ray
DTD
Dogs
Flower
Fine-tuning
72.33 + 1.13
55.55 +: 0.54
77.15 +: 0.52
67.56 +: 0.52
62.53 +: 0.57
88.78 +: 0.65
RocAdam
70.76 + 1.25
55.22
+ 1.29
77.29 + 1.32
67.59
1.03
61.65 + 0.92
88.97 : 0.44
MARS
72.74 0.57
55.35 0.73
77.28 + 1.80
66.79 0.90
62.24 0.96
87.93 + 1.21
RAF
72.95 + 0.46
55.91
+ 0.79
76.86 + 0.97
65.32 + 1.25
62.15 + 0.48
88.92 + 0.78
Further Pre-training
73.38 1: 0.64
53.72 1: 0.46
77.79
1: 2.06
63.55 1: 1.12
62.34 1: 0.39
88.63 1: 0.35
Self-Distilaatin
74.37 + 0.43
58.06 + 0.90
79.68 + 1.05
68.51 + 0.51
63.55 + 0.39
90.28 + 0.44
Table 2: Average and standard deviation of Fl score with 5 runs for text classification datasets.
Method
SCTERC
ACL.ARC
Chemprot
Twitter-mmotion
Fine-tuning
76,63 + 2.06
64.09 + 4.13
80.59 +: 1.15
77.61 + 0.83
RecAdam
79.45 + 1.92
59.70 2.68
82.73 + 0.28
78.26 + 0.88
MARS
74.69 + 1.25
53.57 = 6.64
0
77.48 + 1.69
R3F
75.61 + 2.49
60.13 + 2.55
79.25 2.16
77.79 + 0.81
Further Pre-training
80.32 1: 1.25
69.73 + 2.40
82.33 1 0.46
78.71 + 0.40
Self-Distillation
81.79 + 0.75
73.17 + 2.19
82.87 + 0.35
79.77 + 0.79
with AdamW optimizer for 10 epochs with batch size 32. In terms of further pre-training and self- distillation, we further pre-train RoBERTA for 100 epochs with batch size 128. We evaluate the models with macro F1 for SCIERC, ACL-ARC, and Twitter-Emotion dataset, and micro F1 for Chemprot dataset.
Baselines We compare our method against the following baselines targeting for fine-tuning pre- trained models. All the models are initialized with the pre- trainedee weights finit and @init.
1. Fine-tuning: The model fine-tuned on target labeled dataset D without any further pre-training or regularization except dropout and weight decay.
2. RecAdam (Chen et al., 2020a): The model trained with RecAdam optimizer which is a variant of Adam optimizer (Kingma Ba, 2015) and additionally penalizes L2 distance between the fine-tuned and the initial pre-trained weight.
3. MARS (Gouk al., 2021): The model trained to minimize the cross-entropy loss along with the regularization projecting the fine-tuned weight to lie within a sphere centered on the initial pre- trained weights. For each layer, the distance induced by Maximum Absolute Row Sum (MARS) matrix norm (ıaxx, Σ; W 1.1 U) is used for the regularization.
R3F (Aghajanyan et al., 2021): The model trained to minimize the cross-entropy loss as well as symmetric Kl.-divergence between softmax output of the original input and that of the input perturbed by Gaussian noise.
5. Further Pre-training (Gururangan et al., 2020): Task adaptive pre-training where we further pre-train the model on the unlabeled target dataset DV with masked auto-encoding objective and fine-tune it on the target labeled dataset D
This is our model which is further pre-trained on unlabeled target dataset D with equation 3 and fine-tuned on the target labeled dataset D
5.1 MAIN RESULTS
As shown in Table I, self-distillation consistently outperforms all the regularization methods and the further pre-training method on image datasets. Notably, our method significantly improves the per- formance of the Chest X-ray dataset consisting of grey-scaled images for diagnosis of pneumonia. In addition, self-distillation effectively tackles the Flower dataset which contains only 2,040 labeled examples. In contrast, the other baselines do not show consistent improvement across all the image datasets. For instance, further pre-training is effective for the Aireraft dataset, but significantly de- grades the test accuracy the DTD dataset. Regularization methods such as RecAdam, MARS, and R3F barely improve generalization performance on most datasets or underperform the simple fine-tuning strategy certain datasets. This empirical evidence supports that the regularizations enforcing the fine-tuned models close to the initial pre- pre-trained weight are not effective for adapting pre-trrined model to the target datasets of specific domains.
Furthermore, as shown in Table 2, we provide additional experimental results for text classification tasks. Again, self-distillation significantly outperforms all of the baselines across all four datasets,
7

---

Published as a conference paper at ICLR 2023
53,04
Table 3: Ablation on CUB and SCIERC.
50.3
Modd
CUB
SCIERC
Full Model
S8.06 -: 0.90
81.79 a 0.75
58.5
weo CusE
57.60 4: 0.81
50.06 4: 0.62
wio fecsesal
0.46
1.25
Parker Pootinn
Further Pre-tranxx2
0.75
0.99
Prodiccio-MMatching
58.27 4: 1.07
81.09
4: 1.07
Weight Matching (f:)
53.70
1: 0.91
80.54
4: 1.21
of reetsane
MatthiiM (MARS)
54.82
1 0.60
80,05 1. 0.71
Disitatii (0)
Figure 3: Accuracy with varying the number of training data.
Figure 4: Test accuracy with varying self-distillation round.
except RecAdam in the Chemprot dataset. In contrast to the previous experiment, the further pre- training method improves the test F1 score of the simple fine-tuning method, yet it still underper- forms our model. For regularization methods - RecAdam, MARS, and R3F, they do not achieve consistent improvement across all three datasets. RecAdam moderately improves the FI score on the SCIERC and Chemprot dataset but significantly degrades the generalization performance ARC dataset. Both MARS and R3F show poor performance on SCIERC and ACL-ARC datasets, and their performance slightly is worse than Fine-tuning method on the Chemprot dataset.
ACL-
Result for Low Resource Data We further perform experiments to show how self-distillation effectively handles low resources of labeled data. Given full CIFAR-100 dataset (Krizhevsky al., 2009) which contains 50,000 training pairs of an image and corresponding label, we plot the test accuracy of each model by varying the number of training instances. Note that we also reduce the number of unlabeled images used for further pre-training or delf-distillation. As shown in Figure 3, self-distillation consistently improves the generalization performance of both fine-tuning method and the model which is further pre-trained on the images from the CIFAR- 100 dataset. Notably, the gain by self-distillation becomes larger when the models trained with extremely small number of instances. For example, self-distillation achieves 13% and 6% improvement of test accuracy compared to the model with simple fine-tuning when there are 1,000 and 2,500 labeled examples, respectively. These empirical results verify that self-distillation can effectively adapt the pre-trained model the target dataset even if there are extremely small amounts of labeled data.
Ablation Study We perform ablation study to verify the effectiveness of each component of self-distillation. In Table 3, we show empirical results on both the CUB dataset and SCI- ERC data set while removing or replacing various components of self-distillation Firstly, we remove masked auto-encoding objective CyaE and train the model with only distillation loss Cristill before fine-tuning. On image dataset CUB, it does not make a significant differ- ence, however, removing the masked auto-encoding objective degrades the generalization per- formance of the language model on text classification dataset SCIERC. Alternatively, we re- move the distillation loss Lnistri1 in equation 3, which results in further pre-training method. Furthermore, we continue to pre-train the model for twice longer steps as the original further pre-training method, denoted as Further Pre-traimx2. to show that higher test accuracy of self- distillation is not a consequence of longer pre-training. Both of the models significantly un- derperform self-distillation, which shows the effectiveness of the self-distillation loss. Lastly, we perform experiments for variants of distillation loss Lpiseill in equation 3. Instead of matching representation of teacher and student, we enforce the reconstruction of masked in- puts by teacher and student to be consistent, i.c., minimizeg,d llgs o fo(x) 9d. o feo (x) for VIT or minimizog, Σt= DKL (po,.on (2, (x)((x) for RoBERTA, denoted as Prediction- Matching. Furthermore, we replace the distillation loss with the one minimizing L2 or MARS distance between the parameters of student and teacher, denoted Weight-Matching. As shown in Table 3, all these variants are not effective compared to the one minimizing the distance between hidden representations of the student and teacher.
Multi-Round of Self-Distillation Lastly, we empirically show that the first round of self- distillation plays the most significant role in improving generalization performance. Specifically, we fine-tune each model after t round of self-distillation and plot the test accuracy on Oxford 102 Flower dataset, where 0 round of self-distillation (t 0) denotes the model with further pre-training. As shown in Figure 4, the first round of self distillation significantly improves the test accuracy of the model with further pre- pre-training and the gain by self-distillation becomes marginal after the first round. Considering the extra computational cost and marginal improvement of multi-round self- distillation, we perform single round of self-distillation for all the experiments.
8

---

Published as a conference paper at ICLR 2023
150
Gesseralauttnn Gag
L. distanne
1200
1000
100
Pirtaer Pre train
Pirther Pre tralo
Xingy Froer
CDn
Round Set Dacillatt
(a)
(c)
Figure 5: (a) Generalization gap: Difference between supervised test loss and training loss. (b) Effect of self-distillation on distance: Distance between the initial pre- pre-trained weights and the final fine-tuned weights of further pre-training and self-distillation. (c) Effeet of multi-round self distillation: Distance between the initial pre-trained weights and the final fine-tuned weights for each round of self-distillation & e N+.
5.2 FURTHER ANALYSIS
In this subsection, present numerical experiments analyze why aif-distillation can potentially help improve the generalization performance of downstream tasks compared to further pre-training and empirically show that Theorem and 2 can be extended to deep neural networks transformers.
(a) Generalization gap: In Figure 5a, we plot the generalization gap, which is test loss minus train- ing loss on each labeled dataset, of self-distillation and further pre-training method. Self-distillation improves the generalization gap of the further pre-training method across all the datasets. It is con- sistent with Theorem showing that self distiillaatiin with a simplified model strictly decreases the generalization bound on the supervised loss of the fine-tuning stage.
(b) Effect of celf-distillation on distance: To empirically validate Theorem 2 about regularization effects by self-distillation on L2 distance between the initial pre-trained weight finit and the final weight after fine-tuning, we plot the distance obtained from self-distillation and further pre-training. Specifically, we compare the distance ||iinst 01.r ll and ||0ini: 0,.rl1, where 0G.- is the pa- rameter after round of self-distillation and r steps of gradient descent for fine-tuning. As shown in Figure 5b, self-distillation consistently decreases the distance and the reduced distance correlates with the better generalization gap in Figure 5a. These empirical results confirm the connection be- tween the L2 distance from the initialization and generalization bound (Nagarajan & Kolter, 2019).
(c) Effect of multi-round self-distillation: Lastly, we empirically verify part of Theorem 2 which shows that the first round of self-distillation plays the most critical role of regularization L2 distance between the initial pre-trained weight Ainst and the final weight 0tr denoted as the parameter after t round of self-distillation and T steps of gradient descent for fine-tuning on VGG flower 102 dataset. As shown in Figure 5c, self-distillation significantly decreases the distance at the first round (t = 1) and the regularization effect on the distance diminishes afterward, where 0 round of self oef-dditillation (t = 0) denotes the model with further pre-training but without self delf-distillation.
the
6 CONCLUSION
To effectively adapt pre-trained transformers to a target domain, we proposed self-distillation regularization for further pre-training. Specifically, we first took the initial pre-trained transformer and continued pre-train it with the masked auto-encoding objective on the target unlabeled dataset and considered the encoder part of the model as a teacher for celf-distillation. Then we took the student and enforced representations of the student
copy of the same initial pre-trained model to be close to those of the teacher while optimizing the student with the masked auto-encoding objective on the target unlabeled dataset. Finally, we fine-tuned the self-distilled student on the target labeled dataset. Our empirical evaluation on various image and text classification benchmark datasets showed that self-distillation consistently improved generalization performance compared to relevant baselines. Lastly, provided the theoretical analysis of the proposed method with a simplified model to understand how delf-distillation for further pre-training can potentially help improve the generalization performance of the downstream tasks.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We use Pytorch (Paszke et al., 2019) and transformers library (Wolf et al., 2020) from Huggingface implement all the baselines and our proposed method in the experiments. We have described our method of self-distillation for further pre-training in Algorithm and specified all the experimental setup including hyperparameters in Section and Appendix For theoretical analysis, have provided all the proofs in Appendix A.
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government(MSIT) (No.2019.0-00075, Artificial In- telligence Graduate School Program(KAIST), the Engineering Research Center Program through the National Research Foundation Korea (NRF) funded by the Korean Government MSIT (NRF- 2018RIAS 1059921), Institute of Information & communications Technology Planning & Evalua- tion (ITPP) grant funded by the Korea government( MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. 2022-0-00184 Development and Study of AI Technologies Inexpensively Conform Evolving Policy on Ethics), Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea goveramen((MSIT) 116.2022--00071333 KAIST-NAVER Hypercreative AI Center, and Samsung
Electronics (I0201214-08145-00)),
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. International Conference on Learning Representations, 2021.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9163-9171, 2019.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Peter I. Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normaaizze margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 3615-3620, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 7870- 2020a..
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
10

---

Published as a conference paper at ICLR 2023
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-157555 2021.
M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HIT,, 4171- 4186, 2019.
41771
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616. PMLR, 2018.
Henry Gouk, Timothy Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. In International Conference on Learning Representations, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271 2020.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342 8360, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 9729-977 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NIPS 2014 Deep Learning Workshop, 2014.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Com- putational Linguistics, 6:391-406, 2018
Minki Kang, Moonsu Han, and Sung Ju Hwang. Neural mask generator: Learning to generate adaptive word maskings for language model adaptation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 6102-6120, 2020.
Kenji Kawaguchi. On the theory of implicit deep learning: Global convergence with implicit layers. In International Conference on Learning Representations, 2021.
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diag- noses and treatable diseases by image-based deep learning. Cell, 172(5): 122-1111,, 2018.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine Grained Visual Categorization, IEEE Conference Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.

---

Published as a conference paper at ICLR 2023
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-188888 18673, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Jens Kringelum, Sonny Kim Kjacrulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. Chemprot-3.0: global chemical biology diseases mapping. Datahase, 2016, 2016.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. arXiv preprint, 2009.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, (1144):22441222) 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692,, 2019.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3219-3232, 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained
visual classification of aircraft. arXiv preprint arXiv: 2013.
Andreas Maurer. A vector-contraationn incquality for rademacher complexities. In International
Conference on Algorithmic Learning Theory, pp. 3-17. Springer, 2016.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-33611 2020.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval- 2018 task I: Affect in tweets. Proceedings of the 12th international workshop on semantic evaluation, pp. 1-17, 2018.
Vaishnavh Nagarajan and Zico Kolter. Generalization in deep networks: The role of distance from initialization. arXiv preprint arXiv:1901.0167. 2019.
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv:2111.10055: 2021.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014.
Dawei Sun, Anbang Yao, Aojun Zhou, and Hao Zhao. Deeply -upervsuee knowledge synergy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 6997-7006, 2019.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre- pre-training framework for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8968-8975 2020.
12

---

Published as a conference paper at ICLR 2023
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-UCSD 20ds--00-2201, Tech- nical report, California Institute of Technology, 2011.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing:
system demonstrations, pp. 38-45. 2020.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L Yuille. Training deep neural networks in gen- erations: A more tolerant teacher educates better students. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5628-5635, 2019.
Qinyuan Ye, Belinda Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, and Madian Khabsa. On the influence of masking policies in intermediate pre-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7190-7202, 2021.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Self-supervised
Linfeng Zhang, Jiebo Song. Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3713-3722, 2019.
Linfeng Zhang, Xin Chen, Junbo Zhang, Runpei Dong, and Kaisheng Ma. Contrastive deep super- vision. In European Conference on Computer Vision, pp. 1-19. Springer, 2022.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A PROOFS
We also define the model output vector ft E Ruxp by (ftliz f(z wi)j. For example, fo the initial teacher label matrix. Let [n] [l....,m} nn. Denote the rank of 4 + by
ran @ 1) S np. Define = [1i, ...,, ur] E Rdpxr and P, = 1 - ÜÜT, which is the projection matrix onto the null space of ÜT. We first prove the following lemma, which will be used the proofs of Theorem and Theorem later:
Lemma 1. For any t No. Wt,0 Ei=1 as + L(t Olu, where as, 1. (14(25/075) ) R. ùg D.u, E Rdd, Di (VT vec[fol) E R. and P,
Proof of Lemma 1. Define wi wto for N+. The necessary condition the solution w,
step \ is that V ((w)) = 0. Thus, by solving V((w;) = O for wt, we have that wi (1,, & 1 @
T) + (nni)-1(1 @ Φ) vec[St-1] By using the singular value decomposition [T & + = UEVT,
since UTT U = I and V V I, we have that
w/ = (UEETU T + = vec|fi-1] = (U(EET + -NNIJUT)-UVV vec[fi-1]
-U(ET + + vec[fi-i]
Therefore, U(EN' + --AA)-"EVV vec[fi-1]. Using this and [4 @ Φ = UEVT,
vec[ft] = vec[$ T W. 4p] L4p & Φ] wi = 14 2 T U + +nA) - Ty vec[ft-1]
=V' + + vec[ft-1]-
Therefore, vec[ft] = VAVT vec[fi- where A ΣΣ (EΣT + and) -.. Repeating this process for ve since V I,
vec[ft] = VAV T VAV VAV vec[fo] = VA'VT vec
Plugging this equation of vec[fi-1] = VA'-iVT vecifo] into the equation of wa U(ET + nd7) VVV voe[ft-1], we have that
Wi U(-ET n VV V [vc = -UAA-VV vec
where
-EET + = Here, we can rewrite the matrix B Rddxxp
B
=
(dp-np)
where 0 (ddp-np) Xnp the (dp mp) by np matrix with all entries being zero, and B E pupxup is
diagonal matrix defined by
Bü = ( + =A
Using this B in the above equation w, UBA vec[fol.
B
W/ U
T vec[fol
(dp np)
BA--1 ((dp-np)xmp
[w]
W2
udpl
V vec[fo]
UBA voc[fol
where 0 [Uz, Ug ... Uppl E dppxpp Since the matrix ΣT SEEEET- n' E npxpp is diagonal matrix with its entry being Au (o nA) this can be further simplified
14

---

Published as a conference paper at ICLR 2023
wy = -BA vec[fo]
of aitRA
of
t-1
1/1
aitn
1/2
U
:
o
op +nx
10m
( + (o -
02 + + - + nA) -
[u]
U2
Unp
Onp ( up + nn) -1 (o ( nd) 1 1 Dnp
ap
Σ o.(0; + nA) (o + nA) 1 1/14
Σ oo + nA) = (0? - i:
where the last line follows from the fact that (o? (o?
0 for all i>>>> r.
Since (o + (of(o nd)-1)(-1 implies that
(o? + md)= =
or - ([(ttax)
for S this
W/ = Σ ==1 0, of - nX
D.u. Σ
I+ (n1/o3) y:u:
i=1
Since & E N. was arbitrary, this holds for any / N,. This proves the first statement of the theorem for any / N4. For 0, since
JA = (V vec = (V T [L, & 1)
wo,o); (VTVVVT wo,o):
= Twn.o))
UT wo.o); o.u, wp.0.
we have that
Σ == oi 1 (nd/o7) )
) h:u = Σ uuuf
w0,0 = ŪŪT Wo,o-
Thus,
Wo,0 :Ü Wo.0 + (I-U0) )wo, = ΣΣ
(1+
(1+
1
U.u, + (I Ūū )wooo-
1=1
Since (I 11-00 --ÜTT)Wo = w,oo,, this completes the first statement of the theorem for any E No-
A.1 PROOF OF THEOREM 1
Proof. Define Z = 14 8 T E gnxd where ń = np and d = dp. Then,
((w) [[( vill2 1 _ Y12
where Y veo[[u....... [T] R Since V((Wer)) = 2T (ZWo,, Y).
dwi,r
- (Zwe, YY
dr
Since rank( = and d 2 n, we have (ank(Z) = πi by the property of the Kronecker product with the identity matrix. Since rank(Z) there exists I& such that Y = Zv. Thus, dwa,r dr
-% (Zwer - Zu)
-T 7(WE. - v)
--ZT Z(we, v)-
15

---

Published as a conference paper at ICLR 2023
Since Z" UEVT, have z=
UUE -E==1 oiu,u Thus,
dwa.r Louue (wer 0) Σ of (Wi,r - v).
Since the columns U forms the basis of Rd and w, v E Rd, we can write Wi,r = Ek=z ck ( Uk
and Σd Ek=1 Uh for some (ltt) and gk. Thus,
dwt.r dr
o uu
Ck 91 -gk)uk Ykk
((,T)
k=1
2 (1,7)
CCk
que )usu
k=1
(c (1.7)
a.)u.
Using Wt,r = Ek=1
Ck ug for the right-hand side too, we have that
(1,T)
d
(1T)) u, Σ of (t)r) u.
dr
This implies that for all i {L...., nJ.
d (1,7) = -o (1,7) -q::
dr c
and (t.77) 0 for all & {l...., n). This can be also seen by the fact that dwisz -ZT di
di Zv) with _ = UEVT and thus the dynamics only adds components of u; for E {l..... : nJ. and not for i {l...., n... Thus, for components of u; for i & {l...., Ri. the initial values stays. In other words, for $ {1. u..
(t,)) (c.0)
c c
On the other hand, for E fl,..., m.. since dr 91 = 0,
dr (c - -i)= -9:)
(tT)
(t,T)
(c (1.7) - -i)).
dr
Solving this for (c(1.) 4:). we have that for i E (1...., n)
(1,7) -: = [c, ee ofr
(t,0)
This implies that
(1TT) (tT) = + (c((
qi)e
oft 1:)e-of
9: - oofr + +
(1,0)
Combining these with W4,T Ei=r Ck
(1TT) (1.T) uk.
d W u4
d
(1.
Σ
q - -ofT )u: +
(1.0) - otT u +
Σ
(1.0)
u.
(5)
(=++1
Therefore, for any particular E S, since U = =|u1, U2 ....... ...,, Udpl E dpd is an orthogonal matrix,
[A(() 11 < Σ (i((1 (9 (a(1-e-07T) +
(c
(2 207T +
Σ (q
(1.0133
(6)
1=1
1=
where (i o, and (1.0) all depend c:
16

---

Published as a conference paper at ICLR 2023
By using Lemma 4 of (Pham al., 2021) and taking union bound with P(ss S) I 8, with
probability at least 1 - 6, we have that wr, E Fi and the following holds:
1n(2//)
Eg Ee 2, v)1 -
( Tii U4) + 2R, (Fi) + My
2n
where (Fi) Ea,e (SuP 1 E Ei,..., En are independent uniform random variables taking values in {-1,1}. .. By using Corollary 4 of (Maurer, 2016), there exits constant (only depending on M) such that,
- ((r;, v1)) 1=1
ve[[WT] and
(F) < - -B,s sup wEFi i=1 k=1
Ea (x)((
- E.,e(sun sup n
W&
(xp(r;)
WEFi km
C B. sup w [h]
n
wEFi
where We is the k-th row of W, Ei are independent uniform random variables taking values in {-1,1}, h = vec[Hf] E Rdd,, and E Rd with Hfs = E.= Exp((z))) Thus,
RR) S - E. sup || c(supmee, [1wl12) E. [1//]]2] wEFi
Here,
B.g [11b111]]] = BR,E
Σ Eusp(es)))
5
E
Σ Sap(aa))
k=
(v((T:),)²
(8)
Ø=
km
1=1
Eg Σ
(x(x::))²
k=
j=1
Es 11x((z))11
1=1
Equation 8 holds since
Ey,e (()n)(Z))))) (Eu=((=))))] = E, [1 = 1)(())))((m))))
for all i, E [m).
Thus,
Rn(Fi) 5 cRyplsupweet we (|wl12)
Vn
Define
(s(s) (q - (((-)))?TT) +
c
(1.0) 1100)e 207T +
Σ
(cf (2000)y2
1=
where 9:, o, and (t.0) all depend on s. With this, we define
((t) = sup (s).
S
17

---

Published as a conference paper at ICLR 2023
Then, by combining equation 6, equation 7, and equation 9, with probability at least following holds:
8, the
4c RCD
(n
E En.l x, v)1 S T
E (we, Vi, vi, + {(t))
M
n
2n
Finally, from Lemma 1, for any E No and E {l....,
2
(c )2 = 1 (nN/of)
91
Since [+ < (because nd/o, > 0), the value of [+(nA/of) strictly decreases as t
j² 0 and 0? 2 0, this implies that (c. (2
increases. Since o?
strictly decreasing in E Ng unless
(1.0) Moreover, from Lemma 1, have that
Wr,0
ar + 11- -
Woo.
Since (u1,..., Ug is a orthonormal basis for Rd with inner product (xr, y) = y x, we get
Wo,0 Σ (u: wo.0)u.
Since ÜÜT w0,0 Ei=1 (2," wb,))u, Uu, have that
d Σ
(I (--Ū Jwo,o
(u wo,o)4 (u, wo,o)u.
(u. wo,o)u,
i=
which implies that the u, component to span wno for i E (n + 1....,d d) is only present in (T ÜÜT FUTT)aoo In other words,
d
w1o a +
Σ
1[t=0((u wo,o)u.
1
1==-1
Thus, for any t E No and i E {n + 1, 1..... d), we have that
(c 22 = 1ft = (u wo.o)²
These implies that ((1) is strictly decreasing in / E No unless wp.o = 0.
A.2 PROOF OF THEOREM 2
Proof. In this proof, we continue to use the results and the notation from the proof of Theorem By using equation in the proof of Theorem 1, we have that
||iit
Wrrrll2 Wmit 2ell2
where
n (: -0, )u, Σ
1, Σ
(t.0) o U1
Σ
(1,0)
c
u4-
imn+1
If Winit
Oit, for some 0, then
Wintt
ue 2
||u, + av: 112
= (1 + a) |
||va 112 + |aaell2
d Σ
( c- c-ofTT2 +
Σ
(c
2 207T +
(c.
(1.00) +
|| 1
1=
18

---

round of the self-distillation and Theorem holds Published as a conference paper at ICLR 2023
On the other hand, for any Wini Rd,,
Winit
0el12 _ 110e ll + ||Wmitll2
8 - - c0o---
d Σ
Σ
(((440))--2001
+
(c)
(2
+ W| 12
i=++1
Thus, setting (t) to be the following function satisfies conditions (1) and (2) in the statement:
(t) (1- ell-e-ofr
(t.0) (1)0))2ee
20,T 2
Σ
| l2
i=++1
Finally, from Lemma 1, for any E No and i E ,.....;m...
(c (2 1 1 (nA/o?) (1+
2
which is strictly decreasing in E No unless (t,0) 0 for all {l,.... n. as shown in the proof of Theorem 1. Moreover, from Lemma 1, for any E No and E {n + I.....d).
(c(:0))2 = 1ft (u wo.o)
That is,
(bt) |
Gr + +1)+++++++
Σ 1(t ()? wo G2,
*=
where
G =
9i (1 - -ofTy²
vi(1)= =
Σ
1
207T
1 o
0,
1=1
and
G2 Winitlla
Since -0 > a constant in f and we have previously shown that ( (IF(nd/os)) )
Ú:
strictly decreasing in No unless Wo,o 0. It implies that both 2 (t) and (t) strictly decreasing in L E No unless wo.o = O.
Remark 1. Note that Theorem also shows the distance between the weight of the teacher W4-1.1
and the initial pre-trained weight Winis for all t E N+. Since the teacher at t' E N4 round of self-distillation used to student the for all non-negative integer E No, the distance between the initial weight and the teacher weight I| w,-1,7ll strictly decreases for all N_. For instance, at 1, we obtain the following inequality
| - Wo.r S (b)))
where Wa,r is the weight of the initial teacher without self- self-distillation.
B ADDITIONAL EXPERIMENTS
In this section, we perform additional experiments to better analyze the proposed method, self- distillation for further pre-training
19

---

Published as a conference paper at ICLR 2023
Table 4: Training and test loss.
Method
Split
Aircralt
CUB
Chest-Xray
DTD
Dogs
Flower
Further Pre-training
Train (log)
-9.13 0.09
-8.58 = 0.05
-8.10 1.90
-3.70 0.37
-5.41
-9.70 0.32
Test
0.03
2.44 0.02
2.20
0.22
2:49 0.08
1.96 +
0.63 +: 0.04
Self DittDiitillationn
Train (log)
Test
-9.17 =: 0.08
0.01
-8.16 0.31
2.26
-6.08 1: 0.50
0.17
-4.08 =: 0.30
2.15
-5.50 2: 0.06
1,90 = 0.01
-7.98 =: 1.28
0.50 +:
Supervised Contrastive Learning For the main exper- iments Section 5, we use the same objective function for pre-training and further pre-training. One may won- der what happens if we pre-training stage. In this experiment, we continue to pre- train the encoder of the pre- pre-trained transformer (He et al., 2022) with supervised contrastive loss (Khosla et al., 2020) while fine-tuning randomly initialized linear classifier with cross-entropy loss. As shown in Table 5, supervised contrastive learning (SupCon) significantly degrades the generalization perfor- mance of Vision Transformer on both X-ray and Flower datasets. Based on this experimental result, we conjecture that the transformer pre-trained with a masked auto-encoding objective might not be compatible with the contrastive loss and thus we may use the same objective for pre- vee-traiinng and further pre-traine pre-training.
Table 5: Comparison with supervised con- trastive learning.
Model
X-ray
Flower
different objective at further
SupCon
75.30 1: 1.33
76.70 a: 1.76
Fine-tuning
77.15 + 0.52
88.78 = 0.65
Farther Pre-training
77.79 2.00
88.63 0.35
Self. delf-Distillation
79.68 + 1.08
79.68 + 0.44
Generalization Gap As shown in Figure 5a, self-distillation decreases generalization gap com- pared to further pre-training. Additionally, we separately report training and test loss Table 4. Both of further pre-training and self-distillation reach near zero training loss, but self-distillation achieves way lower test loss than further pre-training as consequence regularization induced by self-distillation.
Table 6: Down weight- ing MAF objective.
Down Weighting of Masked Auto-Encoding Although we fix both weight of self-distillation and masked auto-encoding objective I in equa- tion 3, we vary A E (0, 1], the weight of masked auto-encoding objective (ACrs de D") + Fuistill(( (0; fg D))) and report test accuracy of Vi- sion Transformer with self seff-distillation on the CUB dataset. As shown in Table 6, our proposed method is insensitive to the value of and thus there is no benefit to fine-tuning the weight of masked auto-encoding objective.
CUB
1.0
58.06 + 0.90
0.5
57.76 + 0.17
58.21
0.42
0.1
57.76 = 0.33
Extra Training Time To better analyze extra computational cost for further pre-training and self-distillation, we report training wall clock time for fine-tuning, further pre-training, and self- distillation, respectively. We train a Vision Transformer (Dosovitskiy et al., 2021) on CUB dataset with 3090 RTX GPU and Intel(R) Xcon(R) Silver 4210R CPU. It takes 32 minutes and 18 sec- onds for fine-tuning the transformer. We need an extra 1 hour 29 minutes 33 seconds for further pre-training and hours 13 minutes 22 seconds for self-distillation.
C MASKED AUTO-ENCODING
In this section, we describe the masked auto-encoding objective from equation 2 in more detail. Given a sequence x (Zi,...,,3g) with length K, we sample mask a Binomial distribution Py,K with probability for success 7 E (0. 1) and the number of trials K. For each 2g. we replace with the special token "mask" if 2 same Ty. for an masked input. Let & (æ1,..., .xk) be a masked input and let fo. be en- coder and decoder, respectively. We want to compute the ligliiill of the reconstructed input ΣK= 25 log (x)
(ZI,..., from
Otherwise we use the
For language models, reconstruction of the masked input dt is predicting which token is masked out of pre-defined vocabulary with its size V, where each token is represented as an integer from (1....,V V). Thus the conditional probability of as (1...., V) given & is parameterized as
20

---

Published as a conference paper at ICLR 2023
follows:
exp( Moxz ))
(0,6(2x(3))
f=1 exp(u1)
where (u.....,,uy uv) = I (h:) E RY,
h1
hx fo( E B
For ViT, the sequence x consists of image patches and the reconstruction of the masked input is pre- dicting pixel values for each masked patches, which is a regression problem. Thus, we parameterize the conditional probability of patch Th (2R la ................................ , E R given follows:
1
(kk.s pk.i)
(0,0(xx(0)
II
V2TT2
exp
20²
where μı = (Hk.1 pk.m) E IR"
pk
fo(x)) E R=
Since a > and are constants with respect to f and Ø,
K
K
arg min
log Po. ( = arg min
20
(Tk - mk.j)² mlog
V2n02
0.0
1=1
0,0
k=1
j=1
K
arg min 0,0
(Ikj - um.s)
k=1
K
arg min
|
14x: 12.
0,0
k=1
D DATASET
We describe statistics of all the image and classification datasets used for our experiments in Table 7 and 8.
Table 7: The number of training instances and classes for each image classification dataset.
Aircraft
CUB
Chest X-ray
DTD
Dogs
Flower
instances
6,667
5,594
5,216 2
4,230
12,000
2,040
# of classes
100
200
47
120
102
Table The number of training instances and classes for each text classification dataset.
SCIERC
ACT-ARC
Chemprot
Emotter-Emmtion
of instances
3,219 7
1,688
4,169
4,230
of classes
6
13
47
21

---

Published as a conference paper at ICLR 2023
E HYPERPARAMETERS
In Table 9, we summarize all the hyperparameters for Vision Transformer and RoBERTA.
Table 9: Hyperparameters for Vision Transformer and RoBERTA
Hyperparameters
Vision Transformer
RoBERTA
learning rate for pre-training learning rate for fine-tuning weight decay coefficient batch-size for pre-training batch-size for fine-tuning learning rate scheduler fine-tuning steps pre-training steps round of self-distillation
1.5 10
1-10-
1 1.10-*
2 10-6
1002
1 10
64
128
32 linear-decay
16
lincar-decay epochs 100 epochs
10,000
20,000
22


Reviews:
[{"review_id": "nurdmvEdjP", "reviewer": "Reviewer_5qJ3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the usage of pretrained vision transformers on downstream tasks. The authors point out the weakness of the current further pretraining strategy about overfitting. To solve that problem, they propose a new pipeline which includes a self-distillation process to distill the knowledge from a further-pretrained model to an initial model.", "summary_of_the_review": "As mentioned above, this paper is overall good in my opinion. Although the novelty is to some extent limited by applying commonly used technique, the authors present insightful analysis and comprehensive experiments.", "strength_and_weaknesses": "Strength:\n1. The proposed method is simple yet effective.\n2. The authors provide comprehensive theoretical analysis which clearly shows the role of the proposed algorithm.\n\nWeaknesses:\n1. I wonder if it is safe to say the improvement of self-distillation is solely from solving the overfitting problem. In fact besides the original self-distillation paper which adopts this method for better generalization, many other works also use it for better performance without considering overfitting. The authors present the gap between training and testing loss in the experiments, which is great. It would be better if the authors can show that with similar training loss/training accuracy the model train with the proposed method has lower testing loss.\n2. This experiments in this paper only focuses on MAE-based model. I think the authors may have some discussion about the effect of the proposed method on other types of self-supervised algorithms, for example, the contrastive learning based methods like SimCLR and DINO.\n3. Since the proposed method requires two steps of training, I am not sure if this method is efficient enough.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eAw2jNecSQ", "reviewer": "Reviewer_7okZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use self-distillation as a regularization method to improve transfer learning performance on image classification tasks. The paper first observes that further fine-tuning on a pre-trained model on unlabelled target dataset would result overfitting. And then it proposes the self-distillation method to resolve this issue. The general idea of self-distillation method is to first further pre-train a teacher model on target unlabelled data with a MAE objective, and then guide a student model from the original pre-trained model for the same representation learned from the teacher model and additionally trained with the same MAE objective. And finally to fine-tune the student model with the labelled target data. The paper provides theorical analysis on the generalization bound for the proposed method and showed an improved performance over a range of fine-grained image classification benchmarks. ", "summary_of_the_review": "The paper overall is very good with theoretical analysis and very comprehensive experiment support. There are some minor issues listed in the limitation section and I would consider raising my score if the authors could clarify these issues.", "strength_and_weaknesses": "Strength.\n+ I found the paper is very well written with a clear structure and motivation.\n+ The design of self-distillation method is very clean and general. I can imagine it can be useful beyond image classification tasks.\n+ The paper also provides theoretical analysis showing self-distillation acts as a regulariser on the distance between the initial pre-trained weights and final fine-tuned weights.\n+ Ablative analysis is very comprehensive showing different design strategies within the self-distillation method.\n\nLimitation.\n- Further pre-training induces overfitting? The paper has highlighted multiple times that further pre-training would hurt transfer learning performance and overfits on the target dataset. However, from the Table 1 and 2, the further pre-training strategy actually performs typically the second right after the proposed method, which is a bit contractor to what the paper has highlighted, and it is always better than direct fine-tuning? I am wondering is this because this further pre-training is well tuned or some other reasons? A bit more information on this baseline would be important.\n- A fairer comparison to the baseline. From Figure 2, we can see that the proposed self-distillation method requires 3 times longer than fine-tuning and 1.5 times longer than the further pre-training method.  Comparing with further pre-training might be okay since the further pre-training on the teacher model did not account to be included in the final performance evaluation. However, we may argue that it could be possible that the improved performance was due to longer training time. I would suggest having one small additional experiment showing that the self-distillation method works was due to the design rather than a longer training time.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SheO-zjP3x", "reviewer": "Reviewer_PVpp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript studies the problem of unsupervised fine-tuning on a small target dataset before the actual supervised fine-tuning on it. The insight is that even unsupervised fine-tuning (MAE here) on a small dataset causes biased representation that hurts generalization. So the proposed recipe is to use another unsupervised fine-tuned network as a feature distillation regularization term. The authors also provide a theoretical justification using a single-layer network. Evaluations on a set of vision and language tasks show the effectiveness. The method is easy to implemented and codes are provided.", "summary_of_the_review": "A good empirical study to me.", "strength_and_weaknesses": "Strengths:\n+ This is a somewhat interesting but understandable technique as the distillation loss can function as an ensemble regularization as it is also trained with a MAE loss. This kind of things usually work but knowing that it works for an MAE loss is new to me.\n+ The method is well-benchmarked and shows consistent improvements on a set of fine-grained classification tasks (and some language tasks that are out of my scope).\n\nWeaknesses:\n- I believe the method works well and I understand that using MAE is preferable due to its simplicity. But I still wonder how it compares with SupCon [A] in the direct fine-tuning setting.\n- Missing references to a line of highly-related self-distillation methods that works on a single network [B][C][D].\n\n[A] Supervised Contrastive Learning, NeurIPS 2020.\n[B] Deeply-supervised knowledge synergy, CVPR 2019\n[C] Be your own teacher: Improve the performance of convolutional neural networks via self distillation, ICCV 2019\n[D] Contrastive Deep Supervision, ECCV 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HpoaH3rJKr", "reviewer": "Reviewer_RiKW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-distillation approach to transferring a pre-trained model to a new task.  The method first applies further pre-training to arrive at a set of teacher weights that is then used to guide the subsequent student training starting from original pre-trained weights.  The authors show self-distillation with further pre-training to be an effective method for regularizing further pre-training for downstream tasks through a variety of empirical studies on image and text classification tasks.  Finally, theoretical analysis shows how self-distillation acts as a form of distance-based regularization on the initial pre-trained weights. ", "summary_of_the_review": "The authors introduce a simple yet effective way to transfer a pre-trained model to a new downstream task via self-distillation with further pre-training.  The experimental results are strong and show self-distillation to be state-of-the-art across multiple text and vision classification tasks.  Ablations also show self-distillation to be a critical component of these empirical results. Aside from a couple of questions and some concern regarding limited novelty, I would recommend the paper to be accepted due to its relevance and high potential for impact in practical workloads adapting a pre-trained model to a new task.\n", "strength_and_weaknesses": "Pros:\n- The empirical results for self-distillation relative to other forms of regularized fine-tuning is state-of-the art on both text and image classification tasks.\n- The authors conduct a series of empirical studies and ablation studies to verify the regularization benefits of self-distillation.\n- The writing is clear and well organized.\n\nCons:\n- The approach is limited in novelty but I still think the proposed method is important.\n- Figure 1 shows that even with self-distillation, the model regresses to no benefit from further pre-training after 60k steps.\n\nQuestions:\n- From the text, it appears that self-distillation does not have a tunable parameter to trade off between MAE and Distill loss. Is there any benefit to tuning the weight for the two terms in the training objective? \n- The theoretical analysis shows bounded distance between self-distilled weights and initial weights but does not comment on distance to teacher weights.  Is there a breakdown you can provide for that bound?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8V_mIfltlh", "reviewer": "Reviewer_yUym", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for finetuning transformers on new datasets. By an additional self-distillation process between the pre-training and finetuning stages, the authors show that transformers can achieve better generalization on downstream datasets. The authors provide a theoretical analysis of the proposed method with a simplified model. Experiments on ten image and text classification datasets show the efficacy of the proposed method.\n", "summary_of_the_review": "I like the clear and novel approach. Although the performance boost is not huge, I think the paper provides enough useful insights and analysis to be accepted.\n", "strength_and_weaknesses": "Strength\n1. The writing is clear and the proposed method seems straightforward and easy to implement.\n2. The authors have provided a theoretical analysis of the effect of the self-distillation process.\n3. Experiments are thorough and cover both image classification and text classification with transformers. The experiments are repeated multiple runs to test for statistical significance.\n\nWeaknesses\n1. The results, although significantly different, are not better by a large margin compared to fine-tuning, especially for image classification (1-2 points different). It would be great if the authors also provide an analysis of how much additional computational cost there is compared to the finetuning and further pre-training method, which could be a useful reference for people that want to use the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster