Improved Regularization and Robustness for Fine-tuning in Neural Networks
Authors: Dongyue Li
Hongyang R. Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: fine-tuning
transfer learning
robustness

Abstract: A widely used algorithm for transfer learning is fine-tuning, where a pre-trained model is fine-tuned on a target task with a small amount of labeled data. When the capacity of the pre-trained model is much larger than the size of the target data set, fine-tuning is prone to overfitting and "memorizing" the training labels. Hence, an important question is to regularize fine-tuning and ensure its robustness to noise. To address this question, we begin by analyzing the generalization properties of fine-tuning. We present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the fine-tuned model. We empirically measure these quantities. Based on the analysis, we propose regularized self-labeling---the interpolation between regularization and self-labeling methods, including (i) layer-wise regularization to constrain the distance traveled in each layer; (ii) self label-correction and label-reweighting to correct mislabeled data points (that the model is confident) and reweight less confident data points. We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for a few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average in two noisy settings.

Improved Regularization and Robustness for Fine-tuning in Neural Networks
Dongyue Li
Hongyang Zhang Northeastern University wwaag@@orrceasterm..cmmm edu
Northeastemn University www.ggu@nnrthhsseern.ccmmm
Abstract
A widely used algorithm for transfer learning is fine-tuning, where a pre-trained model is fine-tuned on a target task with small amount of labeled data. When the capacity of the pre-trained model is much larger than the size of the target data set, fine-tuning is prone to overfitting and "memorizing" the training labels. Hence, important question to regularize fine-tuning and ensure its robustness to noise. To address this question,
begin by analyzing the generalization properties of fine-tuning. We present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the fine-tuned model. We empirically measure these quantities. Based on the analysis, we propose regularized self-labeling-ther interpolation between regularization and self-labeling methods, including (i) layer-wise regularization to constrain the distance traveled in each layer; (ii) self label-correction and label-reweightinn to correct mislabeled data points (that the model is confident) and reweight less confident data points. We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average two noisy settings.
Introduction
Learning from limited labeled data is a fundamental problem in many real- world applications (Ratner al., 2016, 2017). common approach to address this problem is fine-tuning large model that has been pre-trained on publicly available labeled data (He et al., 2019). Since fine-tuning is typically applied to a target task with limited labels, this algorithm prone to overfitting or "memorization" issues (Tan et al., 2018 8). These issues worsen when the target task contains noisy labels (Zhang et al., 2016). this paper, we analyze regularization methods for fine-tuning from both thcoretical and empirical perspectives. Based the analysis, propose a regularized self-labeling approach that improves the generalization and robustness properties of fine-tuning.
Previous works (Li et al., 201 ba have proposed regularization methods to constrain the distance between fine-tuned model and the pre-trate pre-trained model in the Euclidean norm. et al. (2020) provides extensive study to show that the performance of fine-tuning and the benefit of adding regularization depend on the hyperparameter choices. Salman et al. (2020) empirically find that performing adversarial training during the pre-training phase helps learn pre-trained models that transfer better to downstream tasks. The work Gouk (2021) generalizes the above ideas to various norm choices and finds that projected gradient descent methods perform well for implementing distance-based regularization. Additionally, they derive generalization bounds for fine-tuning using Rademacher complexity. These works focus settings where there is no label noise in the target data set. When
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

label noise is present, for example, due to applying weak supervision techniques (Ratner et al., 2016), an important question is to design methods that are robust such noise. The problem of learning from noisy labels has a rich history of study in supervised learning (Natarajan et al., 2013). In contrast, little is known in the transfer learning setting. These considerations motivate us to analyze the generalization and robustness properties of fine-tuning.
In Section 4.1, we begin by conducting PAC-Bayesian analysis of regularized fine-tuning. This is inspired by recent works that have found PAC-Bayesian analysis correlates with empirical perfor- mance better than Rademacher complexity (Jiang et al., 2020). We identify two critical measures for analyzing the generalization performance of fine-tuning The first measure is the lg norm of the distance between the pre-trained model (initialization) and the fine-tuned model. The second measure is the perturbed loss of the fine-tuned model, i.e. its loss after the model weights get perturbed by random noise. First, we observe that the fine-tuned weights remain closed to the pre-trained model. Moreover, the top layers travel much further away from the pre-trained model than the bottom layers. Second, we find that fine-tuning from a pre-trained model implies better noise stability than training from randomly initialized model. In Section 4.2, evaluate regularized fine-tuning for target tasks with noisy labels. We find that fine-tuning is prone to "memorizing the noisy labels", and regularization helps alleviate such memorization behavior. Moreover, we observe that the neural network has not yet overfitted to the noisy labels during the early phase of fine-tuning. Thus, its prediction could be used to relabel the noisy labels.
We propose an algorithm that incorporates layer- wise regularization and self-labeling for im- proved regularization and robustness based on our results. Figure 1 illustrates the two compo- nents. First, we encode layer-wise distance con-
0.6
1900
Faae-tiiin::
Fls ttaing Tor.
Number of data points
Prediction accuracy
Regulasiao adl labelis; Trsining
Regulaziol set lsbelig, Test
0.4
Comoeled training lohels
1800
straints to regularize the model weights at dif- ferent levels. Compared to (vanilla) fine-tuning, our algorithm reduces the gap between the train- ing and test accuracy, thus alleviating overfitting. Second, we add salf-labeling mechanism that corrects and reweights "noisy labels" based on the neural network's ' predictions. Figure I shows that our algorithm effectively hinders the model from learning the incorrect labels by relabeling them to correct ones.
0.2
0.0
10
20
Number of epochs
Figure 1: Red: Layer-wise regularization closes generalization gap. Magenta: lebfllabeling rela- bels noisy data points to their correct label.
In Section 5, we evaluate our proposed algorithm for both transfer learning and few-shot classification tasks with image and text data sets. First, using ResNet- (He et al., 2016) as the pre-trained model, our algorithm outperforms previous fine-tuning methods on seven image classification tasks by 1.76% on average and 3.56% when their labels are noisy. Second, we find qualitatively similar results for applying our approach medical image classification tasks (ChestX-ray 14 (Wang et al., 2017: Rajpurkar et al., 2017)) and vision transformers (Dosovitskiy et al., 2020). Finally, we extend our approach to few-shot learning and sentence classification. For these related but different tasks and data modalities, we find an improvement of 0.75% and 0.46% over previous methods, respectively.
In summary, our contributions are threefold. First, we provide PAC-Bayesian analysis of regularized fine-tuning. Our result implies empirical measures that explain the generalization performance of regularized fine-tuning. Second, we present a regularized self-labeling approach to enhance the generalization and robustness properties of fine-tuning. Third, we validate our approach on an extensive collection of classification tasks and pre-trained model architectures.
2
Related work
Fine-tuning is widely used in multi-task and transfer learning, meta-learning, and few-shot learning. Previous works (Li et al., 2018b,a) find that injecting & regularization helps improve the performance of fine-tuning. Li et al. (2018b) propose a E2 distance regularization method that penalizes the & distance between the fine-tuned weights and the pre-trained weights. Li et al. (2018a) penalize the distance between the feature maps as opposed to the layer weights. Chen et al. (2019) design regularization method that suppresses the spectral components (of the feature maps) with small

---

singular values to avoid negative transfer. Gouk et al. (2021) instead encode distance constraints in constrained minimization and use projected gradient descent ensure the weights are close the pre-trained model. Salman et al. (2020) show that fine-tuning from adversarially robust pre-trained models outperforms fine-tuning from (standard) pre-trained models.
The robustness of learning algorithms in the presence of label noise has been extensively studied in supervised learning (Natarajan et al., 2013). Three broad ideas for designing robust algorithms include defining novel losses, identifying noisy labels, and regularization methods. Zhang and Sabuncu (2018) 8)
design the robust Generalized Cross Entropy (GCE) loss which is a mixture of Cross Entropy (CE)
and mean absolute enror. The Symmetric Cross Entropy (SCE) (Wang et al., 2019) loss combines reverse cross- coss-entropy with the CE loss. Ma al. (2020) proposes APL, which normalizes the loss to be robust to noisy labels and combines active and passive loss functions. Thulasidasan et al. (2019) propose DAC, which identifies and suppresses the signals of noisy samples by abstention- training. Liu et al. (2020) introduce ELR, an early learning regularization approach to mitigate label "memorization". Huang et al. (2020) propose self-adaptive training method that corrects noisy labels and reweights training data to suppress erroneous signals. These works primarily concern the supervised learning setting. To the best of our knowledge, fine-tuning algorithms under label noise are relatively under-explored in transfer learning. Our approach draws inspiration from the semi-supervised learning literature, which has evaluated pseudo-labeling and self teainiraaning approaches given limited amount of labeled data, and large amount of unlabeled data (Zou et al., 2019; Xu et al., 2021: Tai et al., 2021). A very recent work considered a sharpness-awwnn aware approach and evaluated its performance for fine-tuning from noisy labels (Foret al., 2021). It would be interesting to see if combining their approach with our ideas could lead to better results.
From theoretical perspective, the seminal work of Ben-David et al. (2010) considers a setting where labeled data from many source tasks and a target task available. They show that minimizing weighted combination of the source and target empirical risks leads to the best result. In the supervised setting, Arora et al. (2019) provide data-dependent generalization bounds for neural networks. Nagarajan and Kolter (2018) and Wei and Ma (2019a,b) provide improved generalization bounds that depend only polynomially on the depth of the neural network characterized by a margin condition. Recent work has found that the PAC-Bayes theory provides generalization measures that correlate with empirical generalization performance better than other alternatives (Jiang et al., 2020). We defer more extensive review of PAC-Bayesian generalization theory to Section A.
3 Preliminaries
Problem setup. We begin by formally introducing the setup. Suppose we would like to solve a target task. We have a training data set of size (1) Let (x(²) "/""),,, (x...,(x(d ((0) (2,on) be the feature vectors and the labels of the training data set. We assume that every x; lies in a d-dimensional space denoted by X C Rdd Following standard terminologies in statistical learning, we assume all data drawn from some unknown distribution supported on X' x V, where y IR. is the label space. Denote the underlying data distribution as p(()
For our result in Section 4.1, we consider feedforward neural networks, and our results can also be extended to convolutional neural networks (e.g.. using ideas from Long and Sedghi (2020)). Consider an L layer neural network. For each layer from to L, let d/; be the activation function and let W; be the weight matrix at layer i. Given an input to the i-th layer, the output is then denoted as (1(=) = (W,,), Thus, the final output of the network is given by
fw) = ΦL &LL1 -.. (z), for any input E X.
where we use W [W1.... ...., Wt] to include all the parameters of the network for ease of notation. The prediction error of fw is measured by a loss function E(-) that is both convex and 1-Lipschitz.
c (fw) (an))>pt))
E
(x),
(I)
Suppose have access to pre-trained source model W7(+) Using vi((e) as an initialization, we then fine-tune the layers W+(s) W-(*) to predict the target labels.
Applications. We consider seven image classification data sets described in Table 1. We use ResNets prentrained on ImageNet as the initialization (+(+) (Russakovsky et al., 2015; He et al., 2016). We
3

---

Table 1: Basic statistics for seven image classification tasks.
Datasets
Training
Validation
Test
Classes
Airerafts (Maji et al., 2013)
3334
3333
3333
100
200-2011 (Wah al., 2011)
5395
599
5794
200
Caltech-256 (Griffin et al., 2007)
7680
5120
5120
256
Stanford-Car (Krause al., 2013)
7330
814
8441
196
Stanford-Doge (Khosla et al., 2011)
10800
1200
8580
120
Flowers (Nilsback and Zisserman, 2008)
1020
1020
6149
102
MIT-Indoor (Sharif Razavian al., 2014)
4824
536
1340
67
perform fine-tuning using the pre-trained network the above data sets. See Section 5.1 for further description of the training procedure.
Modeling label noise. In many settings, fine-tuning is applied target task whose labels may contain noise; for example, if the target labels are created using weak supervision techniques (Ratner al., 2019; Saab et al., 2021). To capture such settings,
denote noisy data set as (a] 11 (1)( )... ...... (1 2000 ( ), where v/( is noisy version of y; (1) We consider two types of label
noise: independent noise and correlated noise. We say that the label noise is independent if it is independent the input feature vector
Pr(O,)
klyl") -., Pr(O kly, J) 7j.k
for some fixed 7j.k between 0 and 1. On the other hand, say that the label noise is correlated if depends on the input feature vector (i.e. the above equation does not hold).
Our proposed approaches
Given the problem setup described above, next, study the generalization and robustness properties of regularized fine-tuning methods. First, we present PAC-Bayes generalization bound for regular- ized fine-tuning. This result motivates us evaluate two empirical measures: the fine-tuned distance in each layer and the perturbed loss of the fine-tuned model. Second, we consider fine-tur tuning from noisy labels. We show that layer-wise regularization prevents the model from memorizing the noisy labels. We then suggest injecting predictions of the model during training. similar to self-training and pseudo-labeling. Finally, incorporate both components into regularized self-labeling approach, blending the strengths of both to improve the generalization performance and the robustness of fine-tuning.
4.1 Fine-tuning and regularization
We consider the following regularized fine-tuning problem, which constrains the network from traveling too far from the pre-trained initialization ((s) (Li et al., 201 Ba,b; Gouk et al., 2021).
W f arg min (fw)
(2)
s.t. I|; ( S Dr, =i= L....,L. L.
(3)
Above, D, is hyperparameter that constrains how far the i-th layer W. can travel from the pre-traine traited initialization W7(()) Previous works have observed that stronger regularization reduces the generaliza- tion gap during fine-tuning (cf. Figure 2). Next, we analyze the generalization error of W. that is. ((t) (09) (f where (f(-) the test loss of für according to equation (1) and (fw) the empirical loss of f on the training data set.
Prodiction accuracy
0.9
Finslaiing Training
Finsstiinng Teat
Regnlariee fin-ttiiing Training,
Regnlarined tine siming Tost
0.5
Number of epochs
Figure 2: The training and test accuracy of fine- tuning using early stopping vs. optimizing (2) and (3) on the Indoor data set.
-ayysayn analysis. We begin by analyzing the generalization error of f using PAC- AC-Bayesian

---

Ratio to pre-trainey model (82
CUB-200-2011
0.6
Stanford: Dogs
Random
Pre-trained
Adversarial
Calterh 256
0.5
Stanford-Cars
10 10-2
2
3.77+0042
1.45+0.13
1.76=00.99
0.4
CUB 200 2011
0.82.10.07 10 10-3
0,62.10.03
0.04
MIT-Indoor
0.8110.04 10"4
0.61
0.01
0.3
Aircrafts
Indoor Pre-trained
0.2
Flowers
0
Random
Adversarial
0.1
10 10-²
2.51110.34
1.1110.09
0.97
10 10-3
0.49
0.36+0.05
0.32
0.0
10
0.44+0.03
0.33+0.02
0.30=0.00
2
4
12 Number of layers
8
10
14
16
(a) Fine-tuned distances.
(b) Perturbed loss.
Figure 3: Left: Ratio between the -nomm of the fine-tuned distances and the pre-trained network. Right: Perturbed loss of fine-tuned model from random initializations (Random), pre-trained model initializations (Pre-trained), adversarially trained model initializations (Adversarial). Results shown for the CUB-200-2011 and the Indoor data sets using ResNet-18; Averaged over 10 runs.
tools (McAllester, 1999a.b). This departs from the previous work of Gouk et al. (2021), which bases on their analysis using Rademacher complexity. This change of perspective is inspired by several recent works that have found PAC-Bayesian bounds to better correlate with empirical performance than Rademacher complexity bounds (Jiang al., 2020). We refer the interested reader Section A for further references from this line of work. After presenting our result, we provide empirical measures of our result and discuss the practical implication.
Theorem 4.1 (PAC-Bayes generalization bound for fine-tuning). Suppose for every i 1,.... L,
[ |12 < B,, for fixed B, > L. Suppose the feature vectors in the domain X are all bounded: | S C. every x E X. for some C1 Z Finally, suppose the loss function 2(-) is I-Lipschitz and bounded from above by fixed constant Cz. Under these conditions, let fw be the minimizer of regularized fine-tuning. solved from problem (3). Let > be an arbitrary small value. Let be the maximun over the width over all the L layers and the input dimension d. Then, with probability at least 1 28 for some small
the expected loss (fw) upper bounded by
nh. (R, (R)+ Di
# bgl(LLCC)
Dis D2
43
& (w))
B,12; 1e
Relation to prior works. Compared to the result of Gouk et al. (2021), we instead proceed by factoring the generalization error into noise error (i.c. the term in equation (4)) and a KL-divergence between Wi(+) and W (i.e. the final term equation (4)). The proof of Theorem 4.1 is based on Neyshabur et al. (2018) and is presented in Appendix A. The difference between Theorem 4.1 and the result of Neyshabur et al. (2018) is that our result is stated for the (e.g. cross-entropy) loss function t(+) whereas Neyshabur et al. (2018) states the result using the soft margin loss.
Our result suggests that the fine-tuned distances (Di)er and the perturbed loss (more precisely Eu (e(f+u (x), v)] where every entry of drawn from (0,, o²)) are two important measures for fine-tuning. Next, we empirically evaluate these two measures real-world data sets.
First, present empirical finding for the fine-tuned distances in each layer. We fine-tune a ResNet-18 model (pre-trained on ImageNet) on seven data sets and calculate the Frobenius distance between (x(()) and W, for every layer i.
Fine-tuned distances {D.)_
Figure 3a shows the ratio of the fine-tuned distances the pre- pre-trained weights in each layer. First, we observe that the fine-tuned distances are relatively small compared to the pre-trained network. This means that fine-tuning stays within a small local region near the pre-trained model. Second, we find that D, varies across layers. D, is smaller at lower layers and is larger at higher layers. This observation aligns with the folklore intuition that different layers in Convolutional neural networks play different role (Guo et al., 2019). Bottom layers extract higher-level representations of features
5

---

0.6
Fizettzing: Thaning
Prediction accurcy
Predlctioon accurayy
Fineetring:
Hogula las lietiing Trining
D.4
Regulaaion fistunnn Ted
tuing Tramiig
Firastainng The:
0.2-
0.2
Regiintod tining Traming
Regriaroo
30
Number spodus
Number of spochs
(a) Noise rate 60%.
(b) Noise rate rj 80%.
Figure 4: Training and test accuracy of fine-tuning using early stopping and optimizing equations (5) and (6) on the Indoor data set with different levels of noise rate. Stronger regularization reduces the generalization gap in both settings.
and stay close to the pre-trained model. Top layers extract class-specific features and vary among different tasks. Thus, top layers travel further away from the pre-trained model.
Based on the above empirical finding, we propose layer-wise constraints as regularization. In particular, the constraints for the bottom layers should be small, and the constraints for the top layers should be large. In practice, we find that an exponentially increasing scheme involving base distance parameter D and a scale factor 7 works well. This is summarized in the following regularized fine-tuning problem:
W t arg min (Sw)
(5)
s.t. W - W/ S DD DD Vi where the scale factor ? according to Figure 3a. The previous work of Gouk al. (2021) the same distance parameter in equation (6), i.e., D; D, vi = L..... L. In our experiments, we have observed that such constant regularization constraints are only active at the top layers. With layer-wise regularization, we instead extend the constraints to the bottom layers as well. In Table 8 (ef. Section B.3), we compare the value of Σ Dē between constant regularization and layer-wise regularization. We find that layer-wise regularization leads to smaller value of Dí= D?. thus indicating tighter generalization bound according to Theorem 4.
1,.=,...,,A
(6)
Perturbed loss Ey (o v) Second, we compare the perturbed loss of models fine-
tuned from random, pre-trained, and adversarially robust pre-trained model initializations. The perturbed loss measured by the average loss on the training set, after perturbing each entry of the layer weights by Gaussian random variable N(O, o2).
In Table 3b, we report the perturbed losses for both the The results show that models fine-tuned from pre-trained initializations are more stable than models fine-tuned from random initializations.
and the Indoor data set.
Explaining why adversarial robustness helps fine-tuning. Recent work (Salman et al., 2020) found that performing adversarial training in the pre- pre-training phase leads to models that transfer better to downstream tasks. More precisely, the adversarial training objective is defined as
min W
(=))) Σ
max ((f((x)) + +)). p8llase
adv Let (s) be a fine-tuned neural network using pre-trained model from adversarial training. We
measure the perturbed loss of (e( adv using the pre-trained model provided by Salman et al. (2020) on the CUB-200-2011 and Indoor data set. Table 3b shows that firto) often incurs lower perturbed losses
than models fine-tuned from random and pre-trained initializations. In Section B.3, additionally observe that the layers of (*) the improved noise stability and the smaller layer-wise norms together imply tighter generalization
adv generally have lower Frobenious norms compared to W)(+), Thus,
bound for adversarial training. We leave a thorough theoretical analysis of adversarial training the context of fine-tuning to future works.

---

0.8
0.17
With repaloriaatoon
Without regairitatinn
Average weight
Precision
0.6
0.12
Coneetll leboled training data
Lacorreolyy labeed training data
0.07
0.1
0.02
0.2
0
0
10
Number spocles
Number of epocias
(a) Precision of self lobel-correction.
(b) Average weight injected by self label-eweigghing
Figure 5: Left: Comparing the precision of self-labeling with vs. without regularization. Precision is defined as the number of data points whose labels are corrected (cf. Line 7 in Alg. 1) divided by the number of data points whose labels are changed every epoch. Right: Comparing the average normalized weight (cf. Line 10 in Alg. 1) of training data points with the correct label vs. data points with an incorrect label.
4.2 Fine-tuning and robustness
Next, extend our approach to fine-tuning from noisy labels. We compare fine-tuning with and without regularization on a target task with independent label noise to motivate our approach. In Figure 4, we plot the training and test accuracy curves the Indoor dataset with two noise rates. We observe that the test accuracy increases at first, meaning the network learns from the correct labels during the first few epochs. As the learning process progresses, the test accuracy of fine-tuning decreases, implying the network is "overfitting" to noisy labels. Furthermore, the gap between the training and test accuracy curves is more than 20%. Similar observations have been presented in prior works (Zhang et al., 2016; Li et al., 201 8c; Liu et al., 2020), where wver-parametrized neural networks One way to mitigate memorization
memorize the entire training capacity (Wu et al., 2020; Yang et al., 2021). Another way is via explicit regularization since the distance constraint significantly reduces the training-test accuracy gaps and improves test accuracy by a significant margin. Based on our empirical observation, the neural network has some discriminating power during the early fine-tuning phase. Thus, can leverage the model predictions relabel incorrect labels and reweight data points with incorrect labels. We describe the two components next.
explicitly reduce model
Self label-correction. We propose a label correction step to augment the number of correct labels. We leverage the discriminating power of a network during the early phase and correct data points for which the model has high confidence. Concretely, let p; be the prediction of the i-th data point. Given a confidence threshold pt at epoch 6, if max(p:) 2 Ps and arg max(p;) H (t) we identify data point noisy and relabel it to
(1) ara max(()).
Figure 5a illustrates the precision of this step. We define precision as the number of correctly relabeled data points divided by the total number data points whose labels are relabeled. We observe that precision is high at the beginning (around 60%) and gradually decreases in later epochs. Moreover, we find precision increases with regularization. This suggests an intricate interaction between regularization and lell-labeling during fine-tuning, We elaborate on the role of regularization in label correction by looking at the number of relabeled data points in Section B.3.
Self label-rewiighting We incorporate soft data removal step prevent the model from overfitting to noisy labels. We identify data points as noisy if their loss values are large and thus them during training. The model will prioritize data points with correct labels in its gradient by
assigning smaller weights large-loss data points. More precisely, given data point (x (t) ((")).
we reweight it by = exp(-L([w) (2: vi )/T) where is temperature parameter. In the implementation, we normalize the weights every data point in every mini-batch B:
(o(Sw)=
1
Σ wi Ew--e(w(uuf e(w((x()))
Eier Wi
7

---

Algorithm Regularized self-labeling (REGSL)
Input nosiy data (x1 M/1 )....... (1 mees k network model fw. constraint distance D, distance scale factor y, re-weight start step Er, re-weight temperature т, label correction start step Fe, label correction threhold pt, and fine-tuning steps T
Input:
I: Initialize model parameters with pre-trained weights W- W and the start step t 0
2: while < T do
Fetch batnh data { 3. )
4: for each data point in the batch do
Calculate the predicted probability p: = (f
6:
if > Ee and max (Pt) ) > ps and arg ma.) # V/ then Correct the label yi arg max(p:)
7:
8: end if
9:
Calculate loss & = ((f(([)) (x:"), )) v.
10: if Er, then Calculate the weight wi exp(-Ix/T) else Set the weight w;
end for
12: Update fw by SGD on L =
Ei D wil, and = L +
13:
Project the weights W, of each layer inside the layer- wise constraint region as in Equation 6
14:
end while
Figure 5b compares the average weight of data points with the correct label to data points with an incorrect label. We find that with our reweighting scheme, the average weight of incorrectly labeled data points is much lower than the average weight of correctly labeled data points. Thus, the reweighting scheme expands the gradient of correctly labeled data points, thus reducing the fraction of noisy data points in the training set.
The pseudo-code for our final approach, which combines both layer-wise regularization and self- labeling, presented in Algorithm
5 Experiments
We evaluate proposed algorithm in wide range of tasks and pre-trained networks. First, show that our algorithm outperforms the baselines by 1.75% on average over seven transfer learning tasks and generalize to more distant task of medical images. Second, our algorithm improves the robustness of fine-tuning in the presence of noisy labels compared to previous methods. Our algorithm improves
numerous baselines by 3.56% on average under both independent and correlated label noise. Moreover, our approach can achieve a similar performance boost for fine- tuning vision transformer models on noisy labeled data. Finally, we show our algorithms improve over fine-tuning baselines by 0.75% on average for the related task of few-shot classification. Our extensive evaluation confirms that our approach applies algorithmic insights. Due to space constraints, we defer the ablation studies to Section B.3. Our code available at https://github. www.loooo//o/aeeeeeee StatsMMLRRssearchhPPe Rewee///eeeeaaeeeeeSfefeee Searredd-Se---abbeee Self-Labeling.
broad range of settings, validating our
5.1 Experimental setup
Data sets. First, we evaluate fine-tuning on seven image classification data sets and one medical image data set. The statistics of the seven image data sets described in Table 1. For the medical imaging task, we consider the ChestX-ray11 14 data set contains 112120 frontal-view chest X-ray images labeled with 14 differrntt diseases (Wang et al., 2017: Rajpurkar al., 2017). Second, we evaluate Algorithm under two different kinds of label noise to test the robustness of fine-tuning. We selected the MIT-Indoor data (Sharif Razavian al., 2014) as the benchmark data set and randomly flipped the labels of the training samples. The results for the other data sets are similar, which can be found in Section B.2. We consider both independent random noise and correlated noise in our experiments. We generate the independent random noise by flipping the labels uniformly with a given noise rate. We simulate the correlated noise by using the predictions of an auxiliary network as noisy labels. Section B. describes the label flipping process. For few-shot image classification, we conduct experiments on the minilmageNet benchmark (Vinyals al., 2016) following the setup of Tian et al. (2020).
8

---

Table 2: Top-1 test accuracy for fine-tuning ResNet- 101 pre-trained on the ILSVRC-2012 subset of ImageNet. Results are averaged over random seeds.
Airerafts
CUB-200-200
Caltech 256
Stanford-Can
Stanford Dogs
Flowers
MIT-Indoor
Fine-tuning e--oom
73.96+0034 34
80.31=0.226
79 79.41::0.22
89. 00.28:20.1 14
82 $2.89-1-0:35
92.92:1:015
74.78:0.977
74.84+00.15
80.80=0.116
79.67=00311
88 96966=011
13.00=+0.10
92.76=0.22
76
LS
74.19:20.41
82.22
81.10:2:.113
89.66=1.110
84.14:10.21
93.72:100044
76.84=0.31
74.09::0.99
01349:10.36
84.13:100.99
88, 08.96.00.09
88, 08,95.00.15
93.06.1:.28
78.11
e PGM
74.90.10.26
81.23
83.25.10.33
88, 08.92.10.39
86.48.10.28
93.23.10.34
77.31 1..30
REGSL (ouns)
75.32:1:0.23
0.224:10.21
84.90:1-.16 0.16
89.14::0.22
89.58:10.13
93.82:1:0.35
79.30:10.31
Architectures. We use the ResNet-101 (He et al., 2016) network for transfer learning tasks and ResNet-18 (He et al., 2016) network for ChestX-ray data set. We use ResNet-18 network for the label- noise experiments and extend the our algorithm to the vision transformer (ViT) model (Dosovitskiy et al., 2020). The ResNet models are pre-trained on ImageNet (Russakovsky et al., 2015) and the ViT model is pre-trained on ImageNet-221 Ik data set (Dosovitskiy et al., 2020). For the few-shot learning tasks, we use ResNet-12 pre-trained on the meta-training data set as in the previous work (Tian et al., 2020). We set four different values of D, for the four blocks of the ResNet models in our algorithm. We describe the fine-tue mne-tuning procedure and the hyperparameters in Section B.
Baselines. For the transfer learning tasks, we use the Frobenius norm for distance regularization. We present ablation studies to compare the Frobenius norm and other norms in Section B.3. we compare our algorithm with fine-tuning with early stop (Fine-tuning), fine-tuning with weight decay (62- Norm), label smoothing (LS) method formulated in Müller et al. (2019), C2-SP (Li et al., 20 2018b), and (2-PGM (Gouk et al., 2021). For testing the robustness of our algorithm. in addition to the baselines described above, we adopt several basclines that have been proposed in the supervised learning setting, including GCE (Zhang and Sabuncu, 2018), SCE (Wang al., 2019), DAC (Thulasidasan et al., 2019), APL (Ma et al., 2020), ELR (Liu et al., 2020) and self-adaptive training (SAT) (Huang al., 2020). We compare with the same baselines to transfer learning in few-shot classification tasks. We describe the implementation and hyper-parameters of baselines in Section B.I.
5.2 Experimental results
Improved regularization for transfer learning. We report the test accuracy of fine-tuning ResNet- 101 on seven data sets in Table 2. We observe that our method performs the best among six regularized fine-tuning methods on average. Our proposed layer-wise regularization method provides an improvement of 1.76% on average over distance- based regularization (Gouk et al., 2021). In particular, our approach outperforms P2-PGM by 2 3% on both Stanford-Dogs and MIT-Indoor data sets. This suggests that adding appropriate distance constraints for all the layers is better than applying only one constraint the top layers. In Table 10 (cf. Section B.3), we note that using the MARS norm of Gouk et al. (2021) yields comparable results to using the l norm with our approach.
Next, we apply our approach to medical image classification, a more distant transfer task (relative to the source data set ImageNet). We report the mean AUROC (averaged over predicting all 14 labels) on the ChestX-ray -ay 14 data set in Table 6a (ef. Section B.2). With layer- regularization, we an 0.39% improvement over the baseline methods. Finally, we extend our approach to text classification. The results are consistent with the above and can be found in Table 7 (cf. Section B.2).
Improved robustness under label noise. Next, we report the test accuracy of our approach on the Indoor data set with independent and correlated noise in Table 3. We find that our approach consistently outperforms bascline methods by 3% for various settings involving label noise. First, method (REGSL) improves the performance by over 4% on average compared to distance-based regularization (Gouk et al., 2021). This result implies our method is more robust to label noise in the training labels. Second, our method outperforms previous supervised training methods by 3.56% % on average. This result suggests that regularization is critical for fine-tuning. Taken together, our results suggest that regularization and self- labeling complement each other during fine-tuning. This is reinforced by our ablation study in Section B.3; We study the influence of each component of our algorithm and find that removing any component degrades performance.
We further apply our approach to fine-tuning pre-trained VIT (Dosovitskiy et al., 2020) from noisy labels, under the same setting as Table 3. Table 6b shows the result (cf. Section B.2). First, we find that

---

Table 3: Top-1 test accuracy of fine-tuning ResNet- the Indoor data set with various settings of noisy labels in the training set. Results are averaged over random seeds.
independent noise
correlated noise 25.18%
Data sets
Methods
20%
40%
60%
80%
Fine- fine-tuning
65.02:40.39
07.49=0.39
44.60:10.95
27.09:10.19
67.49:10.74 74
LS
67.04=0.58
58.98=0.57
48.56+0..33
25.82:+0.90
68.06=10.76
PPPGM
69.45:10.19
6..74
51.24:1:.155
30.15:+0..4
68.86:+0.27
GCE
70.45:::.40
64.73++0.21
54.10:10.16
29.58++0.26
68.88
SCE
69.43++0.20
64.68+0045
55.07+40.52
29.85+t0.44
69.00=
Indoor
DAC
64.45+0.31
59.73:0.27
47.44:10.09
26.69:10.34
68.06=1132
APL
70.05::0.41
66.22:10.10
52.51:10.66
30.9011..77
68.31
SAT
68.98110.3
63.4311.72
$2.8410.38
29.60:10.53
67.06-10.55
ELR
71,43+00.80
66.34+0048
55.22++0.73
31.24++0.19
69.38+0049
REGSL (ours)
72.51:10.46
68.13110.16
57.5910.55
34.08
70.12:10.83
our approach improves upon the best regularization methods by 1.17% averaged over two settings. Second, we find that our approach also improves upon self-labeling by 13.57% averaged over two settings. These two results again highlight that both regularization and self-labeling contribute to the final result. While regularization prevents the model from overfitting to the random labels, self-labeling injects the belief of the fine-tuned model into the noisy data set.
Extension to few-shot classification. We extend our approach to a few-shot image classification task. We compare approach the baseline regularization methods. In Table 4, we report the average accu- racy of 600 sampled tasks from the meta-test split of the minilmageNet benchmark (Vinyals et al., 2016). We find that our layer-wiss wise regularization method achieves O. 75 % improvement on average compared to previous regularization methods. Additionally. reg- ularization methods generally improve the perfor- mance of fine-tuning by over 1%.
Table 4: Test accuracy with 95% confidence interval for fine-tuning ResNet-12 over 600 meta-test splits minilmageNet.
minilmageNet
Methods
5-way-1-shot
5-way-5-shot
Fine-tuning
60.56 + 0.78
76.42 + 0.36
e
60.93 1 0.81
76.57 1:: 0.55
LS
61.31 0.77
76.73 = 0.62
SP
61.48 + 0.76
77.02 + 0.62
e -PGM
61.35 0.83
77.33 = 0.59
We study the influence of remov- ing each component from our algorithm. Results shown in Table 11 (cf. Section B.3) suggest that they all degrade performance. Thus, all three components in Algorithm I contribute the final performance. The importance of each component depends on the noise rate. In particular, if the noise rate is higher than 40%, label-correction is the most critical component. Other ablation studies we present include: comparing the & norm and the MARS norm in our algorithm and comparing the distance parameters between layer-wis wise and constant regularization. We leave the details Section B.3.
Ablation studies.
REGSL (ours)
61.71 11 0.77
78.03 1 0.54
Conclusion
This paper studied regularization methods for fine-tuning as well as their robustness properties. We investigated the generalization error of fine-tuning using PAC-Bayesian techniques. This leads to two empirical measures, which we empirically computed. The analysis inspired us to consider layer-wise regularization for fine-tuning. This approach performs well on both pre-trained ResNets and ViTs; we have found the fine-tuned distance using vanilla fine-tuning and encoded the distance patterns in layer-wise wise regularization. We then evaluated the performance of regularized fine-tuning from noisy labels. We proposed self label-correction and rabei-rewiggtting approach in the noisy setting. We found that regularization and self-labeling complement each other in our experiments, leading to significant improvement over previous methods, which either use regularization methods or use self-labeling methods but not both. We discuss the limitation of our work in Section C.
Negative societal impacts. Our work concerns the theory and the empirical performance of fine- tuning for transfer learning. Due to its technical nature, there is very little negative societal impact. Since transfer learning is commonly used in practice, one avenue that deserves attention for these applications is to carefully evaluate bias or faimess metrics before applying our proposed approaches.
10

---

Acknowledgment.. Thanks to the anonymous reviewers and the area chairs for carefully reading through our work and providing constructive feedback. Thanks the program committee chairs for communicating with us regarding an omitted factor in equation (14), which we have added in the current version. HZ grateful Sen Wu and Chris Ré for stimulating discussions. HZ is supported by Northeastern University's s Khoury peed/proof-of-coneee grant.
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In International Conference on Knowledge Discovery and Data Mining, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-ge Eneeggrined analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Learning. PMLR, 2019.
Peter Bartlett, Dylan J Foster, and Matus Telgarsky. networks. arXiv preprint arXiv: 100d:::0000000 10X.0097066 2017.
margin bounds for neural
Peter L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1) (022:255-2331,, 2019.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from diffferent domains. Machine learning, 2010.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. 2019.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint arXiv: 02729, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv: 1108, 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021.
Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. ICLR, 2021.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Yunhui Guo, Honghui Shi, Abhishek Kumar. Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Conference on computer vision and pattern recognition, pages 770-778, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In International Conference Computer Vision, 2019.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In ACM SIGKDD interna- tional conference on Knowledge discovery and data mining, 2004.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in Neural Information Processing Systems, 2020.
11

---

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. ICLR, 2020.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for GraenG Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, 2011.
ine-grained image categorization. In Workshop
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69850 12,00, 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In Workshop 3D Representation and Recognition, 2013.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint 1777 2020.
Xin Li and Dan Roth. Learning question classifiers, In International conference on Computational linguistics, 2002.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations, 2018a.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. arXiv preprint arXiv:1802.014555 2018b.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over wer-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pages 2-47, PMLR, 2018c.
Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Germadde-Grrnndee Early-learning regularization prevents memorization of noisy labels. arXiv preprint arXiv:2007.0015), 2020.
Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. ICLR, 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Normal- ized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 130.55130 2013.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pages 164-170, 1999a.
David A McAllester. Some pac-bayesian theorems. Machine Learning. 37(3):355-363, 1999b.
Rafael Müller, Simon Korbblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv:1906.02660 2019.
Vaishnavh Nagarajan and Zico Kolter. Deterministic pac-bayesian generalization bounds for deep net- works via generalizing noise-resilience. In International Conference on Learning Representations, 2018.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. Conference on Neural Information Processing Systems, 2013.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-hayesian approach to spectrally- normalized margin bounds for neural networks. ICLR, 2018.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008
12

---

Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. In Anmual meeting on Association for Computational Linguisties, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Annual meeting on association for computational linguistics, 2005.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding. Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint orXiv:1711.05225, 2017.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data program- ming: Creating large training sets, quickly. Advances in neural information processing systems, 2016.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In International Conference on Very Large Data Bases. NIH Public Access, 2017.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 4763 47771, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
Khaled Saab, Sarah M Hooper, Nimit S Sohoni, Jupinder Parmar. Brian Pogatchnik, Sen Wu, Jared A Dunnmon, Hongyang R Zhang, Daniel Rubin, and Christopher Ré. Observational supervision for medical image classification using gaze data. In International Conference Medical Image Computing and Computer-Assister Intervention, pages 603-614. Springer, 2021.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversari- ally robust imagenet models transfer better? arXiv preprint arXiv:2007.084889 2020.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Conference on computer vision and paltern recognition workshops, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D) Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on empirical methods in natural language processing, 2013.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self self-training. arXiv preprint arXiv:2102.088 0862222 2021.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, 2018.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. arXiv preprint 4. 2019.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good emhedding is all you need? arXiv preprint 1553, 2020.
Joel A Tropp. An introduction to matrix concentration inequalities. arXiv preprint 1rX1011501.01571, 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- 2016.
ing networks for one shot learning. arXiv preprint
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie, The Caltech-UCSD. Birds-200-2011 Dataset. Technical Report ANNS-TR-2011-01 California Institute of Technology, 2011.
13

---

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: dospital-scale chest x-ray database and benchmarks on weakly-supervised classifica- tion and localization of common thorax diseases. In Conference on computer vision and pattern recognition, 2017.
Yisen Wang, Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Internationall Conference Computer Vision, pages 322-330, 2019.
Colin Wei and Tengyu Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation. arXiv preprint arXiv: Nekiiv:/00..0 1905.03684, 2019a.
Colin Wei and Tengyu Ma. Improved sample complexities for deep networks and robust classification via an all-layer margin. arXiv preprint arXiv:1910.04288 2019b.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 2005.
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. ICLR, 2020.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning, pages 1525-11536. PMLR, 2021.
Fan Yang, Hongyang R Zhang, Sen Wu, Weijie Su, and Christopher Ré. Analysis of information transfer from heterogeneous sources via precise nigh-dimensional asymptotics. arXiv preprint arXiv:2010.11550, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 161.v:1611.03530. 2016.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv: 1805.07836, 2018
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds the imagenet scale: pac-hayesian compression approach. arXiv preprint arXiv:1804.05862, 2018
Yang Zou, Zhiding Yu. Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982-5991, 2019.
14


Reviews:
[{"review_id": "wkMTyaZwItv", "reviewer": "Reviewer_FQxu", "summary": "Inspired by the empirical findings on fine-tuned distance in each layer and noise stability of model, this paper introduces a fine-tuning algorithm that incorporates layer-wise distance constraint and self label correction and removal. To prove the effectiveness of their algorithm on regularization and robustness, the authors provide experimental results on fine-tuning on different image datasets and fine-tuning on noisy label respectively. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1.\tThe claims in this work are well supported by the PAC-Bayes generalization bound and the experimental results. The authors also provide codes in the supplementary material.\n2.\tThis paper is well written and easy to follow. Their algorithm is related to distance-based regularization and noisy labels problems. The relations are clearly illustrated. \n\nWeaknesses\n1.\tOverall, the paper is highly related to the previous work (Gouk et al. 2020) in terms of the distance-based regularization techniques and PAC-Bayes generalization bound, which slightly degrades the novelty of this work.\n2.\tThe algorithm is the combination of three components: distance regularization, label correction and label removal. It is unclear which part is most important. To my understanding, the distance regularization in this work is $l^2$-pgm (Gouk et al. 2020) + layer wise scale factor. From the ablation study in appendix c.2, the accuracy of distance regularization + label removal is worse than the accuracy of $l^2$-pgm for most setting of independent noise. Seems like by adding the scale factor and label removal, the result is even worse.\n3.\tThe authors only provide experiments on image dataset with relatively small size. Is this algorithm more effective on the convolutional layers? It might be better to add experiments in other application domains that involves more non-convolutional layers. There are six hyper-parameters in the algorithm. It might be challenging to set them with limited labeled data, especially for the label correction threshold and re-weighting temperature factor.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours."}, "extra_text": {}}, {"review_id": "nkuYMai07Rs", "reviewer": "Reviewer_YhjQ", "summary": "\nThe authors present a new analysis of PAC-Bayes generalization in the finetuning with regularization setting and analyze it’s properties in terms of layer wise L2 distances and noise robustness.  They propose a new modified algorithm for finetuning based on having different constraints on different layers as well as self-labeling.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality/Clarity:\nThe paper is overall clearly presented. To the best of reviewers knowledge the work is original, including the bounds, analysis (e.g. motivating usefulness of robustness), and fine-tuning approach. The experiments are reasonably well done, using relevant baselines and datasets.\n\nSignificance:\nFine-tuning is still not fully understood from a theory point of view and this work contributes to improving our understanding. Furthermore, the proposed method shows promising results.\n\nGeneral Comments:\n-Most of the tasks studied are closely related to the source task. It would be interesting to study the behavior of the proposed algorithm when more distant tasks are used (e.g. medical images).\n\n-The connection to “explaining robustness” is a bit loose as the authors point out that adversarially robust models have lower operator norm. Is it possible to go a bit further than this and  illustrate this theoretically\n\n-The authors illustrate how naturally the L2 distance for lower layers is smaller than higher layers, so why is there a need/advantage to incorporate this into the constraints if it already occurs? Related to that the authors motivate this phenomenon by referencing the standard interpretation of feature roles in lower and upper layers, I wonder if this is as well explained by the reduction in gradient norm with depth that typically occurs  in backprop.\n\n-The ablations should include a baseline that shows the effect of layer-wise constraints without self-label \n\n\n\nOverall, the paper has useful theoretical and empirical findings. Some further clarity and ablations can further improve it. \n\nPost-rebuttal: The authors have reasonable addressed my primary concerns.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact and limitations (future work) are adequately covered", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "hwIF9W8ldaK", "reviewer": "Reviewer_a4M8", "summary": "The paper analyses the fine-tuning of neural networks. It provides a PAC-Bayes bound for the generalization of a fine-tuned network. The paper then provides three tricks to improve finetuning: layerwise regularization, label correction, and label removal. The resulting fine-tuning algorithm improves performance on various image classification tasks, including those with artificially-added label noise.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Summary*\n\nOverall the paper is well written. The theoretical analysis provides some interesting motivation. I am not an expert on PAC-Bounds, so I cannot comment whether the result is of significant interest to the theoretical community, nor can I comment on whether the bound is tight or even non-vacuous. The resulting algorithm is intuitive and does seem to offer some improvements on real world datasets.\n\nHowever, the main weaknesses for me are\n* The claim \"outperforms state-of-the-art\" appears far too strong.\n* The connection between the theoretical analysis and the algorithm appears quite weak.\n* There is not sufficient detail on the theoretical analysis for someone unfamiliar with PAC-bounds to understand the implications.\n\nDetails below\n\n*Originality*\n\nThe heuristic algorithmic improvements seem nice, offering modest improvements in performance. However, these are relatively small extensions of existing approaches. \n\n*Quality*\n\nOverall the contribution both of a theoretical result, and solid empirical numbers is convincing. However, I have a few concerns\n\n1) The claim of \"state-of-the-art on seven transfer tasks\" seems to be hugely exaggerated, taking two examples\n\nCUB-200-2011: the paper reports 82.24%, while [1] reports 91.7%, and many algorithms have reported over 90%.\nFlowers: the paper reports 93.82%, while [2] reports 99.74%, and many algorithms have reported over 99%.\n\nPerhaps the authors mean \"state-of-the-art if limited to a particular ResNet101 pre-trained on ImageNet\", however, \"baseline\" would then be more appropriate.\n\nI do not believe that the method needs to attain state-of-the-art to demonstrate its validity, but the claim is too strong, and the fact that the scores are so far behind state-of-the-art (fine-tuning a stronger public pre-trained network is not expensive) brings into doubt the validity of the conclusions.\n\n2) The connection between the PAC-Bayes bound and the algorithm appears weakly motivated. For example, the bound indicates that larger distances moved indicate a lower generalization error (which is intuitive given the Gaussian prior/posterior and KL[P|Q] term). However, the motivation for layerwise regularization (as opposed to the existing methods of constant regularization) is empirical. The bound does not appear to speak to layerwise differences.\n\n3) The algorithm introduces many hyperparameters. While the hyparameter sweeps are listed in the Appendix, it was not clear exactly how many hyperparameters are swept for each algorithm, including the baselines. Could the authors provide a concise summary of the size of the search space for each method?\n\n*Clarity*\n\nThe paper is generally well written. One part that is hard to follow is Section 4.1. For example, the \"noise stability l(...)\" does not appear in Eqn (4), so it is not clear how these are connected. This section could be expanded and more details provided.\n\nA few typos spotted\n* genrealization\n* disturbation\n* over-fitiing\n* regualrized\n* line 227: max(p) <= t,  should this be  >= ?\n\n*Significance*\n\nThe set of tricks to improve fine-tuning could be useful to practitioners. However, it does not constitute a very significant algorithmic advance. \n\nWithout further context, the theoretical result does not appear significant (is the bound tight/trivial?), however, as mentioned above I am not an expert on theory, so I will leave it to others to assess the significance of this result.\n\n\n[1] [TransFG: A Transformer Architecture for Fine-grained Recognition](https://arxiv.org/abs/2103.07976)\n\n[2] [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929v2)\n\n\n\nPOST REBUTTAL\n\nThe response provided additional context and experiments, and the claims of SOTA will be tempered. Increasing my score.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and impact are adequately acknowledged.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "CKR-N6e7P0Z", "reviewer": "Reviewer_uNxM", "summary": "This paper studies the aspects of fine-tuning a model (pre-trained on a separate dataset) in order to learn a new task. It has the following contribution:\n\n(i) Generalization properties of fine-tuning explained by the PAC-Bayes generalization bound that depends on two terms: (a) distance between each layer from the pre-trained model and (b) noise stability of the pre-trained model.\n\n(ii) Inspired by the generalization analysis, the paper presents an algorithm to perform fine-tuning. This algorithm includes (a) regularization term that computes the distance between the current parameters and the pre-trained model, and (b) iteratively corrects mislabeled examples where model has high confidence and down-weighting less confident examples.\n\n(iii) Proposed algorithm is evaluated on a suite of benchmark tasks including transfer learning and few shot classification tasks.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n---------------\n\n- PAC-Bayes generalization bound for fine-tuning explains the generalization performance of the model obtained from fine-tuning. In hindsight, the terms appearining in the bounds look intuitive (i.e. distance from the pre-trained model and the noise stability of the model). It also explains why models trained with adversarial noise result in better fine-tuned performance.\n\nWeakness:\n---------------\n\n- PAC-Bayes bound shows two terms. First one is the distance from the pre-trained model, which the proposed algorithm incorporates as it is. But the second term that is the noise stability of the model, is handled in a heuristic way. Instead a principled approach should have been to incorporate the noise stability by adding some sort of penalty term (similar to the prior used in the PAC-Bayes analysis or some sort of adversarial noise).\n\n- Heuristic to incorporate the label correctness seems to have some issues. (see below)\n\nQuestions for Authors:\n---------------\n\n- Label correction is done by thresholding the confidence of the classifier, in this particular case the class scores less than  a threshold. Did you try any other heuristics for label correction like entropy, etc.?\n\n- What is the rational in down-weighting low confident examples? Why is it assumed that the network cannot learn these examples at all? \n\n- Similarly, iteratively correcting the mislabeled examples does not seem like a good idea. Did the authors verify qualitatively how many times these \"corrections\" were indeed correct?\n\n\nWriting Clarity:\n---------------\n\nPaper is clearly written and easy to understand.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers and myself expressed a positive opinion on the paper following the discussion phase (which greatly helped reach a decision), and its interest to the NeurIPS community and significance of the results. The reviewers however asked for a number of clarifications and I would like to stress that the authors *must* revise significantly their initial submission according to the reviews and the discussion.
Decision: Accept (Poster)