Improved Regularization and Robustness for Fine-tuning in Neural Networks
Year: 2021

Abstract:
A widely used algorithm for transfer learning is fine-tuning, where a pre-trained model is fine-tuned on a target task with a small amount of labeled data. When the capacity of the pre-trained model is much larger than the size of the target data set, fine-tuning is prone to overfitting and “memorizing” the training labels. Hence, an important question is to regularize fine-tuning and ensure its robustness to noise. To address this question, we begin by analyzing the generalization properties of fine-tuning. We present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the fine-tuned model. We empirically measure these quantities. Based on the analysis, we propose regularized self-labeling—the interpolation between regularization and self-labeling methods, including (i) layer-wise regularization to constrain the distance traveled in each layer; (ii) self label-correction and label-reweighting to correct mislabeled data points (that the model is confident) and reweight less confident data points. We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for a few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average in two noisy settings.

heading: 1 Introduction
text: Learning from limited labeled data is a fundamental problem in many real-world applications (Ratner et al., 2016, 2017). A common approach to address this problem is fine-tuning a large model that has been pre-trained on publicly available labeled data (He et al., 2019). Since fine-tuning is typically applied to a target task with limited labels, this algorithm is prone to overfitting or “memorization” issues (Tan et al., 2018). These issues worsen when the target task contains noisy labels (Zhang et al., 2016). In this paper, we analyze regularization methods for fine-tuning from both theoretical and empirical perspectives. Based on the analysis, we propose a regularized self-labeling approach that improves the generalization and robustness properties of fine-tuning.
Previous works (Li et al., 2018a,b) have proposed regularization methods to constrain the distance between a fine-tuned model and the pre-trained model in the Euclidean norm. Li et al. (2020) provides extensive study to show that the performance of fine-tuning and the benefit of adding regularization depend on the hyperparameter choices. Salman et al. (2020) empirically find that performing adversarial training during the pre-training phase helps learn pre-trained models that transfer better to downstream tasks. The work of Gouk et al. (2021) generalizes the above ideas to various norm choices and finds that projected gradient descent methods perform well for implementing distance-based regularization. Additionally, they derive generalization bounds for fine-tuning using Rademacher complexity. These works focus on settings where there is no label noise in the target data set. When
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
label noise is present, for example, due to applying weak supervision techniques (Ratner et al., 2016), an important question is to design methods that are robust to such noise. The problem of learning from noisy labels has a rich history of study in supervised learning (Natarajan et al., 2013). In contrast, little is known in the transfer learning setting. These considerations motivate us to analyze the generalization and robustness properties of fine-tuning.
In Section 4.1, we begin by conducting a PAC-Bayesian analysis of regularized fine-tuning. This is inspired by recent works that have found PAC-Bayesian analysis correlates with empirical performance better than Rademacher complexity (Jiang et al., 2020). We identify two critical measures for analyzing the generalization performance of fine-tuning. The first measure is the `2 norm of the distance between the pre-trained model (initialization) and the fine-tuned model. The second measure is the perturbed loss of the fine-tuned model, i.e. its loss after the model weights get perturbed by random noise. First, we observe that the fine-tuned weights remain closed to the pre-trained model. Moreover, the top layers travel much further away from the pre-trained model than the bottom layers. Second, we find that fine-tuning from a pre-trained model implies better noise stability than training from a randomly initialized model. In Section 4.2, we evaluate regularized fine-tuning for target tasks with noisy labels. We find that fine-tuning is prone to “memorizing the noisy labels”, and regularization helps alleviate such memorization behavior. Moreover, we observe that the neural network has not yet overfitted to the noisy labels during the early phase of fine-tuning. Thus, its prediction could be used to relabel the noisy labels.
We propose an algorithm that incorporates layerwise regularization and self-labeling for improved regularization and robustness based on our results. Figure 1 illustrates the two components. First, we encode layer-wise distance constraints to regularize the model weights at different levels. Compared to (vanilla) fine-tuning, our algorithm reduces the gap between the training and test accuracy, thus alleviating overfitting. Second, we add a self-labeling mechanism that corrects and reweights “noisy labels” based on the neural network’s predictions. Figure 1 shows that our algorithm effectively hinders the model from learning the incorrect labels by relabeling them to correct ones.
In Section 5, we evaluate our proposed algorithm for both transfer learning and few-shot classification tasks with image and text data sets. First, using ResNet-101 (He et al., 2016) as the pre-trained model, our algorithm outperforms previous fine-tuning methods on seven image classification tasks by 1.76% on average and 3.56% when their labels are noisy. Second, we find qualitatively similar results for applying our approach to medical image classification tasks (ChestX-ray14 (Wang et al., 2017; Rajpurkar et al., 2017)) and vision transformers (Dosovitskiy et al., 2020). Finally, we extend our approach to few-shot learning and sentence classification. For these related but different tasks and data modalities, we find an improvement of 0.75% and 0.46% over previous methods, respectively.
In summary, our contributions are threefold. First, we provide a PAC-Bayesian analysis of regularized fine-tuning. Our result implies empirical measures that explain the generalization performance of regularized fine-tuning. Second, we present a regularized self-labeling approach to enhance the generalization and robustness properties of fine-tuning. Third, we validate our approach on an extensive collection of classification tasks and pre-trained model architectures.

heading: 2 Related work
text: Fine-tuning is widely used in multi-task and transfer learning, meta-learning, and few-shot learning. Previous works (Li et al., 2018b,a) find that injecting `2 regularization helps improve the performance of fine-tuning. Li et al. (2018b) propose a `2 distance regularization method that penalizes the `2 distance between the fine-tuned weights and the pre-trained weights. Li et al. (2018a) penalize the distance between the feature maps as opposed to the layer weights. Chen et al. (2019) design a regularization method that suppresses the spectral components (of the feature maps) with small
singular values to avoid negative transfer. Gouk et al. (2021) instead encode distance constraints in constrained minimization and use projected gradient descent to ensure the weights are close to the pre-trained model. Salman et al. (2020) show that fine-tuning from adversarially robust pre-trained models outperforms fine-tuning from (standard) pre-trained models.
The robustness of learning algorithms in the presence of label noise has been extensively studied in supervised learning (Natarajan et al., 2013). Three broad ideas for designing robust algorithms include defining novel losses, identifying noisy labels, and regularization methods. Zhang and Sabuncu (2018) design the robust Generalized Cross Entropy (GCE) loss which is a mixture of Cross Entropy (CE) and mean absolute error. The Symmetric Cross Entropy (SCE) (Wang et al., 2019) loss combines reverse cross-entropy with the CE loss. Ma et al. (2020) proposes APL, which normalizes the loss to be robust to noisy labels and combines active and passive loss functions. Thulasidasan et al. (2019) propose DAC, which identifies and suppresses the signals of noisy samples by abstention-based training. Liu et al. (2020) introduce ELR, an early learning regularization approach to mitigate label “memorization”. Huang et al. (2020) propose a self-adaptive training method that corrects noisy labels and reweights training data to suppress erroneous signals. These works primarily concern the supervised learning setting. To the best of our knowledge, fine-tuning algorithms under label noise are relatively under-explored in transfer learning. Our approach draws inspiration from the semi-supervised learning literature, which has evaluated pseudo-labeling and self-training approaches given a limited amount of labeled data, and a large amount of unlabeled data (Zou et al., 2019; Xu et al., 2021; Tai et al., 2021). A very recent work considered a sharpness-aware approach and evaluated its performance for fine-tuning from noisy labels (Foret et al., 2021). It would be interesting to see if combining their approach with our ideas could lead to better results.
From a theoretical perspective, the seminal work of Ben-David et al. (2010) considers a setting where labeled data from many source tasks and a target task is available. They show that minimizing a weighted combination of the source and target empirical risks leads to the best result. In the supervised setting, Arora et al. (2019) provide data-dependent generalization bounds for neural networks. Nagarajan and Kolter (2018) and Wei and Ma (2019a,b) provide improved generalization bounds that depend only polynomially on the depth of the neural network characterized by a margin condition. Recent work has found that the PAC-Bayes theory provides generalization measures that correlate with empirical generalization performance better than other alternatives (Jiang et al., 2020). We defer a more extensive review of PAC-Bayesian generalization theory to Section A.

heading: 3 Preliminaries
text: Problem setup. We begin by formally introducing the setup. Suppose we would like to solve a target task. We have a training data set of size n(t). Let (x(t)1 , y (t) 1 ), . . . , (x (t) n(t) , y (t) n(t)
) be the feature vectors and the labels of the training data set. We assume that every x(t)
i lies in a d-dimensional space
denoted by X ⇢ Rd. Following standard terminologies in statistical learning, we assume all data are drawn from some unknown distribution supported on X ⇥ Y , where Y ✓ R is the label space. Denote the underlying data distribution as P(t).
For our result in Section 4.1, we consider feedforward neural networks, and our results can also be extended to convolutional neural networks (e.g., using ideas from Long and Sedghi (2020)). Consider an L layer neural network. For each layer i from 1 to L, let i be the activation function and let Wi be the weight matrix at layer i. Given an input z to the i-th layer, the output is then denoted as i(z) = i(Wiz). Thus, the final output of the network is given by
fW (x) = L L 1 · · · 1(x), for any input x 2 X , where we use W = [W1, . . . ,WL] to include all the parameters of the network for ease of notation. The prediction error of fW is measured by a loss function `(·) that is both convex and 1-Lipschitz.
L (t)(fW ) = E (x,y)⇠P(t) [`(fW (x), y)] . (1)
Suppose we have access to a pre-trained source model Ŵ (s). Using Ŵ (s) as an initialization, we then fine-tune the layers Ŵ (s)1 , . . . , Ŵ (s) L to predict the target labels.
Applications. We consider seven image classification data sets described in Table 1. We use ResNets pre-trained on ImageNet as the initialization Ŵ (s) (Russakovsky et al., 2015; He et al., 2016). We
perform fine-tuning using the pre-trained network on the above data sets. See Section 5.1 for further description of the training procedure.
Modeling label noise. In many settings, fine-tuning is applied to a target task whose labels may contain noise; for example, if the target labels are created using weak supervision techniques (Ratner et al., 2019; Saab et al., 2021). To capture such settings, we denote a noisy data set as (x(t)1 , ỹ (t) 1 ), . . . , (x (t) n(t) , ỹ (t) n(t) ), where ỹ(t) i is a noisy version of y(t) i
. We consider two types of label noise: independent noise and correlated noise. We say that the label noise is independent if it is independent of the input feature vector
Pr(ỹ(t) i = k|y(t) i = j, x(t) i ) = Pr(ỹ(t) i = k|y(t) i = j) = ⌘j,k
for some fixed ⌘j,k between 0 and 1. On the other hand, we say that the label noise is correlated if it depends on the input feature vector (i.e. the above equation does not hold).

heading: 4 Our proposed approaches
text: Given the problem setup described above, next, we study the generalization and robustness properties of regularized fine-tuning methods. First, we present a PAC-Bayes generalization bound for regularized fine-tuning. This result motivates us to evaluate two empirical measures: the fine-tuned distance in each layer and the perturbed loss of the fine-tuned model. Second, we consider fine-tuning from noisy labels. We show that layer-wise regularization prevents the model from memorizing the noisy labels. We then suggest injecting predictions of the model during training, similar to self-training and pseudo-labeling. Finally, we incorporate both components into our regularized self-labeling approach, blending the strengths of both to improve the generalization performance and the robustness of fine-tuning.

heading: 4.1 Fine-tuning and regularization
text: We consider the following regularized fine-tuning problem, which constrains the network from traveling too far from the pre-trained initialization Ŵ (s) (Li et al., 2018a,b; Gouk et al., 2021).
Ŵ argmin L̂(t)(fW ) (2)
s.t. kWi Ŵ (s) i kF  Di, 8 i = 1, . . . , L. (3)
Above, Di is a hyperparameter that constrains how far the i-th layer Wi can travel from the pre-trained initialization Ŵ (s)
i . Previous works have observed
that stronger regularization reduces the generalization gap during fine-tuning (cf. Figure 2). Next, we analyze the generalization error of Ŵ , that is, L (t)(f
Ŵ ) L̂(t)(f Ŵ ), where L(t)(f Ŵ ) is the test
loss of f Ŵ according to equation (1) and L̂(t)(f Ŵ ) is the empirical loss of f Ŵ on the training data set.
PAC-Bayesian analysis. We begin by analyzing the generalization error of f
Ŵ using PAC-Bayesian
tools (McAllester, 1999a,b). This departs from the previous work of Gouk et al. (2021), which bases on their analysis using Rademacher complexity. This change of perspective is inspired by several recent works that have found PAC-Bayesian bounds to better correlate with empirical performance than Rademacher complexity bounds (Jiang et al., 2020). We refer the interested reader to Section A for further references from this line of work. After presenting our result, we provide empirical measures of our result and discuss the practical implication.
Theorem 4.1 (PAC-Bayes generalization bound for fine-tuning). Suppose for every i = 1, . . . , L, kŴ
(s) i k2  Bi, for a fixed Bi > 1. Suppose the feature vectors in the domain X are all bounded: kxk2  C1 for every x 2 X , for some C1 1. Finally, suppose the loss function `(·) is 1-Lipschitz and bounded from above by a fixed constant C2. Under these conditions, let fŴ be the minimizer of regularized fine-tuning, solved from problem (3). Let " > 0 be an arbitrary small value. Let H be the maximum over the width over all the L layers and the input dimension d. Then, with probability at least 1 2 for some small > 0, the expected loss L(t)(f Ŵ ) is upper bounded by
L (t)(f Ŵ )  L̂(t)(f Ŵ ) + "+ C2
s 36 "2 ·C21H log(4LHC2) ⇣ PL i=1 QL j=1(Bj+Dj) Bi+Di ⌘2⇣PL i=1 D 2 i ⌘ +3 ln n (t) +8
n(t) . (4)
Relation to prior works. Compared to the result of Gouk et al. (2021), we instead proceed by factoring the generalization error into a noise error (i.e. the " term in equation (4)) and a KL-divergence between Ŵ (s) and Ŵ (i.e. the final term in equation (4)). The proof of Theorem 4.1 is based on Neyshabur et al. (2018) and is presented in Appendix A. The difference between Theorem 4.1 and the result of Neyshabur et al. (2018) is that our result is stated for the (e.g. cross-entropy) loss function `(·) whereas Neyshabur et al. (2018) states the result using the soft margin loss.
Our result suggests that the fine-tuned distances {Di}Li=1 and the perturbed loss (more precisely EU ⇥ `(f Ŵ+U (x), y) ⇤
where every entry of U is drawn from N (0, 2)) are two important measures for fine-tuning. Next, we empirically evaluate these two measures on real-world data sets.
Fine-tuned distances {Di}Li=1. First, we present our empirical finding for the fine-tuned distances in each layer. We fine-tune a ResNet-18 model (pre-trained on ImageNet) on seven data sets and calculate the Frobenius distance between Ŵ (s)
i and Ŵi for every layer i.
Figure 3a shows the ratio of the fine-tuned distances to the pre-trained weights in each layer. First, we observe that the fine-tuned distances are relatively small compared to the pre-trained network. This means that fine-tuning stays within a small local region near the pre-trained model. Second, we find that Di varies across layers. Di is smaller at lower layers and is larger at higher layers. This observation aligns with the folklore intuition that different layers in Convolutional neural networks play a different role (Guo et al., 2019). Bottom layers extract higher-level representations of features
and stay close to the pre-trained model. Top layers extract class-specific features and vary among different tasks. Thus, top layers travel further away from the pre-trained model.
Based on the above empirical finding, we propose layer-wise constraints as regularization. In particular, the constraints for the bottom layers should be small, and the constraints for the top layers should be large. In practice, we find that an exponentially increasing scheme involving a base distance parameter D and a scale factor works well. This is summarized in the following regularized fine-tuning problem:
Ŵ argmin L̂(t)(fW ) (5)
s.t. kWi Ŵ (s) i kF  D · i 1 , 8 i = 1, . . . , L. (6)
where the scale factor > 1 according to Figure 3a. The previous work of Gouk et al. (2021) uses the same distance parameter in equation (6), i.e., Di = D, 8i = 1, . . . , L. In our experiments, we have observed that such constant regularization constraints are only active at the top layers. With layer-wise regularization, we instead extend the constraints to the bottom layers as well. In Table 8 (cf. Section B.3), we compare the value of P L
i=1 D 2 i between constant regularization and layer-wise regularization. We find that layer-wise regularization leads to a smaller value of P L
i=1 D 2 i , thus
indicating a tighter generalization bound according to Theorem 4.1. Perturbed loss EU h `(fŴ+U(x), y) i . Second, we compare the perturbed loss of models finetuned from random, pre-trained, and adversarially robust pre-trained model initializations. The perturbed loss is measured by the average loss on the training set, after perturbing each entry of the layer weights by a Gaussian random variable N(0, 2).
In Table 3b, we report the perturbed losses for both the CUB-200-2011 and the Indoor data set. The results show that models fine-tuned from pre-trained initializations are more stable than models fine-tuned from random initializations.
Explaining why adversarial robustness helps fine-tuning. Recent work (Salman et al., 2020) found that performing adversarial training in the pre-training phase leads to models that transfer better to downstream tasks. More precisely, the adversarial training objective is defined as
L̃ (s) adv(✓) = min
W
1
n(s)
n (s)X
i=1
max k k2"
`(fW (x (s) i + ), y(s) i ). (7)
Let Ŵ (s)adv be a fine-tuned neural network using a pre-trained model from adversarial training. We measure the perturbed loss of Ŵ (s)adv using the pre-trained model provided by Salman et al. (2020) on the CUB-200-2011 and Indoor data set. Table 3b shows that f
Ŵ (s) adv often incurs lower perturbed losses than models fine-tuned from random and pre-trained initializations. In Section B.3, we additionally observe that the layers of Ŵ (s)adv generally have lower Frobenious norms compared to Ŵ
(s). Thus, the improved noise stability and the smaller layer-wise norms together imply a tighter generalization bound for adversarial training. We leave a thorough theoretical analysis of adversarial training in the context of fine-tuning to future works.

heading: 4.2 Fine-tuning and robustness
text: Next, we extend our approach to fine-tuning from noisy labels. We compare fine-tuning with and without regularization on a target task with independent label noise to motivate our approach. In Figure 4, we plot the training and test accuracy curves on the Indoor dataset with two noise rates. We observe that the test accuracy increases at first, meaning the network learns from the correct labels during the first few epochs. As the learning process progresses, the test accuracy of fine-tuning decreases, implying the network is “overfitting” to noisy labels. Furthermore, the gap between the training and test accuracy curves is more than 20%. Similar observations have been presented in prior works (Zhang et al., 2016; Li et al., 2018c; Liu et al., 2020), where over-parametrized neural networks can memorize the entire training set. One way to mitigate memorization is to explicitly reduce model capacity (Wu et al., 2020; Yang et al., 2021). Another way is via explicit regularization since the distance constraint significantly reduces the training-test accuracy gaps and improves test accuracy by a significant margin. Based on our empirical observation, the neural network has some discriminating power during the early fine-tuning phase. Thus, we can leverage the model predictions to relabel incorrect labels and reweight data points with incorrect labels. We describe the two components next.
Self label-correction. We propose a label correction step to augment the number of correct labels. We leverage the discriminating power of a network during the early phase and correct data points for which the model has high confidence. Concretely, let pi be the prediction of the i-th data point. Given a confidence threshold pt at epoch t, if max(pi) pt and argmax(pi) 6= ỹ (t) i
, we identify data point i as noisy and relabel it to
ỹ (t) i = argmax(pi).
Figure 5a illustrates the precision of this step. We define precision as the number of correctly relabeled data points divided by the total number of data points whose labels are relabeled. We observe that precision is high at the beginning (around 60%) and gradually decreases in later epochs. Moreover, we find precision increases with regularization. This suggests an intricate interaction between regularization and self-labeling during fine-tuning. We elaborate on the role of regularization in label correction by looking at the number of relabeled data points in Section B.3.
Self label-reweighting. We incorporate a soft data removal step to prevent the model from overfitting to noisy labels. We identify data points as noisy if their loss values are large and thus down-weight them during training. The model will prioritize data points with correct labels in its gradient by assigning smaller weights to large-loss data points. More precisely, given a data point (x(t)
i , ỹ (t) i
), we reweight it by !i = exp( `(fW (x (t) i ), ỹ(t) i )/⌧) where ⌧ is a temperature parameter. In the implementation, we normalize the weights of every data point in every mini-batch B:
L̂B(fW ) = 1P
i2B !i
X i2B !i · `(fW (x (t) i ), ỹ(t) i ).
Algorithm 1 Regularized self-labeling (REGSL) Input: Input nosiy data (x(t)1 , ỹ (t) 1 ), . . . , (x (t) n(t) , ỹ(t) n(t) ), network model fW , constraint distance D, distance scale factor , re-weight start step Er , re-weight temperature ⌧ , label correction start step Ec, label correction threhold pt, and fine-tuning steps T 1: Initialize model parameters with pre-trained weights W = Ŵ s and the start step t = 0 2: while t < T do 3: Fetch a mini-batch of data {(x(t)i , ỹ (t) i )} m i=1
4: for each data point i in the batch do 5: Calculate the predicted probability pi = softmax(fW (x(t)i )) 6: if t > Ec and max(pi) > pt and argmax(pi) 6= ỹ(t)i then 7: Correct the label ỹ(t)i = argmax(pi) 8: end if 9: Calculate loss `i = `(fW (x(t)i ), ỹ (t) i )
10: if t > Er then Calculate the weight !i = exp( `i/⌧) else Set the weight !i = 1 11: end for 12: Update fW by SGD on L = 1P
i !i
Pm i !i`i and t = t+ 1
13: Project the weights Wi of each layer inside the layer-wise constraint region as in Equation 6 14: end while
Figure 5b compares the average weight of data points with the correct label to data points with an incorrect label. We find that with our reweighting scheme, the average weight of incorrectly labeled data points is much lower than the average weight of correctly labeled data points. Thus, the reweighting scheme expands the gradient of correctly labeled data points, thus reducing the fraction of noisy data points in the training set.
The pseudo-code for our final approach, which combines both layer-wise regularization and selflabeling, is presented in Algorithm 1.

heading: 5 Experiments
text: We evaluate our proposed algorithm in a wide range of tasks and pre-trained networks. First, we show that our algorithm outperforms the baselines by 1.75% on average over seven transfer learning tasks and can generalize to a more distant task of medical images. Second, our algorithm improves the robustness of fine-tuning in the presence of noisy labels compared to previous methods. Our algorithm improves over numerous baselines by 3.56% on average under both independent and correlated label noise. Moreover, our approach can achieve a similar performance boost for finetuning vision transformer models on noisy labeled data. Finally, we show our algorithms improve over fine-tuning baselines by 0.75% on average for the related task of few-shot classification. Our extensive evaluation confirms that our approach applies to a broad range of settings, validating our algorithmic insights. Due to space constraints, we defer the ablation studies to Section B.3. Our code is available at https://github.com/NEU-StatsML-Research/Regularized-Self-Labeling.

heading: 5.1 Experimental setup
text: Data sets. First, we evaluate fine-tuning on seven image classification data sets and one medical image data set. The statistics of the seven image data sets are described in Table 1. For the medical imaging task, we consider the ChestX-ray14 data set contains 112120 frontal-view chest X-ray images labeled with 14 different diseases (Wang et al., 2017; Rajpurkar et al., 2017). Second, we evaluate Algorithm 1 under two different kinds of label noise to test the robustness of fine-tuning. We selected the MIT-Indoor data set (Sharif Razavian et al., 2014) as the benchmark data set and randomly flipped the labels of the training samples. The results for the other data sets are similar, which can be found in Section B.2. We consider both independent random noise and correlated noise in our experiments. We generate the independent random noise by flipping the labels uniformly with a given noise rate. We simulate the correlated noise by using the predictions of an auxiliary network as noisy labels. Section B.1 describes the label flipping process. For few-shot image classification, we conduct experiments on the miniImageNet benchmark (Vinyals et al., 2016) following the setup of Tian et al. (2020).
Architectures. We use the ResNet-101 (He et al., 2016) network for transfer learning tasks and ResNet-18 (He et al., 2016) network for ChestX-ray data set. We use ResNet-18 network for the labelnoise experiments and extend the our algorithm to the vision transformer (ViT) model (Dosovitskiy et al., 2020). The ResNet models are pre-trained on ImageNet (Russakovsky et al., 2015) and the ViT model is pre-trained on ImageNet-21k data set (Dosovitskiy et al., 2020). For the few-shot learning tasks, we use ResNet-12 pre-trained on the meta-training data set as in the previous work (Tian et al., 2020). We set four different values of Di for the four blocks of the ResNet models in our algorithm. We describe the fine-tuning procedure and the hyperparameters in Section B.1.
Baselines. For the transfer learning tasks, we use the Frobenius norm for distance regularization. We present ablation studies to compare the Frobenius norm and other norms in Section B.3. we compare our algorithm with fine-tuning with early stop (Fine-tuning), fine-tuning with weight decay (`2- Norm), label smoothing (LS) method formulated in Müller et al. (2019), `2-SP (Li et al., 2018b), and ` 2-PGM (Gouk et al., 2021). For testing the robustness of our algorithm, in addition to the baselines described above, we adopt several baselines that have been proposed in the supervised learning setting, including GCE (Zhang and Sabuncu, 2018), SCE (Wang et al., 2019), DAC (Thulasidasan et al., 2019), APL (Ma et al., 2020), ELR (Liu et al., 2020) and self-adaptive training (SAT) (Huang et al., 2020). We compare with the same baselines to transfer learning in few-shot classification tasks. We describe the implementation and hyper-parameters of baselines in Section B.1.

heading: 5.2 Experimental results
text: Improved regularization for transfer learning. We report the test accuracy of fine-tuning ResNet101 on seven data sets in Table 2. We observe that our method performs the best among six regularized fine-tuning methods on average. Our proposed layer-wise regularization method provides an improvement of 1.76% on average over distance-based regularization (Gouk et al., 2021). In particular, our approach outperforms `2-PGM by 2 ⇠ 3% on both Stanford-Dogs and MIT-Indoor data sets. This suggests that adding appropriate distance constraints for all the layers is better than applying only one constraint to the top layers. In Table 10 (cf. Section B.3), we note that using the MARS norm of Gouk et al. (2021) yields comparable results to using the `2 norm with our approach.
Next, we apply our approach to medical image classification, a more distant transfer task (relative to the source data set ImageNet). We report the mean AUROC (averaged over predicting all 14 labels) on the ChestX-ray14 data set in Table 6a (cf. Section B.2). With layer-wise regularization, we see an 0.39% improvement over the baseline methods. Finally, we extend our approach to text classification. The results are consistent with the above and can be found in Table 7 (cf. Section B.2).
Improved robustness under label noise. Next, we report the test accuracy of our approach on the Indoor data set with independent and correlated noise in Table 3. We find that our approach consistently outperforms baseline methods by 1 ⇠ 3% for various settings involving label noise. First, our method (REGSL) improves the performance by over 4% on average compared to distance-based regularization (Gouk et al., 2021). This result implies our method is more robust to label noise in the training labels. Second, our method outperforms previous supervised training methods by 3.56% on average. This result suggests that regularization is critical for fine-tuning. Taken together, our results suggest that regularization and self-labeling complement each other during fine-tuning. This is reinforced by our ablation study in Section B.3; We study the influence of each component of our algorithm and find that removing any component degrades performance.
We further apply our approach to fine-tuning pre-trained ViT (Dosovitskiy et al., 2020) from noisy labels, under the same setting as Table 3. Table 6b shows the result (cf. Section B.2). First, we find that
our approach improves upon the best regularization methods by 1.17% averaged over two settings. Second, we find that our approach also improves upon self-labeling by 13.57% averaged over two settings. These two results again highlight that both regularization and self-labeling contribute to the final result. While regularization prevents the model from overfitting to the random labels, self-labeling injects the belief of the fine-tuned model into the noisy data set.
Table 4: Test accuracy with 95% confidence interval for fine-tuning ResNet-12 over 600 meta-test splits of miniImageNet.
Methods miniImageNet
5-way-1-shot 5-way-5-shot
Fine-tuning 60.56 ± 0.78 76.42 ± 0.36 `2-Norm 60.93 ± 0.81 76.57 ± 0.55 LS 61.31 ± 0.77 76.73 ± 0.62 `2-SP 61.48 ± 0.76 77.02 ± 0.62 `2-PGM 61.35 ± 0.83 77.33 ± 0.59 REGSL (ours) 61.71 ± 0.77 78.03 ± 0.54
Extension to few-shot classification. We extend our approach to a few-shot image classification task. We compare our approach to the baseline regularization methods. In Table 4, we report the average accuracy of 600 sampled tasks from the meta-test split of the miniImageNet benchmark (Vinyals et al., 2016). We find that our layer-wise regularization method achieves 0.75% improvement on average compared to previous regularization methods. Additionally, regularization methods generally improve the performance of fine-tuning by over 1%.
Ablation studies. We study the influence of removing each component from our algorithm. Results shown in Table 11 (cf. Section B.3) suggest that they all degrade performance. Thus, all three components in Algorithm 1 contribute to the final performance. The importance of each component depends on the noise rate. In particular, if the noise rate is higher than 40%, label-correction is the most critical component. Other ablation studies we present include: comparing the `2 norm and the MARS norm in our algorithm and comparing the distance parameters between layer-wise and constant regularization. We leave the details to Section B.3.

heading: 6 Conclusion
text: This paper studied regularization methods for fine-tuning as well as their robustness properties. We investigated the generalization error of fine-tuning using PAC-Bayesian techniques. This leads to two empirical measures, which we empirically computed. The analysis inspired us to consider layer-wise regularization for fine-tuning. This approach performs well on both pre-trained ResNets and ViTs; we have found the fine-tuned distance using vanilla fine-tuning and encoded the distance patterns in layer-wise regularization. We then evaluated the performance of regularized fine-tuning from noisy labels. We proposed a self label-correction and label-reweighting approach in the noisy setting. We found that regularization and self-labeling complement each other in our experiments, leading to significant improvement over previous methods, which either use regularization methods or use self-labeling methods but not both. We discuss the limitation of our work in Section C.
Negative societal impacts. Our work concerns the theory and the empirical performance of finetuning for transfer learning. Due to its technical nature, there is very little negative societal impact. Since transfer learning is commonly used in practice, one avenue that deserves attention for these applications is to carefully evaluate bias or fairness metrics before applying our proposed approaches.
Acknowledgment. Thanks to the anonymous reviewers and the area chairs for carefully reading through our work and providing constructive feedback. Thanks to the program committee chairs for communicating with us regarding an omitted factor in equation (14), which we have added in the current version. HZ is grateful to Sen Wu and Chris Ré for stimulating discussions. HZ is supported by Northeastern University’s Khoury seed/proof-of-concept grant.

Review summaries:
This paper studies the generalization and robustness of regularized fine-tuning. They present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the pre-trained model. They empirically measure these quantities to analyze the behavior of fine-tuning. They present an algorithm that interpolates between two components: (i) regularization over the amount of distance traveled in each layer; (ii) iteratively correcting mislabeled data points that the model is highly confident and down-weighting less confident data points. 

This paper includes a PAC-Bayesian analysis of distance-based regularisation during fine-tuning. Using this analysis and supplementary experiments, the authors propose a new means for tuning hyperparameters for distance-based regularisation with constraints and heuristics for dealing with label noise during fine-tuning. The experiments show that the method works slightly better than previous approaches designed for both fine-tuning and training with label noise.

The paper proposes a new regularization method for training and fine-tuning large neural networks. The paper proposes to constrain the maximum change in network parameters during fine-tuning. This constraint is different in each layer. Experiments demonstrate that the proposed method outperforms previous approaches.

The paper analyses the fine-tuning of neural networks. It provides a PAC-Bayes bound for the generalization of a fine-tuned network. The paper then provides three tricks to improve finetuning: layerwise regularization, label correction, and label removal. The resulting fine-tuning algorithm improves performance on various image classification tasks, including those with artificially-added label noise.

This paper studies the aspects of fine-tuning a model (pre-trained on a separate dataset) in order to learn a new task. It has the following contribution:  (i) Generalization properties of fine-tuning explained by the PAC-Bayes generalization bound that depends on two terms: (a) distance between each layer from the pre-trained model and (b) noise stability of the pre-trained model.  (ii) Inspired by the generalization analysis, the paper presents an algorithm to perform fine-tuning. This algorithm includes (a) regularization term that computes the distance between the current parameters and the pre-trained model, and (b) iteratively corrects mislabeled examples where model has high confidence and down-weighting less confident examples.  (iii) Proposed algorithm is evaluated on a suite of benchmark tasks including transfer learning and few shot classification tasks.