Why pre-training is beneficial for downstream classification tasks?

## Abstract

Abstract Pre-training has exhibited notable benefits to downstream tasks by boosting accuracy and speeding up convergence, but the exact reasons for these benefits still remain unclear.
To this end, we propose to quantitatively and explicitly explain effects of pre-training on the downstream task from a novel game-theoretic view, which also sheds new light into the learning behavior of deep neural networks (DNNs).
Specifically, we extract and quantify the knowledge encoded by the pre-trained model, and further track the changes of such knowledge during the fine-tuning process.
Interestingly, we discover that only a small amount of pre-trained model’s knowledge is preserved for the inference of downstream tasks.
However, such preserved knowledge is very challenging for a model training from scratch to learn.
Thus, with the help of this exclusively learned and useful knowledge, the model fine-tuned from pre-training usually achieves better performance than the model training from scratch.
Besides, we discover that pre-training can guide the fine-tuned model to learn target knowledge for the downstream task more directly and quickly, which accounts for the faster convergence of the fine-tuned model.

## 1 Introduction

Pre-training is prevalent in nowadays deep learning, as it has brought great benefits to downstream tasks, including
improving the accuracy [, ], boosting the robustness [], speeding up the convergence [], and etc.
Naturally, a fundamental question arises: why pre-training is beneficial for downstream tasks?
Previous works have tried to answer this question from different perspectives.
For example, [, , ] attributed the benefits of pre-training to a flat loss landscape.
[] concluded that the improved accuracy was a result of unsupervised pre-training acting as a regularizer.

Unlike above perspectives for explanations, we aim to present an in-depth analysis to answer the above question from a new perspective.
That is, we quantify the knowledge encoded by the pre-trained model, and further analyze the effects of such knowledge on the downstream tasks.
In this way, we can provide insightful and accurate explanations for the benefits brought by pre-training, which also sheds new light into the fine-tuning/learning behavior of DNNs.

To this end, we extract the knowledge encoded in the pre-trained model based on the interaction between different input variables [, , ], because the DNN usually lets different input variables interact with each other to construct concepts for inference, rather than utilize each single variable for inference independently.
As Fig. 1(a) shows,
the DNN encodes the co-appearance relationship (interaction) between different image patches in $S=\{\text{{mouth}},\text{{ear}},\text{{eye}}\}$ of the input image $\bm{x}$ to form the dog face concept $S$ for inference.
Only when all three patches in $S$ are all present, the interaction is activated and makes a numerical contribution $I(S|\bm{x})$ to the network output $y$.
The absence/masking(^1^1footnotemark: 1) of any image patch will deactivate the interaction, and the numerical contribution is removed, i.e., $I(S|\bm{x})=0$.

More crucially,
[, ] have empirically verified and
[] have theoretically proven the sparsity property and the universal-matching property of interactions, i.e.,
given an input sample $x$, a well-trained DNN usually encodes a small number of interactions between different input variables, and the network output $y$ can be well explained as the numerical contributions of these interactions, $y=\sum_{S}I(S|\bm{x})$, as shown in Fig. 1(a).
Thus,
these two properties mathematically enables us to take interactions as the knowledge encoded by the DNN for inference.
Apart from these two properties,
the considerable discrimination power and high transferability across different models of interactions [] also provide supports for the faithfulness of using interactions to represent knowledge encoded in a DNN.
Please see Section 3.1 for detailed discussions.

Figure: Figure 1: (a) We use the interaction between different input variables to represent knowledge encoded by a DNN, because the network output is proven to be well explained as the sum of numerical contributions $I(S|\bm{x})$ of interactions. (b) Explaining benefits of pre-training by analyzing effects of pre-trained model’s knowledge on the downstream task.
Refer to caption: x1.png

In this way,
we use interactions to precisely quantify and comprehensively analyze how pre-trained model’s knowledge impacts the downstream classification task, so as to provide insightful explanations for two widely-acknowledged benefits of pre-training, i.e.,
boosting the classification performance and speeding up the convergence.
The following explanations may also guide some interesting directions on pre-training for future studies.

$\bullet$
Quantifying explicit changes of pre-trained model’s knowledge during the fine-tuning process.
We propose metrics to measure how pre-trained model’s knowledge is discarded and preserved by the fine-tuned model for the inference of the downstream task, in order to provide comprehensive analyses for the benefits of pre-training.
In experiments, we surprisingly discover that the fine-tuned model discards a considerable amount of pre-trained model’s knowledge,
especially extremely complex knowledge.
In contrast, the fine-tuned model only preserves a modest amount of pre-trained model’s knowledge that is discriminative for the inference of the downstream task.

$\bullet$
Explaining the superior classification performance of the fine-tuned model.
We discover that only little preserved knowledge can be successfully learned by a model training from scratch merely using a small-scale downstream-task dataset, because the preserved knowledge from the pre-trained model is acquired from an extremely large-scale dataset.
Thus, pre-training makes the fine-tuned model encode more exclusively-learned and discriminative knowledge for inference, which
partially responses to the better accuracy of the fine-tuned model.

$\bullet$
Explaining the accelerated convergence of the fine-tuned model.
Interestingly, we also observe that compared to the model training from scratch, pre-training guides the fine-tuned model more quickly and directly to encode target knowledge used for the inference of the downstream task, by proposing metrics to evaluate the learning speed of target knowledge and the stability of learning directions.
Thus, this answers faster convergence of the fine-tuned model.

Contributions of this paper are summarized as follows.
(1) We propose several theoretically verifiable metrics to quantify the knowledge encoded by the pre-trained model from a novel game-theoretic view.
(2) Based on the quantification of knowledge, we present an in-depth analysis to explain two benefits of pre-training.
(3) Experimental results on various DNNs and datasets verify our explanations, which reveals new insights into pre-training.

## 2 Related work

Explanation of pre-training.
Fine-tuning pre-trained models on downstream tasks to speed up convergence and boost performance has become a conventional practice in deep learning [, , , ].
Many works have attempted to analyze why pre-training is beneficial for downstream tasks from different perspectives.
Specifically,
[] discovered that the unsupervised pre-training acted as a regularizer, which improved the generalization power of the DNN.
Alternatively, a lot of studies explained the high accuracy [, ], the fast convergence speed in federated learning [, ], and the reduced catastrophic forgetting in continual learning [] of the fine-tuned models from the perspective of a flat loss landscape.
Additionally, [, ] explained the transferability of the pre-trained model to downstream tasks from the perspective of the feature space by performing the singular value decomposition.
In comparison, we present a comprehensive analysis to systematically unveil the essential reasons behind different benefits of pre-training, by quantifying the explicit effects of pre-trained model’s knowledge on the downstream task from a game-theoretic perspective.

Using interactions to explain the DNN.
In recent years, employing game-theoretic interactions to explain DNNs has become a newly emerging direction.
Specifically, [, , ] quantified interactions between different input variables to formulate the knowledge encoded by a DNN, whose faithfulness was further experimentally verified and theoretically ensured by [, , ].
Besides, a series of studies utilized the interaction to explain the representation capacity of DNNs, including the generalization power [, , ], adversarial robustness [], adversarial transferability [], the learning difficulty of interactions [, ], and the representation bottleneck [].
In comparison, this paper aims to provide insightful explanations for the benefits of pre-training to downstream tasks.

Quantifying the knowledge encoded by the DNN.
So far, there does not exist a formal and widely accepted method to quantify the knowledge encoded by a DNN.
A series of studies [, , ] employed the mutual information between input variables and the network output to quantify the knowledge in the DNN, but precisely measuring the mutual information was still significantly challenging [].
Besides, other studies employed human-annotated semantic concepts [, ] or automatically learned concepts [] to explain the knowledge in the DNN, but these works could not quantify the exact
changes of knowledge (i.e., the preservation of task-relevant knowledge and the discarding of task-irrelevant knowledge)
during the fine-tuning/training procedure.
In comparison, we use theoretically verifiable interactions to represent knowledge in the DNN, which enables us to explicitly quantify the exact effects of pre-trained model’s knowledge on the downstream task, so as to provide detailed explanations for the benefits of pre-training.

## 3 Explaining why pre-training is beneficial for downstream tasks

### 3.1 Preliminaries: using interactions to represent knowledge in DNNs

In this section, let us introduce the interaction metric, together with a set of interaction properties [, , ] as convincing evidence for the faithfulness of interaction-based explanations, so as to provide a straightforward and concise way to understand why pre-training is beneficial for downstream tasks.

Definition of interactions.
Given a DNN $v$ trained for the classification task and an input sample $\bm{x}=[x_{1},x_{2},\dots,x_{n}]$ composed of $n$ input variables, let $N=\{1,2,\dots,n\}$ represent the indices of all $n$ variables.
Let $v(\bm{x})\in\mathbb{R}$ denote the scalar output of the DNN or a certain output dimension of the DNN, where people can apply different settings for $v(\bm{x})$.
Here, we follow [] to set $v(\bm{x})$ as the confidence of classifying $\bm{x}$ to the ground-truth category $y^{\text{truth}}$ for multi-category classification tasks, as follows.

$$ $\displaystyle v(\bm{x})=\mathrm{log}\frac{p({y=y^{\text{truth}}|\bm{x}})}{1-p( {y=y^{\text{truth}}|\bm{x}})}.$ (1) $$

Then,
the contribution of the interaction between a subset $S\subseteq N$ of input variables to the network output $v$ is calculated by the Harsanyi Dividend [], a typical metric in game theory, as follows.

$$ $\displaystyle I(S|\bm{x})=\sum\nolimits_{T\in S}(-1)^{|S|-|T|}\cdot v(\bm{x}_{ T}),$ (2) $$

where $\bm{x}_{T}$ denotes a masked input sample crafted by masking variables in $N\setminus T$ to baseline values(^1^11We follow the widely-used setting in [] to set the baseline value of each variable as the mean value of this variable over all samples in image classification, and follow [] to set the baseline value of each word as a special token (e.g., [MASK] token) in natural language processing.) and keeping variables in $T$ unchanged.
Let us take the sentence $\bm{x}=$"he has a green thumb" as a toy example to understand (2).
The DNN encodes the interaction between words in a subset $S=\{\text{green},\text{thumb}\}$ with a numerical contribution $I(S)$ to push the DNN’s inference towards the meaning of a “good gardener.”
This numerical contribution is computed as $I(S|\bm{x})=v(\{\text{green},\text{thumb}\})-v(\{\text{green}\})-v(\{\text{
thumb}\})+v(\bm{x}_{\emptyset})$, where $\bm{x}_{\emptyset}$ denotes all words in $\bm{x}$ are masked.

Understanding the physical meaning of interactions.
Each interaction with a numerical contribution $I(S|\bm{x})$ represents a collaboration (AND relationship) between input variables in a subset $S$.
As in the aforementioned example, the co-appearance of two words in $S=\{\text{green},\text{thumb}\}$ constructs a semantic concept of “good gardener,” and makes a numerical contribution $I(S|\bm{x})$ to the network output.
The absence (masking) of any words in $S$ will inactivate this semantic concept and remove its corresponding interaction contribution, i.e., $I(S|\bm{x})=0$.

Quantifying the knowledge encoded by the DNN.
The proven sparsity property and universal-matching property of interactions enable us to use interactions to represent knowledge encoded by the DNN.
Specifically,
[] have proven that under some common conditions(^2^22Please see Appendix for the detailed introduction of common conditions.), a well-trained DNN usually encodes very sparse interactions for inference, which is also experimentally verified by [, ].
In other words, although there exists $2^{n}$ different subsets(^3^33To reduce the computational cost, we select a relatively small number of input variables (image patches or words) to calculate interactions in experiments. Please see Appendix for details.) $S\subseteq N$ in total, only a small set $\Omega_{\text{salient}}$ of interactions make salient contributions to the network output, i.e.,
$\Omega_{\text{salient}}=\{S\subseteq N,|I(S|\bm{x})|>\tau\}$(^4^44$\tau$ is a small constant to select salient interactions, and we set $\tau=0.05\cdot\max_{S}|I(S|\bm{x})|$ in experiments.), subject to $|\Omega_{\text{salient}}|\ll 2^{n}$.
Whereas, a large number of interactions contribute␣‌negligibly $I(S|\bm{x})\approx 0$ to the network output, which can be considered as noisy patterns.
Thus, the network output $v(\bm{x})$ can be well approximated by a small number of salient interactions, i.e.,

$$ $\displaystyle v(\bm{x})=\sum\nolimits_{S\subseteq N}I(S|\bm{x})\approx\sum \nolimits_{S\in\Omega_{\text{salient}}}I(S|\bm{x}).$ (3) $$

###### Theorem 3.1 ( universal-matching property of interactions ) .

Given an input sample $\bm{x}$, there are $2^{n}$ differently masked samples $\{\bm{x}_{T}|T\subseteq N\}$.
[] have proven that network outputs $v(\bm{x}_{T})$ on all $2^{n}$ masked samples $\bm{x}_{T}$ can be universally matched by a small number of salient interactions.

$$ $\displaystyle v(\bm{x}_{T})=\sum\nolimits_{S\subseteq T}I(S|\bm{x})\approx\sum \nolimits_{S\subseteq T\&S\in\Omega_{\text{salient}}}I(S|\bm{x}).$ (4) $$

Theorem 3.1 indicates we can use a small set of salient interactions to well explain
the network output $v(\bm{x}_{T})$ on anyone $\bm{x}_{T}$ of all $2^{n}$ masked samples.
Thus, according to the Occam’s Razor [], we can roughly consider each salient interaction as the knowledge encoded by the DNN for inference, rather than a mathematical trick with unclear physical meanings.

Faithfulness of using interactions to represent the knowledge of the DNN.
Although nowadays there exist various methods to define/quantify the knowledge encoded by the DNN,
a set of theoretically proven and empirically verified interaction properties ensure the faithfulness of the interaction-based explanation.
Specifically, the universal-matching property in Theorem 3.1 and the sparsity property in (3) have mathematically guaranteed that interactions can faithfully explain the output of DNNs.
Besides, [] have experimentally verified the transferability property and the discriminative property of interactions.
That is, interactions exhibit considerable transferability across samples and across models, and have remarkable discrimination power in classification tasks.
Additionally, [] have proven that interactions satisfy seven mathematical properties.
Please see Appendix for detailed discussions.

###### Theorem 3.1 ( universal-matching property of interactions ) .

### 3.2 Quantifying the effects of pre-training on downstream tasks

Despite the ubiquitous utilization and great success of pre-trained models, it still remains mysterious why such models can help the fine-tuned model achieve superior classification performance and converge faster(^5^55Experimental results in Appendix verify that the fine-tuned model achieves higher classification accuracy and converges to a lower loss more quickly than the model training from scratch.), compared to training from scratch.
Thus, to systematically and precisely unveil the reasons behind these two benefits, we propose several metrics based on interactions to explicitly quantify the knowledge of the pre-trained model that is utilized for the inference of the downstream task, and further explain effects of such knowledge on the fine-tuning process.
These explanations also provide some new insights into the learning/fine-tuning behavior of the DNN.

#### 3.2.1 Quantifying changes of pre-trained model’s knowledge during the fine-tuning process

Explaining the precise effects of pre-training on downstream tasks still remains a significant challenge, because interactions (knowledge) directly extracted from the pre-trained model’s output $v$ cannot be used for explanation.
This is due to that the pre-trained model is usually trained on an extremely large-scale dataset with extensive training samples, whose network output often encodes a vast amount of diverse knowledge.
Such knowledge can be further categorized into knowledge that can be used for inference of the downstream task (e.g., some general and common knowledge), and knowledge that cannot be applicable to the downstream task (e.g., knowledge only related to the inference of the pre-trained task).
Thus, we need to extract and quantify the knowledge of the pre-trained model that is used for the inference of the downstream task for explanation, so as to ensure our explanation will not be affected by other irrelevant knowledge.

To this end, we employ the linear probing method [, , , ], a commonly used technique, to extract pre-trained model’s knowledge that is used for the downstream task.
Specifically, given an input sample $\bm{x}$ and a pre-trained model, we freeze all its network parameters, and use the feature $f(\bm{x})$ of its penultimate layer (i.e., the layer preceding the classifier of the pre-trained model) to train a new linear classifier $W^{T}f(\bm{x})+b$ for the same downstream task as the fine-tuned model(^6^66Please see Appendix for the details of training the linear classifier.).
Then, we define the following function $v_{\text{pretrain}}$ to quantify the pre-trained model’s knowledge used for the inference of the downstream task $I(S|\bm{x},v_{\text{pretrain}})$, where $y_{\text{pretrain}}$ denotes the label predicted by the linear classifier.

$$ $\displaystyle v_{\text{pretrain}}=\mathrm{log}\frac{p({y_{\text{pretrain}}=y^{ \text{truth}}|\bm{x}})}{1-p({y_{\text{pretrain}}=y^{\text{truth}}|\bm{x}})}.$ (5) $$

In this way, the classification score $v_{\text{pretrain}}$ enables us to provide a thorough insight into the effects of the pre-trained model on the downstream task, by quantifying the changes of its knowledge $I(S|\bm{x},v_{\text{pretrain}})$ during the fine-tuning process.
Specifically, we disentangle the knowledge $I(S|\bm{x},v_{\text{pretrain}})$ into two components, including the knowledge preserved by the fine-tuned model for inference and the discarded knowledge.
In this way, we define the preserved knowledge $K_{\text{preserve}}$ as the strength of the interaction shared by both the pre-trained model and the fine-tuned model.
The discarded knowledge $K_{\text{discard}}$ is defined as the strength of the interaction that is encoded by the pre-trained model, but discarded by the fine-tuned model, as follows.

$$ $\displaystyle I(S|\bm{x},v_{\text{pretrain}})$ $\displaystyle=\text{sign}(I(S|\bm{x},v_{\text{pretrain}}))\cdot(K_{\text{ preserve}}(S|\bm{x})+K_{\text{discard}}(S|\bm{x})),$ (6) $\displaystyle K_{\text{preserve}}(S|\bm{x})$ $\displaystyle=\mathbbm{1}(\Gamma_{\text{pretrain}}^{\text{finetune}}(S|\bm{x}) >0)\cdot\min(|I(S|\bm{x},v_{\text{pretrain}})|,|I(S|\bm{x},v_{\text{finetune}} )|),$ $\displaystyle K_{\text{discard}}(S|\bm{x})$ $\displaystyle=|I(S|\bm{x},v_{\text{pretrain}})|-K_{\text{preserve}}(S|\bm{x}),$ $$

where $\Gamma_{\text{pretrain}}^{\text{finetune}}(S|\bm{x})=I(S|\bm{x},v_{\text{
pretrain}})\cdot I(S|\bm{x},v_{\text{finetune}})$ measures whether the interaction encoded by the pre-trained model $I(S|\bm{x},v_{\text{pretrain}})$ and the interaction encoded by the fine-tuned model $I(S|\bm{x},v_{\text{finetune}})$ have the same effect.
$v_{\text{finetune}}$ is calculated based on the fine-tuned model according to (1).
$\mathbbm{1}(\cdot)$ is the indicator function. If the condition inside is valid, $\mathbbm{1}(\cdot)$ returns $1$, and otherwise 0 0.

Similarly, we also disentangle the knowledge encoded by the fine-tuned model into two components, including the knowledge inherited from the pre-trained model $K_{\text{preserve}}(S|\bm{x})$, and new knowledge learned by the fine-tuned model itself to adapt the downstream task.
Such a disentanglement helps us gain an insightful understanding of the fine-tuning behavior of the DNN, and also enables us to
seek a deep exploration of the superior classification performance of the fine-tuned model in Section 3.2.2.
Specifically, we define the knowledge $K_{\text{new}}(S|\bm{x})$ newly learned by the fine-tuned model as the strength of the interaction that is encoded by the fine-tuned model, but is not present in the pre-trained model.

$$ $\displaystyle I(S|\bm{x},v_{\text{finetune}})$ $\displaystyle=\text{sign}(I(S|\bm{x},v_{\text{finetune}}))\cdot(K_{\text{ preserve}}(S|\bm{x})+K_{\text{new}}(S|\bm{x})),$ (7) $\displaystyle K_{\text{new}}(S|\bm{x})$ $\displaystyle=|I(S|\bm{x},v_{\text{finetune}})|-K_{\text{preserve}}(S|\bm{x}).$ $$

Figure: Figure 2: The preserved knowledge (interaction) $K^{(i)}_{\text{preserve}}$, the discarded knowledge $K^{(i)}_{\text{discard}}$, and the newly-learned knowledge $K^{(i)}_{\text{new}}$. For each subfigure, the total length of the blue bar and the orange bar equals to the knowledge encoded by the pre-trained model $K^{(i)}_{\text{pretrain}}$, and the length of the green bar and the orange bar equals to the knowledge encoded by the fine-tuned model $K^{(i)}_{\text{finetune}}$.
Refer to caption: x2.png

Experiments.
We conducted experiments to analyze changes of pre-trained model’s knowledge during the fine-tuning process, in order to provide in-depth explanations for the effects of pre-training on downstream tasks.
To this end, we employed off-the-shelf VGG-16 [], ResNet-50 [], ViT-Small, and ViT-Base [] pre-trained on the ImageNet-1K dataset [], and further fine-tuned these models on the CUB200-2011 [], CIFAR-10 [], and Stanford Cars [] datasets for image classification, respectively.
We also fine-tuned the pre-trained $\text{BERT}_{\text{BASE}}$ [] and DistillBERT [] models on the SST-2 [] dataset for binary sentiment classification.

For a detailed explanation, we further quantified the preservation and the discarding of the knowledge of different complexities.
The complexity of the knowledge was defined as the order of the interaction, i.e., the number of input variables involved in the interaction, $\textit{\text{complexity}}(S)=\textit{\text{order}}(S)=|S|$.
Thus, a high-order interaction denoted the interaction among a large number of input variables, which usually represented complex knowledge (interaction).
In comparison, a low-order interaction among a small number of input variables was often referred to as simple and general knowledge.

Fig. 2 reports the average strength of the $i$-th order preserved interactions $K^{(i)}_{\text{preserve}}=\mathbb{E}_{\bm{x}}\mathbb{E}_{S\subseteq N,|S|=i}[K
_{\text{preserve}}(S|\bm{x})]$, discarded interactions $K^{(i)}_{\text{discard}}=\mathbb{E}_{\bm{x}}\mathbb{E}_{S\subseteq N,|S|=i}[K_
{\text{discard}}(S|\bm{x})]$, and newly-learned interactions $K^{(i)}_{\text{new}}$.
Note that according to (6) and (7), the sum of $K^{(i)}_{\text{preserve}}$ and $K^{(i)}_{\text{discard}}$ equalled to the average strength of $i$-th order interactions encoded by the pre-trained model $K^{(i)}_{\text{pretrain}}=\mathbb{E}_{\bm{x}}\mathbb{E}_{S\subseteq N,|S|=i}[|
I(S|\bm{x},v_{\text{pretrain})}|]$, and the sum of $K^{(i)}_{\text{preserve}}$ and $K^{(i)}_{\text{new}}$ equalled to the average strength of $i$-th order interactions encoded by the fine-tuned model $K^{(i)}_{\text{finetune}}=\mathbb{E}_{\bm{x}}\mathbb{E}_{S\subseteq N,|S|=i}[|
I(S|\bm{x},v_{\text{finetune})}|]$.
We discovered that even among different network architectures on different datasets, pre-training exhibits the similar effect on the downstream task, as follows.

$\bullet$
We surprisingly observed that during the fine-tuning process, only a small amount of pre-trained model’s knowledge was preserved for the inference of the downstream task, while a considerable amount of knowledge was discarded, i.e., the amount of the discarded knowledge was more than twice that of the preserved knowledge.

$\bullet$
Interestingly, we also discovered that each fine-tuned model discarded more complex knowledge (reflected by high-order interactions) than simple and general knowledge (reflected by low-order interactions).
This indicated that complex knowledge encoded by the pre-trained model usually
was not discriminative enough for the classification of the downstream task (e.g., memorizing large-scale background patterns), thus the fine-tuned model discarded it, and re-learned discriminative knowledge for inference during the fine-tuning process.

$\bullet$
Correspondingly, the fine-tuned model learned a large amount of new knowledge for the inference of the downstream task, especially complex knowledge.

#### 3.2.2 Why the fine-tuned model can achieve superior classification performance?

Based on the quantification of pre-trained model’s knowledge in the preceding section, here, we provide an insightful explanation for
why pre-training can benefit the fine-tuned model in achieving superior classification performance(^5^5footnotemark: 5).
Intuitively, we consider that compared to training from scratch, the fine-tuned model can preserve some discriminative knowledge from the pre-trained model, which is beneficial for making inference, such as classifying hard samples.
This is due to that the preserved knowledge is usually acquired using a large-scale dataset with numerous training samples, thus it contains sufficiently discriminative information.
More crucially, this knowledge preserved from the pre-trained model is very difficult to be learned by a DNN training from scratch merely using a small-scale downstream-task dataset.
Thus,
pre-training makes the fine-tuned model encodes more exclusively-learned and discriminative knowledge than the model training from scratch for inference, which accounts for the superior performance of the fine-tuned model.

To this end, we propose the following metric to examine whether the model training from scratch can only successfully learns a little preserved knowledge $K_{\text{preserve}}(S|\bm{x})$ for verification.
Specifically, given a pre-trained model and its corresponding fine-tuned model, we train a randomly initialized DNN $v_{\text{random}}$ from scratch for the same downstream task, where we set it has the same network architecture as the fine-tuned model for fair comparisons.
We quantify the ratio of pre-trained model’s knowledge preserved by the fine-tuned model $K_{\text{preserve}}(S|\bm{x})$
that can be successfully learned by the model training from scratch, as follows.

$$ $\displaystyle\text{{ratio}}(S|\bm{x})=\frac{\mathbbm{1}(\Gamma_{\text{pretrain }}^{\text{random}}(S|\bm{x}))\cdot\min(|I(S|\bm{x},v_{\text{random}})|,K_{ \text{preserve}}(S|\bm{x}))}{K_{\text{preserve}}(S|\bm{x})},$ (8) $$

where $\Gamma_{\text{pretrain}}^{\text{random}}(S|\bm{x})=I(S|\bm{x},v_{\text{
pretrain}})\cdot I(S|\bm{x},v_{\text{random}})$ measures whether interactions $I(S|\bm{x},v_{\text{pretrain}})$ and $I(S|\bm{x},v_{\text{random}})$ have the same effect to the network output.
Only when interactions $I(S|\bm{x},v_{\text{pretrain}})$, $I(S|\bm{x},v_{\text{finetune}})$ and $I(S|\bm{x},v_{\text{random}})$ have the same effect, the metric $\text{{ratio}}(S|x)$ is non-zero; Otherwise, $\text{{ratio}}(S|x)=0$.
A small value of $\text{{ratio}}(S|\bm{x})$ indicates that the model training from scratch can merely learn a little
preserved knowledge $K_{\text{preserve}}(S|\bm{x})$.

Figure: Figure 3: The ratio of the preserved knowledge that can be learned by the model training from scratch. This figure verifies that pre-training makes the fine-tuned model encodes more exclusively-learned and discriminative knowledge for inference than the model training from scratch, which responses to the superior performance of the fine-tuned model.
Refer to caption: x3.png

Experiments.
We conducted experiments to verify that the fine-tuned model encoded more exclusively-learned and discriminative knowledge than training from scratch.
To this end, we trained randomly initialized VGG-16, ResNet-50, ViT-Small, and ViT-Base models on the CUB200-2011, CIFAR-10, and Stanford Cars datasets from scratch for image classification, respectively.
We also trained randomly initialized $\text{BERT}_{\text{BASE}}$ and DistillBERT models on the SST-2 dataset from scratch for binary sentiment classification.
Please see Appendix for more training details.

Fig 3 reports the average ratio of the preserved knowledge that the model training from scratch was able to learn, $\text{{Ratio}}=\mathbb{E}_{\bm{x}}\mathbb{E}_{S\subseteq N}[\text{{ratio}}(S|
\bm{x})]$.
We discovered that the average ratio for each DNN was very low, i.e., ranging from $13\%$ to $45\%$.
This indicated that only a little preserved knowledge could be successfully learned by the model training from scratch, while most of it was extremely difficult to be acquired.
Thus, compared to training from scratch,
pre-training enabled the fine-tuned model to encode more exclusively-learned and discriminative knowledge for inference, resulting in its better performance.

#### 3.2.3 Why the fine-tuned model converges faster?

Apart from the improved performance, pre-training can also benefits the fine-tuned model in speeding up the convergence(^5^5footnotemark: 5) [].
In this section, we present an in-depth analysis to explain this benefit.
Specifically, according to the information-bottleneck theory [, ], when training from scratch, the DNN usually tries to encode various knowledge in early epochs and discarding task-irrelevant knowledge in later epochs.
In comparison, pre-training guides the fine-tuned model to directly and quickly learn target knowledge, without temporarily modeling and discarding knowledge unrelated to the inference of the downstream task, which is responsible for the faster convergence of the fine-tuned model.

Explicitly speaking, whether or not a DNN can quickly and directly learn target knowledge can be analyzed as whether the amount of learned target knowledge increases fast and stably along with the epoch number, respectively, where we define the target knowledge as the interaction encoded by the finally-learned DNN.
To this end, we propose the following metrics to examine whether the fine-tuned model encodes target knowledge more directly and quickly for verification.
Specifically,
let the vectors $\bm{I}_{\text{finetune},e}(\bm{x})=[I(S_{1}|\bm{x},v_{\text{finetune},e}),I(S_
{2}|\bm{x},v_{\text{finetune},e}),\cdots,I(S_{d}|\bm{x},v_{\text{finetune},e})
]\in\mathbb{R}^{d}$ and $\bm{I}_{\text{finetune},E}(\bm{x})$ represent the distribution of all interactions encoded by the model fine-tuned after $e$ epochs and $E$ epochs, respectively, where $E$ denotes the total epoch number.
Accordingly, the vector $\bm{I}_{\text{random},E}(\bm{x})$ and the vector $\bm{I}_{\text{random},E}(\bm{x})$ represent the distribution of all interaction encoded by the model training from scratch after $e^{\prime}$ epochs and $E^{\prime}$ epochs, respectively.
Then, we calculate the Jaccard similarity between interactions encoded by the DNN learned after certain epochs and those encoded by the finally-learned DNN.

$$ $\displaystyle\text{{Jaccard}}_{\text{finetune}}$ $\displaystyle=\mathbb{E}_{\bm{x}}\left[{\|\min(\tilde{\bm{I}}_{\text{finetune} ,e}(\bm{x}),\tilde{\bm{I}}_{\text{finetune},E}(\bm{x}))\|_{1}}/{\|\max(\tilde{ \bm{I}}_{\text{finetune},e}(\bm{x}),\tilde{\bm{I}}_{\text{finetune},E}(\bm{x}) )\|_{1}}\right],$ (9) $\displaystyle\text{{Jaccard}}_{\text{random}}$ $\displaystyle=\mathbb{E}_{\bm{x}}\left[{\|\min(\tilde{\bm{I}}_{\text{random},e ^{\prime}}(\bm{x}),\tilde{\bm{I}}_{\text{random},E^{\prime}}(\bm{x}))\|_{1}}/{ \|\max(\tilde{\bm{I}}_{\text{random},e^{\prime}}(\bm{x}),\tilde{\bm{I}}_{\text {random},E^{\prime}}(\bm{x}))\|_{1}}\right],$ $$

where we extend the $d$-dimension vector $\bm{I}_{\text{finetune},e}(\bm{x})$ to into a $2d$-dimension vector $\tilde{\bm{I}}_{\text{finetune},e}(\bm{x})=[(\bm{I}^{+}_{\text{finetune},e}(
\bm{x}))^{\mathrm{T}},(-\bm{I}^{-}_{\text{finetune},e}(\bm{x}))^{\mathrm{T}}]^
{\mathrm{T}}=[\max(\bm{I}_{\text{finetune},e}(\bm{x}),0)^{\mathrm{T}},-\min(
\bm{I}_{\text{finetune},e}(\bm{x}),0)^{\mathrm{T}}]^{\mathrm{T}}\in\mathbb{R}^
{2d}$ without negative elements.
Accordingly, vectors $\tilde{\bm{I}}_{\text{finetune},E}(\bm{x})$, $\tilde{\bm{I}}_{\text{random},e^{\prime}}(\bm{x})$, and $\tilde{\bm{I}}_{\text{random},E^{\prime}}(\bm{x})$ are constructed on $\bm{I}_{\text{finetune},E}(\bm{x})$, $\bm{I}_{\text{random},e^{\prime}}(\bm{x})$, and $\bm{I}_{\text{random},E^{\prime}}(\bm{x})$ to contain non-negative elements, respectively.
Thus, a sharp increase of the similarity at early epochs indicates that the DNN encodes target knowledge quickly.
Besides, a stable increase of the similarity along the epoch number, without significant fluctuations, demonstrates that the DNN encodes target knowledge directly.

Figure: Figure 4: Changes of the Jaccard similarity $\text{{Jaccard}}_{\text{finetune}}$ and $\text{{Jaccard}}_{\text{finetune}}$ along with the epoch number. The similarity $\text{{Jaccard}}_{\text{finetune}}$ of the fine-tuned model exhibits a more sharp and stable increase with the epoch number than that of training from scratch $\text{{Jaccard}}_{\text{finetune}}$. This verifies the fine-tuned model learns target knowledge more quickly and directly, which accounts for its faster convergence.
Refer to caption: x4.png

Experiments.
We conducted experiments to examine whether pre-training guided the fine-tuned model to encode target knowledge more quickly and directly than training from scratch.
To this end, we employed fine-tuned DNNs and DNNs training from scratch introduced in the experiment paragraph of section 3.2.2 for evaluation.
Fig. 4 reports the change of the similarity $\text{{Jaccard}}_{\text{finetune}}$ and $\text{{Jaccard}}_{\text{random}}$ along with the epoch number.
We discovered that pre-training exhibited similar effects on guiding the fine-tuned model to learn target knowledge across different network architectures and datasets, as follows.

$\bullet$
Fig. 4 shows that the similarity $\text{{Jaccard}}_{\text{finetune}}$ first increased sharply in early epochs, then rose gradually and eventually saturated in later epochs, while the similarity $\text{{Jaccard}}_{\text{random}}$ usually exhibited the opposite trend, i.e., first increasing gradually and then increasing rapidly in later epochs.
This indicated that pre-training enabled the fine-tuned model to learn target knowledge more quickly.

$\bullet$
Fig. 4 also illustrates that the similarity $\text{{Jaccard}}_{\text{finetune}}$ usually increased stably along with the epoch number without significant fluctuations, while the similarity $\text{{Jaccard}}_{\text{random}}$ increased with ups and downs.
This demonstrated that pre-training guided the fine-tuned model to straightforwardly learned target knowledge, while the DNN training from scratch temporarily learned various knowledge and discarded task-irrelevant one later.

## 4 Conclusion and discussion

In this paper, we present an in-depth analysis to explain the benefits of pre-training, including the boosted accuracy and the accelerated convergence, from a game-theoretic view.
To this end,
we use interactions to explicitly quantify the knowledge encoded by the pre-trained model, and further analyze the effects of such knowledge on the downstream task, where
the faithfulness of treating interactions as essential knowledge encoded by the DNN for inference has been theoretically ensured by a set of properties of interactions.
We discover that compared to training from scratch, pre-training enables the fine-tuned model to encode more exclusively-learned and discriminative knowledge for inference, and to learn target knowledge more quickly and directly, which accounts for the superior classification performance and faster convergence of the fine-tuned model.
This provides new insights into understanding pre-training, and may also guide new interesting directions on the fine-tuning behavior of the DNN for future studies.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses part.
soundness: 3
strengths: - The study covers various datasets (CUB, Stanford Cars, CIFAR-10, etc), models (both CNN and transformer) and different domains (vision and language), providing a complete analysis of the knowledge-based understanding of pre-training/fine-tuning paradigm.
- This work is logically organized and covers several interesting understandings, like why pre-training/fine-tuning framework generally achieves better performance and fast convergence than training from scratch.
- Key takeaways are highlighted throughout the manuscripts, making it easy to understand.
summary: This study analyzes how pre-training a model helps the fine-tuning process through knowledge-based decomposition. The authors define knowledge as the intersection of certain concepts within the input and empirically identify that: (i) the fine-tuning process utilizes only a small fraction of the pre-trained model's knowledge, with substantial portions being discarded; (ii) fine-tuning incorporates a significant amount of new, especially complex, knowledge; (iii) pre-training enables the model to acquire certain knowledge during fine-tuning that cannot be learned when training from scratch; and (iv) pre-training accelerates the knowledge learning process during fine-tuning.
weaknesses: - For section 3.2.2, whether there is any knowledge (intersections) that can only be learned through training from scratch and whether this knowledge is beneficial for the final classification. Analyzing the ratios of preserved knowledge acquired solely from training from scratch may only partially explain why pre-training and fine-tuning achieve better performance. Demonstrating that there are limited intersections unique to training from scratch would further strengthen this analysis.
- Reporting the final accuracy of different models in Figure 3 would be beneficial to verify whether the ratios of preserved knowledge are related to the final performance.
- Will different types of pre-training, such as self-supervised or adversarial pre-training, exhibit different properties in how knowledge changes during fine-tuning?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: Please refer to the weakness for my questions.
soundness: 2
strengths: 1. The paper investigates a longstanding question in the transfer learning area and applies recently proposed game-theoretic interaction analyses to track the behavior of DNNs during the fine-tuning process.
2. It proposes metrics to explicitly quantify the preserved and discarded knowledge in pre-trained models, as well as the preserved and newly learned knowledge after fine-tuning.
summary: The authors explore how pre-training is beneficial for downstream tasks, addressing a longstanding fundamental question in transfer learning. They connect recent progress in explaining DNN behavior through game-theoretic interactions to track changes in learned knowledge during the fine-tuning process. The authors propose methods to identify preserved, discarded, and newly acquired knowledge by tracking these changes. Equipped with these results, the paper concludes that, compared to training from scratch, pre-training allows the fine-tuned model to encode more exclusively-learned and discriminative knowledge for inference and to acquire target knowledge more quickly and directly. This accounts for the superior classification performance and faster convergence of the fine-tuned model.
weaknesses: 1. Despite ambitious claims in the title and abstract, the manuscript primarily applies existing techniques to transfer learning, leading only to widely recognized conclusions without offering new theoretical insights or surprising experimental results. The manuscript falls short of the ICLR acceptance standard.
2. Given that fine-tuning commonly suffers from catastrophic forgetting and that improvements often stem from leveraging common pre-trained knowledge, the presented experimental results are predictable and lack significance.
3. The empirical analysis uses a broad array of target pre-trained models, including VGG-16, ResNet-50, ViT-small, ViT-base, BERT-base, and DistillBERT. These models differ significantly in terms of parameters, architectures, and pre-training strategies, spanning CNNs and transformers with varying datasets. The conclusions drawn from these comparisons merely confirm that the proposed metrics show consistent trends across different models.
4. Following previous point, the paper would benefit from a more detailed analysis, such as comparing supervised versus unsupervised pre-trained ResNet-50 models on ImageNet. It should investigate how variations in epochs and other hyperparameters might affect behavior, and whether the proposed metrics can detect these differences.
5. The domain gap between the target domain and the pre-trained dataset is a critical factor in fine-tuning behavior, yet the choice of fine-tuning datasets in the study seems arbitrary. The paper should clarify how the proposed metrics handle behaviors influenced by transfer gaps.
6. The experimental setup requires refinement; for example, previous research [1] indicates that linear probing is more challenging than full fine-tuning (requiring a much larger learning rate). The study treats these settings as equivalent. Note that different learning rates and training durations could alter conclusions regarding knowledge forgetting or not.
7. The paper focuses on classification problems. But transfer learning applications extend beyond this scope. The relationship between pre-text tasks and downstream tasks is also a critical factor in fine-tuning behavior.  Can these metrics extend to border transfer scenarios?

[1] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." CVPR 2020.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please address the questions in the Weakness section.
soundness: 3
strengths: 1. The paper is well-written, presenting complex ideas clearly and intuitively.

2. The use of game theory and an 'interaction metric' to study pre-training effects is both novel and insightful.

3. Extensive experiments across various architectures validate the proposed methods, underscoring their effectiveness and applicability.
summary: The paper proposes a novel game-theoretic approach to explain the benefits of pre-training on downstream classification tasks by studying the knowledge that is preserved, discarded, or newly acquired during the fine-tuning process. The authors introduce interaction metrics to assess knowledge encoded by a pre-trained model and examine how it impacts downstream performance and convergence speed. They show that pre-training enables fine-tuned models to retain discriminative knowledge, leading to superior classification accuracy and faster convergence compared to models trained from scratch.
weaknesses: 1. Lack of clear motivation: The paper lacks clear motivation for using a game-theoretic approach, as similar conclusions about pre-training benefits have been reached through alternative methods, such as feature space analysis (e.g., Deng et al., 2023). Additionally, it does not provide a discussion comparing the advantages or unique insights offered by the game-theoretic approach over existing methods, leaving its added value unclear.

2. Lack of Practical Utility and Actionable Recommendations: Although the paper suggests the advantages of using pretrained models for downstream classification, it doesn’t provide concrete recommendations for how practitioners could adjust pre-training or fine-tuning protocols based on this insight of preserved or newly added knowledge. For example, it remains unclear how to retain more useful information or optimize the number of fine-tuning epochs.

3. Computational Complexity and Scalability Concerns: The paper does not address whether these metrics are feasible for large-scale models or datasets, raising concerns about scalability for real-world datasets.

4. Neglect of Discarded Knowledge’s Role in Robustness and Generalization: The paper assumes that discarded knowledge is irrelevant but does not analyze if this information affects model robustness, generalization to out-of-distribution data, or resilience to adversarial attacks. Understanding this could provide a more nuanced view of pre-training benefits and clarify whether certain discarded knowledge could enhance model robustness.

5. Continual and Online Learning Analysis: The paper does not explore scenarios where data arrives incrementally, as in continual or online learning. Research such as [1] suggests that in these cases, training from scratch may outperform using a pre-trained model. Examining the impact of preserved and discarded knowledge in such settings would extend the paper’s relevance.

6.  Evaluation with self-supervised pretrained model: Self-supervised learning has shown strong generalizability for classification and could yield different knowledge preservation dynamics compared to supervised pre-training. Experiments comparing self-supervised and supervised pre-training could reveal how each approach affects preserved and discarded knowledge, especially for tasks with limited labeled data.

7. Comparison of CNNs vs. Transformers :
Since CNNs and transformers represent data differently, they may encode and preserve different types of knowledge. It will be interesting to explore further whether CNNs or transformers retain more useful knowledge for downstream tasks, which could inform architecture selection for different scenarios.

8. The approach to annotating semantic parts for calculating interactions in this paper appears to be limited by arbitrary selection, as they randomly choose 2 images for each category to annotate semantic parts and compute interactions. While this strategy reduces computational complexity, it might introduce potential biases and undermines the rigor of interaction measurement by possibly excluding critical semantic parts. This might weaken the reliability of the results, as the randomly chosen annotations may not accurately represent the model's true interactions with significant features. Please elaborate on this.

References: [1]  Ash, J. and Adams, R.P., 2020. On warm-starting neural network training. Advances in neural information processing systems, 33, pp.3884-3894.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: How are the patch sizes for the images determined? Would this not change the types of interactions that are able to be found? Furthermore, the spatial coverage of each patch is unstructured with respect to the image content, would this method find interactions that  are smaller than the patch size? It's unclear how this would be possible.

Could counterfactual experiments be performed to validate the interactions determined in Figure 2 (e.g. Preserved knowledge) are indeed correctly identified?
soundness: 2
strengths: This paper provides a systematic way to analyze pre-training benefits. The framework moves the field beyond just observing that pre-training works to understanding how and why it works.
summary: The paper proposes a novel game-theoretic approach to explain why pre-training is beneficial for downstream tasks by analyzing and quantifying the knowledge encoded in pre-trained models. They represent knowledge in neural networks through "interactions" between input variables. An interaction measures how different input variables work together to contribute to the network's output.

The method relies on the following assumptons:

Sparsity property: Well-trained DNNs encode only a small number of significant interactions
Universal-matching property: Network outputs can be well explained by these sparse interactions

The authors propose metrics to quantify three types of knowledge during fine-tuning:
Preserved knowledge: Knowledge from pre-trained model that is kept during fine-tuning
Discarded knowledge: Pre-trained knowledge that is eliminated during fine-tuning
Newly-learned knowledge: Additional knowledge acquired during fine-tuning

The method is validated across multiple architectures (VGG-16, ResNet-50, ViT variants, BERT models) and datasets (CUB200-2011, CIFAR-10, Stanford Cars, SST-2), showing consistent patterns in how pre-training benefits downstream tasks.
weaknesses: The paper's empirical verifications focus more on validating their hypotheses about how pre-training helps downstream tasks, rather than directly verifying if the interactions they find are meaningful. While the authors thoroughly verify their hypotheses about pre-training benefits, they rely more on theoretical justification and prior work for the validity of their interaction analysis method itself.

The motivation for why this procedure should work is difficult to follow. Much of the text uses the justification of a proven sparsity property, but there isn't a clear intuition of why this property should hold for DNNs. Some of it is discussed in Appendix A and B, but would be helpful to have toy experiments to justify both this property but also the method itself recovers the correct ground truth of interactions in a controlled environment.

The method appears extremely similar to the SHAP (SHapley Additive exPlanations) approach. Much of this is mentioned in the citations of the related work, but the main contribution of this work appears to be the application of the SHAP approach to study the influence of training on fine-tuning.

Minor things:

Please specify the pre-training datasets for the language models.

Grammar Issue, Line 100

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Does the proposed method only applicable to single-label classification tasks?

- In Eq. (2), the meaning of the indicator $(-1)^{|S|-|T|}$ needs clarification. How should we interpret the impact of (-1) or (+1) when S contains many elements, like 10? Also, should $T \in S$ be written as $T \subset S$ or $T \in 2^S$?

- What is the approximation error when using the salient interaction? Is this salient interaction universally applicable, or does it depend on the testing distribution?

- Since image variables are defined as patches, there will naturally be correlations between different patches. How does the method address this correlation? 

- Is it possible that the pre-trained model learns redundant interactions, where pruning some interactions still leaves it strong enough for downstream tasks?

- It would help to calculate the correlation between the preservation ratio and model performance to support the claim that “pre-training makes the ﬁne-tuned model encodes more exclusively-learned and discriminative knowledge than the model training from scratch for inference.”

- Figure 4 is unclear; if all lines converge to 1.0 at the same time, how does this support the faster convergence rate of fine-tuning? Additionally, how can we verify that this relationship is causal and not just correlational? How do we ensure that an improvement in the interaction score actually enhances model capacity, rather than some latent factor improving both measures?
soundness: 3
strengths: Instead of measuring learning by classification performance, the authors use the interaction between elements within an image. This approach offers a more interpretable view of the learning process.
summary: This paper proposes a metric to measure the knowledge learned by classification models, which can help track how knowledge changes during pre-training and fine-tuning. Using this measurement, the authors analyze the behavior of classification models, and claim that: a small portion of knowledge gained in pre-training is retained in fine-tuning. This retained knowledge is important for downstream tasks and cannot be learned from scratch. Also, pre-training makes fine-tuning faster and more direct in learning the target knowledge.
weaknesses: Some concepts in the paper are not clearly defined mathematically and lack theoretical or experimental support. For example, in Line 340, the authors mention that pre-trained high-order knowledge is not “discriminative” for downstream tasks. It would help to clarify this—such as by explaining if low interaction scores or incorrect predictions caused by the discarded pre-trained knowledge support this idea.

The authors should prove the relationship between the proposed metric and the model performance is causal.