MULTI-STAGE INFLUENCE FUNCTION
Year: 2019

Abstract:
Multi-stage training and knowledge transfer from a large-scale pretraining task to various finetuning tasks have revolutionized natural language processing (NLP) and computer vision (CV), with state-of-the-art performances constantly being improved. In this paper, we develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task. The proposed multistage influence function generalizes the original influence function for a single model in Koh & Liang (2017), thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks. We test our proposed method in various experiments to show its effectiveness and potential applications.

text: Multi-stage training and knowledge transfer from a large-scale pretraining task to various finetuning tasks have revolutionized natural language processing (NLP) and computer vision (CV), with state-of-the-art performances constantly being improved. In this paper, we develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task. The proposed multistage influence function generalizes the original influence function for a single model in Koh & Liang (2017), thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks. We test our proposed method in various experiments to show its effectiveness and potential applications.

heading: 1 INTRODUCTION
text: Multi-stage training has become increasingly important and has achieved state-of-the-art results in many tasks. In NLP applications, it is now a common practice to first learn word embeddings (e.g., word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014)) or contextual representations (e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2018)) from a large unsupervised corpus, and then refine or finetune the model on supervised end tasks. In computer vision applications, it is common to use a pretrained CNN as a feature extractor and only finetune top-layer networks through training on the end task. Also, it has been demonstrated that pretraining ResNet (He et al., 2016) with large hashtag data can greatly benefit many end tasks (Mahajan et al., 2018). Intuitively, the successes of these multi-stage learning paradigms are due to knowledge transfer from pretraining tasks to the end task. However, current approaches using multi-stage learning are usually based on trial-and-error and many fundamental questions remain unanswered. For example, which part of the pretraining data/task contributes most to the end task? How can one detect “false transfer” where some pretraining data/task could be harmful for the end task? If a testing point is wrongly predicted by the finetuned model, can we trace back to the problematic examples in the pretraining data? Answering these questions requires a quantitative measurement of how the data and loss function in the pretraining stage influence the end model, which has not been studied in the past and will be the main focus of this paper.
To find the most influential training data responsible for a model’s prediction, the influence function was first introduced by Cook & Weisberg (1980) from a robust statistics point of view. Recently, as large-scale applications become more challenging for influence function computation, Koh & Liang (2017) proposed to use a first-order approximation to measure the effect of removing one training point on the model’s prediction, to overcome computational challenges. More broadly, there are many works using influence functions to investigate the impact of training data on models in various machine learning applications, such as tracing back the origins of bias in the word embeddings generated by GloVe (Brunet et al., 2019), and understanding and mitigating disparate impact to improve model fairness (Wang et al., 2019). However, all of the existing influence score computation algorithms study the case of single-stage training – where there is only one model with one set of training/prediction data in the training process. To the best of our knowledge, the influence of pretraining data on a subsequent finetuning task and model has not been studied, and it is nontrivial to apply the original influence function in (Koh & Liang, 2017) to this scenario.
In this work, we derive the influence function from pretraining data to the end task in multi-stage training. Since the computation involves several expensive Hessian vector products, we also show how to compute the influence function efficiently in large-scale problems. Based on this technique, we show that
• in real datasets and experiments across various vision and NLP tasks, predictions using the technique and actual influence for the pretraining data to the finetuned model are highly correlated (Pearson’s r score to be around 0.6). This shows the effectiveness of our proposed technique for computing the influence scores in multi-stage models; • the influence for the pretraining data to the finetuned model can be split into two parts: the
influence of the pretraining data on the pretrained model, and influence of the pretraining data on the finetuned model. Therefore the testing data from the finetuning task will be impacted by changes in the pretraining data, which can be quantified using our proposed technique; • the influence of the pretraining data on the finetuning task is highly dependent on 1) the
similarity of two tasks or stages, 2) and the number of training data in the finetuning task. Thus our proposed technique provides a novel way to measure how the pretraining data helps or benefits the finetuning data.

heading: 2 RELATED WORK
text: Multi-stage model training that trains models in many stages on different tasks to improve the endtask has been used widely in many machine learning areas, such as in transfer learning (Ando & Zhang, 2005) and zero-shot learning (Larochelle et al., 2008). Recently, multi-stage model training has achieved state-of-the-art performance by learning large embeddings or data representation as the pretraining step on a very large pretraining dataset, which is then followed by a finetune step with further training on the end-task. Examples include the recently proposed BERT (Devlin et al., 2018), which learns contextual embeddings on a large corpus with the pretraining tasks chosen to be predicting the masked words in a sentence and predicting whether one sentence is after another sentence. This contextual embedding is then used in finetuning tasks, such as a question answering task. ELMo (Peters et al., 2018) is widely used in multi-stage model training as a sentence feature extractor to benefit the end-task. Similarly, there are some works in computer vision that train an image representation model on a large number of images as the pretraining step, and then use the resulting features to finetune another task, such as particular image classification tasks. For example, (Mahajan et al., 2018) uses ResNet in the pretraining step, and the finetuning task is based on hashtag data. The rationale for this multi-stage model is that the pretraining task can learn some common or latent representation which could benefit the end task.
Another related line of research is on understanding machine learning models. One category of research is to explain predictions with respect to model variables, and trace back the contribution of variables to the prediction. For example, Oramas et al. (2019) automatically detects internal features in the set of classes in the pretrained model that are relevant to interpreting the prediction, and shows for various vision tasks that the proposed scheme can produce detailed explanations based on the features that are relevant to the targeted classes. Guo et al. (2019) aims to interpret variable-wise hidden states in an LSTM model to quantify variable importance and variable-wise temporal importance in the model.
Closely related research has sought to connect model prediction and training data, and trace back the most influential training data that are most responsible for the model prediction. Among them, the influence function (Cook & Weisberg, 1980; Koh & Liang, 2017), which aims to model the prediction changes when training data is added/removed, has been shown to be effective in many applications. There is a series of work on influence functions, including investigating the influence of a group of data on the prediction (Koh et al., 2019), using influence functions to detect bias in word embeddings (Brunet et al., 2019), and using it in preventing data poisoning attacks (Steinhardt et al., 2017), etc. All of these works only consider a single stage training procedure, and it is not straightforward to apply the existing influence functions to multi-stage models. In this paper, we propose to analyze the influence of pretraining data on predictions in the subsequent finetuned model and end task.

heading: 3 ALGORITHMS
text: In this section, we detail the procedure of multi-stage training, show how to compute the influence score for the multi-stage training, and then discuss how to scale up the computation.

heading: 3.1 MULTI-STAGE MODEL TRAINING
text: Multi-stage models, which train different models in consecutive stages, have been widely used in various ML tasks. Mathematically, letZ be the training set for pretraining task with data size |Z| = m, and X be the training data for the finetuning task with data size |X | = n. In pretraining stage, we assume the parameters of the pretrained network have two parts: the parameters W that are shared with the end task, and the task-specific parameters U that will only be used in the pretraining stage. Note that W could be a word embedding matrix (e.g., in word2vec) or a representation extraction network (e.g., Elmo, BERT, ResNet), while U is usually the last few layers that corresponds to the pretraining task. After training on the pretraining task, we obtain the optimal parameters W ∗, U∗. The pretraining stage can be formulated as
Pretrain Stage: W ∗, U∗ = arg min W,U
1
m ∑ z∈Z g(z, W, U) := arg min W,U G(W,U), (1)
where g(·) is the loss function for the pretrain task. In the finetuning stage, the network parameters are W,Θ, where W is shared with the pretraining task and Θ is the rest of the parameters specifically associated with the finetuning task. We will initialize the W part by W ∗. Let f(·) denote the finetuning loss, there are two cases when finetuning the end-task:
• Finetuning Case 1: Fixing embedding parameters W = W ∗, and only finetune Θ:
Θ∗ = arg min Θ
1
n ∑ x∈X f(x, W ∗, Θ) := arg min Θ F (W ∗,Θ). (2)
• Finetuning Case 2: finetune both the embedding parameters W (initialized from W ∗) and Θ. Sometimes updating the embedding parameters W in the finetuning stage is necessary, as the embedding parameters from the pretrained model may not be good enough for the finetuning task. This corresponds to the following formulation:
W ∗∗,Θ∗ = arg min W,Θ
1
n ∑ x∈X f(x, W, Θ) := arg min W,Θ F (W,Θ). (3)

heading: 3.2 INFLUENCE FUNCTION FOR MULTI-STAGE MODELS
text: We derive the influence function for the multi-stage model to trace the influence of pretraining data on the finetuned model. In Figure 1 we show the task we are interested in solving in this paper. Note
that we use the same definition of influence function as (Koh & Liang, 2017) and discuss how to compute it in the multi-stage training scenario. As discussed at the end of Section 3.1, depending on whether or not we are updating the shared parameters W in the finetuning stage, we will derive the influence functions under two different scenarios.

heading: 3.2.1 CASE 1: EMBEDDING PARAMETERS W ARE FIXED IN FINETUNING
text: To compute the influence of pretraining data on the finetuning task, the main idea is to perturb one data example in the pretraining data, and study how that impacts the test data. Mathematically, if we perturb a pretraining data example z with loss change by a small , the perturbed model can be defined as
Ŵ , Û = arg min W,U G(W,U) + g(z,W,U). (4)
Note that choices of can result in different effects in the loss function from the original solution in (1). For instance, if we set = − 1n , then we are removing the training data z in the pretraining dataset.
For the finetuning stage, since we consider Case 1 where the embedding parameters W are fixed in the finetuning stage, the new model for the end-task or finetuning task will thus be
Θ̂ = arg min Θ F (Ŵ ,Θ). (5)
The influence function that measures the impact of a small perturbation on z to the finetuning loss on a test sample xt from finetuning task is defined as
Iz,xt := ∂f(xt, Ŵ , Θ̂ )
∂
∣∣ =0
(6)
= ∇Θf(xt,W ∗,Θ∗)T · Iz,Θ +∇W f(xt,W ∗,Θ∗)T · Iz,W . (7)
with Iz,Θ := ∂Θ̂ ∂ ∣∣ =0 and Iz,W := ∂Ŵ ∂ ∣∣ =0 , (8)
where Iz,Θ measures the influence of z on the finetuning task parameters Θ, and Iz,W measures how z influences the pretrained model W . Therefore we can split the influence of z on the test sample into two pieces: one is the impact of z on the pretrained model Iz,W , and the other is the impact of z on the finetuned model Iz,Θ. It is worth mentioning that, due to linearity, if we want to estimate a set of test example influence function scores with respect to a set of pretraining examples, we can simply sum up the pair-wise influence functions, and so define
I{z(i)},{x(j)t } := ∑ i ∑ j I z(i),x (j) t . (9)
where {z(i)} and {x(j)t } contain a set of pretraining data and finetuning test data. Next we will derive these two influence scores Iz,Θ and Iz,W (see the detailed derivations in the appendix) in Theorem 1 below. Theorem 1. For the two-stage training procedure in (1) and (2), we have
Iz,W := ∂Ŵ ∂ ∣∣ =0 = − [(∂2G(W ∗, U∗) ∂(W,U)2 )−1 ( ∂g(z,W ∗, U∗) ∂(W,U) ) ] W
(10)
Iz,Θ := ∂Θ̂ ∂ ∣∣ =0 = ( ∂2F (W ∗,Θ∗) ∂Θ2 )−1 · (∂ 2F (W ∗,Θ∗) ∂Θ∂W ) · [(∂2G(W ∗, U∗) ∂(W,U)2 )−1 ( ∂g(z,W ∗, U∗) ∂(W,U) ) ] W
where [·]W means taking the W part of the vector.
By plugging (10) into (7), we finally obtain the influence score of pretraining data z on the finetuning task testing point xt, Iz,xt as
Iz,xt =
[ − ∂f(x,W ∗,Θ∗)T
∂Θ ·(∂
2F (W ∗,Θ∗)
∂Θ2 )−1 · ∂
2F (W ∗,Θ∗) ∂Θ∂W + ∂f(x,W ∗,Θ∗)T ∂W
] Iz,W
(11) Algorithm 1 shows how to compute the influence score in (11). The pseudocode for computing the influence function in (11) is shown in Algorithm 1.
Algorithm 1: Multi-Stage Influence Score with Fixed Embedding 1 Input: pretrain and finetune models with W ∗, Θ∗, and U∗; pretrain and finetune training data Z and X ; test example xt; and a pretrain training example z; 2 Output: Influence function value Iz,xt ; 3 Compute fintune model’s gradients∂f(xt,W
∗,Θ∗) ∂Θ and ∂f(xt,W ∗,Θ∗) ∂W ;
4 Compute the first inverse Hessian vector product Vihvp1(xt) := ( ∂2F (W∗,Θ∗) ∂Θ2 ) −1 ∂f(xt,W∗,Θ∗) ∂Θ ; 5 Compute finetune loss’s gradient w.r.t W : ∂f(xt,W ∗,Θ∗)T
∂W = V T ihvp1 ∂2F (W∗Θ∗) ∂Θ∂W − f(xt,W ∗,Θ∗)
∂W and concatenate it with 0 to make it the same dimension as (W, U);
6 Compute and save the second inverse Hessian vector product
V Tihvp2(xt) := [ ∂f(xt,Θ
∗,W∗)T ∂W , 0]( ∂2G(W∗,U∗) ∂(W,U)2 ) −1 ;
7 Compute influence function score Iz,xt = V T ihvp2(xt) ∂g(z,W∗,U∗) ∂(W,U) ;

heading: 3.2.2 CASE 2: EMBEDDING PARAMETER W IS ALSO UPDATED IN THE FINETUNING STAGE
text: For the second finetuning stage case in (3), we will also further train the embedding parameter W from the pretraining stage. When W is also updated in the finetuning stage, it is challenging to characterize the influence since the pretrained embedding W ∗ is only used as an initialization. In general, the final model (W ∗∗,Θ∗) may be totally unrelated to W ∗; for instance, when the objective function is strongly convex, any initialization of W in (3) will converge to the same solution.
However, in practice the initialization of W will strongly influence the finetuning stage in deep learning, since the finetuning objective is usually highly non-convex and initializing W with W ∗ will converge to a local minimum near W ∗. Therefore, we propose to approximate the whole training procedure as
W̄ , Ū = arg min W,U G(W,U) (12)
W ∗,Θ∗ = arg min W,Θ {α‖W − W̄‖2F + F (W,Θ)},
where W̄ , Ū are optimal for the pretraining stage, W ∗,Θ∗ are optimal for the finetuning stage, and 0 ≤ α 1 is a small value. This is to characterize that in the finetuning stage, we are targeting to find a solution that minimizes F (W,Θ) and is close to W̄ . In this way, the pretrained parameters are connected with finetuning task and thus influence of pretraining data to the finetuning task can be tractable. The results in our experiments show that with this approximation, the computed influence score can still reflect the real influence quite well.
Similarly we can have ∂Θ̂ ∂ , ∂Ŵ ∂ , and ∂W̄ ∂ to measure the difference between their original optimal solutions in (12) and the optimal solutions from perturbation over the pretraining data z. Similar to (7), the influence function Iz,xt that measures the influence of perturbation to pretraining data z on test sample xt’s loss is
Iz,xt : = ∂f(xt, Ŵ , Θ̂ )
∂
∣∣ =0 = ∂f(xt,W ∗,Θ∗)
∂(W,Θ)
T [ ∂Ŵ ∂ ∣∣ =0
∂Θ̂ ∂ ∣∣ =0
] . (13)
The influence function of small perturbation of G(W,U) to W̄ ,W ∗,Θ∗ can be computed following the same approach in Subsection 3.2.1 by replacing W̄ for W ∗ and [Θ∗,W ∗] for Θ∗ in (10). This will lead to
∂W̄ ∂ ∣∣ =0 =− [(∂2G(W̄ , Ū) ∂(W,U)2 )−1 ( ∂g(z, W̄ , Ū) ∂(W,U) ) ] W
(14)[ ∂Θ̂ ∂ ∣∣ =0
∂Ŵ ∂ ∣∣ =0
] = [ ∂2F (W∗,Θ∗) ∂Θ2 ∂2F (W∗,Θ∗)
∂Θ∂W ∂2F (W∗,Θ∗)
∂Θ∂W ∂2F (W∗,Θ∗) ∂W 2 + 2αI
]−1 [ 0 −2αI ] [(∂2G(W̄ , Ū) ∂(W,U)2 )−1 ( ∂g(z, W̄ , Ū) ∂(W,U) ) ] W
After plugging (14) into (13), we will have the influence function Iz,xt . Similarly, the algorithm for computing Iz,xt for Case 2 can follow Algorithm 1 for Case 1 by replacing gradient computation.

heading: 3.3 IMPLEMENTATION DETAILS
text: The influence function computation for multi-stage model is presented in the previous section. As we can see in Algorithm 1 that the influence score computation involves many Hessian matrix operations, which will be very expensive and sometimes unstable for large-scale models. We used several strategies to speed up the computation and make the scores more stable.
Large Hessian Matrices A Hessian matrix H has a size of p × p, where p is the number of parameters in the model. For large deep learning models with thousands or even millions of parameters, it is almost impossible to fit a p × p Hessian into memory. Also, to invert a Hessian requires O(p3) operations. Similar to Koh & Liang (2017), we avoid explicitly computing and storing the Hessian matrix and its inverse, and instead compute product of the inverse Hessian with a vector directly. Every time we need an inverse Hessian vector product v = H−1b, we invoke conjugate gradients (CG), which transforms the linear system problem into an quadratic optimization problem H−1b ≡ arg minx{ 12x
THx− bTx}. In each iteration of CG, instead of computing H−1b directly, we will compute a Hessian vector product, which can be efficiently done by backprop through the model twice with O(p) time complexity. The aforementioned conjugate gradient method requires the Hessian matrix to be positive definite. However, in practice the Hessian may have negative eigenvalues, since we run a SGD and the final H may not at a local minimum exactly. To tackle this issue, we solve
arg min x {1 2 xTH2x− bTHx}, (15)
whose solution can be shown the same as arg minx{ 12x THx − bTx} since the Hessian matrix is symmetric. H2 is guaranteed to be positive definite as long as H is invertible, even when H has negative eigenvalues. If H2 is not ill-conditioned, we can solve (15) directly. The rate of convergence of CG depends on √ κ(H2)−1√ κ(H2)+1 , where κ(H2) is the condition number of H2, which can be very large if H2 is ill-conditioned. When H2 is ill-conditioned, to stabilize the solution and to encourage faster convergence, we add a small damping term λ on the diagonal and solve arg minx{ 12x T (H2 + λI)x− bTHx}.
Time Complexity As mentioned above, we can get an inverse Hessian vector product in O(p) time. We assume there are p1 parameters in our pretrained model and p2 parameters in our finetuned model. Since F and G are summation of loss with respect to all pretraining or finetuning examples, it takes O(mp1) or O(np2) to compute a Hessian vector product, where m is the number of pretraining examples and n is the number of finetuning examples. We may also subsample the pretraining examples to estimate G(W,U) when the number of pretraining examples is gigantic such as pretraining using One-Billion-word dataset (Chelba et al., 2013) for ELMo etc. For the two inverse Hessian vector products, the time complexity is O(np2r) and O(mp1r), where r is the number of iterations in CG. For other operations in computing the influence score, vector product has a time complexity of O(p1) or O(p1), and computing the gradients of all pretraining examples has a complexity of O(mp1). So computing the total time complexity of computing a multi-stage influence score is O((mp1 + np2)r)

heading: 4 EXPERIMENTS
text: In this section, we will conduct experiment with real datasets on both vision and NLP tasks to show the effectiveness of our proposed influence function. We will show the results in the main text on the vision tasks, and some qualitative results on NLP task related to ELMo are presented in Section C in Appendix.

heading: 4.1 INFLUENCE FUNCTION CORRELATION WITH REAL SCORE
text: To show that our proposed influence score are a good approximation, we evaluate our proposed multi-stage influence function on two CNN models with CIFAR and MNIST datasets. The model structures are shown in Table A in Appendix. We use Tanh for all activations. For both MNIST and CIFAR models, CNN layers are used as embeddings and fully connected layers are task-specific. At the pretraining stage, we train the models with examples from only two classes (“bird" vs. “frog") for
CIFAR and four classes (0, 1, 2, and 3) for MNIST. The resulting embedding is used in the finetuning tasks, where we finetune the model with the examples from the remaining eight classes in CIFAR or the other 6 numbers in MNIST task for classification. In order to make the experiments closer to typical real finetuning situations, we reduce the size of the training set in the finetuning task by subsampling.
In this experiment we test the correlation between individual pretraining example’s multi-stage influence function and the real loss difference when the pretraining examples are removed. We test two cases (as mentioned in Section 3.1) – where the pretrained embedding is fixed, and where it is updated during finetuning. For both MNIST and CIFAR, we first train the embedding with the binary classification for Tpretrain steps. The embedding is then used in the finetuned model. Then we train the finetuned model for Tfinetune steps. Depending on different scenarios which will be explained below, the embedding may be fixed or updated in the finetuning task training. For a given example in the pretraining data, we calculate its influence function score with respect to each test example in the finetuning task test set using the method presented in Section 3. To evaluate this pretraining example’s contribution to the overall performance of the model, we sum up the influence function scores across the whole test set in the finetuning task.
To validate the score, we remove that pretraining example and go through the aforementioned process again by updating the model. In the updating process we further train the pretrained and finetuned models for T ′pretrain and T ′ finetune steps from the original model checkpoints. Note that in this process the pretraining is conducted with the new leave-one-out pretraining training set, while the training set for the finetuning task is intact. Due to computation constraints, we only use the top 100 pretraining examples with the largest influence function absolute values in this experiments to get 100 scoredifference pairs. Then we run a linear regression between the true loss difference values obtained and the influence score computed to show their correlation. The detailed hyperparameters used in these experiments are presented in Appendix B.

heading: 4.1.1 EMBEDDING IS FIXED
text: In Figure 2 we show the correlation results of CIFAR and MNIST models when the embedding is fixed in finetuning task training. Though we make many approximations in our formulation, from Figures 2a and 2b we can see that there is a clear linear correlation between the true loss difference and the influence function scores obtained. The correlation is evaluated with Pearson’s r value. This supports our argument that we can use this score to detect the examples in the pretraining set which contributes most to the model’s performance. In Figure 3 we demonstrate the misclassified test
images in the finetuning task and the images with the largest positive influence function score in the pretraining dataset. Examples with large positive influence score are expected to have negative effect on the model’s performance since intuitively when they are added to the pretraining dataset, the loss of the test example will increase. From Figure 3 we can indeed see that the identified examples are with low quality, and they can be easily misclassified even with human eyes.
One may doubt the effectiveness of the expensive inverse Hessian computation in our formulation. As a comparison, we replace all inverse Hessians in (11) with identity matrices to compute the
influence function score for the MNIST model. The results are shown in Figure 4 with a much smaller Pearson’s r of 0.17. This again shows effectiveness of our proposed influence function.

heading: 4.1.2 EMBEDDING IS UPDATED IN FINETUNE
text: Practically, the embedding can also be updated in the finetuning process. In Figure 5 we show the correlation between true loss difference and influence function score values using (13). We can see that even under this challenging condition, our multi-stage influence function from (13) still has a strong correlation with the true loss difference, with a Pearson’s r = 0.40.

heading: 4.2 THE FINETUNING TASK’S SIMILARITY TO THE PRETRAINING TASK
text: In this experiment, we explore the relationship between influence function score and finetuning task similarity with the pretraining task. Specifically, we study whether the influence function score will increase in absolute value if the finetuning task is very similar to the pretraining task. To do this, we use the CIFAR embedding obtained from a “bird vs. frog" classification and test its influence function scores on two finetuning tasks. The finetuning task A is exactly the same as the pretraining “bird vs. frog" classification, while the finetuning task B is a classification on two other classes (“automobile vs. deer"). All hyperparameters used in the two finetuning tasks are the same. In Figure 6, for both tasks we plot the distribution of the influence function values with respect to each pretraining example. We sum up the influence score for all test examples. We can see that, the first finetuning task influence function has much larger absolute values than that of the second task. The average absolute value of task A influence function score is 0.055, much larger than that of task B, which is 0.025. This supports the argument that if pretraining task and finetuning task are similar, the pretraining data will have larger influence on the finetuning task performance.

heading: 4.3 INFLUENCE FUNCTION SCORE WITH DIFFERENT NUMBER OF FINETUNE EXAMPLES
text: We also study the relationship between the influence function scores and number of examples used in finetuning. In this experiment, we update the pretrained embedding in finetuning stage. We use the same pretraining and finetuning task as in Section 4.1. In Figure 7, model C is the model used in
Section 4.1.2 while in model D we triple the number of finetuning examples as well as the number of finetuning steps. Figure 7 demonstrates the distribution of each pretraining examples’ influence function score with the whole test set. The average absolute value of influence function score in model D is 0.15, much less than that of model C. This indicates that with more finetuning examples and more finetuning steps, the influence of pretraining data to the finetuning model’s performance will decrease. This makes sense as if the finetuning data does not have sufficient information for training a good finetuning task, then pretraining data will have more impact on the finetuning task.

heading: 5 CONCLUSION
text: We introduce a multi-stage influence function to evaluate pretraining examples’ contribution to finetuned model’s prediction. Two different cases are studied: the pretrained embedding is fixed in finetuning or the pretrained embedding is updated in finetuning. We test our method on both CV and NLP tasks. Our experimental results show strong correlation between the proposed multistage influence function scores and the true loss difference when an example is removed from the pretraining data. We believe this is a promising way to connect finetuned model’s performance with pretraining data directly.

heading: A PROOF OF THEOREM 1
text: Proof. Since Θ̂ , Û , Ŵ are optimal solutions, and thus satisfy the following optimality conditions:
0 = ∂
∂Θ F (Ŵ , Θ̂ ) (16)
0 = ∂
∂(W,U) G(Ŵ , Û ) +
∂
∂(W,U) g(z, Ŵ , Û ), (17)
where ∂(W,U) means concatenate the U and W as [W,U ] and compute the gradient w.r.t [W,U ]. We define the changes of parameters as ∆W = Ŵ − Ŵ , ∆Θ = Θ̂ − Θ̂, and ∆U = Û − Û . Applying Taylor expansion to the rhs of (17) we get
0 ≈ ∂ ∂(W,U)
G(W ∗, U∗) + ∂2G(W ∗, U∗)
∂(W,U)2
[ ∆W ∆U ] + ∂g(z,W ∗, U∗) ∂(W,U) + ∂2g(z,W ∗, U∗) ∂(W,U)2 [ ∆W ∆U ] (18)
Since W ∗, U∗ are optimal of unperturbed problem, ∂∂(W,U)G(W ∗, U∗) = 0, and we have[
∆W ∆U
] ≈ − ( ∂2G(W ∗, U∗)
∂(W,U)2 +
∂2g(z,W ∗, U∗)
∂(W,U)2
)−1 ( ∂g(z,W ∗, U∗)
∂(W,U) ) (19)
Since → 0, we have further approximation[ ∆W ∆U ] ≈ ( ∂2G(W ∗, U∗)
∂(W,U)2
)−1 ( ∂g(z,W ∗, U∗)
∂(W,U) ) (20)
Similarly, based on (16) and applying first order Taylor expansion to its rhs we have
0 ≈ ∂F (W ∗,Θ∗) ∂Θ + ∂2F (W ∗,Θ∗) ∂Θ∂W ·∆W + ∂2F (W ∗,Θ∗) ∂Θ2 ∆Θ . (21)
Combining (21) with (20) we have
∆Θ ≈ ( ∂2F (W ∗,Θ∗)
∂Θ2 )−1 · (∂
2F (W ∗,Θ∗) ∂Θ∂W ) · [(∂2G(W ∗, U∗) ∂(W,U)2 )−1 ( ∂g(z,W ∗, U∗) ∂(W,U) ) ] W
where [·]W means taking the W part of the vector. Therefore,
Iz,W := ∂Ŵ ∂ ∣∣ =0 = − [(∂2G(W ∗, U∗) ∂(W,U)2 )−1 ( ∂g(z,W ∗, U∗) ∂(W,U) ) ] W
(22)
Iz,Θ := ∂Θ̂ ∂ ∣∣ =0 = ( ∂2F (W ∗,Θ∗) ∂Θ2 )−1 · (∂ 2F (W ∗,Θ∗) ∂Θ∂W ) · [(∂2G(W ∗, U∗) ∂(W,U)2 )−1 ( ∂g(z,W ∗, U∗) ∂(W,U) ) ] W
(23)

heading: B MODELS AND HYPERPARAMETERS FOR THE EXPERIMENTS IN
text: SECTIONS 4.1, 4.2 AND 4.3
The model structures we used in Sections 4.1, 4.2 and 4.3 are listed in Table A. As mentioned in the main text, for all models, CNN layers are used as embeddings and fully connected layers are task-specific. The number of neurons on the last fully connected layer is determined by the number of classes in the classification. There is no activation at the final output layer and all other activations are Tanh.
• For MNIST experiments in Section 4.1.1, we train a four-class classification (0, 1, 2, and 3) in pretraining. All examples in the original MNIST training set with with these four labels are used in pretraining. The finetuning task is to classify the rest six classes, and
we subsample only 5000 examples to finetune. The pretrained embedding is fixed in finetuning. We run Adam optimizer in both pretraining and finetuning with a batch size of 512. The pretrained and finetuned models are trained to converge. When validating the influence function score, we remove an example from pretraining dataset. Then we re-run the pretraining and finetuning process with this leave-one-out pretraining dataset starting from the original models’ weights. In this process, we only run 100 steps for pretraining and finetuning as the models converge. When computing the influence function scores, the damping term for the pretrained and finetuned model’s Hessians are 1× 10−2 and 1× 10−8, respectively. We sample 1000 pretraining examples when computing the pretraind model’s Hessian summation. • For CIFAR experiments in Section 4.1.1, we train a two-class classification (“bird" vs “frog")
in pretraining. All examples in the original CIFAR training set with with these four labels are used in pretraining. The finetuning task is to classify the rest eight classes, and we subsample only 5000 examples to finetune. The pretrained embedding is fixed in finetuning. We run Adam optimizer to train both pretrained and finetuned model with a batch size of 128. The pretrained and finetuned models are trained to converge. When validating the influence function score, we remove an example from pretraining dataset. Then we re-run the pretraining and finetuning process with this leave-one-out pretraining dataset starting from the original models’ weights. In this process, we only run 6000 steps for pretraining and 3000 steps for finetuning. When computing the influence function scores, the damping term for the pretrained and finetuned model’s Hessians are 1× 10−8 and 1× 10−6, respectively. Same hyperparameters are used in experiments in Sections 4.2 and 4.3. We also use these hyperparameters in Section 4.1.2’s experiments, except that the pretrained embedding is updated in finetuning and the number of finetuning steps is reduced to 1000 in validation. The α constant in Equation 14 is chosen as 0.01. We sample 1000 pretraining examples when computing the pretrained model’s Hessian summation.
Dataset MNIST CIFAR
Embedding
CONV 32 5×5+1 CONV 32 3×3+1 MAX-POOL 2×2 +2 CONV 64 4×4+1
CONV 64 5×5+1 MAX-POOL 2×2 +2 MAX-POOL 2×2 +2 CONV 128 2×2+1
MAX-POOL 2×2 +2 CONV 128 2×2+1 MAX-POOL 2×2 +2
Task specific FC <# classes> FC 1500FC <# classes>
Table A: Model Architectures. “CONV k w×h+s” represents a 2D convolutional layer with k filters of size w×h using a stride of s in both dimensions. “MAX-POOL w×h+s” represents a 2D max pooling layer with kernel size w×h using a stride of s in both dimensions. “FC n” = fully connected layer with n outputs. All activation functions are Tanh and last fully connected layers do not have activation functions. The number of neurons on the last fully connected layer is determined by the number of classes in the task.

heading: C ELMO EXPERIMENT
text: In this section we show influence function score results with ELMo. The finetune task is a binary sentiment classification of twitter1 and the ELMo model is pretrained on a one-billion-word dataset Chelba et al. (2013). For the finetuned model, we add a hidden layer with 64 neurons and an output layer to build the classifier. The activation function is Tanh. For simplicity, we use the original pretrained ELMo embedding and the embedding is fixed in finetuning. We randomly sample a subset of 1000 sentences from one-billion-word dataset. For a test sentence, we list the pretrain sentences with the largest and the smallest absolute influence function score values in one-billion-word dataset.
1 https://datahack.analyticsvidhya.com/contest/linguipedia-codefest-natural-language-processing-1/#data_dictionary
Test Sentence Max abs. score Sentence in Pretrain Min abs. score Sentence in Pretrain
Finally a transparent silicon case Thanks to my uncle :) #yay #Sony #Xperia #S #sonyexperias. . .
-0.0049 Prof Slobodchikoff details the experiments he has done to reveal the hidden structure of the prairie dog ’s language within the BBC natural history programme " Prairie dogs , talk of the town , " broadcast as part of the Natural World documentary series .
6.74× 10−9 He will be there to help you . "
Bout to go shopping again listening to music #iphone #justme #music #likeforlike #followforfollow. . .
0.0014 We are seeing the first big systematic investment in dance .
−6.30× 10−9 The war seemed to energize her , and she began to hang out with the American journalists based in London .
Ha! Not heavy machinery but it does what I need it to. @Apple really dropped the ball with that design. #drinkyourhaterade
0.00052 He ’s been on the move since his comeback victory over Juan Manuel Marquez more than two weeks ago , a brutally efficient boxing display that generated a staggering 1 million pay-per-v iew buys . −2.25× 10−9 Taylor said the syndicated TV psychologist broached the idea of the show to Spears ’ handlers , who eventually decided that such a show would be " detrimental " to the family .
Table B: The list of test sentences and pretraining sentences with the largest and the smallest absolute influence function score values in our subset of pretraining data. The subset consists of 1000 random sentences from one-billion-word, which is used to pretrained ELMo embedding.

Review summaries:
The authors derive the influence function of models that are first pre-trained and then fine-tuned. This extends influence functions beyond the standard supervised setting that they have been primarily considered in. To do so, the authors make two methodological contributions: 1) working through the calculus for the pre-training setting and deriving a corresponding efficient algorithm, and 2) adding $L_2$ regularization to approximate the effect of fine-tuning for a limited number of gradient steps.

This is an analysis paper of pretraining with the tool “influence function”. First, the authors calculate the influence score for the models with/without pretraining, and then propose some implementation details (i.e., use CG to estimate the inversed Hessian). To calculate the influence function of a model with pretraining, the authors use an approximation f(w)+||w-w*||, where w* is pretrained.