MULTI-STAGE INFLUENCE FUNCTION
Authors: Hongge Chen
Si Si
Yang Li
Ciprian Chelba
Sanjiv Kumar
Duane Boning
Cho-Jui Hsieh
Venue:  (2020)
Keywords: influence function
multistage training
pretrained model

Abstract: Multi-stage training and knowledge transfer from a large-scale pretrain task to various fine-tune end tasks have revolutionized natural language processing (NLP) and computer vision (CV), with state-of-the-art performances constantly being improved. In this paper, we develop a multi-stage influence function score to track predictions from a finetune model all the way back to the pretrain data. With this score, we can identify the pretrain examples in the pretrain task that contribute most to a prediction in the fine-tune task. The proposed multi-stage influence function generalizes the original influence function for a single model in Koh et al 2017, thereby enabling influence computation through both pretrain and fine-tune models. We test our proposed method in various experiments to show its effectiveness and potential applications.

Under review as a conference paper at ICLR 2020
MULTI-STAGE INFLUENCE FUNCTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Multi-stage training and knowledge transfer from a large-scale pretraining task to various finetuning tasks have revolutionized natural language processing (NLP) performances constantly being
and computer vision (CV), with improved. In this paper. develop multi-stage influence function score track predictions from finetuned model all the way back to the pretraining data. identify the pretraining examples in the pretraining task
With this score, that contribute most to a prediction in the finetuning task. The proposed multi- stage influence function generalizes the original influence function for single model in Koh & Liang (2017), thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks. We test our proposed method in various experiments to show its effectiveness and potential applications.
INTRODUCTION
Multi-stage training has become increasingly important and has achieved state-of-the-an results in many tasks. In NLP applications, it is now common practice to first learn word embeddings (e.g.. word2vee (Mikolov et al. 2013), GloVe (Pennington et al.) 2014)) or contextual representations (e.g., ELMo (Peters et al. 2018), BERT (Devlin et al., 2018)) from large unsupervised corpus, and then refine or finetune the model on supervised end tasks. In computer vision applications, is common use pretrained CNN as a feature extractor and only finetune top-layer networks through training on the end task. Also, it has been demonstrated that pretraining ResNet (He et al. (016)) with large hashtag data can greatly benefit many end tasks (Mahajan al., 2018). Intuitively, the successes of these multi-stage learning paradigms are due to knowledge transfer from pretraining tasks to the end task. However, current approaches using multi-stage learning are usually based on trial-and-error and many fundamental questions remain unanswered. For example, which part of the pretraining data/task contributes most to the end task? How can one detect "false transfer" where some pretraining data/task could be harmful for the end task? If testing point is wrongly predicted by the finetuned model, can we trace back to the problematic examples in the pretraining data? Answering these questions requires quantitative measurement of how the data and loss function in the pretraining stage influence the end model, which has not been studied in the past and will be the main focus of this paper.
To find the most influential training data responsible for model's prediction, the influence function was first introduced by Cook & Weisberg (1980) from robust statistics point of view. Recently, as large-scale applications become more challenging for influence function computation, Koh & Liang (2017) proposed to use a first-order approximation to measure the effect of removing one training point on the model's prediction, to overcome computational challenges. More broadly. there are many works using influence functions to investigate the impact of training data on models in various machine learning applications, such as tracing back the origins of bias in the word embeddings generated by GloVe (Brunet et al. 2019). and understanding and mitigating disparate impact to improve model fairness (Wang et al. 2019). However, all of the existing influence score computation algorithms study the case of single-stas stage training where there is only one model with one set of suiiing/prediction data in the training process. To the best of our knowledge, the influence of pretraining data on a subsequent finetuning task and model has not been studied, and it is nontrivial to apply the original influence function in (Koh & Liang 2017) to this scenario.

---

Under review as a conference paper at ICLR 2020
In this work, we derive the influence function from pretraining data to the end task in multi-stage training. Since the computation involves several expensive Hessian vector products, we also show how to compute the influence function efficiently in large-scale problems. Based this technique, we show that
in real datasets and experiments across various vision and NLP tasks, predictions using the technique and actual influence for the pretraining data to the finetuned model are highly correlated (Pearson's r score to be around 0.6). This shows the effectiveness of our proposed technique for computing the influence scores in multi-stage models;
the influence for the pretraining data to the finetuned model can be split into two parts: the influence of the pretraining data on the pretrained model, and influence of the pretraining data on the finetuned model. Therefore the testing data from the finetuning task will be impacted by changes in the pretraining data, which can be quantified using our proposed technique;
the influence of the pretraining data on the finetuning task is highly dependent on 1) the similarity of two tasks or stages, 2) and the number of training data in the finetuning task. Thus proposed technique provides novel way to measure how the pretraining data helps or benefits the finetuning data.
2
RELATED WORK
Multi-stage model training that trains models in many stages on different tasks to improve the end- task has been used widely in many machine learning areas, such as in transfer learning (Ando & Zhang 2005) and shot learning (Larochelle et al. 2008). Recently, multi-stage model training has achieved state-of-the-an performance by learning large embeddings or data representation as the pretraining step very large pretraining dataset, which is then followed by a finetune step with further training on the end-task. Examples include the recently proposed BERT (Devlin et al.) 2018), which learns contextual embeddings on large corpus with the pretraining tasks chosen to be predicting the masked words in a sentence and predicting whether one sentence is after another sentence. This contextual embedding then used in finetuning tasks, such as a question answering task. ELMo (Peters et al., 2018) is widely used in multi-stage model training as a sentence feature extractor to benefit the end-task. Similarly, there are some works in computer vision that train an image representation model on large number of images as the pretraining step, and then use the resulting features to finetune another task, such as particular image classification tasks. For example, (Mahajan et al.) 2018) uses ResNet in the pretraining step, and the finetuning task is based on hashtag data. The rationale for this multi-stage model is that the pretraining task can learn some common or latent representation which could benefit the end task.
Another related line of research is on understanding machine learning models. One category of research is to explain predictions with respect to model variables, and trace back the contribution of variables to the prediction. For example, Oramas et al. (2019) automatically detects internal features in the set of classes in the pretrained model that relevant to interpreting the prediction, and shows for various vision tasks that the proposed scheme can produce detailed explanations based on the features that are relevant to the targeted classes. Guo al. (2019) aims to interpret variable-wise hidden states in an LSTM model to quantify variable importance and variable-wise temporal importance in the model.
Closely related research has sought to connect model prediction and training data, and trace back the most influential training data that are most responsible for the model prediction. Among them, the influence function (Cook& & Weisberg 1980; Koh & Liang, 2017). which aims to model the prediction changes when training data added/removed, has been shown be effective in many applications. There is a series of work on influence functions, including investigating the influence of a group of data on the prediction (Koh et al., 2019), using influence functions to detect bias in word embeddings (Brunet et al. 2019), and using it in preventing data poisoning attacks (Steinhardt et al. 2017). etc. All of these works only consider a single stage training procedure, and it is not straightforward to apply the existing influence functions to multi-stage models. In this paper, propose to analyze the influence of pretraining data on predictions in the subsequent finetuned model and end task.
2

---

Under review as a conference paper at ICLR 2020
3 ALGORIHHMS
In this section, we detail the procedure of multi-stage training, show how to compute the influence score for the multi-stage training, and then discuss how scale up the computation.
Proteim boa
Rio sine keef
Inst lca tol
pr parsrstor sut
towssiver lask pasinster
Pre-training
+/-Z
Fine tune data (t.
Figure I: The setting for influence functions in multi-stage models. We consider a two-stage model, where we have first pretrained model, and a second finetuned model for desired end task. We seek to compute the influence of the pretraining data on predictions using testing data in the finetuning task.
3.1 MULTI-STAGE MODEL TRAINING
Multi-stage models, which train different models in consecutive stages, have been widely used in various ML tasks. Mathematically, let Z be the training set for pretraining task with data size [2] = m, and X be the training data for the finetuning task with data size | assume the parameters of the pretrained network have two parts: the parameters W that are shared with the end task, and the task task-specific parameters V that will only be used in the pretraining stage. Note that W' could be a word embedding matrix (e.g., g. in word2vec) or a representation extraction network (e.g., Elmo, BERT, ResNet), while U is usually the last few layers that corresponds to the pretraining task. After training on the pretraining task, we obtain the optimal parameters W U* The pretraining stage can be formulated as
In pretraining stage, we
1 Eg(z, W.U m =E2
Pretrain Stage: W U
arg min
g(2, W, U) min G(W, U); W.U
where g(-) is the loss function for the pretrain task.
In the finetuning stage, the network parameters are W, θ where W is shared with the pretraining task and ④ is the rest of the parameters specifically associated with the finetuning task. We will initialize the W part by W*. Let f-) denote the finetuning loss, there are two cases when finetuning the end-task:
Finetuning Case I: Fixing embedding parameters W W and only finetune O:
θ* arg min 1 Σ fzz W", )) == arg min F(W*, O).
TE
Finetuning Case 2: finetune both the embedding parameters W (initialized from W*) and O. Sometimes updating the embedding parameters W in the finetuning stage is necessary, the embedding parameters from the pretrained model may not be good enough for the finetuning task. This corresponds to the following formulation:
1 ">
W 0 = arg min we
-
f(x,
W, θ) = arg min F(W, θ).
(3)
n ZEN
W.
3.2 INFLUENCE FUNCTION FOR MULTI-STAGE MODELS
We derive the influence function for the multi-stage model to trace the influence of pretraining data on the finetuned model. In Figure I we show the task we interested in solving in this paper. Note
3

---

Under review as a conference paper at ICLR 2020
that use the same definition of influence function as (Koh & Liang 2017) and discuss how to compute it in the multi multi-stage training scenario. As discussed at the end of Section3.11 depending on whether or not
updating the shared parameters W in the finetuning stage, we will derive the influence functions under two different scenarios.
3.2.1 CASE 1: EMBEDDING PARAMETERS W ARE FIXED FINETUNING
To compute the influence of pretraining data on the finetuning task, the main idea is perturb one data example in the pretraining data, and study how that impacts the test data. Mathematically. if we perturb a pretraining data example with loss change by small c, the perturbed model can be defined as
We Ü. arg min CW eg(z, W,U).
(4)
W.U
Note that choices of result in different effects in the loss function from the original solution in (T) For instance, if we set e = # then we are removing the training data z in the pretraining dataset.
For the finetuning stage, since we consider Case where the embedding parameters W are fixed in the finetuning stage, the new model for the end-task or finetuning task will thus be
b. = arg min F' F(W., θ).
(5)
The influence function that measures the impact of a small perturbation = the finetuning loss on a test sample xy from finetuning task is defined as
I 2,2s 0f We, 6.) 1--
(6)
dc
Vef(Ie, W",0"" 10,0 + Vwf(I,, W 0-7T Lew ad.
(7)
ow. where 1,00 measures the influence of the finetuning task parameters θ and L.w measures how e influences the pretrained model W. Therefore we can split the influence of = on the test sample into pieces: one is the impact of the pretrained model I.,w, and the other is the impact of on the finetuned model 12 It is worth mentioning that, due to linearity, if we want to estimate set of test example influence function scores with respect a of pretraining examples, we can simply sum up the pair-wise influence functions, and so define
with I,0: de - and Is,w de 1
ΣΣ' 200 (())
(120)
where {2 and {x contain a set of pretraining data and finetuning test data. Next we will derive these two influence scores I., and [z,w (see the detailed derivations in the appendix) Theorem below.
Theorem 1. For the two-stage training procedure in ① and 2b. we have
aW,
(PG(W.
0g(2, 2g(2,W*,U W*.U*
1,w = de
(10)
0(W,U)2
0(W,U)
w
ad.
8 FFF(W",0")
8 FF 0*)
00 U*)
1,,0 = dE c==
002
D0ew
a(W,U)2
O(W.U)
where l:lw means taking the W part of the vector
By plugging (O into
we finally obtain the influence score of pretraining data the finetuning
task testing point Tt. 12,0r as
0f(z, W OT)T
OFF("* 0*) -- FFFW"" _
0f(z, W (T,)0)
I2.01
Law
00
202
20aw
aw
(11)
Algorithm shows how to compute the influence score in (IT). The pseudocode for computing the influence function in (TT is shown in Algorithm
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Multi-Stage Influence Score with Fixed Embedding
Input: pretrain and finetune models with W Or, and U*; pretrain and finetune training data 2 and X'; test example 24; and pretrain training example z;
Output:
Influence function value Tg,
Compute fintune model's gradients Wf(x,.W'.e" . 89
and ([<=."'8"))
8W
8 8FF(W",6")
) (as, ([/[ss/...6.
w.,"").
Compute the first inverse Hessian vector product Vihop1 (2ie) ==
8132
86
Compute finetune loss's gradient w.r.t W: Of(xs,W". W' """""""""" =Vrr 8 DFF(W"e")
Vhupp
and
aW concatenate it with 0 to make it the same dimension as (W, U); Compute and save the second inverse Hessian vector product
deaw
OW
(#x) o'" ( G'G ( )-1;
B(W,))=
Compute influence function score 12, = VIT
Vhhpp( S(W,))
Bgla.W U')
3.2.2 CASE 2: EMBEDDING PARAMETER W IS ALSO UPDATED IN THE FINETUNING STAGE
For the second finetuning stage case in (3), we will also further train the embedding parameter W from the pretraining stage. When W is also updated in the finetuning stage, it is challenging to characterize the influence since the pretrained embedding W is only used as an initialization. In general, the final model (W**, (²*) may be totally unrelated to W*; for instance, when the objective will converge to the same solution.
function is strongly convex, any initialization of W
However, in practice the initialization of W will strongly influence the finetuning stage in deep learning, since the finetuning objective is usually highly non-convex and initializing W with W* will converge local minimum near W Therefore, we propose to approximate the whole training procedure as
W.U U arg min G(W,U) W.
(12)
W - = arg min W + F(W,
we
where W, U are optimal for the pretraining stage, W O" optimal for the finetuning stage, and 0 S a & 1 is a small value. This is to characterize that in the finetuning stage, we are targeting to find a solution that minimizes F(W, Q) and is close W. In this way, the pretrained parameters are connected with finetuning task and thus influence of pretraining data to the finetuning task can be tractable. The results in our experiments show that with this approximation, the computed influence score can still reflect the real influence quite well.
dé_ aw, Similarly we can have 8c Br and oW. measure the difference between their original optimal solutions in (12) and the optimal solutions from perturbation over the pretraining data z. Similar to 77. the influence function Iz,z, that measures the influence of perturbation to pretraining data on test sample 2,'s 's loss is
Of(a,, We, ②.) ==
0f(2,, W*, "²')"") a(W, O)
Tow. de, Be
I2.0
(13)
de
The influence function of small perturbation of G(W, U) to W, W* O* can be computed following the same approach in Subsection 3.2.1 by replacing W for W and j0*, W*] for @* in (T0. This will lead to
aW.
O(W.U) PGWW,D) U) -1 0g(2, NO(2,W,U, W,U), a(W,U)²
(14)
0=0
W
86,
FFF(W",0")
FF(W"
0 -2aT
0 PG(W, aW Uy (aww.))
-2 W.
2eaW
SWG dc
e=
F 08OW
F(W"
OW2
+ 2aI
a(W.U)
W
After plugging 14 into 13 we will have the influence function Iz...- Similarly, the algorithm for computing 7 T2,z, for Case can follow Algorithm for Case by replacing gradient computation.
5

---

Under review as a conference paper at ICLR 2020
3.3 IMPLEMENTATION DETAILS
The inftuence function computation for multi-stage model is presented in the previous section. As we can see in Algorithm that the influence score computation involves many Hessian matrix operations. which will be very expensive and sometimes unstable for large-scale models. We used several strategies speed up the computation and make the scores more stable.
Large Hessian Matrices A Hessian matrix H has size of p x p. where p is the number of parameters in the model. For large deep learning models with thousands or even millions of parame- ters, it is almost impossible fit a p p Hessian into memory. Also, to invert a Hessian requires O(p") operations. Similar to Koh & Liang (0117), we avoid explicitly computing and storing the Hessian matrix and its inverse, and instead compute product of the inverse Hessian with vector directly. Every time we need an inverse Hessian vector product v = H H- we invoke conjugate gradients (CG), which transforms the linear system problem into an quadratic optimization problem -- = arg min, { fx= Hx - z. In each iteration of CG, instead of computing H- directly, we will compute a Hessian vector product, which can be efficiently done by backprop through the model twice with Opp) time complexity. The aforementioned conjugate gradient method requires the Hessian matrix to be positive definite. However, in practice the Hessian may have negative eigenvalues, since we run SGD and the final H may not at local minimum exactly. To tackle this issue, we solve
arg min! x H2 x- b" Hx),
(15)
2
whose solution can be shown the same as arg min, 2 11=TTT Hx bT x} since the Hessian matrix symmetric. H2 is guaranteed be positive definite as long as H is invertible, even when H has negative eigenvalues. If 2 is not ill-conditioned, we can solve (15) directly. The rate of
convergence of CG depends on V=(HP)11 where K² k((1²) is the condition number of H22 which V Vx(H")+1
can be very large if H2 is ill-conditioned. When H is ill-conditioned, to stabilize the solution and to encourage faster convergence, we add a small damping term on the diagonal and solve arg min, x (H2 + A1)x= - 6d Hx).
Time Complexity As mentioned above, we can get an inverse Hessian vector product in (pp) time. We assume there are Pi parameters our pretrained model and p2 parameters in our finetuned model. Since F and G are summation of loss with respect to all pretraining or finetuning examples, it takes O(mp1) or O(p22) to compute Hessian vector product, where m is the number of pretraining ex- amples and is the number of finetuming examples. We may also subsample the pretraining examples to estimate G(W, U) when the number of pretraining examples is gigantic such as pretraining using One- Billion-word dataset (Chelba et al., 2013) for ELMo etc. For the two inverse Hessian vector products, the time complexity is O(np2r) and Omppr)) where r is the number of iterations in CG. For other operations in computing the influence score, vector product has time complexity of O(pi) or O(pi), ), and computing the gradients of all pretraining examples has a complexity of O(mp1): So computing the total time complexity of computing multi-stage influence score is O((mp) n122rr)
EXPERIMENTS
In this section, we will conduct experiment with real datasets on both vision and NLP tasks to show the effectiveness of our proposed influence function. We will show the results in the main text on the vision tasks, and some qualitative results on NLP task related to ELMo are presented in Section C in Appendix.
4.1 INFLUENCE FUNCTION CORRELATION WITH REAL SCORE
To show that our proposed influence score are good approximation, we evaluate our proposed multi-stage influence function on two CNN models with CIFAR and MNIST datasets. The model structures shown in Table il Appendix. We use Tanh for all activations. For both MNIST and CIFAR models, CNN layers are used as embeddings and fully connected layers are task-specific. At the pretraining stage, we train the models with examples from only two classes ("bird" vs. "frog") for

---

Under review as a conference paper at ICLR 2020
CIFAR and four classes (0, 1, 2, and 3) for MNIST. The resulting embedding is used in the finetuning tasks, where we finetune the model with the examples from the remaining eight classes in CIFAR or the other 6 numbers in MNIST task for classification. In order to make the experiments closer to typical real finetuning situations, we reduce the size of the training set in the finetuning task by subsampling.
In this experiment we test the correlation between individual pretraining example's multi-stage influence function and the real loss difference when the pretraining examples are removed. We test two cases (as mentioned in Section 3.1) ) - where the pretrained embedding is fixed, and where it updated during finetuning. For both MNIST and CIFAR, we first train the embedding with the binary classification for pretrain steps. The embedding is then used in the finetuned model. Then we train the finetuned model for linetuue steps. Depending on different scenarios which will be explained below, the embedding may be fixed or updated in the finetuning task training. For a given example in the pretraining data, we calculate its influence function score with respect to each test example in the finetuning task test set using the method presented in Section 3 To evaluate this pretraining example's s contribution to the overall performance of the model, we sum up the influence function scores across the whole test set in the finetuning task.
To validate the score, we remove that pretraining example and go through the aforementioned process again by updating the model. In the updating process further train the pretrained and finetuned models for T and Tieetume steps from the original model checkpoints. Note that in this process the pretraining is conducted with the new leave-one-out pretraining training set, while the training set for the finetuning task is intact. Due to computation constraints, we only use the top 100 pretraining examples with the largest influence function absolute values in this experiments to get 100 score- difference pairs. Then we run linear regression between the true loss difference values obtained and the influence score computed to show their correlation. The detailed hyperparameters used in these experiments are presented in Appendix B
4.1.1 EMBEDDING FIXED
In Figure 2 we show the correlation results of CIFAR and MNIST models when the embedding is fixed in finetuning task training. Though we make many approximations in our formulation, from Figures 2a and 2b we can see that there is clear linear correlation between the true loss difference and the influence function scores obtained. The correlation evaluated with Pearson's value. This use this score to detect the examples in the pretraining set which we demonstrate the misclassified test
supports argument that
contributes most to the model's performance. In Figure 3
6006810
Scote Valae
Score Value
6.00999
100004
0.003
Jo0ces1s
-60000
True Loss Difference
Truc Loss Differencce
(a) CIFAR Pearson's r 0.62.
(b) MNIST Pearson's 's r 0.47.
Figure 2: CIFAR and MNIST model true loss difference vs. the influence function scores by 8). The
loss is calculated the sum of all test examples.
images in the finetuning task and the images with the largest positive influence function score in the pretraining dataset. Examples with large positive influence score are expected to have negative effect on the model's performance since intuitively when they are added to the pretraining dataset, the loss of the test example will increase. From Figure we can indeed see that the identified examples are with low quality, and they can be easily misclassified even with human eyes.
One may doubt the effectiveness of the expensive inverse Hessian computation in our formulation. As comparison, we replace all inverse Hessians in [1] with identity matrices compute the
7

---

Under review as a conference paper at ICLR 2020
influence function score for the MNIST model. The results shown in Figure 4 with much smaller Pearson's 's r of 0.17. This again shows effecctivneess of our proposed influence function.
4.1.2 EMBEDDING IS UPDATED IN FINETUNE
Practically, the embedding can also be updated in the finetuning process. In Figure S we show the correlation between true loss difference and influence function score values using [3 We can see
that even under this challenging condition, our multi-stage influence function from 13 still has
strong correlation with the true loss difference, with a Pearson's r 0.40.
4.2 THE FINETUNING TASK'S SIMILARITY TO THE PRETRAINING TASK
In this experiment, we explore the relationship between influence function score and finetuning task similarity with the pretraining task. Specifically, study whether the influence function score will increase in absolute value if the finetuning task is very similar to the pretraining task. To do this, we the CIFAR embedding obtained from "bird frog" classification and test its influence function scores on two finetuning tasks. The finetuning task A is exactly the same as the pretraining "bird vs. frog" classification, while the finetuning task B is classification on two other classes ("automobile vs. deer") )) All hyperparameters used in the two finetuning tasks are the same. In Figure O for both tasks we plot the distribution of the influence function values with respect each pretraining example. We sum up the influence score for all test examples. We can see that, the first finetuning task influence function has much larger absolute values than that the second task. The average absolute value of task A influence function score is 0.055, much larger than that of task B, which is 0.025. This supports the argument that if pretraining task and finetuning task are similar, the pretraining data will have larger influence the finetuning task performance.
4.3 INFLUENCE FUNCTION SCORE WITH DIFFERENT NUMBER OF FINETUNE EXAMPLES
We also study the relationship between the influence function scores and number of examples used in finetuning. In this experiment, we update the pretrained embedding in finetuning stage. We use the same pretraining and finetuning task in Section 4.1 In Figure model C is the model used in
P pretrain example influence score=91.6 true label=2
test example prediction=6 true label=5
test example prediction= true label=" automobile"
pretrain example
influence score=1060. true label=" "bird"
Figure 3: Identifying the pretrain example with largest influence function score which contributes an eтror in the finetune task.
6001
800000
Score Value
#70300
Score
-6001
-10000
-S.8003 -BXELOX -uukio -1.2244 -0.000
-5.75h
-1200
-E.IA
True
True
Figure MNIST model true loss differ- ence and influence function with all Hes- sians replaced by identity matrices. Pear- son's 0.17.
Figure 5: CIFAR example using lz reg- ularization formulation in (33 when em- bedding not fixed in finetune. Pearson's 0.40.
8

---

Under review as a conference paper at ICLR 2020
bird frog
10"
C:
B: automobile vs. deer
D: 15k finetune data
Probability Density
12-1
20-
20-2
10-9
-255
-150
-75
-5.1
-25
56
-28
-30
20 Influence Function Score Value
Influence Function Score Value
Figure 6: Two different finetuning task dis- tribution of influence function scores with respect to each pretrain example. Each pre- training example's s influence score value summed up for all test examples. The pre- trained embedding is fixed in finetuning. For both finetuning tasks, the pretrained model the same, and is trained using "bird frog" in CIFAR. For model A, finetun- ing task and pretraining task are the same. The average absolute values of influence function scores for models A and are 0.055 and 0.025, respectively.
Two different finetuning task dis- tribution of influence function scores which respect to each pretraining example. Each pretraining example's influence score value is summed up for all test examples. The pre- trained embedding also updated in fine- tuning. The pretraining and finetuning tasks are the same as in Section 4.1.2 Model D's number of finetuning examples and finetun- ing steps are 3X of model C's. The average absolute values of influence function scores for Models and D are 0.22 and 0.15, re- spectively.
Figure 7:
Section 4.1.2 while in model D triple the number of finetuning examples well the number of finetuning steps. demoostrates the distribution of each pretraining examples' influence function score with the whole test set. The average absolute value of influence function model 15, much less than that model C. This indicates that with more finetuning examples and more finetuning steps, the influence of pretraining data the finetuning model's performance will decrease. This makes sense as if the finetuning data does not have sufficient information for training a good finetuning task, then pretraining data will have more impact on the finetuning task.
in
CONCLUSION
We introduce a multi-stage influence function to evaluate pretraining examples' contribution to finetuned model's prediction. Two different cases are studied: the pretrained embedding is fixed in finetuning or the pretrained embedding is updated in finetuning. We test our method both CV and NLP tasks. Our experimental results show strong correlation between the proposed multi- stage influence function scores and the true loss difference when an example is removed from the pretraining data. We believe this is promising way to connect finetuned model's performance with pretraining data directly.
REFERENCES
Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. J. Mach. Learn. Res., 6:1817-1853, 2005.
Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. Understand- ing the origins of bias in word embeddings. In ICML, pp. 803-811, 2019.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. 2013.
arXiv preprini
R. Dennis Cook and Sanford Weisberg. Characterizations of empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-508, 1980.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18:0::81000000 04.004 2018.

---

Under review as a conference paper at ICLR 2020
Tian Guo, Tao Lin, and Nino multi-variable data. In Proceedings of the 36th International Conference on Machine Learning, pp. 2494-2504, 2019.
Exploring interpretable LSTM neural networks
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1885-1894. JMLR. org, 2017.
Pang Wei Koh, Kai-Siang Ang, Hubert H. K. Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In NeurlPS, 2019.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In Proceed- ings of the 23rd National Conference on Artificial Intelligence Volume AAAL00,, pp. 646-651, 2008.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference Computer Vision (ECCV), pp. 181-196, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-3119, 2013.
Jose Oramas, Kaili Wang, and Tinne Tuytelaars. Visual explanation by interpretation: Improving visual feedback capabilities of deep neural networks. In International Conference on Learning Representations, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv: 1802.05.65, 2018.
Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 3520-3532, 2017.
Hao Wang, Berk Ustun, and Flávio P. Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In Proceedings the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 661 1-666727, 2019.
10

---

Under review as a conference paper at ICLR 2020
A PROOF OF THEOREM
Proof. Since 6e, Uc. We are optimal solutions, and thus satisfy the following optimality conditions:
0 r(W., 6.)
(16)
de
G(W + a(w, u) W. W., a 0 ()
(17)
where U(W means concatenate the U and W as [W, U and compute the gradient w.r.t [W.
We define the changes of parameters as AW, W. W. AO, Ô. -, and AU, Ü. U.
Applying Taylor expansion to the rhs of (17) we get
,U)++ + PGWW, U*)
AW. O(W,U)² AU.
Øg(z, W*,
8z W WW*)))
AW
0 ~ D(W.U G(W*
+
0(W.U)
d(W
AU. (18)
Since W*, are optimal of unperturbed problem, DWW,)
8 G(W", U*) = 0, and we have
Aw. [PG(WW U*) AU.
-
8 f2, "o,,W"U)) .U))
W
0g(2, W U))
(19)
a(W.U)²
a(W,U)²
a(W.U)
Since - O, we have further approximation
AW:
(DO(W)))))
-1 ag(z, W U*)
O(W,U) )
(20)
AU.
O(W,U)²
Similarly, based (16) and applying first order Taylor expansion to its rhs we have
OF(W*, O*)
0² FFF(W=,
8 F²F(WW, O0))
AW,
002 AO.
(21)
ae
Deow
Combining 21 with 20
we have
2 F(W" O*)
OFF(W"
(PGG -1 a(W,U)
11,0000000 0g(2, W' UU)
AO, R3
002
DOaW
O(W.U)
W
where Fl means taking the W part of the vector. Therefore,
- O0(2,W" U aθ, 88 F(W",, 0*)
aw.
-
(PG(W"U")
I,W
(22)
c=
a(W,U)²
a(W.U)
w FF 07).
o(W.Up ( ,U*) 09(2, 00,0),,,, w0 ,U))
I,,0
de c==0
002
deew
O(W.U)
w
(23)
B MODELS AND HYPERPARAMETERS FOR THE EXPERIMENTS IN SECTIONS 4.1 4.2 AND 4.3
The model structures we used in Sections 4.2 and 4.3 are listed in Table A As mentioned in the main text, for all models, CNN layers are used as embeddings and fully connected layers are task-specific. The number of neurons on the last fully connected layer determined by the number of classes in the classification. There is no activation at the final output layer and all other activations are Tanh.
For MNIST experiments in Section 4.1.1 train four-class classification (0. 1, 2, and 3) in pretraining. All examples in the original MNIST training set with with these four labels used in pretraining. The finetuning task is to classify the rest six classes, and
11

---

Under review as a conference paper at ICLR 2020
we subsample only 5000 examples to finetune. The pretrained embedding is fixed in finetuning. We run Adam optimizer in both pretraining and finetuning with a batch size of 512. The pretrained and finetuned models are trained converge. When validating the influence function score, we remove an example from pretraining dataset. Then we re-run the pretraining and finetuning process with this leave-one-out pretraining dataset starting from the original models' weights. In this process, we only run 100 steps for pretraining and finetuning as the models converge. When computing the influence function scores, the damping term for the pretrained and finetuned model's 's Hessians are 1 x 10 and 1 x 10 respectively. We sample 1000 pretraining examples when computing the pretraind model's Hessian summation.
For CIFAR experiments in Section 4.1.1 we train two-class classification ("bird" vs "frog") in pretraining. All examples in the original CIFAR training with with these four labels are used in pretraining. The finetuning task is to classify the rest eight classes, and we subsample only 5000 examples to finetune. The pretrained embedding is fixed in finetuning. We run Adam optimizer to train both pretrained and finetuned model with batch size of 128. The pretrained and finetuned models are trained to converge. When validating the influence function score, we remove an example from pretraining dataset. Then we re-run the pretraining and finetuning process with this leave-one-out pretraining dataset starting from the original models' weights. In this process, only rum 6000 steps for pretraining and 3000 steps for finetuning. When computing the influence function scores, the damping term for the pretrained and finetuned model's Hessians are 1 10 and 1 10 6 respectively. Same hyperparameters are used in experiments in Sections 4.2 2 and 4.3 We also these hyperparameters in Section 4.1.2 s experiments, except that the pretrained embedding is updated in finetuning and the number of finetuning steps is reduced to 1000 in validation. The a constant Equation [4]s chosen as 0.01. We sample 1000 pretraining examples when computing the pretrained model's ' Hessian summation.
Dataset
MNIST
CIFAR
CONV 32 5x5+1
CONY 3x3+1
MAX-POOL 2x2 +2
CONV 64 4x4+1
CONV 64 6x5+1
MAX-POOL 2x2 -2
Embedding
MAX-POOL. 2x2 +2
CONV 128 2x2+1
MAX-POOL 2x2 #2
CONV 128 2x2+1
MAX-POOL 2x2
FC <t classes>
FC 1500 FC <# classes>
Task specific
Table A: Model Architectures. "CONV i w x x represents a 2D convolutional layer with k filters of size w x h using a stride of s in both dimensions. "MAX-POOL w xh++ represents a 2D max pooling layer with kernel size w x h using stride of in both dimensions. "FC n" = fully connected layer with outputs. All activation functions are Tanh and last fully connected layers do not have activation functions. The number of neurons on the last fully connected layer is determined by the number of classes in the task.
C ELMO EXPERIMENT
In this section show influence function score results with ELMo. The finetune task is bi- nary sentiment classification of twitter" and the ELMo model is pretrained on a one-billion-word dataset dataset[Chelba Chelba et al. (2013). For the finetuned model, add hidden layer with 64 neurons and an output layer to build the classifier. The activation function is Tanh. For simplicity, we use the original pretrained ELMo embedding and the embedding is fixed in finetuning. We randomly sample a subset of 1000 sentences from billion-won dataset. For a test sentence, we list the pretrain sentences with the largest and the smallest absolute influence function score values in me--biliin-woo dataset.
dictionady
12

---

Under review as a conference paper at ICLR 2020
Test Sentence
Max abs. score
Sentence in Pretrain
Min abs. score
Sentence in Pretrain
Finally a transparen silicon Thanks uncle tyay #Sony #Xperta #5 #sonyexpe exprxeeeris....
-0.0049
Prof Stobodchikedf details the experiment he has done to reveal the hidden structure of prairic dog language within the BBC natural history programime Prairie dogs, salk of the brodcast as part of the Natural World documentary series.
6.74 10-9
He will be there help you.
Bout go shopping again listoning to music Hiphone #justme #music #ilkeforlike ffollowforfollow....
0.0014
We seeing the first big gystematic investment dance.
-6.30 x 10
The seemed to energize her, and she began to hong with the American journalists based in London.
Ha! Not heavy machinery but is does what need to. @ Apple veally dropped the ball with that design.
0.00052
He been the hit comeback victory over Juan Manuel Marquez than two weeks ago a brutully efficient boxinn di display that generated
2..2 x 10
Taylor said the syndicated TV psychelogist broachd the idea the show Spears handlers who eventually would be detrimentel family.
Sdringouuhateradee
stoggering million pay-per-v iew buys.
Table The test sentences and pretraining sentences with the largest and the smallest absolute influence function score values in our subset of pretraining data. The subset consists of 1000 random sentences from ane-billion-we world, which is used to pretrained ELMo embedding.
13


Reviews:
[{"review_id": "rkxd5xEcFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors derive the influence function of models that are first pre-trained and then fine-tuned. This extends influence functions beyond the standard supervised setting that they have been primarily considered in. To do so, the authors make two methodological contributions: 1) working through the calculus for the pre-training setting and deriving a corresponding efficient algorithm, and 2) adding $L_2$ regularization to approximate the effect of fine-tuning for a limited number of gradient steps.\n\nI believe that these are useful technical contributions that will help to broaden the applicability of influence functions beyond the standard supervised setting. For that reason, I recommend a weak accept. I have some questions and reservations about the current paper:\n\n1) Does pretraining actually help in the MNIST/CIFAR settings considered? These seem to be non-standard pretraining settings. More generally, can we relate influence to some objective measure that we care about (say test accuracy), for example by showing that removing the top X% of influential pretraining data hurts test accuracy as much as predicted? Minor: section 4.2 also seems non-standard. Are the exact same bird vs. frog examples being used for both pretraining and finetuning?\n\n2) In what situations might we want to examine the influence of pretraining data, and can we design experiments that show those situations? For example, perhaps we're wondering if different types of sentences in the one-billion-word dataset might be more or less useful. Can we verify those claims using these multi-stage influence functions? It is otherwise difficult to assess the utility of the qualitative results (e.g., Figure 3 and Appendix C).\n\n3) It'd be helpful to get a better understanding of the technical contributions of this paper. Specifically, \na. What is the impact of $\\alpha$ in equation 12 and how does it interact with the number of fine-tuning steps taken?\nb. If the Hessian has negative eigenvalues, we can still take $H^{-1}b$ by solving CG with $H^2$, but what does this correspond to? Is the influence equation well defined (or the Taylor approximation justified) if $H$ is not positive definite? \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rJlosyF6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an analysis paper of pretraining with the tool “influence function”. First, the authors calculate the influence score for the models with/without pretraining, and then propose some implementation details (i.e., use CG to estimate the inversed Hessian). To calculate the influence function of a model with pretraining, the authors use an approximation f(w)+||w-w*||, where w* is pretrained. \nThe experiments are conducted on MNIST and CIFAR. \n\n1.\tThe idea of converting a pre-trained model with f(w)+||w-w*|| is interesting. But I do not think the conclusion is very promising and convincing. The authors leverage Pearson correlation to measure the similarity between “true loss difference” and “score value”. However, i do not think the value $0.62$ is significant. As shown in Figure (2), intuitively, the linear correlation between these two values do not hold. Also, I am not quite sure about the practical value of calculating influence scores.\n2.\tThe experiments are conduct on small-scale datasets. I am not sure whether the conclusion holds for larger dataset.\n3.\tPage 7, last paragraph, “we replace all inverse Hessians in (11) with identity matrice”=>why?\n4.\tIn figure 3, what is the relationship between the two MNIST images, and the relationship between the two CIFAR images?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJgdpYkL9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-stage influence function for transfer learning to identify the impact of source samples to the performance of the learned target model on the target domain. It considers two cases: fixed pretrained parameters and fine-tuned parameters.\n\nWhy not to directly add a scaled identity matrix to problem (15) to avoid the non-PSD issue?\n\nHow to use the proposed method to identify source samples that cause negative transfer as discussed in the introduction?\n\nEven using the conjugate gradient method to reduce the complexity, the total complexity is still high as the number of parameters in a deep neural network is large. It is better to report the running time to see the efficiency of the proposed method.\n\nIn transfer learning, there is a setting that source data are not accessible due to, for example, the purpose of the privacy protection. In this case, can influence function be used?\n\nFor presentation, I think it is not correct to use ‘pretrain’ or ‘finetune’ before a noun. They should be replaced with ‘pretrained’ and ‘finetuned’.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the idea of influence functions (aka the implicit function theorem) to multi-stage training pipelines, and also adds an L2 penalty to approximate the effect of training for a limited number of iterations.

I think this paper is borderline.  I also think that R3 had the best take and questions on this paper.

Pros:
 - The main idea makes sense, and could be used to understand real training pipelines better.
 - The experiments, while mostly small-scale, answer most of the immediate questions about this model.

Cons:
 - The paper still isn't all that polished.  E.g. on page 4: "Algorithm 1 shows how to compute the influence score in (11). The pseudocode for computing the influence function in (11) is shown in Algorithm 1"
 - I wish the image dataset experiments had been done with larger images and models.

Ultimately, the straightforwardness of the extension and the relative niche applications mean that although the main idea is sound, the quality and the overall impact of this paper don't quite meet the bar.
Decision: Reject