Influence Functions in Deep Learning Are Fragile
Authors: Samyadeep Basu
Phil Pope
Soheil Feizi
Venue: ICLR 2021 Poster (2021)
Keywords: Influence Functions
Interpretability

Abstract: Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation. A commonly-used (first-order) influence function can be implemented efficiently as a post-hoc method requiring access only to the gradients and Hessian of the model. For linear models, influence functions are well-defined due to the convexity of the underlying loss function and are generally accurate even across difficult settings where model changes are fairly large such as estimating group influences. Influence functions, however, are not well-understood in the context of deep learning with non-convex loss functions.  In this paper, we provide a comprehensive and large-scale empirical study of successes and failures of influence functions in neural network models trained on datasets such as Iris, MNIST, CIFAR-10 and ImageNet. Through our extensive experiments, we show that the network architecture, its depth and width, as well as the extent of model parameterization and regularization techniques have strong effects in the accuracy of influence functions. In particular, we find that (i) influence estimates are fairly accurate for shallow networks, while for deeper networks the estimates are often erroneous; (ii) for certain network architectures and datasets, training with weight-decay regularization is important to get high-quality influence estimates; and (iii) the accuracy of influence estimates can vary significantly depending on the examined test points. These results suggest that in general influence functions in deep learning are fragile and call for developing improved influence estimation methods to mitigate these issues in non-convex setups.

Published as a conference paper at ICLR 2021
INFLUENCE FUNCTIONS IN DEEP LEARNING ARE FRAGILE
Samyadeep Basu, Phillip Pope & Soheil Feizi Department of Computer Science University of Maryland, College Park (sbasul2, 2, pepope, sfeizi @css umd. edu
ABSTRACT
Influence functions approximate the effect of training samples in test-time predic- tions and have a wide variety of applications in machine learning interpretabil- ity and uncertainty estimation. A commonly-used (first-order) influence function can be implemented efficiently as a post-hoc method requiring access only to the gradients and Hessian of the model. For linear models, influence functions are well-defined due to the convexity of the underlying loss function and are gener- ally accurate even across difficult settings where model changes are fairly large such as estimating group influences. Influence functions, however, are not well- understood in the context of deep learning with non-convex loss functions. In this paper, we provide comprehensive and large-scale empirical study of successes and failures of influence functions in neural network models trained on datasets such Iris, MNIST, CIFAR-I0 and ImageNet. Through our extensive experi- ments, we show that the network architecture, its depth and width, as well as the extent of model parameterization and regularization techniques have strong ef- fects in the accuracy of influence functions. In particular, we find that (i) influence estimates are fairly accurate for shallow networks, while for deeper networks the estimates are often erroneous; (ii) for certain network architectures and datasets, training with weight-decay regularization important get high-quality influ- ence estimates; and (iii) the accuracy of influence estimates can vary significantly depending on the examined test points. These results suggest that general in- fluence functions in deep learning are fragile and call for developing improved influence estimation methods to mitigate these issues in non-convex setups.
INTRODUCTION
In machine learning, influence functions (Cook & Weisberg 1980) can be used to estimate the change in model parameters when the empirical weight distribution of the training samples is per- turbed infinitesimally. This approximation is cheaper compute compared to the expensive process of repeatedly re-training the model to retrieve the exact parameter changes. Influence functions could thus be used to understand the effect of removing an individual training point (or, groups of training samples) on the model predictions at the test-time. Leveraging a first-order Taylor's approx- imation of the loss function, (Koh Liang 2017) has shown that a (first-order) influence function, computed using the gradient and the Hessian of the loss function, can be useful to interpret machine learning models, fix mislabelled training samples and create data poisoning attacks.
Influence functions are in general well-defined and studied for models such as logistic regression (Koh & Liang] (007)), where the underlying loss-function is convex. For convex loss functions, fluence functions are also accurate even when the model perturbations are fairly large (e.g. in the group influence case (Koh et al. 2019b [asu et al.) (0200). However, when the convexity assump- tion of the underlying loss function is violated, which the case in deep learning, the behaviour of influence functions is not well understood and is still an open area research. With recent advances in computer vision (Szeliski 2010). natural language processing (Sebastiani) 2002). high-stakes ap- plications such as medicine Lundervold & Lundervold 2018), it has become particularly important
Authors contrihuted equally

---

Published as a conference paper at ICLR 2021
to interpret deep model predictions. This makes it critical to understand influence functions in the context of deep learning, which is the main focus of our paper.
Despite their non-convexity, it is sometimes believed that influence functions would work for deep networks. The excellent work of (Koh & Liang 2017) successfully demonstrated one example of influence estimation for deep network, small (2600 parameters), "all-convolutional" network Springenberg et al. 2015). To the best of our knowledge, this is the one of the few cases for deep networks where influence estimation has been shown work. A question of key importance to practitioners then arises: for what other classes of deep networks does influence estimation work? In this work, we provide comprehensive study of this question and find a pessimistic answer: inftuence estimation is quite fragile for a variety of deep networks.
In the case of deep networks, several factors might have an impact on influence estimates: (i) due to non-convexity of the loss function, different initializations of the perturbed model can lead to significantly different model parameters (with approximately similar loss values); (ii) even the initialization of the model is fixed, the curvature values of the network (i.e. eigenvalues of the Hes- sian matrix) at optimal model parameters might very large in very deep networks, leading to a substantial Taylor's approximation error of the loss function and thus resulting in poor influence esti- Vector product, required
mates; (iii) for large neural networks, computing the exact in computation of influence estimates, can be computationally very expensive. Thus, one needs to Vector product techniques which might be erroneous; resulting have different loss
use approximate
in low quality influence estimates; and finally (iv) different architectures landscape geometries near the optimal model parameters, leading to varying influence estimates.
In this paper, we study aforementioned issues of using influence functions in deep learning through an extensive experimental study on progresivvelyggroviii start our analysis with case study of a small neural network for the Iris dataset where the exact Hessian matrix can be computed. We then progressively increase the complexity of the network and analyse CNN architecture (depth of 6) trained on 10% of MNIST dataset, similar to (Koh & Liang, 2017). Next, we evaluate the accuracy of influence estimates for more complex deep architectures (e.g. ResNets) trained on MNIST and CIFAR-10. Finally, we compute influence estimates on the ImageNet dataset using ResNet-50.
complex models and datasets. We first
We make the following observations through our analysis:
We find that the network depth and width have strong impact on influence estimates. In particular, we show that influence estimates are fairly accurate when the network is shallow, while for deeper models, influence estimates are often erroneous. We attribute this partially to the increasing curvature values of the network as the depth increases.
We observe that the weight decay regularization is important to obtain high quality influ- ence estimates in certain architectures and datasets.
We show that the Lnverse-Hessine Vector product approximation techniques such as stochastic estimation (Agarwal et af.) 2016) are erroneous, especially when the network is deep. This can contribute to the low quality of influence estimates in deep models.
We observe that the choice of test-point has substantial impact the quality of influence estimates, across different datasets and architectures.
In very large-scale datasets such as ImageNet, we have found that even ground-truth influ- ence estimates (obtained by leave-one-out re-training) can be inaccurate and noisy partially due to the model's training and convergence.
These results highlight sensitivity of current influence functions in deep learning and call for devel- oping robust influence estimators to be used in large-scale machine learning applications.
2 RELATED WORKS
Influence functions are primarily used to identify important training samples for test-time predic- tions and debug machine learning models (Koh & Liang 2017). Similar to influence functions, (Chaudhuri & Mykland (993) tackles the problem of approximating dataset using subset of the dataset. In recent times, there is an increase in the applications influence functions for tasks other than interpretability. For e.g. (Schulam & Saria 2019) has used influence functions audit
2

---

Published as a conference paper at ICLR 2021
the reliability of test-predictions. In NLP, influence functions have been used to detect biases in word-embeddinng (Brunet et al. (018)) whereas in the domain of ML security, influence functions have been shown to be effective in crafting stronger data-poisoning attacks (Koh et al.] (019a). In- fluence functions are also effective in the identification of important training groups (rather than an individual sample) (Basu al. 2019 Kohh al.) 2019b). Prior theoretical work (Giordano al.) 2018 2019) have focused on quantifying finite sample error-bounds for influence estimates when compared the ground-truth re-training procedures. Recently, alternative methods to find influ- ential samples in deep networks have been proposed. In (Yeh et al., 2018), test-time predictions are explained by kernel function evaluated the training samples. Influential training examples can also be obtained by tracking the change in loss for test-prediction through model-checkpoints. which are stored during the training time (Pruthi et al. 2020). While these alternative methods (Yeh) et al. 2018 Pruthi et al.) (020)) work well for deep networks in interpreting model predictions, they lack the "jackknife" " like ability of influence functions which makes it useful in multiple applications other than interpretability (e.g. uncertainty estimation).
3 BASICS OF INFLUENCE FUNCTION
Consider h to be a function parameterized by which maps from an input feature space to an output space denoted by V. The training samples are denoted by the set S = { (a,, y.) while the loss function is represented by ((h) for a particular training example 2. The standard empirical risk minimization solves the following optimization problem:
1 ((ho(z))).
0 arg min 12
Up-weighting a training example by an infinitesimal amount e leads a new of model param- eters denoted by 1zp This set of new model parameters (xx) is obtained by solving:
(i arg min TI
Σ
(ho()z)) cz)))
(2)
Removing training point z similar to up- up-wethtting its corresponding weight by c <= -1/n in Equation The main idea used by (Koh & Liang 2017) is to approximate ((2) by the first-order Taylor series expansion around the optimal model parameters represented by 0", which leads to:
(1)) = 0" cH VoE(ho- (chh-- (2)),
where Ho- represents the Hessian with respect model parameters 0* Following the classical
result of (Cook& & Weisherg. 1980), the change in the model parameters (A0 = (-)) (²) on up-
weighting the training example can be approximated by the influence function (I(z)) as follows:
d8c
I(z)
= de le-0 - H H Vel (ha-(z)).
(4)
The change in the loss value for a particular test point z₁ when a training point 2 is up-weighted can be approximated as a closed form expression by the chain rule (Koh & Liang 2017):
I(2, = V (z VV (2)).
(S)
I(z, 2/)/n is approximately the change in the loss for the test-sample when a training sample is removed from the training set. This result is, however, based on the assumption that the underlying loss function is strictly convex in the model parameters 0 and the Hessian Hg- is a positive-definite matrix (Koh & Liang 2017). For large models, inverting the exact Hessian Hg- expensive. In such cases, the Hessian Vector product can be computed efficiently with a combination of Hessian-vector product (Pearlmutter 1994) and optimization techniques (see Appendix for details).
4 WHAT CAN Go WRONG FOR INFLUENCE FUNCTIONS IN DEEP LEARNING?
First-order influence functions (Koh & Liang 2017) assume that the underlying loss function is convex and the change in model parameters is small when the empirical weight distribution of the training data is infinitesimally perturbed. In essence, this denotes the Taylor's gap in Equation (3)
3

---

Published as a conference paper at ICLR 2021
With Waight Docay
Wegy
Depth
Corelattion With Witth
Nom C3 Roperoae
Panuretar Change
Opcomeee Correet
Sprermnn Comtaston
04
00-
Nom Of Esazt Parmmttrr Change
Norm Exset Paramolar Change
Númber Layors
Witth Layerrs
(a)
(9)
(0)
Figure I:
Iris dataset experimental results (a.b) Comparison of norm of parameter changes com-
puted with influence function re-training; (a) trained with weight-decay; (b) trained without
weight-decay. (c) Spearman correlation vs. network depth. (d) Spearman correlation vs. network
width.
to be small for accurate influence estimate. However in the case of non-convex loss functions, this assumption not generally true. Empirically, we find that the Taylor's gap is strongly affected by common hyper-parameters for deep networks. For example, in Fig. (a.b), we find that for networks trained without a weight-decay regularization on Iris, the Taylor's s gap is large resulting in low quality influence estimates. In a similar vein, when the network depth and width is considerably large (i.e. the over-parameterized regime), the Taylor's 's gap increases and substantially degrades the quality of influence estimates (Fig. 2 Empirically this increase in Taylor's gap strongly correlates with the curvature values of the loss function evaluated at the optimal model parameters as observed in Fig. (b)).
Further complications may arise for larger models, where influence estimations in such settings require an additional approximation to compute the lnverse-Hessian vector product. Nonetheless, we observe in Fig. (a), that on Iris this approximation has only marginal impact on the influence estimation. These results show that that network architecture, syper-parametent and loss curvatures are important factorr for proper influence estimations. In the next section, we discuss these issues in details through controlled experiments on datasets and models of increasing complexity.
EXPERIMENTS
Datasets: We first study the behaviour of influence functions in a small Iris dataset (Anderson, 1936), where the exact Hessian can be computed. Further, we progressively increase the complexity of the model and datasets: we use small MNIST (Koh & Liang, 2017) to evaluate the accuracy of influence functions in a small CNN architecture with a depth of 6. Next, we study influence functions on modern deep architectures trained on the standard MNIST (LeCun et al., 1998) and CIFAR-10 10 (Krizhevsky et al., 2000) datasets. Finally, to understand how influence functions scale to large datasets, we use ImageNet (Deng et al. 2009) to compute the influence estimates.
We evaluate the accuracy of influence estimates at a given test point 2; us- ing both Pearson (Kirch) 2008) and Spearman rank-order correlation (Spearman) 1904) with the ground-truth (obtained by re-training the model) across set of training points. Most of the ex- isting interpretability methods desire that influential examples ranked in the correct order of their importance (Ghorbani et al.) (017). Therefore, to evaluate the accuracy of influence estimates, Spearman correlation is often a better choice.
Evaluation Metrics:
5.1 UNDERSTANDING INFLUENCE FUNCTIONS WHENN THE EXACT HESSIAN CAN BE COMPUTED
Setup: Computing influence estimates with the exact Hessian has certain advantages in our study: a) bypasses inverse-Hessian Vector product approximation techniques which induce errors com- puting influence estimates. Thus, we can compare influence estimates computed with exact vs. approximate inverse-Hessian Vector products to quantify this type of error; b) The deviation of the parameters computed with the influence function from the exact parameters can be computed ex- actly. This information can be useful to further quantify the error incurred by (first-order) influence estimates in the non-convex setup. However, computations of the exact Hessian matrix and its in- verse are only computationally feasible for models with small number of parameters. Thus, we use the Iris dataset along with small feed-forward neural network to analyse the behaviour of influence
4

---

Published as a conference paper at ICLR 2021
Accuracy Of Stochaskic Approsimation
Top Elgenvalue incraases With Dxpth
Taylor Expansion Gep Increasies With Depth
1.8
Stochaatc Estimation
Exact Invers INP
Topp Figaruatu Hacean
2000
Speamer Consaatinn
Spearman Correaaton
08-
1500
06-
3000
05
04
Number Layers (a)
Number Of
Number Of Layers
(D)
(C)
Figure 2: Iris dataset experimental results; (a) Spearman correlation of influence estimates with the ground-truth estimates computed with stochastic estimation vs. exact evvrse-Hessian vector product. (b) Top eigenvalue of the Hessian the network depth. (c) Spearman correlation between the norm of parameter changes computed with influence function vs. re-training.
function computed with the exact Hessian in a non-convex setup. We train models to convergence for 60k iterations with full-batch gradient descent. To obtain the ground-truth estimates, we re- train the models for 7.5k steps, starting from the optimal model parameters. For our analysis, we choose the test-point with the maximum loss and evaluate the accuracy of influence estimates with the ground-truth amongst of the top 16.6% of the training points. Through our experiments with the exact Hessian, we answer some relevant questions related to how properties of the network such as depth, width and regularizers (e.g. weight-decay) affect the influence estimates.
The Effect of Weight-Decay: One of the simpler and common regularization techniques used to train neural networks is weight-decay regularization. In particular, a term x||0112. penalizing the scaled norm of the model parameters is added to the objective function, during training, where is a hyperparameter which needs to be tuned. We train a simple feed-forward network with and without weight-decay regularization. For the network trained with weight-decay, we observe a Spearman correlation of 0.97 between the influence estimates and the ground-truth estimates. In comparison, for the network trained without a weight-decay regularization, the Spearman correlation estimates decrease 0.508. In this case, we notice that the Hessian matrix is singular, thus a damping factor of 0.001 is added to the Hessian matrix, to make it invertible. To further understand the reason for this decrease in the quality of influence estimates, we compare the following metric across all training examples: a) Norm of the model parameter changes computed by re- re-triini b) Norm of the model parameter changes computed using the influence function (i.e. [Hz=' (a.)). We observe that when the network is trained without weight-decay, changes in model parameters computed with the influence function have a substantially larger deviation from those computed using re- ce-triining. This essentially suggests that the gap in Taylor expansion, using (first- order) influence estimates is large, when the model is trained without weight-decay. We observe similar results with smooth activation functions such tanh (see the Appendix for details).
Vi E [1, ni) (Fig.
The Effect Of Network Depth: From Fig. (c), we see that network depth has a dramatic effect on the quality of influence estimates. For example, when the depth of the network is increased to 8, notice a considerable decrease in the Spearman correlation estimates. To further our understanding about the decrease in the quality of influence estimates when the network is deeper, we compute the gap in the approximation between the ground truth parameter changes (computed by re-training) and the approximate parameter changes (computed using the influence function). To quantify the error gap, we compute the Spearman correlation estimates between the norm of true and approximate parameter changes across the top 16.6% of the influential examples. We find that with increasing depth, the Spearman correlation estimates between the norm of the true and approximate parameter changes decrease. From Fig. (c.,, we see that the approximation error gap is particularly large when the depth of the network is more than 5. We also notice consistent increase in the curvature
of the loss function (Fig. (b)). as the network becomes deeper. This possibly suggests that the curvature information of the network can be an upper bound in the approximation error gap between
With width of 5, depth of and ReL. activations
5

---

Published as a conference paper at ICLR 2021
Port
Nart Al Gecnas
Poarzon Spourass
Comelattion
Comelction
Corresation
-45
Porseeilaag Douael
spcimnrreccge Docay!
Nansontlo Weight Decay)
Poasenthe Woggh Docas!
Spearnr tho Decpl
Different Test-Paints
Difierant Test:
Weight-Decay Factor
(a)
(b)
(c)
Figure 3:
Experiments on small MNIST using CNN architecture. (a) Estimation of influence function with and without weight decay on (a) the top influential points, (b) training points 30th
percentile of influence score distribution. (c) Correlation vs the weight decay factor (evaluated on the top influential points).
the true parameters and the computed using the influence function. Even in case of non-smooth activation functions like ReLU, we have similar observation. (see the Appendix for more details).
The Effect Of Network Width: To the effect of the network width the quality of influence estimates, we evaluate the influence estimates for a feed-forward network of constant depth, by progressively increasing its width. From Fig. (d), we observe that with an increase network width, the Spearman correlation decreases consistently. For example, we find that the Spearman correlation decreases from 0.82 to 0.56, when the width of the network is increased from 8 to 50. This observation suggests that over-paranee parameterizing network by increasing its width has strong impact in the quality of influence estimates.
The Effect of Stochastic Estimation on inverse-Hessian Vector Product: For large deep net- works, the lnverse-Hessian Vector product computed using stochastic AlimatioonAAarwal et al.] 2016), as the exact Hessian matrix cannot be computed and inverted. To understand the effective- ness of stochastic approximation, we compute the influence estimates with both the exact Hessian and stochastic estimation. We observe that across different network depths, the influence estimates computed with stochastic estimation have marginally lower Spearman correlation when compared to the ones computed with the exact Hessian. From Fig. (a), find that the error in the approxi- mation is more, when the network is deeper.
5.2 UNDERSTANDING INFLUENCE FUNCTIONS SHALLOW CNN ARCHITECTURES
Setup: In this section. we perform a case study using a CNN architecture" on the small MNIST dataset (i.e. 10% of MNIST); similar setup used in Kohh & Liang 2017). To assess the accuracy of influence estimates, we select set of test-points with high test-losses computed at the optimal model parameters. For each of the test points, select 100 training samples with the highest influence scores and compute the ground-truth influence by re-training the model. We also select 100 training points with influence scores at the 30th percentile of the entire influence score distribution. These training points have low influence scores and a lower variance in their scores when compared to the top influential points. The model trained with and without weight-decay regularization.
When trained with weight-decay and evaluated based on the top influential points, we find that the correlation estimates are consistently significant (Fig. (a)). This is consistent with the results reported in (Koh & Liang. 2017). However, when the evaluation is done with the set of training samples at the 30th percentile of the influence score distribution, the correlation estimates decrease significantly (Fig. (b)). This shows that influence estimates of only the top influential points are precise when compared to ground-truth re-trainings. Furthermore, without the weight-decay regularization, influence estimates in both cases poor across all the test-points (Fig. (a.b))
To further understand the impact of weight-decay on influence estimates, we train the network with different weight-decay regularization factors. From Fig. (c), see that the selection of weight-
"The model has 2600 parameters and is trained for 500k iterations to reach convergence with the optimal model parameters 0' The ground-truth estimates are obtained by re-training the models from the optimal parameter 8" for 30k iterations. When trained with weight-decay, regularization factor of 0.001 used.

---

Published as a conference paper at ICLR 2021
Dataset
MNIST
CIFAR-10
B
A
A
B
A
(With
(With
(Without Decay)
(With
(With
WWithout
Decay)
Decay)
Decay)
Decay)
Decay)
Architecture
P
S
P
S
P
S
P
S
P
S
P
S
Small CNN
0.95
0.87
0.92
0.82
0.41
0.35
LeNet
0.83
0.51
0.28
0.29
0.18
0.12
0.81
0.69
0.45
0.46
0.19
0.09
VGG13
0.34
0.44
0.29
0.18
0.38
0.31
0.67
0.63
0.66
0.63
0.79
0.73
VGG14
0.32
0.26
0.28
0.22
0.21
0.11
0.61
0.59
0.49
0.41
0.75
0.64
88
0.49
0.26
0.39
0.35
0.14
0.11
0.64
0.42
0.25
0.26
0.72
0.69
ResNet50
0.24
0.22
0.29
0.19
0.08
0.13
0.46
0.36
0.24
0.09
0.32
0.14
A-Test-point with highest loss; B=Test- point at the 50th percentile of test-loss spectrum; P-Pearson correlation; S=Spearman correlation
Table I: Correlation estimates MNIST And CIFAR-I0 10:
decay factor is important in getting high-quality influence estimates. For this specific CNN archi- tecture, we notice that the correlations start decreasing when the weight-decay factor is greater than 0.01. Moreover, from Fig. (a.b), we find that the selection of test-point also has strong impact on the quality of influence estimates. For example, when the network trained with weight-decay and the influence estimates computed for top influential training points, we notice that the Spearman correlation estimates range from 0.92 to 0.38 across different test-points and have a high variance.
These results show that despite some successful applications of influence functions in this non- convex setup, as reported in (Koh & Liang, 2017). their performances are very sensitive to hyper- parameters of the experiment well as to the training procedure. In the next two sections, we assess the quality of influence estimates on more complex architectures and datasets including MNIST, CIFAR-16 10 and ImageNet. In particular, we desire understand, if the insights gained from experi- ments on smaller networks can be generalized to more complex networks and datasets.
5.3 UNDERSTANDING INFLUENCE FUNCTIONS IN DEEP ARCHITECTURES
Setup: In this section, we evaluate the accuracy of influence estimates using MNIST and CIFAR-10 datasets across diffcrent network architectures including small CNN & Liang 2017). LeNet (Lecun al. 1998), ResNets (He et al. 2015). and VGGNets (Simonyan & Zisserman 2015 To compute influence estimates, we choose two test points for each architecture: a) the test-point with the highest loss, and b) the test-point at the 50th percentile of the losses of all test points. For each of these two test points, we select the top 40 influential training samples and compute the correlation of their influence estimates with the ground-truth estimates. To compute the ground-trute -trth influence estimates, we follow the strategy of (Koh & Liang 2017), where we re-train the models from optimal parameters for 6% of the steps used for training the optimal model. When the networks are trained with a weight-decay regularization, we use constant deight-decay factor of 0.001 across all the architectures (see Appendix for more details).
From Table I we observe that for the test-point with the highest loss, the influence estimates in the small CNN and LeNet architectures (trained with the weight-decay regu- larization) have high qualities. These networks have 2. 2.6k and 44k parameters, respectively, and are relatively smaller and less deep than the other networks used in our experimental setup. As the depth of the network increases, we observe consistent decrease in quality of influence estimates. For the test-point with a loss at the thth percentile of test-point losses, we observe that influence estimates only in the small CNN architecture have good qualities.
Results On MNIST:
Results On CIFAR-10: For CIFAR-10, across all architectures trained with the weight-decay regu- larization, we observe that the correlation estimates for the test-point with the highest loss highly significant. For example, the correlation estimates are above 0.6 for a majority of the network ar- chitectures. However, for the test-point evaluated at the 50th percentile of the loss, the correlations decrease marginally across most of the architectures. We find that on CIFAR- 10, even architec- tures trained without weight-decay regularization have highly significant correlation estimates when evaluated with the test-point which incurs the highest loss.
Forr CIFAR-10, evaluations on small CNN have not heen performed due the poor test accuracy.
7

---

Published as a conference paper at ICLR 2021
In case of MNIST, have found that in shallow networks, the influence estimates are fairly accurate while for deeper networks, the quality of influence estimates decrease. For CIFAR-10, although the influence estimates are significant, in deeper networks such as ResNet-50. The improved quality of influence estimates in CIFAR-10 10 can be attributed to the fact that for similar depth, architectures trained on CIFAR-10 10 are less over-parameterized compared to architectures trained on MNIST. Note that, in Section [.1 where the exact Hessian matrix can be computed, we observed that over-paramellrization decreases the quality of influence estimates. From TT) we also observed that the selection of test-point has a sizeable impact on the quality of influence estimates. Furthermore, we noticed large variations in the quality of influence estimates across different architectures. In general we found that influence estimates for small CNN and LeNet are reasonably accurate, while for ResNet-50, the quality of estimates decrease across both MNIST and CIFAR-10. Precise reasons for these variations are difficult to establish. We hypothesize that can be due to the following factors: (i) Different architectures trained different datasets have contrasting characteristics of loss landscapes at the optimal parameters which can have an impact on influence estimates. (ii) The weight-decay factor may need to be set differently in various architectures, obtain high quality influence estimates.
found that the correlations are marginally lower
Results on CIFAR-100: In the case of CIFAR-100,
0.4
we train a ResNet--18 model with a weight-decay regularization factor of 5e The influence esti- mates are then computed for test-points with the highest losses (Index: 6017, 2407, 9383) and test- points around the 50th percentile of the test loss (In- dex: 783, 7106) over multiple model initialisations. Unlike in the case of MNIST and CIFAR-10, 10, from Fig. we observe the correlation estimates to be of substantially poor quality. We provide additional visualizations of the influential training examples in the Appendix section.
02-
0.0
Comelation
-044
-0.6
Pearson
Spearman
6017
2007
9383
7106
Different Tess Points
Figure 4: Influence for CIFAR-100
5.4 Is SCALING INFLUENCE ESTIMATES To IMAGENET POSSIBLE?
The application of influence functions to ImageNet scale models provides an appealing yet challeng- ing opportunity. It is appealing because, if successful, it opens a range of applications to large-scale image models, including interpretability, robustness, data poisoning, and uncertainty estimation. is challenging for a number of reasons. Notable among these is the high computational cost of training and re-training, which limits the number of ground truth evaluations. previously discussed difficulties in influence estimations still remain, including (i) non-convexity of the loss, (ii) selection of scaling and damping hyperparameters in the stochastic estimation of the Hessian, and (iii) the lack of convergence of the model parameters. The scale of ImageNet raises additional questions about the feasibility of leave-one-out retraining as the ground truth estimator. Given that there are 1.2M images in the training set, it even possible that the removal of one image can significantly alter the model? In other words, we question whether or not reliable ground truth estimates may be obtained through leave-one-out re-training at this scale.
addition, all of the
To illustrate this, we conduct an additional influence estimation on ImageNet. After training an initial model to 92.302% top5 test accuracy, we select two test points at random, calculate influence over the entire training set, and then select the top 50 points by their influences as candidates for re-training. We then use the re-training procedure suggested by (Koh & Liang) 2017). which starts leave-one-out re-training from the parameter set obtained after the initial training. We re-train for an additional 2 epochs, approximately 5% of the original training time, and calculate the correlations. We observe that for both test points, both Pearson and Spearman correlations are very low (less than 0.15, see details in the Appendix).
In our experiments, we observe high variability among ground-truth estimates obtained by re- training the model (see the appendix for details). We conjecture that this may be partially due to the fact that the original model has not be fully converged. To study this, we train the original model with all training points for an additional 2 epochs and measure the change in the test loss. We find that the overall top5 test accuracy has improved slightly to 92.336 % (+0.034) and the loss for one
8

---

Published as a conference paper at ICLR 2021
Difference in norm of parameters is small
Correlation is similar
0.0010
1.0
Diferece in in of Paraneter
Retroning (Scratch)
0.0000
0.5
0.0006
0.6
0.0004
0.0
0.0000
2.5
12.5
17.5
Different Training Examples
Figure 5: (a) Difference in norm of parameters obtained by re-training from scratch vs. re-training from optimal parameters. (b) Correlation estimates with re-training from scratch vs. re-training from optimal parameters.
of the considered test points has decreased by relatively a significant amount of 0.679, However, the loss for the other point has increased slightly by 0.066. Such changes in loss values can therefore out-power the effect of leave-one-out re-training procedure. Second, we calculate the 2-norm of the weight gradients, which should be close to zero near an optimal point, and compare it to a standard pre-trained ImageNet ResNet-50 model as baseline. We find these norms to be 20.18 and and 15.89, respectively, showing our model has similar weight gradient norm to the baseline. Although these norms are relatively small given that there are 25.5M parameters, further re- re-training the model still changes loss values for some samples considerably, making the ground-truth estimates noisy. We suggest that one way to obtain reliable ground-truth influence estimates in such large models can be through assessing the influence of group of samples, rather than single one.
DISCUSSION ON GROUND-TRUTH INFLUENCE
In our experimental setup, to obtain the ground-truth influence, we follow the strategy of re-training from optimal model parameters as shown (Koh & Liang, 2017: Koh et al., 2019b). Even for mod- erately sized datasets and architectures, re-training from scratch (instead of re-training from optimal model parameters) computationally expensive. Although re-training from optimal model param- eters is approximation compared to re-training from scratch, we notice that the approximation works quite well in practice. To validate the effectiveness of this strategy, we first compute the norm of the difference in parameters obtained by re-training from scratch re-training from optimal pa- rameters. Next we compute the correlation between the influence estimates and ground-truth using both the re-training strategies. From Fig. 5 we observe the norm of parameter differences using the two re-training strategies to be small. Similarly using both the re-training strategies as ground-truth yield similar correlation estimates. These results highlight that re- traininig from optimal parameters (although an approximation) is close to re-training from scratch.
CONCLUSION
In this paper, present a comprehensive analysis of the successes and failures of influence func- tions in deep learning. Through our experiments on datasets including Iris, MNIST, CIFAR-10, CIFAR-108 100, ImageNet and architectures including LeNet. VGGNets, ResNets, we have demon- strated that influence functions in deep learning are fragile in general. We have shown that several factors such as the weight-decay, depth and width of the network, the network architecture, stochas- tic approximation and the selection of test points, all have strong effects in the quality of influ- ence estimates. general, we have observed that influence estimates are fairly accurate in shallow architectures such small Koh & Liang 2017) and LeNet, while in very deep and wide architectures such as ResNet-50, the estimates are often erroneous. Additionally, we have scaled up influence computations to the ImageNet scale, where have observed influence estimates highly imprecise. These results call for developing robust influence estimators in the non-convex setups of deep learning.

---

Published as a conference paper at ICLR 2021
8 ACKNOWLEDGEMENTS
Authors thank Daniel Hsu, Alexander D' A'oour and Pang Wei Koh for helpful discussions. This project was supported in part by NSF CAREER AWARD 1942230, HR001 1990026-GARD-PP.. 052, AWS Machine Learning Research Award, sponsorship from Capital One, and Simons Fel- lowship on "Foundations of Deep Learning".
REFERENCES
Naman Agarwal, Brian Bullins, and Elad Hazan. Second order stochastic optimization in linear time. ArXiv, (is.0994,,394,, 2016.
Anderson. Iris flower dataset. In 1936.
Samyadeep Basu, Xuchen You, and Soheil Feizi. Second-order group influence functions for black- box predictions. ArXiv, abs/191 111199110000000 1.00418, 2019.
Samyadeep Basu, Xuchen You, and Soheil Feizi. On second-order group influence functions for black-box predictions. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Re- search, pp. 715-724. PMLR, 13-18 Jul 2020. URL h http:::/pprrcceei.gg..s.
v119/basu20b. html
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In in COMPSTAT, 2010.
Mure-EEienne Brunet, Colleen Alkalay- Ashton Anderson, and Richard S. Zemel. Un- derstanding the origins of bias in word embeddings. CoRR, abs/1810.03311, 2018. URL
htttp://arxiv.orgge
03611
Probal Chaudhuri and Per A. Mykland. Nonlinear experiments: Optimal design and inference based on likelihood. Journal of the American Statistical Association, 38(422):538-544 1993. doi: 10. 10.1080/01621455 1993. 10476305. URL https: tand wwwpdoo//oooooooooooooooooo.o. abbs/
10.1080/016210 621116611440
993. 104763033333
R. Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-508 1980. ISSN 0040-1706.
doi: 10. 10.100000400000 198800048670999
Deng, Dong, R. Socher, L.-J. Li, Li, and L.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Amirata Ghorbani, Abubakar Abid, and James Y. Zou. Interpretation of neural networks is fragile. In AAAI, 2017.
Ryan Giordano, Will Stephenson, Runjing Liu, Michael I. Jordan, and Tamara Broderick. A swiss army infinitesimal jackknife. In AISTATS, 2018.
Ryan Giordano, Michael I. Jordan, and Tamara Broderick. A higher-order swiss army infinitesimal jackknife. ArXiv, abs/1907.12116, 2019.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet hour. CoRR, abs/1 hes/1706.02677, 2017. URL tttp::/arxx...org/bbb/// 02677
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. residual learning for image recog- nition. CoRR, abs/1512.00388. 2015. URL, URt.httpp//arriii.or/////////// 03385)
Jeremy Howard et al. fastai. httppp///gtthho.co//aaaaa//aaa
2018.
10

---

Published as a conference paper at ICLR 2021
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nip systems: How should define and evaluate faithfulness? Proceedings of the 58th Annual Meeting of the Association URL http://dx.
for Computational Linguistics, 2020. doi: 10.
doi maiioorg/1... 18653/811 ac -ma in. 386
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Wilhelm Kirch (ed.). Pearson's Correlation Coefficient, pp. 1090-1091. Springer Netherlands, doi: 10.1007/978- 1-4020-5614-722555 URL httpss//doi org. 1007/ 101.11111177799002222000
Dordrecht, 2008. ISBN 978- 978-1-4020-56147,
2569
W. Koh, J. Steinhardt, and P. Liang. Stronger data poisoning attacks break data sanitization defenses. arXiv preprini arXiv: 1.00741, 2019a.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In
Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:// proceedings mwr. wwwpseeessvvee. http:///www....ppssessss//////// koh1 html
Pang Wei Koh, Kai-Siang Ang, Hubert H. K. Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. CoRR, abs/1905.13228 2019b. URL http://arxiv..
org/abs/19055 905.. 13289
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). CVPR, 2000. URL cifar. com
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, volume 86, pp. 2224-2324, 1998. URL 42. 7665
httt::/ccteseeee istteeere...ss.......... ist psu. wwtp/vddwwweeeeee http::///ssammmmmmmdddmodo//
Alexander Selvikvág Lundervold and Arvid Lundervold. An overview of deep learning in medical imaging focusing on MRI. CoRR, abs/ 0t.//8811.1052,, 2018. URL http::t:t////..rrrttttt// abs/
181 0005
Barak A. Pearlmutter. Fast exact multiplication by the hessian. Neural Comput., 6(1):147-160, January 1994. ISSN 0899-7667. doi: 2a0116//neco.1194...118. URL http://dx doi.
orq/10 http:/0.1166//1 www.he................. .9999.9..... 147
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracking gradient descent. ArXiv, abs/2002.2000 2020.
Sebastian Ruder. An overview of gradient descent optimiizzinn algorithms. CoRR, abs/ wabs/1609 1609.04747 2016.
C4747
Peter G. Schulam and Suchi Saria. Can you trust this prediction? auditing pointwise reliability after learning. AISTATS, 2019.
Fabrizio Sebastiani. Machine learning in automated text categorization. ACM Comput. Surv. 34(1): 1-47, March 2002. ISSN 0360-0300. doi: 10. 10.11555555282.555223.. http: acm.
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks, 2018.
Jonathan R Shewchuk. An introduction to the conjugate gradient method without the agonizing pain. -. 1994.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
11

---

Published as a conference paper at ICLR 2021
Leslie N. Smith. A disciplined approach to neural network syper-parameters: Part learning rate, batch size, momentum, and weight decay. CoRR, (bbs/1803.09800 2018. URL http:
(aaxiv..rrg/as///883..00
C. Spearman. The proof and measurement of association between things, American Journal of Psychology, 15:88-103, 1904.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In Yoshua Bengio and Yann LeCun (eds.), 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Truck Proceedings, 2015. / al11/ 6806
DAWNBench: An End-to-End Deep Learning Benchmark and Competition, 2017. Stanford. URL
https: /ttpps://daw.... www.ntrandrr..oo edu
pdf
Richard Szeliski. Computer Vision: Algorithms and Applications. Springer- Verlag, Berlin, Heidel- berg, 1st edition, 2010. ISBN 1848829345, 1781848829333.
Chih-Kuan Yeh, Joon Sik Kim, Ian En-Hsu Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks. CoRR, abs/1811.09722 2018.
URL http:
12

---

Published as a conference paper at ICLR 2021
Deccosses Dopch
Exparakon trcnesss Depth
Sparman Comilation
1009
Spoorman Con eiaiaiiioo
Top Eigenvolue
Number of ayam (a)
Number of Layers
Number of Layers (c)
(b)
Figure Additional Iris experimental results for ReLU networks: (a) Spearman correlation vs. net- work depth; (b) Top eigenvalue of the Hessian vs.network depth; (c) Spearman correlation between the norm of parameter changes computed with influence function vs. re-training.
APPENDIX
9.1 ADDITIONAL EXPERIMENTAL RESULTS ON IRIS DATASET
In this section, we provide additional experimental results to understand the effect of network depth on the correlation estimates for ReLU networks. From Fig. 6 we observe that even in case of architectures trained with non-smooth activation functions such as ReLU, the correlation estimates consistently decrease with depth. Similar to our findings in case of networks trained with tanh acti- vation (as shown in the main text), we observe that the top eigenvalue of the Hessian matrix and the Taylor's approximation gap increases with depth. In the main text, reported that when network with ReLU activation is trained with a weight-decay regularization, the correlation estimates are significant and the Taylor's approximation gap is less. We find similar result even with smoother activation functions such as tanh. From Fig. 7 we observe that when a network with tanh activation trained with a weight-decay regularization, the Taylor's approximation gap less. However when the network is trained without a weight- -egcht-decay regularization, the Taylor's expansion gap is large resulting in poor quality of influence estimates.
With Weight-Decay
Without Weight-Decay
Norm of Approsimate Parsmeter
Paramsters
0.2
60
-8.
-30
Marm of bxac Parsmaters
(b)
Figure 7: Additional Iris experimental results for tanh networks; (a) When trained with weight- decay, the Taylor's approximation gap is small; (b) When trained without weight-decay, the Taylor's expansion gap large. These results are similar our findings for ReLU networks which are reported in the main text.
13

---

Published as a conference paper at ICLR 2021
9.2 WHAT DOES WEIGHT-DECAY DO?
observe that with increasing between the influence estimates and the ground-truth Additionally with increasing depth, the loss curvature values increase. We notice that with a high-value of weight-decay, the loss curvature for deeper networks decrease, which also leads to improvement in corre- lation values between the influence estimates and ground-truth. For e.g. in Fig. g with weight-decay value of 0.03, the Spearman correlation estimates are 0.47. With relatively higher weight-decay factor of 0.075, the correlation values improve to 0.72. Increas- ing the weight-decay factor from 0.03 to 0.075, also de- creases the loss curvature values substantially. These results highlight that the selection of weight-decay fac- tor is crucial to obtain high-quality influence estimates, especially for deeper overparameterized networks.
In experiments,
network depth, the correlation estimates
decrease considerably.
Low Weight High Weight-Docay
160
120
100
Depth of Networks
Figure training samples
Correlations with different
9.3 VISUALISATION OF TOP INFLUENTIAL POINTS
In this section, we visualise the top influential training samples corresponding to a given test-point. In the main text, we noted that the selection of test-points has strong impact on the quality of influence estimates. Additionally, we also observe that the selection of test-points has an impact on the semantic-level similarities between inferred influential training points and the test-points being evaluated. For example, in Fig. 9 observe that out of the top 5 influential points are not from the same class as the test-point with index 1479. However Fig. [00] we observe that all the top 5 influential training samples are semantically similar and from the same class as the evaluated test-point with index 7196.
Test 1479
10759
1:47421
8247
ID: 19901
IC: 14429
Bird
Deer
Bird
Deer
Figure 9: Top 5 influential points for the test point: 1479 (CIFAR-10). The model is ResNet- 18 trained with a weight-decay regularization; Only out of the points are semantically similar to the test-point with class "Bird".
ID:
2006
Airplane
Airplane
Airplane
Airplane
Airplane
Airplane
Figure 10: Top influential points for the test point: 7196 (CIFAR-10), The model is a ResNet- 18 trained with deight-decay regularization; All the 5 training points are semantically similar to the test-point from class "Airplane".
14

---

Published as a conference paper at ICLR 2021
Influence Computation Time (MNIST)
Influence Computation Time (CIFAR-10)
Architecture
Small CNN
141.13 0.51
N/A
LeNet
162.6 = 2.20
136.39 + 3.16
VGG13
3886.23 + 3.45
4416.54 + 2.01
VGGI4
4619.11 + 5.08
4620.69 + 6.11
ResNet-18
960.08 + 4.67
910.58 + 8.49
ResNet-50
1323.13< + 8.26
3857.66 + 21.6
Table 2: Computational running times for influence function across different architectures
9.4 RUNNING TIMES
In this section, we provide computational running times for (first-order) influence function estima- tions. We note that in models with a large number of parameters, the influence computation relatively slow. However, even in large deep models, it is still faster than re-training the model for every training example. our implementation, for a given test-point Stest, we first compute Hg. (Z(ee)(()tet)) once which is the most computationally expensive step. We then com- pute vector dot product i.e. cf (e (2)) Vi E [1, n). In Table 2, we provide the computational running times for estimating influence functions in different network architectures.
9.5 ADDITIONAL EXPERIMENTAL DETAILS ON IMAGENET INFLUENCE CALCULATIONS
In this section we give further details the influence estimation on ImageNet. To help address the high computational cost of training and re re-training, we utilize highly optimized ImageNet training schemes such as those submitted the DAWNBench competition (Sta) 2017). particular we use the scheme published from (Howard et al. 2018 for the ResNet-50 architecture which uses several training tricks including progressive image resizing, weight decay tuning, dynamic batch sizes (Goyal et al.) 2017), learning rates (Smith, 2018), and half-precision floats. Although these techniques are unorthodox, they are sufficient for our purposes since we need only to compare be- tween the fully trained and re-trained models. We replicate this scheme and obtain top-5 validation accuracy of 92.302%.
We give further details on the test points selected. The first has test loss at the 83rd per- centile (loss=2.6344 index = 13,923, class=kit fox), the second has the test loss at the 37th percentile (loss=0.081, index 2,257, class =gila monster), where the indices refer to where they appear in test_loader: loader.... dataset We visualize these test points in Figure [2
Next, for each of these test points, compute influence across the entire dataset and select the top 50 training points by influence scores. We visualize 25 of these points in Figures [3 and 14 We observe that there is qualitative similarity between the test points and some of their respective most influential training points, but not others. Although there is qualitative similarity is some cases, the results are still overall weak quantitatively
We plot the obtained correlations in Figure
For computing the weight gradient norm, we take the mean norm in batches of size 128 over the entire dataset for both our model and standard PyTorch pretrained model as baseline, both of which are 5e models with around 25, 25.5M parameters.
15

---

Published as a conference paper at ICLR 2021
Imagenet Influence Correlations
Test Index
13923
Test Index 2257
0.90
Spearman
0.12
0.14
0.85
0.12
Infhvence
Influerce
0.10
0.65
0.08
0.60
0.06
0.55
0.2
0.3
0.5
0.6
0.8
-0.175-0.150-0.1 100-0.11000001..0 0.5-0 050-0.025 Loss Retrain
A Loss Retrain
ImageNet influence estimation results for the selected test points 13,923 (left) and 2,257 (right). X-axis is change in test loss after removal of training point and retraining described in the text. Y-axis is the change in test loss estimated with influence function. Pearson and Spearman correlations shown in the caption. Correlations low, showing the weakness of this influence estimation.
Figure 11:
Test index = 13,923 Class Kit Fox
Test index = 2,257 Class Gila Monster
Figure 12: Selected test points for influence estimation.
9.6 COMPUTING INVERSE- -EVERSS- VECTOR PRODUCT
In large wver-parameterized deep networks, computing and inverting the exact Hessian Hg- is ex- pensive. In such cases, the Hessian-vector product rule (Pearlmutter) 1994) used along with conjugate-gradie (Shewchuk) 1994) or stochastic estimation (Agarwal et al. 2016) to compute the approximate inverse-Hessian Vector product. More specifically, to compute following optimization problem using snojugate-gradieent
Hg v. we solve the ( ( H If where Vog( (2))). This optimization, however, requires the Hessian Hg- to be positive definite matrix, which not true in case of deep networks due to the presence negative cigenvalues. In practice, the Hessian
= arg min,
be regularized by adding damping factor of X its eigenvalues (i.e. Hø- + X/) to make it positive definite.
In deep models, with large number of parameters and large training set, einjuaategraddien is often expensive as it requires computing the Hessian-vector product (Pearlmutter 1994) for every data sample in the training In those cases, stochastic estimation techniques (Agarwal et al.
16

---

Published as a conference paper at ICLR 2021
Top training Points
influence class
influence 0.863 basenji
influence dhole
influence 0.848 wolf
influence 0.810 class dingo
walt
influence 792 class
influence 788 wallaby
influence 769 class basset
influence 760 class cougar
influence 747 class dhole
influence 723 class walf
influence 0.9 class
influence /14 class iguana
influence 0.710
influence 703 haired
influence 89690 class walt
influence camel
influence 0.686 highorn
influence 0.683 wolf
influence 0.672 highorn
influence
influence wolf
influence
influence husky
influence Afghan
Figure 13: Top 25 ImageNet training points by influence for test point 13,293, kit fox. Many of the identified classes are furred mammals, e.g. red wolf, basenji, and dingo, which have visual similarity the test point. Other examples are questionable, e.g. the common iguana, and African elephant. Although there is qualitative similarity is some cases, the results are still overall weak quantitatively
2016) have been used which are fast as they do not require going through all the training samples. In stochastic estimation, the inverse Hessian computed using recursive reformulation of the Taylor expansion: H,¹ I + (I H) H where is the recursion depth hyperparameter. A training example is uniformly sampled and 22((hg- (z:)) is used estimator for computing H. This
17

---

Published as a conference paper at ICLR 2021
Top training Points
influence 0 1.3 class ant
influence 0 130 class banded gecko
influence 130 class banded gecka
influence 125 class banded gecko
influence 0 109
class banded gecko
class gocko
class
African crocodile
barrow
influence 0.082 class water snake
influence 0.082 class Europcan fire salamander
influence 0.080 class mousetrap
influence 0.078 class Europcan fire salamander
influence 0.077 class alligator lizard
0.075
fly
class
coyoto
influence = 0.073 class common newt
influence 0.072 class
influence 0.071 class coral fungus
influence = 0.070 class eel
influence 0 0.069 class stingray
alligator lizard
Figure 14: Top 25 ImageNet training points by influence for test point 2.257, gila monster. Many of the identified classes are spotted lizards, c.g. banded gecko ad European fire salamander, which have visual similarity to the test point. Other examples are questionable, e.g. the stingray, coral fungus, and barrow. Although there is qualitative similarity is some cases, the results are still overall weak quantitatively.
technique also requires tuning scaling hyperparameter and damping hyperparameter , In
'It is assumed that Vi, / (2 x 0: Koh & Liang 2017) notes that if this is not true, the loss can be scaled down without affecting the parameters. The scaling factor is hyperparameter which helps the convergence of the Taylor series. The damping coefficient is added to the diagonal of the Hessian matrix to make invertible.
18

---

Published as a conference paper at ICLR 2021
experiments with large deep models, use the stochastic estimation method to compute the inverse-Hessian Vector product.
9.7 EFFECT OF INITIALISATION AND OPTIMIZERS INFLUENCE ESTIMATES
To understand the effect of network initialisation the quality of influence estimates, compute the influence scores across different random initialisations. The influence estimates are computed for the small CNN architecture (Koh & Liang 2017) and LeNet (Lecun al. 1998), both trained on the MNIST dataset. Both the architectures are trained with constant weight-decay factor of 0.001.
In [5] we observe that across different net- work iniiialisatinns, although both the Pearson and Spearman correlations between the influ- ence estimates and the ground-truth are incon- sistent, the variance amongst them is particu- larly low. Note that for both the network chitectures, we compute the influence estimates for the test-point with the highest loss at the optimal model parameters. The correlation be- tween the influence estimates and leave-out re- trainings are computed with the top 40 influen- tial training examples. Additionally stand the impact of the selection of optimizer on the influence estimates, we train the LeNet architecture on MNIST with different optimiz- ers namely Adam (Kingma & Ba] 2014), Gradi- ent Descent (Bottou 2010). Nesterov and RM- SProp (Ruder 2016). We notice that the Pearson correlation (0.72 + 0.04) has a marginally lower
10
0.
u0
Pearson Small CNN
Spearman-Smi Small
under-
62
Pearson-LeVes
Spear Letiet
Figure 15: Correlations with different network initialisation
variance when compared the Spearman rank-order correlation (0.56 = 0.11).
9.8 EFFECT OF TRAINING SAMPLE SELECTION FOR GROUND-TRUL TRUTH INFLUENCE
In this section, we understand the effect of selecting different number of training samples on the correlation estimates. We investigate this with case study for CNN architecture trained small MNIST. Keeping test-point with high loss fixed, we sample different sets of training examples with the highest and the lowest influence scores over different network initialisations. Note that in this setting as shown in the main paper, the quality of influence es- timates are relatively good. We observe that when the influence estimates are eval- uated with the top influential points, both the Pearson and Spearman correlations are relevant. This is true across different num- ber of training samples. However when the evaluation carried out with respect the lowest influential training samples, the correlation estimates are of poor quality. These results highlight the importance of the selection the type of training sam- ples with respect to which the correlation estimates are computed.
1.00
0.75
0.50
Correlatio
0.25
0.00
-0.25
Top Training Examples (Pearson)
Tp Training Examples (Spearman)
-0.75
Training Examples (Pearson)
-1.00
Bottom Trining Examples (Spearman)
10
60
80
100
120
Number Training Samples
Figure 16: Correlations with different training samples
9.9 FAITHFULNESS AND PLAUSIBILITY OF INFLUENCE FUNCTIONS
The authors (Jacovi & Goldberg 2020
primarily tackle the importance and trade-offs between plausibility (i.e. if the interpretations are convincing to humans) and faithfulness (i.e. how accurate interpretation is to the "true reasoning process of the model") of existing interpretation methods. To the best of our knowledge such an analysis has not been done for influence functions. We observe that explanations from influence functions for deep networks are sometimes plausible and sometimes
19

---

Published as a conference paper at ICLR 2021
not. For instance, in Appendix Fig. 9 we observe that the selection of test-point with (class bird) leads to training examples with (class deer) amongst the top influential points. On the other hand, in Appendix Fig. To observe many plausible explanations. Influence functions that work faithful because they answer the following question: would this model have done if certain data were excluded?". This class of questions, while exhaustive, have special relevance because they are counterfactuals, which hold both intuitive appeal and for their special status in causal reasoning. However, we must be cautious because they may not be faithful when they incur approximation errors, as highlighted in our paper.
9.10 CIFAR-100 INFLUENTIAL EXAMPLES
That
41277
ID: 3873873
Porcupine
Street
Sea
Bridge
Bridge
Woman
2407
Cloud
Cloud
Sea
Cloud
Cloud
Oloud
Figure 17: Top influential points for the test points: 7106 and 2407 CIFAR-100), The model a ResNet-18 trained with a weight-decay regularization. For the test-point with index 7106, the influential training samples semantically dissimilar from the test-point. However for the test- point with index 2407, 4 out of the top samples share semantic similarity with the test-point.
9.11 PRELIMINARY RESULTS ON GROUP INFLUENCE
Understanding model changes when a group of training samples are up- up-weighted indeed an important research problem. Influence functions (Cook & Weisberg 1980 Koh Liang 2017) in general are accurate when the model perturbation is small. However when a group of samples up-weighted, the model perturbation large, which violates the small pertur- bation assumption of influence functions, Previously it has been shown (Koh et al.) 2019b Basu al. 2019) that group influ- ence functions are fairly accurate for linear and convex models, even when the model perturbation is substantial. In this section, we present some preliminary results on the behaviour of group influence functions for non-convex models. Our main observation is that group influence functions are fairly accurate for small networks. Nonetheless for large and complex net- works, the influence estimates are of poor quality. For e.g. in Fig. 18] we observe that the correla- tion estimates for small group sizes are accurate, whereas for larger group sizes, the estimates are of poor quality. For ResNet- 18 model trained MNIST (with weight-decay regularization factor of 0.001), we observe the correlation estimates across different group sizes to vary from 0.01 to 0.21. Similarly for ResNet- 18 trained on CIFAR- 100, observe the group influence correlation
0.8
0.6
Correlation
0.4
02
0.0
Pearson
-02
Spearman
20
60
70
Size
Figure 18: Group Influence on Iris
20

---

Published as a conference paper at ICLR 2021
0.008
Difference in Norm of Parameters
0.006
0.005
0.004
0.001
0.000
10
20
Training Examples
Figure 19: Norm of difference in parameters obtained by training from scratch vs, re-training from optimal parameters
1.0
0.8
Spearman Correlation
0.6
0.4
0.2
0.0
-0.2
180
250
300
Width Layers
Figure 20: Width vs. Spearman Correlation for a one-layered network
estimates to range from 0.01 to 0.18. We leave the complete investigation of group influence in deep learning as a direction for future work.
9.12 ADDITIONAL EXPERIMENTS WITH MULTIPLE TEST-POINTS
In our experimental setup, we evaluate the correlation estimates with respect to one test-point at time. Although the evaluation of the correlation estimates with multiple test-points is more robust, it comes at the expense of high computational cost. To illustrate the quality of influence estimates with multiple test-points, we compute the influence estimates for small MNIST with different test-points. We sample two test-points each from (a) 100 percentile of the test-loss; (b) 75th percentile of the test-loss; (c) 50th percentile of the test-loss: (d) 25th percentile the test-loss. The Pearson and Spearman correlations are 0.91 and 0.78 respectively. In similar setting, for complex architecture such as ResNet- 18 trained on CIFAR- 100, the Pearson and Spearman correlations are 0.15 and 0.1 respectively.
21

---

Published as a conference paper at ICLR 2021
9.13 IMPACT OF ACTIVATION FUNCTIONS
In our experiments we observe that even with non-smooth activation functions such as ReLU, we obtain high quality influence estimates for certain networks. Understanding influence estimates with ReLU has an additional challenge since there measure zero subsets where the function is non- differentiable. Recently (Serra et al., 2018) has provided improved bounds on the number of linear regions for shallow RcLU networks. Understanding the impact of the number of linear regions in ReLU networks on the influence estimates an interesting research direction, however we defer for future work.
22


Reviews:
[{"review_id": "n2gNeYmGg0b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper does a thorough investigation of influence functions applied to neural networks. Specifically, the authors find that deeper networks result in erroneous influence estimates and weight-decay regularization can help lower the gap in the first-order Taylor expansion of the influence estimates. They also study the empirical effects of network depth and width on the quality of influence estimates. They conclude that influence estimates for smaller networks are quite accurate but are erroneous for larger networks.\n\nStrengths\n- Thank you for the specificity in the empirical setup of all the experiments. This should greatly help with the reproducibility of your results. \n- The paper clearly defines the influence function set up and notes the shortcomings in estimating it. The authors do a thorough job of discussing how each hyperparameter of network training affects the influence estimates.\n\nWeaknesses\n- I would have loved to see some more discussion on why weight decay led to improved influence estimates. Are there some properties of the weight-decayed Hessian that make it more reliable? \n- The fragility of influence on large networks has been folk knowledge in the community, and the authors do a good job of codifying this; however, I feel providing theoretical intuitions for the observed phenomena would strengthen the paper's findings. I.e. does overparametrization always lead to imprecise influence estimates?\n\nQuestion\n- Do you intend to release the code for all experiments? For such an empirical paper, this would be strongly encouraged and beneficial for the community.\n- Can you provide intuition or details for how to scale influence to larger networks? Is it possible to develop some theory to determine when influence estimates will err (perhaps in terms of the number of parameters, number of training points, etc.)?\n- Did you ever run group influence estimation? You mention it in 5.4, but would be nice to see if the same results hold for the influence of a set of points. \n\n*** Post Rebuttal: Thank you so much for your response and for the codebase link. Your additional experiments are convincing for group influence and overparametrization.  I sustain my recommendation to accept this paper :) ***", "extra_scores": {}, "extra_text": {}}, {"review_id": "hZSUGrCtoG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\nThe paper provides an extensive experimental study on using influence functions in deep neural networks. The authors suggest that the estimation via influence functions can be quite fragile depending on various architectures of neural networks. To point out such information, approximation with inverse-Hessian Vector product techniques might be incorrect, which leads to low quality influence estimates. They show several meaningful findings which address the true nature of using influence functions in deep neural networks. One of their findings through various experimental study is that the network depth and width strongly affect influence estimates.\n\n##########################################################################\n\nReasons for score: \nThe introduced through experimental settings seem to be highly interesting and reasonable. I like the idea of addressing weakness of influence functions in deep neural network and the authors’ findings are interesting. I believe it will be better if the authors perform the ablation studies with different parameters and more datasets to generalize their findings.\n\n##########################################################################\n\nPros: \n1. The paper address the important issue of influence functions, which can affect the field of model interpretability. \n2. The introduced experimental settings seem to be reasonable to empirically show that using influence functions in deep neural networks can be very sensitive depending on the architectures of neural networks, the use of weight decay, and the choice of hyper-parameters.\n3. The experiments are well designed with multiple architectures of neural networks and detasets. \n\nIn the case of non-convex loss functions, the assumption of fist-order influence functions is not generally true. They empirically found that the Taylor’s gap is strongly affected by common hyper-parameters for deep networks. \n\n##########################################################################\n\nCons: \nMore ablation studies with different set of hyper parameters and multiple datasets can improve the quality of this paper. \n\n##########################################################################\n\nQuestions during rebuttal period: \nIn the case study using a CNN architecture on the small MNIST, the authors need to perform the experiments with different number of selected training samples with both the highest and the lowest influence scores to amplify their point. \n \n#########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "UNFotJ1fIl0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength: \n\n+ The paper provides an interesting application of Influence functions that were introduced in Koh et al. for studying data poisoning attacks. The main idea of the paper given in Section 3 is that an approximation of the influence or impact of a training sample on the test set can be obtained using second order Taylor's series expansion at the optimal model parameter -- a bilinear form where the matrix of the form is given by the hessian inverse and the vectors are gradients evaluated using training and test sample. However, since this approximation only holds when the Hessian positive definite, the paper identifies certain settings in which the influence functions can be estimated to reasonable approximation.\n\nWeaknesses:\n\n- The \"retraining\" phase to get the ground truth seems like an interesting idea, which in my understanding has not been looked at. However, this retraining phase has been directly borrowed from Koh et al., and in fact, Koh et al propose the method also as an approximation. The paper fails to clarify why such an approximation will provide any insights on the performance of neural networks. Including a summary, and intuition of the approach will be helpful to the reader.\n- Most of the neural networks used have a relu activations (with linear layers), making them a piecewise linear function with respect to the parameters, more importantly, nonsmooth. In fact, in it possible to count the number of linear regions, see (http://proceedings.mlr.press/v80/serra18b.html). In this light, it is unclear what the main message of the paper from an intuitive perspective is. Moreover, the paper misses an important line of work in statistics called Experimental Design in which the goal is to approximate a dataset using a subset with respect to various statistical criterion on the estimators, see https://www.jstor.org/stable/2290334?seq=1.\n- Related to the previous point: It is unclear what the take away is from the experimental results in the paper. Some of the experiments are performed using very shallow networks (width of 5, depth of 1 and relu activations) whereas others have a few more layers. However, two test points are used in all the experiments. Why? All of the datasets used in the experiments have thousands of points in each class, and it is also possible to augment them by simple rotations, translations etc.. The paper will benefit by adding robust estimates of Pearson and Spearman using a fraction of test samples around the highest and 50% test points. \n- Finally, none of the results are provided with confidence intervals over runs. Are the experiments averaged over a few runs? If so, I suggest you include them in the figures to provide an accurate summary of the experiments.\n\nAfter response: Thanks for the clarifications especially the piecewise linearity of deep network as a function of input and not as parameters (like I incorrectly suggested in my review). I think the empirical results have some merit, but are preliminary and I fail to see the bigger picture. For example, more carefully designed ablation studies are needed to determine the practical utility i.e., when the approximations suggested are reliable and when they are not.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0w4PFxiE1pY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors perform an empirical analysis of influence functions in deep learning. Influence functions allow one to estimate the effect of weighting a training point at test time (for instance, dropping a training point from the training set with a weight of $\\frac{-1}{n}$ under assumptions about the convexity of the loss function and the models Hessian $H_{\\theta}$. The authors note that successful applications of influence functions are rare in the literature, despite their potential applications. The authors investigate the effects of model width, depth, regularization schedule, Hessian approximation techniques, and dataset size on the accuracy of influence functions. In general, they find that each of these can cause issues with influence function calculation. As model width and depth increases, the Spearman correlation decreases between the calculated influence functions and the ground truth obtained by retraining. They also find that the top eigenvalue of $H_\\theta$ increases quite dramatically as depth increases. Weight regularization was found to be important for successful influence functions calculation, while factors such as the huge size of the ImageNet dataset proved prohibitive at this point. \n\nStrengths: \n* The paper has excellent presentation. The authors establish the background and relevant literature succinctly but with proper depth \n* I do think that the authors demonstrated that most of the factors they consider do contribute to the difficulty of accurately computing influence functions, but see (weaknesses)\n\n\nWeaknesses: \n* The authors use the term \"significant\" or \"significantly\" 10+ times throughout the paper but I think this word should generally be reserved for when a test of statistical significance is performed. Consider using another word such as \"substantial\" \n* Why are there no error bars on Fig 1c-d, Fig 2, and Fig 3? Could you consider rerun your experiments over many different initializations of the model? Since you mainly analyzed results on MNIST and CIFAR-10 that should be feasible. \n*Fig 1c and 2c seem particularly stochastic and could benefit from multiple initializations to confirm or deny the trends that are claimed. \n\nOverall, I think this is a fairly strong work that performs an important empirical analysis but I have concerns about the lack of repeated experiments and seeing whether these trends reported hold up over many initializations. I rate it a 6 on this basis.  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines under what conditions influence estimation can be applied to deep networks and finds that, among of items, that influence estimates are poorer for deeper architectures, perhaps due to poor inverse Hessian vector approximations for poor for deeper models. The authors provide an extensive experimental evaluation across datasets and architectures, and demonstrates the fragility of influence estimates in a number of conditions. Although the reviewers noted that these issues are now "folk knowledge", there has been less scientific effort in identifying these failures.

Of course, more theoretical understanding would help the community better understand where these fragilities lie, but the experimental evaluation is sufficiently strong to be of broad interest to the community.
Decision: Accept (Poster)