Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: The paper does a thorough investigation of influence functions applied to neural networks. Specifically, the authors find that deeper networks result in erroneous influence estimates and weight-decay regularization can help lower the gap in the first-order Taylor expansion of the influence estimates. They also study the empirical effects of network depth and width on the quality of influence estimates. They conclude that influence estimates for smaller networks are quite accurate but are erroneous for larger networks.

Strengths
- Thank you for the specificity in the empirical setup of all the experiments. This should greatly help with the reproducibility of your results. 
- The paper clearly defines the influence function set up and notes the shortcomings in estimating it. The authors do a thorough job of discussing how each hyperparameter of network training affects the influence estimates.

Weaknesses
- I would have loved to see some more discussion on why weight decay led to improved influence estimates. Are there some properties of the weight-decayed Hessian that make it more reliable? 
- The fragility of influence on large networks has been folk knowledge in the community, and the authors do a good job of codifying this; however, I feel providing theoretical intuitions for the observed phenomena would strengthen the paper's findings. I.e. does overparametrization always lead to imprecise influence estimates?

Question
- Do you intend to release the code for all experiments? For such an empirical paper, this would be strongly encouraged and beneficial for the community.
- Can you provide intuition or details for how to scale influence to larger networks? Is it possible to develop some theory to determine when influence estimates will err (perhaps in terms of the number of parameters, number of training points, etc.)?
- Did you ever run group influence estimation? You mention it in 5.4, but would be nice to see if the same results hold for the influence of a set of points. 

*** Post Rebuttal: Thank you so much for your response and for the codebase link. Your additional experiments are convincing for group influence and overparametrization.  I sustain my recommendation to accept this paper :) ***
review_title: Strong Empirical Investigation of Influence Functions for DL

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strength: 

+ The paper provides an interesting application of Influence functions that were introduced in Koh et al. for studying data poisoning attacks. The main idea of the paper given in Section 3 is that an approximation of the influence or impact of a training sample on the test set can be obtained using second order Taylor's series expansion at the optimal model parameter -- a bilinear form where the matrix of the form is given by the hessian inverse and the vectors are gradients evaluated using training and test sample. However, since this approximation only holds when the Hessian positive definite, the paper identifies certain settings in which the influence functions can be estimated to reasonable approximation.

Weaknesses:

- The "retraining" phase to get the ground truth seems like an interesting idea, which in my understanding has not been looked at. However, this retraining phase has been directly borrowed from Koh et al., and in fact, Koh et al propose the method also as an approximation. The paper fails to clarify why such an approximation will provide any insights on the performance of neural networks. Including a summary, and intuition of the approach will be helpful to the reader.
- Most of the neural networks used have a relu activations (with linear layers), making them a piecewise linear function with respect to the parameters, more importantly, nonsmooth. In fact, in it possible to count the number of linear regions, see (http://proceedings.mlr.press/v80/serra18b.html). In this light, it is unclear what the main message of the paper from an intuitive perspective is. Moreover, the paper misses an important line of work in statistics called Experimental Design in which the goal is to approximate a dataset using a subset with respect to various statistical criterion on the estimators, see https://www.jstor.org/stable/2290334?seq=1.
- Related to the previous point: It is unclear what the take away is from the experimental results in the paper. Some of the experiments are performed using very shallow networks (width of 5, depth of 1 and relu activations) whereas others have a few more layers. However, two test points are used in all the experiments. Why? All of the datasets used in the experiments have thousands of points in each class, and it is also possible to augment them by simple rotations, translations etc.. The paper will benefit by adding robust estimates of Pearson and Spearman using a fraction of test samples around the highest and 50% test points. 
- Finally, none of the results are provided with confidence intervals over runs. Are the experiments averaged over a few runs? If so, I suggest you include them in the figures to provide an accurate summary of the experiments.

After response: Thanks for the clarifications especially the piecewise linearity of deep network as a function of input and not as parameters (like I incorrectly suggested in my review). I think the empirical results have some merit, but are preliminary and I fail to see the bigger picture. For example, more carefully designed ablation studies are needed to determine the practical utility i.e., when the approximations suggested are reliable and when they are not.
review_title: Summary: The paper provides an empirical analysis of using influence functions in the context of neural network architectures. A detailed empirical study on shallow nets is provided to support the findings.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:
The paper provides an extensive experimental study on using influence functions in deep neural networks. The authors suggest that the estimation via influence functions can be quite fragile depending on various architectures of neural networks. To point out such information, approximation with inverse-Hessian Vector product techniques might be incorrect, which leads to low quality influence estimates. They show several meaningful findings which address the true nature of using influence functions in deep neural networks. One of their findings through various experimental study is that the network depth and width strongly affect influence estimates.

##########################################################################

Reasons for score: 
The introduced through experimental settings seem to be highly interesting and reasonable. I like the idea of addressing weakness of influence functions in deep neural network and the authors’ findings are interesting. I believe it will be better if the authors perform the ablation studies with different parameters and more datasets to generalize their findings.

##########################################################################

Pros: 
1. The paper address the important issue of influence functions, which can affect the field of model interpretability. 
2. The introduced experimental settings seem to be reasonable to empirically show that using influence functions in deep neural networks can be very sensitive depending on the architectures of neural networks, the use of weight decay, and the choice of hyper-parameters.
3. The experiments are well designed with multiple architectures of neural networks and detasets. 

In the case of non-convex loss functions, the assumption of fist-order influence functions is not generally true. They empirically found that the Taylor’s gap is strongly affected by common hyper-parameters for deep networks. 

##########################################################################

Cons: 
More ablation studies with different set of hyper parameters and multiple datasets can improve the quality of this paper. 

##########################################################################

Questions during rebuttal period: 
In the case study using a CNN architecture on the small MNIST, the authors need to perform the experiments with different number of selected training samples with both the highest and the lowest influence scores to amplify their point. 
 
#########################################################################
review_title: Review on "Influence functions in deep learning are fragile"

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this work, the authors perform an empirical analysis of influence functions in deep learning. Influence functions allow one to estimate the effect of weighting a training point at test time (for instance, dropping a training point from the training set with a weight of $\frac{-1}{n}$ under assumptions about the convexity of the loss function and the models Hessian $H_{\theta}$. The authors note that successful applications of influence functions are rare in the literature, despite their potential applications. The authors investigate the effects of model width, depth, regularization schedule, Hessian approximation techniques, and dataset size on the accuracy of influence functions. In general, they find that each of these can cause issues with influence function calculation. As model width and depth increases, the Spearman correlation decreases between the calculated influence functions and the ground truth obtained by retraining. They also find that the top eigenvalue of $H_\theta$ increases quite dramatically as depth increases. Weight regularization was found to be important for successful influence functions calculation, while factors such as the huge size of the ImageNet dataset proved prohibitive at this point. 

Strengths: 
* The paper has excellent presentation. The authors establish the background and relevant literature succinctly but with proper depth 
* I do think that the authors demonstrated that most of the factors they consider do contribute to the difficulty of accurately computing influence functions, but see (weaknesses)


Weaknesses: 
* The authors use the term "significant" or "significantly" 10+ times throughout the paper but I think this word should generally be reserved for when a test of statistical significance is performed. Consider using another word such as "substantial" 
* Why are there no error bars on Fig 1c-d, Fig 2, and Fig 3? Could you consider rerun your experiments over many different initializations of the model? Since you mainly analyzed results on MNIST and CIFAR-10 that should be feasible. 
*Fig 1c and 2c seem particularly stochastic and could benefit from multiple initializations to confirm or deny the trends that are claimed. 

Overall, I think this is a fairly strong work that performs an important empirical analysis but I have concerns about the lack of repeated experiments and seeing whether these trends reported hold up over many initializations. I rate it a 6 on this basis.  
review_title: influence functions are indeed fragile, but how significant are these results? 

Meta-review: This paper examines under what conditions influence estimation can be applied to deep networks and finds that, among of items, that influence estimates are poorer for deeper architectures, perhaps due to poor inverse Hessian vector approximations for poor for deeper models. The authors provide an extensive experimental evaluation across datasets and architectures, and demonstrates the fragility of influence estimates in a number of conditions. Although the reviewers noted that these issues are now "folk knowledge", there has been less scientific effort in identifying these failures.

Of course, more theoretical understanding would help the community better understand where these fragilities lie, but the experimental evaluation is sufficiently strong to be of broad interest to the community.
Decision: Accept (Poster)