Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The influence function-based method can analyze the effects of adding/removing/perturbating a single data point on parametric models using a linear approximation at the optimal point of parameters. It was pointed out that this approximation on deep neural networks is poor in previous works. 

In this paper, the authors push it further and analyze the factors that affect the approximation, and argue that the performance of influence function methods is not necessarily reflected in the LOO re-training process, but rather can be measured using proximal Bregman Response Function. In the experiments, the authors thoroughly analyzed how different factors affect the discrepancy between influence functions and LOO retraining and showed that PBRF better aligns with influence function methods.
strengths_and_weaknesses: Strengths:
+ The authors perform analysis and make hypotheses on an important problem (the discrepancy between influence function and LOO), and point out five sources that affect the misalignment
+ The authors pointed out that PBRF better captures the behaviour of influence functions
+ The experiment evaluations are thorough

Weaknesses:
- Although identifying the sources that lead to the misalignment is quite interesting, I'm not very convinced that introducing a different score to solely analyze the influence functions is quite useful or necessary. Influence function methods are initially proposed as an interpretability tool to analyze the model behaviour of removing data points. Introduction and emphasizing on PBRF seem to have lost this goal and are leading to a less meaningful direction.
- In general, I think the structure and writing of this paper need to be improved.
questions: see above
limitations: authors briefly mentioned that in Q/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies influence functions. Influence functions work well with leave-one-out retraining for linear models, but they often have poor performance in neural networks. The authors investigate the specific factors that cause this discrepancy and decompose into five terms. The authors then study the contributions of each term. Furthermore, the authors show that influence functions are often a good approximation to the proximal Bregman response function (PBRF). The experimental results suggest that current algorithms for influence function estimation can still give more informative results than previous error analyses would suggest. In my opinion, the main contribution of this work is a more in-depth analysis of influence functions, especially as it applies to neural networks. Overall, the paper is well organized and well written.
strengths_and_weaknesses: Strengths:
- This work investigates the source of the discrepancy between influence functions and LOO retraining in neural networks, and further decompose the discrepancy into five detailed components.
- The authors evaluate the contributions of each component on binary classification, regression, image classification, and language modeling tasks.
- The experimental results show that influence functions are a much better match to the proximal Bregman response function (PBRF), and also can give more informative results than previous error analyses would suggest.

Weaknesses:
- What is the underlying reason for selecting the 5 components and why are these components considered important? These are not explicitly stated in the paper. 
- The author considers PBRF can be regarded as the gold standard, and I think more rigorous proofs are needed.

questions: Here are some questions for the authors:
- As mentioned in weaknesses, what is the underlying reason for selecting the 5 components?
- Are there any considerations in the selection of experimental datasets?

limitations: The authors adequately addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies what causes influence estimates misalign the leave-one-out retraining. Specifically, the authors decompose discrepancy into five separate terms and empirically show the contributions of each term under a variety of settings (e.g., model architectures and datasets). 
strengths_and_weaknesses: *Originality*: The paper shows the connections between influence estimates and the decomposition of the five terms, and finds out that influence estimates are highly correlated to the proposed proximal Bregman response function (PBRF). The results are overall interesting. 

*Quality*: The decomposition is technically sound and empirical analysis of the decomposition is well conducted. However, I suggest the authors show some use cases of PBRF empirically. As demonstrated in the experiment, PBRF is highly correlated with influence estimates. Does it offer some better advantage over influence estimates in some use cases such as data poisoning, improving fairness and explanablity of the model. I think these are also key questions to answer. 

*Clarity*: The paper is well written and easy to read. However, some technical details of the paper are not well explained. For example, I am a little confused by the introduction of Linearization Error (from Eq. 7 to Eq. 9). I am not sure what are the purposes/usages by performing the second-order expansion of the loss around $y_s$. 

*Significance*: This paper provides an in-depth understanding of influence function by trying to bridge the gap between influence function and LOO re-training . It could impact on the community in the long run. 

questions: 1. What is the purpose of the deviation from Eq. 7 to Eq. 9?
2. In eq. 8, the authors mention “$f_{\text{lin}}(\cdot, \cdot)$ is the linearization of the network outputs with respect to the parameters”. 
3. What do you mean by “the linearization of the network outputs”? Is there any practical advantage of using over PBRF influence estimates?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary:  The paper tackles  a very important topic about the evaluation of influence function based methods. Currently, the effectiveness of influence functions are based on (a) Direct metrics such as Leave-out-retraining OR (b) Indirect metrics such as detection of mislabeled samples etc. The authors propose to use proximal Bregman response function as a way to measure the effectiveness or alignment of influence functions, especially for deep models. 
strengths_and_weaknesses: Strengths: 
- The paper does a comprehensive analysis of the different components (warm start, non-convergence, linearization, addition of iHVP etc. ) involved in the misalignment between the influence function and the LOO training objectives. The analysis is interesting and well-executed.

- The authors acknowledge that in principle the deep models are only partially trained as they always don't converge to the optima. Hence, taking this information into account and designing a gold-standard ground-truth is impactful to understand the effectiveness of influence functions. 

- The array of experiments are solid and cover a wide range of architectures and datasets / domains. The paper is a good extension to Basu et al (ICLR 2021)[1]  and follows up that paper with more in-depth experiments.

Weaknesses:
- While the PBRF metric aligns well with the influence score, I would like to see some more motivation about why the authors thought it’s a good metric to evaluate influence functions on.  The current version of the paper lacks this motivation or information corresponding to it.
 
- If we already know IFs are good at other metrics such as detection of mislabeled examples / relabeling, why do we require a fix to the evaluation procedure of influence functions. It would be good if the authors could substantiate on why alignment of IF with a gold standard ground-truth is important as opposed to testing them on other indirect but more practical metrics such as detection of mislabeled examples / relabeling. 

- While the authors cover a wide range of experiments, I would like to see some experiments on more complex datasets such as Imagenet-1k. A focused analysis on Imagenet-1k would make the paper stronger.


[1]. https://arxiv.org/abs/2006.14651?context=stat.ML
questions: The questions are added in the Weaknesses section. 
limitations: - Lack of motivation on the design of the PBRF metric.

- More analysis is required on Imagenet scale datasets. Although IF is tricky to compute on Imagenet, it would be beneficial to understand the proposed metric at scale.

Overall, I feel the paper is strong and does a well laid out analysis on the alignment of influence estimation and LOO re-training which is lacking in the community. I would also urge the authors to think about in what practical settings and how this metric can be used instead of the commonly used indirect metrics such as mislabeled data points detection.
review_title: null

Meta-review: metareview: All reviewers felt that this paper deserves to be accepted given its interesting analysis of an important problem and compelling empirical evaluations. The authors should consider incorporating the reviewers' comments, particularly around motivating the PBRF as a "gold standard" and clarifying aspects of the presentation, in the final version of the paper.
confidence: Certain
Decision: Accept