Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a generic framework which encapsulates different local explanation techniques as special cases of their LFA framework (linear function approximation). They introduce the no free lunch theorem here within the perspective of local explanations claiming that no single explanation method can perform local function approximation across all neighborhoods. Some of the key experimental results include comparison between Captum's explanation and the explanation from proposed approach by comparing the proximity of these. In addition, perturbation based tests have also been done to reinforce the notion of the no free lunch theorem
strengths_and_weaknesses: **Strengths**

1) Paper is easy to follow and the authors have supported their claims through benchmarked results. Overall coverage of local explainers within their framework is also reasonably exhaustive. 

**Weakness**

1) While the no free lunch theorem here might appear a really novel contribution, I don't find it particularly appealing as papers in the past Dylan et al (Fooling LIME and SHAP - AIES 2020) and several others have demonstrated that neighborhood samplers are always prone to issues such as adversarial attacks, bias, etc. I believe the no free lunch theorem is just a fancier way of presenting something already known in the community.

2) I would have liked to see some more results on how LFA is more robust to issues such as overfitting etc (due to train/dev/test and hyper parameter tuning advantages, etc). Its a simple experiment but can enhance value of the paper possibly.

**Post Rebuttal**

I have changed my score after reading the author response to my queries.

questions: 1) In line with the review above, I would expect authors to add on what insight is this no free lunch theorem providing to the XAI community which was not known before ?
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work presents a framework called local function approximation that can be used to unify local feature importance explanations (based on gradients and perturbations). 
strengths_and_weaknesses: The overall usefulness of the LFA framework is questionable. 

Based on the current work, a framework that unifies perturbation methods and gradient based methods is "good to have" in terms of understanding all the methods together and perhaps designing new methods in future. The idea of characterising explanation methods via model recovery which is also good. However, whether the framework can help practitioners decide or guide which explanation method to use for which dataset is still not clear. 

Some of the ideas discussed related to perturbation methods are partially known in AI explainability community. The fact that one could fit a model alternate to ridge regression for LIME or one could do train/test on the perturbed dataset generated by LIME, etc. or may be use an alternate loss function, etc. 


questions: a) Can you mention a few practical uses of the LFA framework for data scientists or practitioners ? 

b) Is it possible to overcome any limitation in any of the explanation methods proposed in the literature through the LFA framework ?

c) If my use case of explainability is debugging a model or identifying bias. Can you elaborate how the LFA framework could help me decide which explanation method I should use for these use cases. 

d) Could you please elaborate if the LFA framework could help me evaluate the different explanation methods that it unifies and how? 






limitations: See above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a framework for local function approximation explanations (LFA) for generating explanations of black box ML models. They present theoretical results that place several existing explanation methods from the literature within the LFA framework; they provide a variety of theoretical and computational results demonstrating the value of this framework.

strengths_and_weaknesses: The authors very clearly present their conceptual framework (LFA). Their theoretical results are helpful (and intiutive, though this isn't bad). Their theoretical and computational results clearly demonstrate the utility of LFA.

In the context of explaining ML models, their contributions are a bit narrow: this paper focuses on explanations based on local function approximations, and they use fidelity as a measure of explanation "goodness". But the authors are extremely thorough in exploring this region of ML explanations. Overall, I don't have anything bad to say about this paper.
questions: None
limitations: The authors sufficiently addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose the local function approximation (LFA) framework and prove that numerous local explanations are an instance of this framework. In their proposed no-free lunch theorem of explanations, the authors show that no single explanation can outperform the rest of the explanations in all neighbors around the local instances. The claims are shown across numerous datasets and models. 

Overall, I vote for the acceptance of the paper. I do believe that the proposed approach is sound and interesting. However, I suspect that some of the assumptions in the paper need more careful refinement. 

I am open to changing my score in case the authors can provide sound answers to the questions I have written in the questions section. 
strengths_and_weaknesses: Strength: 

* The problem of unifications explanation techniques is an essential and important topic of study
* The authors have built upon numerous other related works and include formalism to strengthen their arguments.
* The evaluation includes numerous explanation techniques and datasets

Limitations:
 
* The paper is dense and was very hard for me to read. I kindly suggest to the authors to try to help readers navigate all the theories in their work by including more explanations and not relying on compact forms of writing as theorems (especially in Section 3)

questions: * I agree with the authors that the loss formulation of LFA resembles LIME (Equation 1). But we also know that LIME does much. LIME does feature selection before training the surrogate model. Can the authors explain whether the claims and findings still apply?  

* Can the authors provide a similar visualization as Figure 3 to show the permutation tests for cases where important features were removed? I think it is important to understand the model recovery in those cases as well. 
limitations: Line 364-370: The authors provide some limitations of their work about including a subset of all explanation techniques. As a reader, I was keen to know a more detailed discussion on 1) limitations of their proposed framework, e.g. where LFA formulation cannot capture explanation techniques 2) Limitations of the assumptions in LFA framework for some application scenarios.
review_title: null

Meta-review: metareview: The decision is to accept the paper.

The authors propose a unifying Local Function Approximation framework for characterizing a number of local explanation methods into a single formalism. Using this formalism, the authors provide a no-free-lunch theorem indicating that no explanation method can dominate all others across all perturbation specifications. The authors then provide a guiding principle regarding global recovery of the underlying function when the explanation function class is complex enough, and give several examples where different explanation methods do/do not satisfy this principle. There are also empirical results to confirm the theory, and suggestions for designing new explanation methods based on the theory.

Overall, there was agreement that the LFA framework provides a useful unified perspective on many different explanation methods. Some of the remaining concerns from reviewers could be addressed by discussing the limitations of the framework more explicitly in the main paper. For example, referencing appendix A.1 and the types of methods that are not characterized by this framework; or discussing how the choice of explanation method may also depend on how the particular use case is mapped to neighborhoods etc, which is out of scope here (in the spirit of the current comment about the choice of G). This is mainly a matter of expanding the discussion a bit, which should be feasible with an extra page.
confidence: Certain
Decision: Accept