Salvage: Shapley-distribution Approximation Learning Via Attribution Guided Exploration for Explainable Image Classification

Authors: Mehdi Naouar, Hanne Raum, Jens Rahnfeld, Yannick Vogt, Joschka Boedecker, Gabriel Kalweit, Maria Kalweit
Primary area: interpretability and explainable AI
Keywords: Explainability, XAI, feature attribution

Abstract:
The integration of deep learning into critical vision application areas has given rise to a necessity for techniques that can explain the rationale behind predictions. In this paper, we address this need by introducing Salvage, a novel removal-based explainability method for image classification. Our approach involves training an explainer model that learns the prediction distribution of the classifier on masked images. We first introduce the concept of Shapley-distributions, which offers a more accurate approximation of classification probability distributions than existing methods. Furthermore, we address the issue of unbalanced important and unimportant features. In such settings, naive uniform sampling of feature subsets often results in a highly unbalanced ratio of samples with high and low prediction likelihoods, which can hinder effective learning. To mitigate this, we propose an informed sampling strategy that leverages approximated feature importance scores, thereby reducing imbalance and facilitating the estimation of underrepresented features. After incorporating these two principles into our method, we conducted an extensive analysis on the ImageNette, MURA, WBC, and Pet datasets. The results show that Salvage outperforms various baseline explainability methods, including attention-, gradient-, and removal-based approaches, both qualitatively and quantitatively. Furthermore, we demonstrate that our explainer model can serve as a fully explainable classifier without a major decrease in classification performance, paving the way for fully explainable image classification.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730216126227

thread_type:
official_review

<thread_object>
creator: reviewer_hzto

<content>
summary:
This article proposes a new explainability method for image classification. Most of the explainability methods target CNNs; instead, Salvage, the proposed method is architecture agnostic. Salvage is a removal-based approach based on ViT-Shapely with further improvements. The method surpasses the current SOTA.

strengths:
The paper is clear and well-explained. The methodology is adequate. The method is well evaluated. The method, despite not being completely novel, builds on SOTA methods (ViT Shapely) and improves their shortcomings.

weaknesses:
The method is incremental with respect to ViT-Shapely but better. So no much concern.

questions:
I find the paper good as is.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535030299

thread_type:
official_review

<thread_object>
creator: reviewer_7jfj

<content>
summary:
This paper introduces Salvage,a removal-based explainability method for image classification. It includes a concept of Shapley-distributions,which offers a more accurate approximation of classification probability distributions and an informed sampling strategy that leverages approximated feature importance scores to reduce imbalance and facilitate the estimation of underrepresented features.

strengths:
1. A  new concept of Shapley-distributions,which offers a
 more accurate approximation of classification probability distributions, is introduced.
2. The comparison methods, datasets, and metrics are quite comprehensive. On some metrics, it has a clear advantage.

weaknesses:
**The presentation should be improved a lot.**

1. The explanation of symbols in the formulas is not clear enough, causing difficulty in understanding, such as Eq.2.
2. In Table 1, there are several obvious typographical errors of the experimental results, for example “68,56”. It should be “68.56”.
3. The bottom line of the Table 2 is not drawn!

**Soundness**
1. There is no theoretical  proof or experimental results can demonstrate that the INFORMATIVE SAMPLING has improved efficiency.
2. In Table 2, on PET, in terms of RRA, Salvage underperforms about 3 point compared with SOTA. There should be appropriate analysis and discussion regarding this.
3. Only make ablation study on INFORMATIVE SAMPLING.

questions:
1.How is the experimental performance when using SHAPLEY DISTRIBUTION ESTIMATION alone?

2. Is this method applicable to other types of tasks (such as object detection or segmentation)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630052218

thread_type:
official_review

<thread_object>
creator: reviewer_sz6d

<content>
summary:
This paper introduces a novel explainability method for image classification known as Salvage. The paper employs a removal-based technique coupled with the concept of Shapley-distributions. These techniques are used to train an explainer model that learns the prediction distribution of the classifier on masked images. The authors address the imbalance between important and unimportant features by devising an informed sampling strategy. This strategy facilitates better approximation of the classifier’s distribution and helps the estimation of underrepresented features. The effectiveness of Salvage is validated through experiments on the ImageNette, MURA, and Pet datasets. The study illustrates that Salvage outdoes various baseline explainability methods and can additionally be used as a fully explainable classifier without a considerable fall in classification performance. The paper concludes by pointing out future optimizations and improvement possibilities for Salvage.

strengths:
The paper presents "Salvage," a novel removal-based explainability method for image classification that tackles unbalanced important features via an informed sampling strategy. The invention of Shapley-distributions for a more accurate approximation of classification probability distributions is impressive. The paper's comprehensive and clear presentation, alongside robust experimental evaluation, underlines the method's potential for explainable classification, with comparable accuracy to standard classifiers.

weaknesses:
1. Although the paper demonstrates a performance comparison with a couple of explainability methods like ViT-Shapley and RISE, more extensive comparison with a wider array of contemporary removal-based explainability methods could provide a more robust evaluation of the Salvage algorithm's efficacy.
2. It would be beneficial to see Salvage's performance with other types of data or in other domains, beyond the ones mentioned in the paper (ImageNette, MURA, and Pet datasets). This would help in evaluating the broad applicability and versatility of the approach.
3. The impacts of temperature parameter changes in the softmax or sigmoid functions during the approximation of the classifier’s distribution could have been explored in more depth. More extensive experimental study in this aspect could enhance the robustness of Salvage.

questions:
1. Could you please elaborate more on why the informative sampling's improvement in performance is less noticeable in the Pet dataset compared to the other datasets?
2. How would different neural architecture designs for the explainer model impact the performance of Salvage?
3. Given that the model has been tested on a limited number of datasets, have you considered testing Salvage on a wider array of datasets, particularly more complex or diverse ones, to evaluate its broad applicability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720401937

thread_type:
official_review

<thread_object>
creator: reviewer_d1q4

<content>
summary:
The authors highlight the shortcomings of existing methods for existing Shapley-based explainers and propose a method called Salvage, which effectively learns and samples based on the Shapley value distribution.

strengths:
1) The issues identified with existing methods appear valid and relevant.
2) The authors demonstrate an improvement in explanation accuracy compared to existing methods across various datasets.

weaknesses:
1) The explainer model is essentially an estimation model for interpreting the behavior of the target classification model; however, this paper does not clearly define what the target model is. Furthermore, there is insufficient evidence to show that the method operates effectively across different target models.
2) Some results seem incomplete, as suggested by Figure 1.
3) The ablation study is lacking. While the proposed method focuses on effectively learning the distribution and sampling, there is no analysis of which aspect is more critical to the overall success of the method.
4) The problems identified with existing methods are described conceptually but lack empirical validation.

questions:
1) What exactly does Figure 3 illustrate?
2) Since the goal is to derive an explainer model for a specific classification model, I am curious not only about the classification performance but also about how well the predictions align with those of the existing classification model (as seen in Table 2).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732304528330

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply (1)

comment:
First and foremost, we thank all the reviewers for their time and effort in reviewing our manuscript. The constructive feedback has significantly contributed to improving our paper. We have submitted a revised version with all changes marked in blue. Below each review, we provide detailed responses to the raised points. The key changes are as follows:

* We added qualitative examples of the missing baselines to Figures 1 and 8. 
* We revised Section 5.4 and Figure 3 to provide clearer ablation studies investigating the individual contribution of each core principle in our method (Shapley distribution estimation and informative sampling).
* An efficiency analysis of Salvage with and without informative sampling has been added to Section A.2 of the appendix.
* We added a new Section A.4 to the appendix providing more insights about the alignment between the predictions of the explainer model and those of the original classification model.
* We improved upon the presentation of the paper.

We hope these revisions address the concerns raised, and we thank the reviewers again for the constructive and fruitful discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732633125860

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply (2)

comment:
We would like to thank all reviewers once again for their valuable feedback and fair assessment. In response, we have submitted a revised version of the manuscript, with all new changes highlighted in green while retaining changes from the first revision in blue. The key updates are as follows:

- **Inclusion of an additional dataset (WBC):** We incorporated the WBC dataset [1]. We believe this dataset significantly enhances the diversity of our evaluations, as it introduces a new type of image data that is markedly different from the other three datasets.  We updated all relevant tables and figures to include its results. This addition aims to further demonstrate the robustness of our method across diverse datasets, addressing a suggestion from reviewer sz6D.

- **Improved presentation of the classification analysis (Section 5.5):** We enhanced the discussion to include both the classification accuracy of Salvage and the alignment of its predictions with the classifier model. This refinement follows reviewer d1Q404's suggestion to provide a more comprehensive analysis.
We hope these updates address the reviewers' suggestions and further clarify the contributions of our work. Thank you again for your constructive feedback and thoughtful evaluation.

[1] Alexandra Bodzas, Pavel Kodytek & Jan Zidek. A high-resolution large-scale dataset of pathological and normal white blood cells. Scientific Data, 10, 07 2023. doi: 10.1038/s41597-023-02378-7.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Salvage, a removal-based explainability method for image classification. The proposed method uses the concept of Shapley-distribution by learning the prediction distribution on masked images. This strategy helps the estimation of underrepresented features. Experiments were conducted on ImageNette, MURA and Pet datasets.

Overall, after the author-reviewer discussion, all the reviewers agreed that this paper exceeds the acceptance threshold. Specifically, the authors' responses include more experimental results on an additional dataset (WBC) and the revised paper's presentation is improved based on the reviewers' comments. I agree with the reviewers and recommend acceptance.