Towards improving saliency map interpretability using feature map smoothing

Authors: Dipkamal Bhusal, Md Tanvirul Alam, Monish Kumar manikya Veerabhadran, Michael Clifford, Sara Rampazzi, Nidhi Rastogi
Primary area: interpretability and explainable AI
Keywords: computer vision; saliency map; explanations;  vanilla gradient; integrated gradient; smoothgrad

Abstract:
Input-gradient-based feature attribution methods, such as Vanilla Gradient, Integrated Gradients, and SmoothGrad, are widely used to explain image classifiers by generating saliency maps. However, these methods struggle to provide explanations that are both visually clear and quantitatively robust. Key challenges include ensuring that explanations are sparse, stable, and faithfully reflect the model’s decision-making. Adversarial training, known for enhancing model robustness, have been shown to produce sparser explanations with these methods; however, this sparsity often comes at the cost of stability. In this work, we investigate the trade-off between stability and sparsity in saliency maps and propose the use of a smoothing layer during adversarial training. Through extensive experiments and evaluation, we demonstrate this smoothing technique improves the stability and faithfulness of saliency maps without sacrificing sparsity. Furthermore, a qualitative user study reveals that human evaluators tend to distrust explanations that are overly noisy or excessively sparse—issues commonly associated with explanations in naturally and adversarially trained models, respectively and prefer explanations produced by our proposed approach. Our findings offer a promising direction for generating reliable explanations with robust models, striking a balance between clarity and usability.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730370653704

thread_type:
official_review

<thread_object>
creator: reviewer_qv4w

<content>
summary:
This paper addresses challenges in the interpretability of saliency maps for image classifiers, especially those based on input gradients. Commonly used saliency map methods like Vanilla Gradient, Integrated Gradients, and SmoothGrad often fail to provide both visually clear and robust explanations. This study examines the trade-offs between stability and sparsity in saliency maps and proposes a feature map smoothing technique during adversarial training to address these issues. Experimental evaluations demonstrate that this method enhances the clarity and usability of saliency maps, making them more comprehensible and trustworthy for users. A qualitative user study supports these findings, showing that human evaluators prefer the proposed approach's explanations due to their balance between sparsity and clarity.

strengths:
1. The paper presents an innovative approach by applying feature map smoothing in the context of interpretability, which has traditionally been studied mainly within adversarial robustness. By exploring its role in saliency map interpretability, the paper introduces a fresh perspective that adds significant value to explainable AI.
2. Given the indirect measures required in explainable AI due to the lack of ground truth, the experimental setup and interpretations in this paper appear extensive and credible. The design allows for robust support of the proposed method’s effectiveness in balancing clarity and stability without sacrificing sparsity.

weaknesses:
1. ($\bf{Major}$) A major flaw lies in the mathematical logic presented in Section 3.1, particularly in equations (5), (6), and (7), where the claim that stability of input-gradient-based attribution ($\Delta$) is a direct result of a predictive model's sensitivity (line 171) seems overly simplified. Rather, this is mathematically false. This is because upper bound does not mean a proportional relationship. The paper lacks a discussion on whether the derived upper bounds are tight or vacuous, which would be crucial for the validity of their approach. A more nuanced analysis is needed to reinforce this section's findings.
2. ($\bf{Major}$) Sections 3.1, 3.2, and 3.3 seem loosely connected, lacking a cohesive framework. For example, if the authors’ claim regarding the direct correlation between stability and model sensitivity were accurate (I believe it's not true at this stage), then Sections 3.2 and 3.3 would need a more rigorous focus on how adversarial training and feature map smoothing concretely reduce the model's sensitivity. Presently, these sections feel more like an introduction to known techniques rather than a focused analysis supporting the main claim.
3. (Minor) The presentation of notations in Section 3 could benefit from refinement, as certain notations are repeated excessively. For example, the notation on $F(x)=H(<w,x>)$ appears too repeatedly.

questions:
How confident are the authors in the tightness of the upper bound in equations (5), (6), and (7)? Are there alternative approaches to validate the proportional relationship between stability and sensitivity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730496360722

thread_type:
official_review

<thread_object>
creator: reviewer_24zf

<content>
summary:
The paper proposes a method to improve the quality of saliency maps by adding a smoothing layer during adversarial training. The paper discusses that existing smoothing techniques can lead to noisy, overly sparse and incomplete model understanding. Proposed methods strike a balance between sparsity and coherency and focusing on the key regions. The paper discusses that explanation sensitivity stems from the model itself and adversarially-trained models address that sensitivity. The paper shows that their approach achieves better performance in terms of stability and faithfulness on the FMNIST, CIFAR-10 and ImageNet datasets.

strengths:
+ Overall the paper the methods and introduction sections of the paper are well-written and easy to follow the story of the paper.
+ The paper evaluates their approach against a wide variety of quality metrics and methods.

weaknesses:
+ In sections 3.2 - 3.3 the paper discusses that adversarial training leads to less input sensitivity and less stability. At this point in the paper it is unclear what stability is. If the model is less sensitive then the explanation will be more stable. Why is stability an issue here ? Stability and sensitivity terms don’t seem to be defined in advance. I recommend clarifying those terms before using those terms, otherwise it is hard to follow a logical chain of thoughts.
+ It is a bit hard to follow different methods compares against each other in the paper:
NG models (non-local gaussian) - what does it really mean ? Figures 4 - 9 have plots for CIFAR, FMNIST and ImageNet but ImageNet is not a dataset and the former 2 are. It is unclear what are the proposed methods and what are other methods that we are comparing against. Are M1 and M2 the methods that the authors propose and the rest are what we are comparing against ? The naming and notations of the methods makes it difficult to understand what is going on with the experimental results. 
+ NG models (non-local gaussian) sounds like a model and at the same time it is evaluated in combination with ImageNet. I really recommend to have more descriptive titles in the figures. Reading the titles of the figures doesn’t tell much about how and what to interpret on the figures. At the same time the figures are full of non-intuitive acronyms and abbreviations. 
Figure 4c: Why are the accuracy trajectories significantly different for M1 and M2 ? At the same time the trajectories almost overlap for  Figure 4b. Why is that the case
+ The paper discusses the limitations of the proposed approach however it is unclear why exactly authors chose 1x1 convolution after the smoothing operation.

questions:
+ It is unclear what layers the smoothing should be applied. Should it be applied on the last fully connect layer ? Would the quality of explanations change if the smoothing was applied in the earlier or later layers ?
+ See weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730555835294

thread_type:
official_review

<thread_object>
creator: reviewer_aesz

<content>
summary:
The paper explores enhancing the interpretability of saliency maps for image classification networks by balancing sparsity and stability. Traditional gradient-based attribution methods, like Vanilla Gradient and SmoothGrad, aim to provide clear explanations but often face issues with either excessive sparsity or instability, particularly under adversarial training. To address this, this paper proposes adding a smoothing layer during adversarial training to maintain sparse yet stable saliency maps. Their experiments show that this approach enhances the stability and faithfulness of explanations without compromising on sparsity. A user study confirms that human evaluators prefer the explanations generated by this method, as they find them clearer and less noisy than those from natural and adversarially trained models.

strengths:
This paper presents two strengths. 
1. It introduces an interesting approach that effectively balances stability and sparsity in saliency maps by applying a smoothing layer during adversarial training. 
2. This paper includes a comprehensive user study that validates the method's effectiveness from a human perspective. The study shows that users find these explanations more interpretable and trustworthy.

weaknesses:
The following questions hinder the contribution of this paper.
1. Though applying feature map smoothing and adversarial training improves the sparsity and stability of the model explanations, it is unclear whether this training technique hinders the accuracy. Besides, this XAI method requires retraining the model, which limits the generalization ability when the models can not be trained.
2. The proposed smoothing technique is simply adapted from the Feature-map Smoothing paper (Xie et al., 2019), hindering the novelty.
3. There are many typos in this paper, and the author is encouraged to go through the paper carefully. For instance, in Line 261, page 5, ImageNette -> ImageNet. Besides, the author misused  /cite, /citep and /citet, see Line 414 in Page 8 for an example.

questions:
Please refer to the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659047740

thread_type:
official_review

<thread_object>
creator: reviewer_exux

<content>
summary:
The authors propose a solution to improve saliency maps by smoothening the feature maps during training.
The solution aims to address instability of the maps incurred by adversarial training.
The solution is tested through experiments with ResNet-18 on three datasets (CIFAR10, FMNIT, ImageNette).

strengths:
- The authors offers new insights about the trade-off between stability and sparsity in saliency maps.
- The solution provided seems helpful to improve the stability (and potentially the faithfulness), with minimal impact on the sparsity.

weaknesses:
- The contribution is limited in terms of depth and impact. 
The method proposed requires adaptation to model training and architecture and hence cannot be used as a generic attribution method. 
- The analysis is limited in terms of architectures tested.
The method only supports computer-vision models. Moreover, there were no tests with ViT, now a major architecture in CV.
- There was no in-depth analysis about why smoothening is required besides showing some experimental results.
My concern is that smoothening incurs significant change to the architecture by adding new layers. Besides an explicit 1x1 convolution added, the smoothening filter is implemented as an implicit KxK conv layer. This makes most comparisons unfair. Even if the parameters of the smoothenng kernel are frozen, however, the kernel itself expands the receptive field. By taking into account all layers with strides, the cumulative expansion of the RF becomes very significant. The perceived advantages might simply be a result of this expansion. The authors did not control for such effects.  

There were frequent writing issues with respect to articles and citation format. For ex:
we demonstrate this smoothing => that
within same object => the same
for the designing models => remove “the”
Please use \citep when the citation is not a natural part of the text and should be inside parentheses, and use \citet when the author names are part of the sentence with sound grammar.

questions:
How would the proposed method work for ViT models? And how could smoothening be implemented in NLP transformers?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers raised several weaknesses: There is insufficient justification for the smoothing technique, and the addition of new layers, especially the smoothing filter, unfairly alters the architecture. Furthermore, the paper's terminology around stability and sensitivity is unclear, and the experimental results are difficult to follow due to poor figure titles and unclear method comparisons. The mathematical logic in Sections 3.1–3.3 is overly simplistic, with unsubstantiated claims about the relationship between stability and model sensitivity. Additionally, the paper suffers from numerous writing issues, such as typos, citation formatting errors, and unclear notations, which affect overall clarity. The proposed method is also not sufficiently novel, as it closely resembles existing work, and its impact on model accuracy and generalization is not fully addressed.