How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations

Authors: Siddhartha Gairola, Moritz Böhle, Francesco Locatello, Bernt Schiele
Primary area: interpretability and explainable AI
Keywords: Interpretability, Explainable AI, Representation Learning, Pre-training

Abstract:
Post-hoc importance attribution methods are a popular tool for “explaining” Deep Neural Networks (DNNs) and are inherently based on the assumption that the explanations can be applied independently of how the models were trained. Contrarily, in this work we bring forward empirical evidence that challenges this very notion. Surprisingly, we discover a strong dependency on and demonstrate that the training details of a pre-trained model’s classification layer (<10% of model parameters) play a crucial role, much more than the pre-training scheme itself. This is of high practical relevance: (1) as techniques for pre-training models are becoming increasingly diverse, understanding the interplay between these techniques and attribution methods is critical; (2) it sheds light on an important yet overlooked assumption of post-hoc attribution methods which can drastically impact model explanations and how they are interpreted eventually. With this finding we also present simple yet effective adjustments to the classification layers, that can significantly enhance the quality of model explanations. We validate our findings across several visual pre-training frameworks (fully-supervised, self-supervised, contrastive vision-language training) and analyse how they impact explanations for a wide range of attribution methods on a diverse set of evaluation metrics.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730232844953

thread_type:
official_review

<thread_object>
creator: reviewer_awie

<content>
summary:
The authors find and support an interesting observation that the method of training the classifier layer of a model has a significant impact on the results of post-hoc attribution methods. Because many post-hoc attribution methods assume that model training does not have an impact, they find that this must be reconsidered, and in fact, simply modifying the method of training the last linear layer(s) can improve model accuracy and explainability.

strengths:
This paper is very well written and planned. Not only are the approaches and findings very clear but the authors provide extensive support of their findings over numerous models, datasets, attribution methods, and metrics. 

The choice to study multiple pre-training approaches adds significant strength to their arguments and findings.  

The overall findings are simple, but impactful for future considerations of interpretable model design, post-hoc explainability, and improving model interpretation.

weaknesses:
There are not significant weaknesses to address. There are minor spelling mistakes, but it does not hurt the delivery of the information.

questions:
Would the authors suggest the development of future classification models take into consideration the information in this paper?  

Do the authors think that a training loss could be created to further improve explainability as the minor differences in CE and BCE have a significant effect?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730235762441

thread_type:
official_review

<thread_object>
creator: reviewer_vgda

<content>
summary:
The main motivation behind this work is that two models using the same training regime and ending at the same loss can produce two extremely different attributions for the same image. The authors demonstrate that the training paradigm for the final classification layer of a network is the most important decider in generating more precise attributions, regardless of the attribution method. They specifically show that a binary cross entropy trained output layer produces better attributions than a cross entropy trained output layer. The increase in attribution quality does typically come at the cost of <10% accuracy reduction when using a linear layer, but the accuracy can be improved by using a more complex output layer.

strengths:
Overall, it is an interesting read and demonstrates some interesting results. The writing is generally clear, the issue is well defined, and the experiments are impactful. It is difficult for me to say exactly what the authors did well, other than that it is a good read. 
    
1. In-depth motivation section, outlining the issues around generating consistently clear attributions
2. Plenty of qualitative results
3. Experiments over a variety of pre-trained models and datasets
4. The authors clearly show that this is an attribution-invariant issue.

weaknesses:
There isn't any discussion of why there is an increase in accuracy and attribution quality with more complex output layers. Is it as simple as the layers being larger, or is there another reason? I assume proper train, test, and validation sets have been used?

questions:
1. I am still confused as to why CE produces worse attribution than BCE. Could the authors explain this again?
    2. Also, why is it that the last output layer is so important? Why is the rest of the model have such little importance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730357390748

thread_type:
official_review

<thread_object>
creator: reviewer_7mwf

<content>
summary:
This paper discovers and demonstrates the strong dependence of post-hoc importance attribution methods on the training details of the classification layer of the pre-trained model. Based on this findings, the paper also proposes a simple but effective adjustment to the classification layer to significantly improve the quality of model explanations.

strengths:
This paper reveals and demonstrates the strong dependence of post-hoc importance attribution methods on the training details of the classification layer in pre-trained models.

weaknesses:
1. The experimental method is limited to ResNet50, and the results are not extensive enough. Thus, experimental results are not convincing enough to verify the effectiveness of their methods.

2. The contribution of this article is not enough. The author discovered the impact of training details on post-processing methods, but the evaluation metrics used and the subsequent B-cos model are not the author's innovation.

3. [minor] Figures in this paper have obvious flaws. It will be better that authors carefully revise their figures.

questions:
1. In the case of backbone freezing, increasing classifier parameters can improve performance. Is the design of B-cos MLP necessary? Is MLP not possible?
2. Can you provide more loss function results to verify Softmax Shift-Invariance? How about the cross-entropy loss?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520584402

thread_type:
official_review

<thread_object>
creator: reviewer_ndzf

<content>
summary:
The paper challenges the tradition notation that model explanations are independent of training methods by demonstrating that the quality of attributions for pre-trained models depends significantly on how the classification head is trained. It shows that using binary cross-entropy (BCE) loss instead of conventional cross-entropy (CE) loss leads to marked improvements in interpretability metrics across several visual pre-training frameworks. Furthermore, it is found that the non-linear B-cos MLP probes boost the class-specific localization ability of attribution methods.

strengths:
1.	Clarity and Organization: The paper is exceptionally well-written and structured, enhancing readability and accessibility of the key finding
2.	The study reveals that training probes using binary cross-entropy (BCE) loss instead of the traditional cross-entropy (CE) loss consistently enhances interpretability metrics. The analysis of the Softmax Shift-Invariance Issue in interesting and insightful. This could have substantial implications for various DNN-based applications.
3.	The improvements in interpretability metrics are shown to be consistent across various training methods for the visual encoder. The robustness of these findings was thoroughly validated using diverse learning paradigms, including supervised, self-supervised and CLIP.

weaknesses:
1.	(Major) Limited Model Diversity: The research exclusively utilizes the ResNet50 model backbone, which  canot adequately represent the behavior across various architectures. Testing additional backbones, especially Vision Transformers (ViTs), and incorporating explanation methods tailored for these models (referenced as [1][2][3]), would provide a more robust validation of the findings.
2.	Inclusion of Additional Methods: The paper could be strengthened by including more population perturbation-based methods, such as RISE [4] and Score-CAM [5], to further substantiate the interpretability improvements.
3.	Selection of Examples: Concerns arise regarding whether the examples shown in Figures 1 and 6 are cherry-picked, especially since the GridPG Score in Figure 5 suggests that the BCE model does not always perform perfectly. Including a broader range of examples, particularly where the BCE model scores lower on the GridPG, would offer a more comprehensive understanding and enhance the paper's credibility. 

I would be happy to improve my rating of the paper if these issues are addressed thoroughly.

[1] Transformer interpretability beyond attention visualization. 

[2] Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.

[3] Vit-cx: Causal explanation of vision transformers.

[4] RISE: Randomized Input Sampling for Explanation of Black-box Models.

[5] Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732524943812

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author’s Response to Common Concern Regarding Limited Model Diversity

comment:
In the following, we address the common concern that was shared by Reviewers (nDZf, 7mwF) regarding limited model diversity.

__We updated our submission with results for ViTs, which fully corroborate our previous findings; we thank the reviewers for this excellent suggestion which we believe to further strengthen our submission.__

In particular, following the reviewer’s suggestion, we evaluated on both the conventional ViTs interpretability using CGW1 [1], ViT-cx (CasualX) [2], Rollout [3] and the inherently-interpretable B-cos ViTs [4]. To summarize we observe the following:

* The interpretability scores for BCE-trained classifiers consistently outperform CE-trained classifiers across all pre-trainings (supervised, MoCo, DINO) and explanation methods (see Table 1,2 below).
* The classification accuracy on ImageNet for models trained with BCE loss matches the accuracy of models trained with CE loss (See Table 1,2 below).
* When probing the ViT backbones with Bcos MLPs, we find improvements in both interpretability scores and accuracy (see Table 3).

__Table 1:Accuracy and GridPG Localization Scores for Conventional ViTs on [1,2,3]__
|               |              | ACC % |      |  |           Localization%    |               |
|---------------|--------------|:-----:|:----:|:-------------:|:-------------:|:-------------:|
| Backbone | Pre-training |   CE  |  BCE |   $\Delta_{bce-ce}^{CGW1}$ [1]  | $\Delta_{bce-ce}^{Rollout}$ [2] | $\Delta_{bce-ce}^{CausalX}$ [3] |
|   ViT-B/16  |      Sup     |  73.2 | 72.8 |     **+15.7**     |      **+0.4**     |      **+0.7**     |
|   ViT-B/16  |     DINO     |  77.2 | 77.6 |     **+11.7**     |      **+4.8**     |      **+5.3**     |
|   ViT-B/16  |    MoCov3    |  76.3 | 75.1 |      **+4.4**     |      **+3.3**     |      **+3.8**     |

__Table2: Accuracy and GridPG Localization Scores for Bcos ViTs [4]__
|  Backbone | Pre-training | Acc$_{CE}$% |  Acc$_{BCE}$ | $\Delta_{bce-ce}^{Bcos}$ [4] |
|:--------------:|:------------:|:-----:|:----:|:----------:|
|  B-ViT$_c$-B/16  |      Sup     |  77.3 | 78.1 |    **+23.5**   |
| B-ViT$_c$-S/16 |     DINO     |  73.2 | 73.4 |     **+32.7**  |
| B-ViT$_c$-B/16 |     DINO     |  77.1 | 77.3 |     **+20.4**  |


__Table 3: Bcos MLP vs Linear Probing—Accuracy and GridPG Localization Scores on B-ViTs [4]__
| Backbone     | Classifier   | Acc% | $\Delta_{mlp-lp}^{acc}$ | Loc.% | $\Delta_{mlp-lp}^{loc}$ |
|--------------|--------------|------|-------------------------|-------|-------------------------|
| B-ViT$_c$-S/16 | Linear Probe | 73.4 | --                      |  79.9 | --                      |
| B-ViT$_c$-S/16 | Bcos-MLP-2   | 74.2 |                     **+0.8** |  80.4 |                     **+0.5**|
| B-ViT$_c$-S/16 | Bcos-MLP-3   | 74.7 |                     **+1.3** |  82.4 |                     **+2.5** |
| B-ViT$_c$-B/16 | Linear Probe | 77.3 | --                      |  80.3 | --      |
| B-ViT$_c$-B/16 | Bcos-MLP-2   | 78.2 |                     **+0.9** |  82.1 |                     **+1.8** |
| B-ViT$_c$-B/16 | Bcos-MLP-3   | 79.7 |                     **+2.4** |  83.4 |                     **+3.1** |

We have updated the results in the revised submission under the Appendix in Section C (BCE vs. CE Probing — Table C1, C2: accuracy and localization scores; MLP Probing—Table C3) and referred to this in Section 5 of the main paper.
 
__In short, the new results complement our previous findings and further highlight the general applicability of the findings to diverse architectures and backbones.__

For individual queries please refer to the respective rebuttal sections.

References:
1. H. Chefer, S. Gur, and L. Wolf. Transformer interpretability beyond attention visualization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 782–791, 2020.
2. S. Abnar and W. Zuidema. Quantifying attention flow in transformers. In Annual Meeting of the Association for Computational Linguistics, 2020.
3. W. Xie, X. hui Li, C. C. Cao, and N. L. Zhang. Vit-cx: Causal explanation of vision transformers. In International Joint Conference on Artificial Intelligence, 2022.
4. M. Boehle, N. Singh, M. Fritz, and B. Schiele. B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732536073073

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author's Summary of Rebuttal Discussion

comment:
We would like to thank all the reviewers for their constructive feedback and are pleased that they find our paper to be __“exceptionally well-written and structured"__ (nDZf, VgDa, Awie), our experiments to be __“well defined, planned, [as well as] thoroughly validated”__, showing __“impactful, interesting and insightful results"__ (nDZf, VgDa, Awie) that can have __“substantial implications for various DNN-based application(s)”__ (nDZf).

This is a summary of main concerns and how we addressed them:

1._"Limited Model Diversity"_ (nDZf, 7mwF): __We updated our submission with results for ViTs, which fully corroborate our previous findings; we thank the reviewers for this excellent suggestion which we believe to further strengthen our submission.__ 
(refer to: https://openreview.net/forum?id=57NfyYxh5f&noteId=xEjpQlnGLg)

2._“Inclusion of Additional Perturbation-based explanation methods” (nDZf)_: __We updated our submission additionally with results for ScoreCAM which are very well consistent with our previous findings.__  (refer to: https://openreview.net/forum?id=57NfyYxh5f&noteId=D3n6dTt028)

3._“Concerns regarding the selection of examples” (nDZf)_: __We have updated our submission to address the reviewer's concern regarding the selection of visual examples by adding many more qualitative examples to better reflect the range of different localization scores.__ (refer to: https://openreview.net/forum?id=57NfyYxh5f&noteId=D3n6dTt028)

Importantly, as the reviewers agree (nDZf, VgDa, Awie), __our findings uncover a new crucial aspect of explainable artificial intelligence (XAI)__ that should be considered when using existing attribution methods or developing new ones in the future. 

We hope our revisions and expanded analysis adequately addresses all concerns from the reviewers, and we would be happy to provide any further clarifications.

_Note_
* _All additions to the content in the revised submission have been made in green text, to clearly highlight the changes._
* _Please refresh your browser tab in case the tables / math does not render properly on OpenReview._

__Updates__

* Reviewer (nDZf) have already taken the time to assess our changes and response. They have increased their rating to a positive score, and find our responses to have "well addressed their concerns".
* Reviewer (Awie) have also acknowledged their satisfaction with all our responses, and have retained their positive score.
* Reviewer (VgDa) are satisfied with our response to their questions, and have retained their positive score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper reveals that post-hoc importance attribution methods are strongly influenced by the training details of the classification layer in a pre-trained model. Building on these findings, the authors propose a simple yet effective adjustment to the classification layer, resulting in significantly improved model explanations. Considering the paper's novelty and the generally positive feedback from reviewers, I am inclined to accept this submission.