How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations
Authors: Siddhartha Gairola
Moritz Böhle
Francesco Locatello
Bernt Schiele
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
Explainable AI
Representation Learning
Pre-training
Primary area: interpretability and explainable AI

Abstract: Post-hoc importance attribution methods are a popular tool for “explaining” Deep Neural Networks (DNNs) and are inherently based on the assumption that the explanations can be applied independently of how the models were trained. Contrarily, in this work we bring forward empirical evidence that challenges this very notion. Surprisingly, we discover a strong dependency on and demonstrate that the training details of a pre-trained model’s classification layer (<10% of model parameters) play a crucial role, much more than the pre-training scheme itself. This is of high practical relevance: (1) as techniques for pre-training models are becoming increasingly diverse, understanding the interplay between these techniques and attribution methods is critical; (2) it sheds light on an important yet overlooked assumption of post-hoc attribution methods which can drastically impact model explanations and how they are interpreted eventually. With this finding we also present simple yet effective adjustments to the classification layers, that can significantly enhance the quality of model explanations. We validate our findings across several visual pre-training frameworks (fully-supervised, self-supervised, contrastive vision-language training) and analyse how they impact explanations for a wide range of attribution methods on a diverse set of evaluation metrics.

Published as a conference paper at ICLR 2025
How TO PROBE: SIMPLE YET EFFECTIVE TECH- NIQUES FOR IMPROVING POST-HOC EXPLANATIONS
Siddhartha Gairola 1.2, Moritz Böhle Francesco Locatello", Bernt Schiele
Max Planck Institute for Informatics, Saarland Informaties Campus, Germany,
Institute of Science and Technology Austria, Kyutai, France.
(agairola, schle Meheelleee Empl-int -inf mpg................... de, moritzekyutai.org
francesco. www.elolesstt...oooooooooooos..t losateelleeisss.a......
Work done while at MPT for Informatics
Traikrr Truck
CE Loss
BCE Loss
(b)
Binoculars
Lineat Probe
3-layer MLP
Figure 1: (a) Using Binary Cross- Cooss-Entropy (BCE, col. 3) instead Cross- Entso-y (CE, col. to train linear probes on identical backbones significantly increases the class- class-specificity of explana- tions. (b) Non-linear probes better extract class-specific features from pretrained representations: moving from a linear to 3-layer BCE probe noticeably improves class localization. These critical findings are observed across a wide range of pre-training and attribution methods. Here, we show B-cos (Böhle al., 2022) explanations for model pre-trained using DINO (Caron et al., 2021).
ABSTRACT
Post-hoc importance attribution methods are popular tool for "explaining" Deep Neural Networks (DNNs) and are inherently based on the assumption that the ex- planations can be applied independently of how the models were trained. Con- trarily, in this work we bring forward empirical evidence that challenges this very notion. Surprisingly, we discover a strong dependency on and demonstrate that the training details a pre-trained model's classification layer (<10% model parameters) play crucial role, much more than the pre-training scheme itself. This is of high practical relevance: (1) as techniques for pre-training models are becoming increasingly diverse, understanding the interplay between these techniques and attribution methods is critical; (2) it sheds light on an im- portant yet overlooked assumption of post-hoe attribution methods which drastically impact model explanations and how they are interpreted eventually, With this finding we also present simple yet effective adjustments to the clas- sification layers, that can significantly enhance the quality of model explana- tions. We validate our findings across several visual pre-training frameworks (fully-supervised, self-supervised, contrastive vision-language training), model architectures and analyze how they impact explanations for a wide range of at- tribution methods a diverse set of evaluation metrics. Code available at: https://gi thub. com/ s sidggiirren ro18 /ttww/hwtttoo to cc--prbbee
INTRODUCTION
Most prominently in image classification models, importance attribution methods have emerged as a popular approach mitigate the 'black box' problem of modern Deep Neural Networks (DNNs) (Samek et al., 2021). These methods assign importance values to input features, such as image pixels, to help humans understand why a DNN arrived a particular classification decision.
While most attribution methods do not take the model training into account (are applied "post-hoc"), given the recent emergence of increasingly diverse pre-training paradigms for representation learn- ing (Chen et al., 2020a;b; He et al., 2019; Chen et al., 2020c; Grill et al., 2020; Caron et al., 2021; Chen & He, 2020; Caron et al., 2020; Bardes et al., 2022; Xie et al., 2023; Li et al., 2023a; Radford et al., 2021) and the popularity of importance attribution methods (Bach et al., 2015; Selvaraju al., 2017: Böhle et al., 2022; Shrikumar et al., 2017) to examine model decisions, a better understanding of the interplay between model explanations and training details is of great practical importance.

---

Published as a conference paper at ICLR 2025
DINO: Boos: Wosd x Input
DENO: Layerwise Belerance Propogation
1.0
Wes Input
DENO: Layerwise Belerance Propagation
+2%
+.5%
+44%
Target Class Monaaiiii
+22%
+%
0..5
0.6
EPG
+35
0.2
0.2
C
15
-255%
<500%
<70%%
<100%
+20%
-550%%
<75%
<100%
% of pizels removed
96 pitels removed
BBOX %
(4)
BBOX
(b)
Figure 2: Impact of Loss (BCE vs. CE). (a) EPG Scores, and (b) Pixel Deletion scores for Boos and LRP attributions for linear probe when trained on frozen pre-trained features (DINO in this case). We find that BCE probes lead to more localized and stable attributions, thus higlighting the significant impact of the loss function on well-established attribution methods. Interestingly, despite the fact that the models differ only by single classification layer, the attributions show stark differences in commonly used metrics for evaluating their quality.
In this work, we present an important finding that raises questions about the underlying assumptions of post-hoc attribution methods and their utility on downstream tasks. In particular, we find that the quality of attributions for pre-trained models can be highly dependent on how the classification head (i.e. the 'probe') is trained, even if the model backbone remains frozen. For example. in figure Figure 2a we show that the localization scores of B-cos (Böhle et al., 2022) and LRP (Bach et al., 2015) attributions significantly differ depending on which loss (BCE / CE) is used, with consistent improvements under the BCE loss; this can also be observed qualitatively, see Figure la. Similar improvements can also be seen under the commonly used pixel deletion paradigm for evaluating attribution methods (Figure 2b), despite the fact that the two models only differ by single linear layer and the linear probes themselves have limited modeling capacity (see discussion Appendix B).
Importantly, we find these effects to be robust across several pre-training paradigms, model archi- tectures and attribution methods on downstream classification tasks. Specifically, we conduct a wide range of common interpretability met-
thorough evaluation of the resulting explanations rics, that include object localization (Zhang et al., 2018; Samek et al., 2017; Wang et al., 2020), pixel deletion (Samek et al., 2017; Hedström et al., 2024), compactness (Chalasani al., 2018) and complexity (Tseng et al., 2020) measures. By this we demonstrate empirically that training the probes via the binary cross-entropy (BCE) loss, as opposed to the conventionally used cross-entropy (CE) loss, leads to consistent and significant gains across several interpretability metrics, which thus might have important implications for many baN-d applications.
To further study the impact of the probes, we investigate the interplay between the probe's 's com- plexity and the properties of the resulting explanations. Interestingly, we find that more complex (i.e. multi-layer) probes can better distill class-specific features from the pre-trained representations and thus significantly increase the localization performance (cf. Figure 1b), in particular when using interpretable B-cos Multi-Layer Perceptrons (MLPs) (Böhle et al., 2024).
In short, we make the following contributions: (1) We identify and analyze a critical yet overlooked problem of importance attribution methods, namely that how models are trained can significantly impact the resulting attributions. (2) We show both quantitatively and qualitatively that even when models differ only in their linear probe, explanations can dramatically differ based the objective used to train the probe on downstream tasks; in particular, we find that using BCE instead of CE leads to significantly improved explanations when evaluated on a diverse set of interpretability metrics. (3) We demonstrate that our findings are independent of how the visual encoder (both convolutional and vision transformer architectures) is trained by conducting a detailed study across supervised, self- supervised (MoCov2 (He et al., 2019), DINO (Caron et al., 2021), 1), B BYO (Grill et al., 2020)), and vision languane-n based learning (CLIP, (Radford et al., 2021)). For this, we use diverse explanation methods (LRP (Bach et al., 2015), IntGrad (Sundararajan et al., 2017). B-cos (Böhle et al., 2022), Inputx (Shrikumar et al., 2017), GradCAM (Selvaraju et al., 2017), ScoreCAM (Wang et al., 2020), LIME (Ribeiro et al., 2016)), CausalX (Xie et al., 2022), CGW1 (Chefer et al., 2020), Rollout (Abnar& & Zuidema, 2020) and assess the quality of the resulting explanations on multiple datasets (ImageNet (Russakovsky et al., 2015), VOC (Everingham et al., 2009), and COCO (Lin et al., 2014)). (4) Furthermore, we find that non-linear B-cos MLP probes further boost downstream performance and 'class-specific' localization ability of attribution methods across pre-trained back- bones. (5) We also show for the first time that the inherently interpretable B-cos (Böhle et al., 2022) models are compatible with SSL approaches, preserving both their performance and interpretability.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Importance Attributions. To understand deep neural networks (DNNs), several post-hoc attribu- tion methods (Selvaraju et al., 2017; Chattopadhyay et al., 2018; Bach et al., 2015; Jiang et al., 2021; Wang et al., 2020; Petsiuk et al., 2018; Shrikumar et al., 2017; Sundararajan et al., 2017; Springenberg et al., 2015; Simonyan et al., 2014; Teney et al., 2020; Fong & Vedaldi, 2017; Ribeiro al., 2016; Dabkowski & Gal, 2017; Xie et al., 2022; Chefer et al., 2020; Abnar & Zuidema, 2020), as well as inherently interpretable models (Chen et al., 2019; Brendel & Bethge, 2019; Böhle al., 2021; 2022) have been developed. The attribution (or explanation) map summarizes the DNN computation and assigns importance values to the pixels that the model has used to make decision. Such attribution methods can be broadly classified into the following categories: (1) backpropagation-bess -aseed (Bach et al., 2015; Sundararajan et al., 2017; Shrikumar et al., 2017; Sprin- genberg et al., 2015) that rely on gradients computed either with respect to the input or intermediate layers, (2) perturbation-bust baasedd (Ribeiro et al., 2016; Petsiuk et al., 2018) 8) that assign importance by noting the change output on perturbing the input while treating the network as 'black-box', (3)
uctivation-basel (Selvaraju et al., 2017; Jiang et al., 2021; Wang et al., 2020; Desai & Ramaswamy, 2020) that leverage the weights of activation maps across different layers to assign importance val- ues to the input, and (4) inherently interpretable models (Chen et al., 2019; Brendel & Bethge, 2019; Böhle et al., 2021; 2022) that have been architecturally designed to be more interpretable.
In this work, we present surprising empirical evidence that shows that both 'post-hoc' and inherently interpretable explanation methods highly depend on the weights of the last classification layer, such as the linear probe for self-supervised methods.
Evaluating Importance Attributions. Recent work has studied important properties of such ex- planation methods, like faithfulness (Adebayo et al., 2018: 8; Hooker et al., 2019; Srinivas & Fleuret, 2018; Rao et al., 2022), robustness to adversarial attacks (Ghorbani et al., 2019; Slack et al., 2020; Dombrowski al., 2019) and fairness (Dai al., 2022). In contrast to this, we focus on another important dimension- the sensitivity of explanations to the model training, which has thus far not been systematically studied. And although a dependence on the training has been reported in a few instances (Caron et al., 2021; Tsipras et al., 2019; Böhle et al., 2024), these are mostly limited to single explanation methods and pre-training paradigms. Moreover, while explanation methods are often developed and evaluated in the context of supervised models, the pre-training paradigms even for classification models have become increasingly diverse. Given the fact that many explanations are applied 'post hoc", it is important to understand whether and to what degree they yield consistent results independent of the pre- pre-training paradigm.
To address this, we systematically study wide range of explanation methods across a variety of pre- trained backbones and find that the results are consistent across explanation methods, suggesting that our conclusions not artifact of a particular explanation method backbone combination.
Non-linear Probes. He et al. (2022) argue that linear probes cannot disentangle non-linear representations, and show improved aceuracy by fine-tuning multiple-layers of pre-trained models. Li et al. (2023b) propose to dynamically choose the complexity of a 'readout' (i.e. probing) module increase performance. Similarly, we also find non-linear probes consistently improve classification accuracy. In contrast to these works, we propose using interpretable MLPs that lead to both improved accuracy and quality of explanations.
3 INTERPRETABLE PROBING OF
REPRESENTATIONS
To test the generality of our findings and isolate the impact of pre- pre-training, we evaluate model ex- planations across a broad range of pre-training paradigms, with a specific focus on commonly used self self-supervised representation learning methods. Linear probing of pre- tre-trained models is a widely adopted approach to evaluate the learned representations on downstream tasks, making it very rel- evant understand how to obtain effective explanations for combined models (backbone + probe).
3.1 SETUP
To holistically evaluate explanations methods, utilize diverse suite of interpretability metrics described below. Note, however, that in contrast to fully supervised models, where output neurons optimized to represent specific classes, this is not the case for self-supervised models.
3

---

Published as a conference paper at ICLR 2025
Step Lincar Probe
Step 2: Fevaluate Explanation Quality
Loss
Model Prediction: Butterlly
Pre-trained
Probe 6()
Pre-trained
Probe 8()
Grid Pointing Game (CrAlPPG)
Inpur Image
Encoder
Explonation
2x2 Input Image
Encoder
J
Figure 3: Setup:
Step 1. Linear or MLP probes are trained on frozen pre-trained models f. Step 2. Explanation methods applied the classification predictions of the trained probes, and evaluated across a wide array of interpretability metrics to assess explanation quality (e.g. localization).
To nonetheless compare the explanations obtained from various pre-trained model backbones, we propose the following experimental setting: (1) we pre-train the models based on various pre-training paradigms and freeze the model parameters: (2) train linear and non-linear classifiers (probes) on those frozen features for downstream image classification; (3) we apply attribution methods the classification predictions the trained probes and evaluate the quality of the generated explanations (see Figure 3).
This allows us compare all pre-trained backbones in a standardized setting and to leverage existing evaluation metrics that were developed in the context of explaining classification models (see below). Further, by freezing the backbone features, we are able to isolate the impact of the pre-training paradigm. Finally, note that with a linear probe, the classification output is the result of simply linear combination of the backbone representations. For most attribution methods (e.g. LRP, B-cos, LIME, IxG, and IntGrad, see Section 4.2), the resulting importance attributions, in turn, are also just a linear superposition of the attributions that would be obtained for individual neurons in the backbones' representations, and therefore direct reflection of the backbones' interpretability itself.
Evaluation metrics. To assess the class-specificity of model explanations, we follow prior work and measure the fraction of positive contributions A, that fall within pre-specified region R of given image vs. the total amount of positive contributions EA, The score s; for each image is thus given by 8; A /EA, We discuss the motivation for our metric selection in Appendix A.
For single-label classification (ImageNet (Russakovsky et al., 2015)), we employ the grid pointing game (GridPG) (Böhle et al., 2021; 2022; Zhang al., 2018; Samek et al., 2017). Here, the trained models are evaluated on a synthetic grid of images of distinct classes and for each of the class logits the region given by its respective position in the synthetic image grid, see also Figure 4.
For multi-label classification (VOC (Everingham et al., 2009), COCO (Lin et al., 2014)), we rely on the bounding box annotations provided in the datasets and use the energy pointing game (EPG) (Wang et al., 2020). L.c. the region R corresponds to the bounding boxes of the class for which the explanations are computed. The ImageNet validation notations, which we use to additionally report the EPG score on this dataset. We report both the GridPG and EPG scores in percentages, with higher scores indicating better localization.
also includes bounding box an-
We further analyze the explanations using the pixel deletion method (Samek et al., 2017; Hedström al., 2024), and also evaluate their compactness (Gini index p.p. as in (Chalasani et al., 201 8)) and complexity (entropy as in (Tseng et al., 2020)), thus ensuring a comprehensive evaluation setting.
3.2 THE IMPACT OF THE PROBES' TRAINING OBJECTIVE
As discussed the previous section, we train linear probes on the frozen, pre-trained representa- tions, apply common explanation methods and metrics.
Following prior work (Caron et al., 2021; Grill et al., 2020; Chen al., 2020c;a;;; He al., 2022), optimize these probes via the cross-entropy (CE) loss LEE, for image which is given by exp (De,))
LCE,
Σ log Σn exp (1/2.)
x te
Here, H..; denotes the probe's 's output logit for class c and input i, and tr, the respective one-hot en- coded label. Interestingly, in our experiments noticed that the explanations of the predicted class
4

---

Published as a conference paper at ICLR 2025
Cross Entropy
Ioput
Probo
Poebe 2
No Cormatid de Atrbascoas
-1.
Explained Class: Bighomn
Loes 4.08
Loss 4.08
GridPG Score 65.7%
GriPG Soore 11.9%
Figure 4: Due to the shift-invariance of softmax, one cannot expect positive and negative attribu- tions to be well calibrated, which can lead to unintuitive model explanations, see also Equation (3). Specifically, one can easily define equivalent linear probes (Probe 1,2) that achieve the same CE loss, but visually dissimilar explanations and GridPG scores (65.7% vs. 11.9%). Col. 2+3 show LRP (Bach et al., 2015) attributions for two equivalent probes explaining the same class (bighorn).
for CE-trained linear probes were highly distributed and failed to localize the class objects effec- tively (see Figure la). We hypothesize that this could be due to the shift-invariance of the CE loss.
Softmax Shift-Invariance Issue. To understand this, note that the CE loss is invariant to adding shift 5 to all output logits, as long as this shift is the same for all classes (Srinivas & Fleuret, 2021); in fact, this shift can even be specific to image (see Figure 4):
exp (je,s + 6:) Ee exp (UN, + 6.) Σn exp (ūk..) exp (6.) Σn exp (ÚR,))
exp (De,)) exp (6.)
exp (Nc,))
Importantly, note that such an image-specific shift can be obtained by shifting the probes' weight vectors Wy for all classes & by a fixed vector w/: =
(w/T 24
exp
exp (w- a, + AwTa. a.) Σn exp exp(wx w a + Aw a.
exp (De, + 0.)) Ee exp (ūk. à 0e)
Ee exp (w' a.)
Dv.i
Here, a, denotes the backbones' frozen input representation. As can be seen from Equations (2) and (3), there are an infinite number of linear probes which achieve the same loss and are thus indistinguishable as far as the optimization concerned.
As most attribution methods, in some form or another, rely on the models' weights to compute the importance attributions, this reliance can have a crucial impact on the resulting explanations, as we show in Figure 4. Specifically, we show the attributions derived via layer-wise relevance propagation (Bach et al., 2015) for two functionally equivalent probes. While the probes give the same predictions and achieve the same loss for every input x by design, they yield vastly different importance attribution maps (cf. Sundararajan et al. (2017)). This, in turn, results in very different GridPG scores (65.7% vs. 11.9%).
Since both probes are equivalent under CE-based optimization, it cannot be expected that for CE- trained probes the attributions are calibrated such that 'positive' attributions will be class-specific. Thus, we also evaluate BCE-based probes, which do not exhibit the shift invariance as CE models.
BCE Probing. The BCE objective has recently been shown to perform well for image classification (Wightman et al., 2021; Böhle et al., 2024); in detail, the BCE loss is given by
LBCE, Σ La log (a (ūe.)) + (1 bo,i) log (1 (ge,a)
with a denoting the sigmoid function. Importantly, in contrast to the CE loss, the BCE loss is not shift-invariant. Specifically, note that for BCE, the linear probe is penalized for adding a con- stant positive shift to non-target classes and thus biased towards focusing on class-specific features. We therefore expect it to result in better calibrated explanations, which is indeed what we observe: specifically, as show in Section 5. BCE probes exhibit significantly higher degree of class- specificity and lend themselves better for localizing class objects. They also lead to more stable pre- dictions under the pixel deletion evaluation, as well more compact and less complex explanations.
5

---

Published as a conference paper at ICLR 2025
3.3 NON-LINEAR PROBING WITH INTERPRETABLE MLPs
In addition to the impact of the probes' loss function on the explanations (Section 3.2), here we discuss the interplay between classifier's complexity and the resulting explanations.
In particular, we note that features computed by self-supervised (Chen et al., 2020c; Grill et al,, 2020: Caron et al., 2021) or ussoonlanguage backbones (Radford al., 2021) are not necessarily optimized to be linearly separable with respect to the classes of any arbitrary downstream task (He et al., 2022). We posit that this lack of linear separability might further diminish the localization abil- ity of explanation methods, as different classes may share certain features in the frozen feature rep- resentations of pre-trained backbones that are not trained with full supervision. To address this, prior work (Li et al., 2023b; Hewitt & Liang, 2019) investigates using non-linear probes, which have been shown to result in improved downstream performance. However, this might come at a cost to model interpretability, as it has been shown (Rao et al., 2022) that explanation methods like GradCAM (Sel- varaju et al., 2017), perform significantly worse at carlier layers than at the last layer of a DNN.
MLP Probes. To mitigate this, we propose use interpretable Multi-Layer Perceptron (MLP) probing technique to improve model accuracy and explanation quality. Specifically, we also train more complex probes on the frozen features, namely two-layer and three-layer conventional and B-cos (Böhle et al., 2022) MLPs and evaluate how this impacts the explanations' class-specificity.
A conventional fully connected MIP f (x;0) 0) with I. layers is given by:
f(x: 0) I 1 o o (x))
(5)
where 1n denotes a linear layer j with parameters W j. and is the set of all parameters within the MLP. For given input a, to layer I, the output is computed as: laa;;; Wi) = o (Wa'), with ø non-linear activation function (e.g. ReLU).
Instead of computing their outputs as a Rel R-acttvvde linear transformation, the B-cos layers I employ the B-cos transformation, which is given by:
B-cos layer (ag Wi) le(a;; W c W.a
Here, @ is row-wise multiplication, i.e. all rows of Wt are scaled by the scalar entries of the vector to its left and e computes the cosine similarity between the input vector ay and each row of W,.
Böhle et al. (2022) showed that the additional cosine factor in Equation (6) induces increased weight-input alignment during optimization, which significantly increase the localization perfor- mance of a linear summary of the B-cos models. Interestingly, we show in Section 5.2, using B-cos MLP probes on conventional backbones can also improve the localization post-hoc explanations.
4 EXPERIMENTAL SCOPE
In the following section, we outline our experimental scope, and discuss the pre-training (Sec- tion 4.1) and explanation methods (Section 4.2) that we evaluate.
4.1 PRE-TRAINING FRAMEWORKS
We aim to have a broad enough representative set that highlights how our evaluation generalizes across differently trained feature extractors, particularly (1) fully-supervised, (2) self-supervised. and (3) contrastive vision-language learning.
(1) Fully-Supervised Learning First, we evaluate the explanation methods on fully supervised backbones. On the one hand, backbones pre- pre-trained in a supervised manner are still often used for transfer learning (Cheng et al., 2022; Xie et al., 2021; Chen et al., 2017). On the other hand, an evaluation of fully supervised models also provides useful reference value, most explanation methods have been developed in this context.
In addition to evaluating the end-to-end trained classifiers, we also evaluate linear probes the frozen representations of these models, in order to increase the comparability to the self-supervised approaches we present in the following.

---

Published as a conference paper at ICLR 2025
(2) Self-Supervised Learning. We consider three popular sulf-supervised pre- pre-training frameworks: MoCov2 (Chen et al., 2020c), BYOL (Grill et al., 2020), and DINO (Caron et al., 2021).
(3) Vision-Language Learning. For this use the modal CLIP (Radford et al., 2021), that is pre-trained on a large-scale dataset comprising of image-text pairs.
To summarize, evaluate across broad spectrum of pre-training mechanisms: contrastive, two relf-distillation-aasse and a multi-modal pre-training paradigm, which cover some of the most popular approaches to sele-sup learning. We contrast our evaluations with fully-supervised trained models (for more details refer to Appendix F.3).
4.2 EXPLANATION METHODS
To evaluate model interpretabillity we apply explanation methods to the classification predictions of the probes trained on the frozen features (cf. Figure and Section 3.1). In the following, we provide short overview on the explanation methods.
Inputx x Gradient (Shrikumar et al., 2017) is a backpropagation-bass attribution method that in- volves taking the element-wise product of the input and the gradient. For given model f(x: 0) and (f(x;))
input x, is denoted by x @ Ex
Integrated Gradients (Sundararajan et al., 2017) follows axiomatic method and is formulated as the integral of gradients over a straight line path from baseline input x' to the given input x. +x)xxx()) ao.
IntGrad for an input x is equal to (x- - x') x fu 8F(x'
8x
Layer-wise Relevance Propagation (Bach et al., 2015) generates attribution maps by propagat- ing relevance scores backwards through the network, thus decomposing the model prediction into contributions from individual input features.
GradCAM (Selvaraju et al., 2017) is activation based explanation method, that generates attri- butions corresponding to the gradient of the class logit with respect to the feature map of the last convolutional layer of DNN.
ScoreCAM (Wang et al., 2020) computes a weight for each activation map based on its impact on the final prediction (instead of relying gradients), leading to more interpretable saliency maps.
LIME (Ribeiro et al., 2016) samples perturbed versions of the input interest and observes the changes in predictions. A linear model is fit to these perturbed instances to provide local explana- tions for a model's 's decision.
B-cos (Böhle al., 2022) attributions generated by the inherently interpretable B-cos networks. Essentially, the attribution map is computed by an element- wise product of the dynamic weights with the input that faithfully encapsulates the contribution of each pixel to given class (W? (x) x).
Rollout (Abnar& Zuidema, 2020) generates attribution maps by aggregating attention scores across layers in a vision transformer (ViT). This is done by recursively multiplying the attention matrices.
CGW1 (Chefer et al., 2020) propagates explanations through self-attention layers, effectively cap- turing token interactions to provide an intuitive understanding of ViT-based models.
CausalX (Xie et al., 2022) identifies important input features by analyzing their causal effects on model predictions through interventions, allowing more robust interpretation of ViT decisions.
In short, we evaluate a wide range of attribution methods, including gradient-busee activation- based, perturbation-buaee post-hoc explanations, well the inherent model explanations of the recently proposed B-cos models, for both convolutional- and vision transformer- based architectures.
For exact implementation details on datasets, models, pre-training please see Appendix F,
RESULTS
We now discuss our experimental findings. Specifically, we analyze the impact of the probes' opti- mization objective (Section 5.1) and the probe complexity (Section 5.2) on the model explanations.
7

---

and for Published as a conference paper at ICLR 2025
Boose WiK) Input
Integrated Gradicuts
Ortdti Soom (ho1)
Accesnoy
Acturaey
Layerwisc Relevance Progapation
Intagrattdd Cradients
LIME
ceteino Sram choll
Acturacy
ACCUIOC3
Accuracy
Figure 5: BCE CE Accuracy and GridPG scores ImageNet for ResNet-50 models. GridPG scores improve significantly with BCE loss over CE loss, and this is consistent across pre-training paradigms for both B-cos models (top row) and conventional models (bottom row).
Table BCE vs. CE Accurryy and GridPG scores on ImageNet for ViT models. consistent improvement in localization score for BCE over CE probes is seen.
Acc. %
Localization %
Backbone
Pre-training
CE
BCE
A
VIT-B/16
Sup.
73.2
72.8
+0.7
+15.7
+0.4
VIT-B/16
DINO
77.2
45.3
+11.7
+4.8
ViT-B/16
MoCov3
76.3
75.1
43.8
44.4
43.3
IMPACT OF BCE VS. CE
Here, we evaluate how the optimization objective of linear probes impacts their accuracy and expla- nations, using metrics in Section 3.1. Results are for ResNet-50 (He et al., 2),6), unless specified.
Localization. Figure 5, we plot the linear probe accuracy versus the GridPG scores for MoCov2 (red), BYOL (green), DINO (blue), CLIP (yellow) and supervised (black) models for B-cos/LRP (col. 1), IntGrad (col. 2) and LIME (col. 3) on ImageNet. We do so for linear probes trained via BCE (filled markers) and CE (hollow markers) for conventional (row 2) and B-cos (row 1) models.
We find significant gains in the explanations' localization for all approaches, models, and expla- nations when using the BCE loss instead of the commonly used CE loss. E.g., for conventional models explained via LRP (Figure 5, col. 1, row 2), the GridPG score for CLIP improves by 32p.p- (19% -151% %). MoCov2 improves 30p.p. (50% ~80% )), for BYOL by 18p.p. (48% -66%) ) and for DINO even by 40p.p. (52 % +).%).
Similarly, the GridPG score for B-cos explanations also significantly increases (Figure 5, col. row 1): for MoCov2, it improves by 33p.p. (48% +81 %), for BYOL by 28p.p.
DINO by 37p-p. (43%~88%)
Interestingly. IxG and GradCAM (see Appendix E) only show consistent improvements for B-cos models. For I,G, this is line with prior work, model gradients for conventional models are known to suffer from 'shattered gradients' (cf. (Balduzzi et al., 2017)).
Notably, the observed improvements localization lead significant qualitative improvements regarding the class-specificity of the explanations, see Figure 6 (left). The B-cos attributions in Figure (left) for probes trained with the BCE loss (top row) are much more localized compared to the probes trained with CE loss (bottom row). We see this behavior is consistent across all pre- training approaches; for results on additional explanation methods, please see the Appendix D. To show B-cos explanations
also compare with supervised models qualitatively, in Figure 6 (right) for DINO (cols. 2+3) and supervised models (cols. 4+5). We find both explanations to visually look similar, and to follow similar trend when trained using either CE (cols. 2+4) or BCE (cols. 3+5). For BCE, the model attributions are better localized for both types of training.
In Figure 2a similar improvements seen for the EPG metric (on ImageNet) when using BCE probes for B-cos and LRP attributions. We specifically see greater improvement on smaller bounding boxes thus highlighting the ability for improved localization compared to CE probes.
8

---

Published as a conference paper at ICLR 2025
MrCond
30B
Socciin
Clase: Popuicle GGMPG
Sombor
3
BCB
CE BCE
Figure 6: (a) BCE CE. The B-cos attributions for linear probe trained with CE loss (bottom
row) leak into nearby cells in the 2x2 GridPG evaluation setting. The attributions for linear probes trained with BCE loss (top row) are consistently much more localized.
(b) SSL Super-
vised. The B-cos attributions for DINO (cols. 2+3) are visually very similar to supervised models (cols. 4+5), despite being optimized very differently, thus highlighting the importance of the training objective of the linear probe (see Appendix D.I for more results).
Backerboll
Bag
Umiorells
Dogonily
Plunger
Iapes
Lincar Probe
3-laye MLP
Figure 7: Qualitative results of MLP probes Iet. We find that explanations for B-cos MLPs trained on the DINO features exhibit better localization than a linear probe. For other pre- training and explanation methods, see Appendix D.2.
Vision Transformer specific explanation methods demonstrate similar improvements (see Table 1).
Pixel Deletion. Under the pixel deletion setup, in Figure 2b observe the BCE probes lead to more stable predictions when least important pixels are successively removed. This consistent across majority of the pre-training methods as well as attribution methods.
Complexity and Compactness, Table El demonstrates a consistent improvement in compactness (Gini index p.p.) and reduction in complexity (entropy) for BCE CE, except for IxG and Grad- CAM in the case of conventional backbones.
We thus note, that in contrast to the loss function, the choice pre-training method has limited impact only on explanation quality, with no particular method consistently outperforming others.
5.2 EFFECT OF MORE COMPLEX PROBES
In this section, we present the experimental results on the effect of training more complex classifiers on top of the frozen features (ResNet-50 unless specified). In particular, we train 2- and 3-layer MLPs and evaluate the impact on accuracy and the quality of explanations.
In Figure 8, we show the results on ImageNet and COCO for B-cos MLPs. It can be seen that for all pre-trained models there not only a steady increase accuracy but also an improvement localization ability (GridPG and EPG scores) of model explanations.
E.g., for standard models explained via IntGrad we observe the following improvements when going from single linear probe to 3-layer MLP in (accuracy, GridPG): MoCov2 (66.9, 69.0) (71.3, 83.0), BYOL (69.3, 83.0) - (72.1, 85.0), and DINO (70.6, 70.0) - (73.1, 89.0). A similar trend is observed for B-cos models, and across explanation methods and datasets (see Figure 8 for results COCO), with GradCAM being an exception (for details and discussion on this, see the Appendix E).
In Figure 9, the EPG scores (for ImageNet with bounding boxes) improve across all pre-training paradigms for B-cos MLP probes, which is especially prominent smaller bounding boxes. Fur- thermore, Figure 7 depicts the observed localization improvements qualitatively. We find ML.P

---

Published as a conference paper at ICLR 2025
W()) Ingut
leput Insegrated
GriPG Score (x.3 3)
ZIVON
4
B-RN50, W(s) Enput
B-RN36, laput Grudient
RN50, Insegrated Gradicnoo
0431
ENG Scone
14
VoLia
Figure 8: Effect of more complex probes on Accuracy vs. GridPG on ImageNet (top) and F1 Score vs. EGP COCO (bottom). A more complex B-cos probe (MLP) not only increases performance on the downstream task (x-axis), but interestingly also the GridPG score (y-axis, top) and the EPG score (y-axis, bottom) of the explanations (individual plots). This is true for B-cos models (left cols.) and for conventional models (right cols.) probed via B-cos MIPs.
probes to better localize the object class of interest, relying less on background. This indicates that they better capture class-specific features, which, in turn, improves their classification performance.
Interestingly, this trend in improvement in both accuracy and localization is only for B-cos MIPs, independently of them being applied to conventional or B-cos models. While an increase in localization measured by the EPG score is observed for conventional MLPs COCO, on ImageNet we see a consistent decrease in the GridPG score (see Appendix E.2). In order get well-localizing attribution maps downstream tasks, thus recommend probe pre-trained models via B-cos probes.
consistently
In short, find that training relatively lightweight B-cos MLPs (~ 10% of entire model parame- ters) on frozen features of SSL-trained models with BCE loss is versatile approach to obtain both highly interpretable and also highly performant classifiers on downstream tasks.
Note: Detailed results for vision transformers and other explanation methods are in Appendix C.E.
McCov2 Bcos: Input
BYOL: W(x) x Input
DINO: Bcos: W(X) x Input
1.0
MLP-3
+1%
MLP-3
+44
MLP-3
0.8
MLP2
+2%
MLP2
+5%
MLP2
+5%
BPG Score
0.6
+2%
+6%
+7%
0.4
+7%
+8%
0.2
<25%
<500%
<75%
<100%
<25%
<50%%
<73%
<100%
-25%
<550%
<755%
<100%
BBOX
BBOX %
BBOX %
Figure 9: Stronger probes lead to more localized attributions. EPG scores on ImageNet.
6 CONCLUSION
We discover an important and surprising finding, that the quality of explanations derived from a wide range of attribution methods for pre-trained models is more dependent on how the classification layer is trained for given downstream task, than on the choice of pre-training paradigm itself. This places important practical considerations on end-users when using 'post-hoc' attribution methods that are typically assumed to be applied independent of model training. Further, we show that employing lightweight multi-layer B-cos probes contributes to enhanced localization performance of the explanations, providing simple and effective improvement. We support our findings through self-supervised, extensive experiments across several pre- pre-training frameworks stully-supervised
langualla pre-training). and analysis the quality of explanations for popular attribution methods on a diverse evaluation setting. Our findings are robust to the pre- pre- vee-training paradigms, therefore they have broad implications for DNN-based applications using XAI methods.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. We sincerely thank Sukrut Rao and Yue Fan for their valuable feedback on the paper and insightful discussions throughout the project. Additionally, we appreciate Sukrut's 's help with some LTTX sorcery. This work was partially supported by ELSA Mobility Program as part of the ELLIS exchange program to the Institute of Science and Technology Austria (ISTA), where a portion of this research conducted.
Reproducibility Statement. We provide the complete code for pre-training, probing and evaluation of the trained models as well for generating the quantitative and qualitative results of the expla- nation methods used. The code is well-documented with helper scripts to run the different parts of the pipeline and help with reproducibility. Additionally, we also make the entire pipeline avail- able to the broader community by sourcing our software and provide the pre-trained model checkpoints which further helps in reproducing the results in the manuscript. We were also careful to only use open-source software, and publicly available datasets, which thus supports reproducible research. This an effort to provide transparency, and encourage further research in the field. Code to reproduce all experiments: https: //g1 thub. com/ s dgairol8 8 httwwwotoott to-probe
Broader Impact and Ethics Statement. The findings and discussion our work opens up new conversation about designing attribution / XAI methods that do take the training details of underly- ing models into account when getting explanations for their decisions. The fact that this also impacts inherently interpretable models, which are designed to intrinsically learn interpretable features dur- ing training, speaks more to the importance of the finding (i.e. we need to be very careful about how to handle such models and methods). The user, really needs to consider such details when leveraging model explanations.
Typically past research has focused on evaluating the interpretability of attribution methods given a fixed model, post-hoc. Yet, in the real world, we often have use-cases where one has a large pre-trained backbone over which probes are trained depending on the downstream task. In this work, we find that, surprisingly, how these probes are trained can have significant impact on how interpretable the attributions of the model is using fixed post hoc attribution method. Given that training such probes is usually much cheaper as compared to the backbone, our findings can be used to guide the training to yield models that are more interpretable post-hoc. Notably, we find that with this simple approach, the improvements hold across model architectures, pre-training paradigms, and even across different attribution methods.
To conclude, we see apparent ethical concerns raised by the scientific discovery presented in our work. However, we do acknowledge the privilige and availability of resources that enable deep learning research, c.g. running large-scale training on GPUs that do result in an increased carbon footprint and efforts must be made to be more careful when using such resources.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers. In ACL, 2020.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In NeurIPS, pp. 9525-9536, 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Rlaur-Robert Müller, and Wojeiech Samek. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer- Wise Relevance Propagation. PLoS ONE, 10, 2015.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The Shattered Gradients Problem: If ResNets are the Answer, then What is the Question? In ICML, pp. 342-350, 2017.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-Ivvaranee-CCvaaaa Regular- ization For Self-Supervised Learning. In ICLR, 2022.
Wieland Brendel and Matthias Bethge. Approximating CNNs with Bag-of- lagg-of-oocl-Feature models works surprisingly well on ImageNet. In ICLR. 2019.
hhttps///lliss /ttp::///llls.ee/phhh ppddppattpdeeee

---

Published as a conference paper at ICLR 2025
Moritz Böhle, Mario Fritz, and Bernt Schiele. Convolutional Dynamic Alignment Networks for Interpretable Classifications. In CVPR, pp. 10029-10038, 2021.
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos Networks: Alignment is All We Need for Interpretability. In CVPR, pp. 10319-10328, 2022.
Moritz Böhle, Navdeeppal Singh, Mario Fritz, and Bernt Schiele. B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers. IEEE TPAMI, 46(6):4504 4518, 2024.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, pp. 9630-9640, 2021.
Prasad R. Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Somesh Jha, and Wu. Concise Explanations of Neural Networks using Adversarial Training. In ICML, 2018.
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. In WACV, pp. 839-847, 2018.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer Interpretability Beyond Attention Visualization. In CVPR, pp. 782-791, 2020.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This Looks like That: Deep Learning for Interpretable Image Recognition. In NeurIPS, 2019.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE TPAMI, 40(4):834 848, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Suuee Models are Strong Supi-Supe Learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In CVPR, pp. 15745-15753, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv preprint arXiv:2003.04297 2020c.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention Mask Transformer for Universal Image Segmentation. In CVPR, 2022.
Piotr Dabkowski and Yarin Gal. Real Time Image Saliency for Black Box Classifiers. In NeurIPS, pp. 6979, 2017.
Jessica Dai, Sohini Upadhyay, Ulrich Aivodji, Stephen H. Bach, and Himabindu Lakkaraju. Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations. In AIES, pp. 203-214, 2022.
Saurabh Desai and Harish G. Ramaswamy, Ablation-CAM: Visual Explanations for Deep Convo- lutional Network via Gradient- freee Localization. In WACV, pp. 972-980, 2020.
Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be Manipulated and Geometry is to Blame. In NewrIPS, 2019.
12

---

Published as a conference paper at ICLR 2025
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJC,, 88:303-308, 2009.
Ruth C. Fong and Andrea Vedaldi. Interpretable Explanations of Black Boxes by Meaningful Per- turbation. In ICCV, pp.
2017.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of Neural Networks Is Fragile. In AAAI, volume 33, pp. 3681-3688, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangging Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training Ima- geNet in Hour. arXiv preprint arXiv: 1706. 02677, 2017.
Baan-Battien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap Your Own Latent New Approach Self-Supervised Learning. In NewrIPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, pp. 9726-9735, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders are Scalable Vision Learners. CVPR, pp. 16000-16009, 2022.
Anna Hedström, Leander Weber, Dilyara Bareeva, Daniel Krakowezyk, Franz Motzkus, Wojeiech Samek, Sebastian Lapuschkin, and Marina M.-C. Höhne. Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond. J. Mach. Learn. Res., 24(1), 2024.
John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In EMNLP, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A Benchmark for Inter- pretability Methods in Deep Neural Networks. In NewrIPS, 2019.
Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. LayerCAM: Exploring Hierarchical Class Activation Maps for Localization. IEEE TIP, 30:5875-5888, 2021.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Un- terthiner, and Xiaohua Zhai. An Image is Worth 16x16 Words: Transformers for Image Recogni-
tion Scale. In ICLR, 2021.
Wei Li, Jiahao Xie, and Chen Change Loy. Correlational Image Modeling for Self-Supervised Visual Pre-Training. In CVPR, 2023a.
Yazhe Li, Jorg Bornschein, and Mareus Hutter. Evaluating Representations with Readout Model Switching. In ICLR, 2023b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, pp. 740-755, 2014.
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and -laus-Robert Müller. Layer- Wise Relevance Propagation: An Overview. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 Lecture Notes in Computer Science, pp. 193-209 Springer, 2019.
13

---

Published as a conference paper at ICLR 2025
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AL. ACM Comput. Surv. 55(13s), 2023.
Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models. In BMVC, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICMI,, pp. 8748-8763, 2021.
Sukrut Rao, Moritz Böhle, and Bernt Schiele. Towards Better Understanding Attribution Methods. In CVPR, pp. 10213-10222, 2022.
Sukrut Rao, Moritz Böhle, Amin Parchami-Araghi, and Bernt Schiele. Studying How to Efficiently and Effectively Guide Models with Explanations. In ICCV, pp. 1922-1933, 2023.
Sukrut Rao, Moritz Böhle, and Bernt Schiele. Better understanding differences attribution meth- ods via systematic evaluations. IEEE TPAMI, 46(6):4090-4100 2024.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier. In KDD, pp. 1135-1144, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJV,, 115(3):211-22 1-222, 2015.
Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the Visualization of What Deep Neural Network Has Learned. IEEE Trans. Neural Netw. Learn. Syst., 18:26600-2673 11 2017.
Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher Anders, and Klaus- Robert Müller. Explaining Deep Neural Networks and Beyond: A Review of Methods and Ap- plications. Proceedings of the IEEE, 109(3):247-278, 2021.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Bare Based Localization. In ICCV, pp. 618-626, 2017.
Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. CASTing your model: Learning to Localize Improves Self-supervised Representations. In CVPR, 2021.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning Important Features through Propagating Activation Differences. ICML, pp. 3145-3153, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In ICLRW, 2014.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In AIES, pp. 180-186, 2020.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for Simplicity: The All Convolutional Net. In ICLRW, 2015.
Suraj Srinivas and Francois Fleuret. Rethinking the Role of Gradient-based Attribution Methods for Model Interpretability. In ICLR, 2021.
Suraj Srinivas and François Fleuret. Knowledge Transfer with Jacobian Matching, In ICML, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In ICMI,, pp. 3319-3328, 2017.
14

---

Published as a conference paper at ICLR 2025
Antti Tarvainen and Harri Valpola. Mean Teachers are Better Role Models: Weight-averaged Con- sistency Targets Improve Semi-supervised Deep Learning Results. In NeurIPS, pp. 1195-1204, 2017.
Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. Learning What Makes Difference from Counterfactual Examples and Gradient Supervision. In ECCV, pp. 580-599, 2020.
Alex M. Tseng, Avanti Shrikumar, and Anshul Kundaje. muuriee-transform-aaaae Attribution Priors Improve the Interpretability and Stability of Deep Learning Models for Genomics. In NeurIPS, 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness May Be at Odds with Accuracy. In ICLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. In NeurlPS, 2017.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang. Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Score-Weights Visual Explanations for Convolutional Neural Networks. In CVPRW, pp. 111-119, 2020.
Ross Wightman, Hugo Touvron, and Herve Jegou. ResNet Strikes Back: An Improved Training Procedure in timm. In NeurIPS 2021 Workshop ImageNet: Past, Present, and Future, 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- Former: Simple and Efficient Design for Semantic Segmentation with Transformers. In NeurIPS, 2021.
Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong. and Chen Change Loy, Masked Frequency Modeling for Self-Supervised Visual Pre-Training. In ICLR, 2023.
Weiyan Xie, Xiao hui Li, Caleb Chen Cao, and Nevin L.Zhang. ViT-CX: Causal Explanation of Vision Transformers. In IJCAI, 2022.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down Neural Attention by Excitation Backprop. IJCV, 12((10):084-1102, 2018
Ke Zhu and Jianxin Wu. Residual Attention: A Simple but Effective Method for Multi-Label Recog- nition. In ICCV, pp. 184-193, 2021.
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
In this appendix to our work on simple yet effective techniques for improving post-hoe explanations,
provide:
(A) Discussion on Evaluation Metries and Co-12 Properties In this section, we provide a discussion on the importance of the interpretability metrics have selected in evaluations, and also place them in context with the recently proposed Co-12 Properties (Nauta et al., 2023).
17
(B) Linear Probing On Frozen Features
18
In this section, we provide short mathematical proof to show that when linear probing on frozen pre-trained backbone features, it simply results in learning weighted linear combination of the backbone features.
(C) Additional Architectures: Vision Transformers
19
In this section, we provide quantitative results Vision Transformers (ViTs, (Kolesnikov et al., 2021)) when evaluated on their related explanation methods.
(D) Additional Qualitative Results
22
In this section, we provide additional qualitative results for different evaluation settings, for both B-cos models well conventional models. Specifically, qualitatively show examples highlighting the impact of using BCE vs. CE loss and the effect of using more complex probes, Further, we provide comparison between different explanation methods for all pre-training (supervised and SSL) frameworks evaluated in our work.
(E) Additional Quantitative Results
33 this section, we provide additional quantitative results: (1) GridPG results of additional explanation methods ImageNet, and (2) EPG results for additional methods on COCO and VOC.
(F) Implementation Details In this section, we provide additional details regarding datasets, training, imple- mentation and evaluation procedures.
52
16

---

Published as a conference paper at ICLR 2025
A DISCUSSION ON THE EVALUATION METRICS AND C0-12 PROPERTIES
In this section, we provide discussion the importance of the interpretability metries have lected in our evaluations, and also place them in context with the recently proposed Co- Properties by Nauta et al. (2023).
Nauta al. (2023) posit that explainability properties that describe the different aspects of explanation quality. Specifically, they introduce these properties as the Co-12 properties that are crucial to be evaluated for a comprehensive assessment of explanation methods. These 12 properties are namely the following:
concept and propose various
Correctnss: denotes how faithful the explanation is with respect to the underlying 'black- box' model.
2. Completeness: measures the extent to which the model is described by the explanation.
3. Consistency: evaluates the degree of determinisim and invariance of the explanation method,
4. Continuity: measures continuity and generalizability of the explanation function.
5. Contrastivity: measures the discriminativity of the explanation with respect to different targets.
6. Covariate Complexity: assesses the complexity (human interpretable) of features in the explanation.
7. Compactess: reports the overall size of the explanation.
8. Composition: describes the presentation format and organization of the explanation.
9. Confidence: measure of confidence of the explanation or model output.
10. Context: measures how relevant is the generated explanation to users.
11. Coherence: evaluates the plausibility of the explanation.
12. Controllability: measures the control or influence users have on the explanation.
In our work we present an important finding for explainable artificial intelligence (XAI), and conduct a systematic study across pre-training schemes, model architectures and heatmap based attribution methods evaluate to what extent the training influences the explanations derived from these attri- bution methods. To quantitatively evaluate the quality of the explanations we assess their ability to localize class-specific features using the grid pointing game (GPG) Böhle (2021; 2022); Zhang et al. (201 8); Samek et al. (2017) and the energy pointing game (EPG) Wang et al. (2020).
The grid pointing game (GPG) is established metric that reflects various of the Co- 12 properties (cf. Table in (Nauta et al., 2023)). First, it constitutes a "controlled synthetic data check" which allows to (approximately) deduce ground truth explanations, thus reflecting the correctness of the explanations. Further, as multiple targets are present in the grid images, GPG reflects the target sensitivity of the explanations and thus their contrastivity. By highlighting relevant regions for the decision, explanations that score highly on GPG can be useful to end users (context).
The energy pointing game (EPG) can be seen as a general case of the pointing game (Hooker et al., 2019), and thus subsumes all the above properties of the GPG.
We also analyze the explanations under the pixel removal method, that has been shown to be a reliable measure of faithfulness (correctness) of an explanation (Samek al., 2017; Hedström al., 2024).
Finally, to have comprehensive evaluation we also report explanation evaluations using the Gini index (as in (Chalasani et al., 2018)) to measure the compactness and the entropy (as in (Tseng et al., 2020)) to measure the complexity.
Importantly, note that other Co- 12 properties might not relate to heatmap- based explanations (com- position, confidence, controllability) or remain unchanged by design as they are intrinsic to the explanation method itself, such consistency, completeness, coherence and continuity.
17

---

Published as a conference paper at ICLR 2025
B LINEAR PROBING ON FROZEN BACKBONE FEATURES
Interpretability of frozen backbone: Linear probing on the frozen pre-trained backbone features is important step when using mre-trained models for downstream tasks and can also inform us about the interpretability of the backbone itself as we explain further. Since the linear probes themselves have limited modeling capacity, these explanations must therefore necessarily reflect the 'knowl- edge' of the backbone model. The different probes (CE BCE), compute their outputs by nothing but a weighted mean of the frozen backbone features.
can be demonstrated mathematically by the following: let z RR represent the output feature vector from the frozen backbone after the global average pooling operation, where the vector has D dimensions. Let W E RCxD be the weight matrix of the fully connected classification layer, where C' is the number of output classes (c.g., 1000 for ImageNet), and let b RC be the bias term of the classification layer. The output prediction vector RC is then given by:
Wz
Here, Wz represents a linear combination of the backbone features 2, with the weight matrix W. The bias term b is added each class output.
Since the feature extractor is frozen (i.e., its weights are not updated during training), the classifica- tion layer performs a simple linear combination of the extracted features, and only the weights W and bias b are learned during training. Thus, essentially when analyzing the explanations generated by the model, it is largely dominated by the backbone features. This is what makes our finding surprising, where BCE trained probes lend themselves to generate better quality explanations.
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL ARCHITECTURES: VISION TRANSFORMERS
In this section, provide quantitative results Vision Transformers (VTTs, Kolesnikov et al. (2021)) and the explanation (attribution) methods developed specifically for ViTs: (1) CGW1 (Chefer al., 2020), (2) ViT-CX (CausalX (Xie et al., 2022)), (3) Rollout (Abnar & Zuidema, 2020), and (4) B-cos (Böhle et al., 2024).
In particular, we first evaluate the impact of the training objective on probing in terms of accuracy and explanation quality using the metrics discussed in Section 3.1 on ImageNet. Next we analyze the effect of probe complexity the explanation localization.
GridPG Scores for Vision Transformers with BCE vs CE Probing
BGE
CE
1.0
+23%
Boa:: Wi22 Ispat
+339%
+20%
+1%
CasselX
+.5%
+4%
00W1
Retbuu
+16%
+12%
+4%
Court
0.4-
0%
+5%
+3%
0.2
SUPP
DINO VT: 5/16
DINO VIT: B/16
SUP BV1
DINO VI W7-8/16
MOCOV3 VIT
SUP B/16
DENO VIT-B/16
MOGUVA VI7-8/16
SUP WT B/16
DINO VIT-B/16
MOCOWA
B/16
WT-B/16
Figure C1: VIT backbones BCE vs. CE Probing GridPG scores on ImageNet. GridPG for (1)
B-cos (Böhle et al., 2024), (2) CausalX (Xie et al., 2022), (3) CGW1 (Chefer et al., 2020), and (4)
Rollout (Abnar & Zuidema, 2020), BCE trained probes consistently outperform CE trained probes
across all pre-training paradigms and explanation method.
C.I IMPACT OF BCE VS. CE
In the following, we evaluate the impact of the training objective of the linear probes on ViTs similar the evaluations for CNNs described in Section 5.1 both supervised (Kolesnikov al., 2021)
and self-supervised (He et al. (2019), Caron et al. ()) pre-trained backbones.
GridPG Localization. In Figure CI, we plot the GridPG scores for ViT backbones when probed with the BCE (orange) vs. CE (blue) training objective for both conventional and B-cos models. We find significant gains in the explanations' localization for all approaches, models, and explanations when using the BCE loss instead the CE loss. E.g., for conventional backbones when explained via CGWI I (Figure C1, col. 3), the GridPG score for DINO improves by 12p.p (31 % +43%) MoCov3 improves by 4p.p. (48%-o -52%), ). and for supervised by 16p.p. (34 % 356-550% )).. For B-cos explanations, the increase in GridPG score is drastic (Figure C1, col. 1): for DINO, it improves by upto 33p.p (47% % -80% -08%) and for supervised by 23p-p (61 %)).%)
In Table C1, we also report the accuracies and when probing with different training objectives and observe that BCE probes achieve similar performance CE probes while achieving significantly greater localization scores.
Table Cl: VIT backbones BCE vs. CE Probing-Accurant and GridPG scores on ImageNet. A consistent improvement in localization score for BCE CE probes seen for both conventional ViTs (Kolesnikov et al., 2021) and inherently interpretable B-cos B-ViTs (Böhle et al., 2024) for supervised and self-supervised pre-training of the backbones with comparable accuracies.
Acc. %
Localization %
Backbone
Pre-training
CE
BCE
Beos hee-ce
CausalX Ice-cc
CGW1 hee-ce
Rollout hce-ce
ViT-B/16
Sup.
73.2
72.8
+0.7
+15.7
+0.4
ViT-B/16
DINO
77.2
77.6
+5.3
+11.7
+4.8
VIT-B/16
MoCov3
76.3
75.1
+3.8
+4.4
+3.3
B-VI
Sup.
77.3
78.1
+23.5
--V
DINO
73.2
73.4
+32.7
B- B-VIT.-B/I
DINO
77.1
77.3
+20.4
19

---

Published as a conference paper at ICLR 2025
Table C2: VIT backbones BCE vs. CE Probing-Accuracy and EPG scores on ImageNet. We see improvement in localization score for BCE over CE probes for majority of cases across Super- vised and self- Supervissd pre-trate pre-training of the backbones and explanation methods with comparable accuracies. Note: We see greater improvement in localization scores for smaller bounding boxes (with size - 50% of image area).
Localization %
BBox size <55%
BBox size
BBox size
BBox size < <100%
Backbone
XAI Method Pre-training Loss
Acc.
<50%
<
VIT-B/16
CGWI
Sup
CE
73.2
26.4
42.8
53.7
71.0
ViT-B/16
CGWI
Sup
BCE
72.8
28.2
(5.2(+1.5) (4.5(+1.7)
72.1+(1.1)
VIT-B/16
CGWI
DINO
CE
77.2
40.2
51.4
61.5
77.5
VIT-B/16
CGW1
DINO
BCE
77.6
41.7 (41.6)
52.5
(+0.))
77.7
VIT-B/16
CGW1
MoCov3
CE
76.3
34.2
50.5
59.5
76.1
VIT-B/16
CGWI
MoCov3
BCE
75.1
(425))
(2.2
60.1 (+0.6)
(+.5)+
VYT-B/16
Rollout
Sup
CE
73.2
28.6
42.8
53.1
70.5
VIT-B/16
Rollout
Sup
BCE
72.8
30.1
30.1
(3.9(41.1)
53.5 +0.4)
71.1(40.6)
VIT-B/16
Rollout
DINO
CE
77.2
41.1
53.3
61.1
76.8
VIT-B/16
Rollout
DINO
BCE
77.6
(2.0.)
52.6 (-0.7)
60.4, 0.7)
76.5(+0.3)
VIT-B/16
Rollout
MoCov3
CE
76.3
36.5
50.6
57.9
75.1
VIT-B/16
Rollout
MoCov3
BCE
75.1
36.
(1.2(+0.6)
(+0.5)
(5.5
VIT-B/16
CausalX
Sup
CE
73.2
14.6
28.6
42.2
61.2
VIT-B/16
CausalX
Sup
BCE
72.8
14.6(0.0)
29.1
(1.6(-0.6)
61.4(+(0.2)
VIT-B/16
CausalX
DINO
CE
77.2
14.5
28.4
42.3
62.2
VIT-B/16
CausalX
DINO
BCE
77.6
(5.2
43.1 (9.3(+0.9)
63.1(40.9)
ViT-B/16
CausalX
MoCov3
CE
76.3
14.2
28.9
43.3
63.2
VIT-B/16
CausalX
MoCov3
BCE
75.1
(4.4()
(0.1(+1.1)
(+0.9)
(3.3
Table C3: VIT backbones MLP Probing-Accurace and EPG scores on ImageNet. A consistent improvement in accuracy Aace and localization score Atoc is seen for B-cos explanations when using more complex 2 or 3 layer B-cos MLPs over a linear probe for DINO pre-trained small and base ViT.s. This demonstrates that more powerful classifier heads are able to distill 'class-specific' information better.
Backbone
Classifier
Acc. %
A acc Loc. %
Aloc
--V16
Linear Probe
73.4
79.9
B-ViT SS/6
Bcos-MLP-22
74.2
+0.8
80.4
+0.5
B-VIT -S11
Bcos-MLP-3
74.7
+1.3
82.4
+2.5
B- -BI16
Linear Probe
77.3
80.3
B- B-ViT.-B/16
Bcos-MLP-2
78.2
+0.9
82.1
+1.8
B- -BV16 Bcos-MLP-3
79.7
+2.4
83.4
+3.1
EPG Localization. In Table C2 and Figure C2, similar improvements are seen for the EPG metric (on ImageNet) when using using BCE probes for CGWI (Chefer al., 2020), Rollout (Abnar & Zuidema, 2020) and CausalX (Xie et al., 2022) attributions. We specifically see greater improve- ment on smaller bounding boxes thus highlighting the ability for improved localization as compared to CE probes.
C.2 IMPACT OF COMPLEX PROBES
In Table C3, we show the results for probing DINO B-ViT (small) and B-ViT.-B (base) with B-cos MIPs. It is seen that for these models there is an increase in both accuracy Aace as well as the localization ability Aloc (GridPG scores) of the B-cos explanations. This does seem to suggest that using more complex probes helps distill 'class-specific' information from the frozen backbone features thus leading to improved localization scores even for transformer based architectures.
These results are consistent with the improvements seen for convolutional backbones (see Section 5) and demonstrate the robustness of our findings across different model architectures (vision trans- formers and convolutional networks).
20

---

Published as a conference paper at ICLR 2025
EPG Scores for Vision Transformers with BCE CE Probing
BCE
CE
CGW1 Explanations
SUP WT B/16
DINO vIE
MOCOVS MY 8/16
1.0
0%
0%
0.8
+1%
+1%
+1%
+2%
+1%
+2%
Store
0.6
+2%
+2%
+2%
0.4
+2%
0.0
<50%
<75%
<100%
-25%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
Rollout Explanations
1.0
+1%
0%
0%
Score
0.6
+1%
0%
+1%
11
-1%
+1%
+1%
0%
0.4
+2%
0.2
<25%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
BBOX
BBOX %
BBOX
CausslX Explanations
SUI VIF-11
DIO WT-8/16
MOCINS WT-811
1.0
0.8
0%
+1%
0%
0.6
-1%
+1%
+1%
+1%
+1%
+1%
0%
+1%
+1%
0.2
0.0
<100%
<2.5%
<50%
<75%
<100%
Figure C2: VIT backbones BCE vs. CE Probing-EPG scores on ImageNet. EPG for (1) CGW1 (Chefer et al., 2020), (2) Rollout (Abnar & Zuidema, 2020), and (3) CausalX (Xie et al., 2022). BCE trained probes outperform CE trained probes for majority cases; with larger and more consistent gains for smaller bounding boxes (occupying 0 50% of the image area).
21

---

Published as a conference paper at ICLR 2025
D ADDITIONAL QUALITATIVE RESULTS
In this section we show additional qualitative results for the explanations of both conventional as well as B-cos models under the different evaluation settings studied in the main paper. We first qual- itatively show the impact of training the probes with binary cross entropy (BCE) vs. cross entropy (CE) loss in Sec. D.I. Then show the effect of more complex probes on localization in Sec. D.2. Additionally, in Sec. D.3 we also provide qualitative comparisons between different explanation methods for all SSL models well fully supervised models. Finally, in Sec. D.4 add more qualitative results showing diverse set of samples for all self-supervised pre-trainings.
D.1 IMPACT OF BCE VS. CE
The probing strategy can have significant influence on the localization ability of model expla- nations. As described Sec. 5.2 of the main paper, probes trained with BCE loss localize more strongly compared to probes trained with CE loss. Figures D1, D2, D3 (cf. Figure 6 in main paper), show the B-cos (Böhle et al., 2022) explanations in the 2 x GridPG evaluation setting for DINO, BYOL and MoCov2 respectively when probed with BCE CE loss. It can be observed that for all SSL-frameworks as well as supervised training, the explanations for linear probes trained with BCE loss (cols. 3+5) are much more localized as compared probing with CE loss. Inter- estingly, however, while the focus region seems to be very similar between the SSL-trained and the fully supervised models, the B-cos explanations of B BYO and MoCov2 show lower saturation.
DINO
Supervised
Input
CE
BCE
CE
BCE
Stocking
Ladybird
Sombrero
Figure D1: DINO vs. Supervised Explanations. The B-cos explanations for DINO (cols. 2+3) are visually very similar to supervised models (cols. 4+5), despite being optimized very differently. We also see that for B-cos models the improvements in attribution localization for probes trained with BCE loss are consistent for both DINO and supervised models.
Results for ResNet-50 backbones, unless specified otherwise.
22

---

Published as a conference paper at ICLR 2025
BYOL
Supervised
Input
CE
BCE
CÉ
BCE
Stocking
Ladybird
Sombrero
Figure D2: BYOL Supervised Explanations. Similar to Figure DI the B-cos explanations for models trained via BYOL (cols. 2+3) exhibit significant improvements in localization when trained BCE. Interestingly. compared the explanations of supervised models (cols. 4+5), we find BOL explanations to exhibit less saturation.
MoCov2
Supervised
Input
CE
BCE
CE
BCE
Stocking
Ladybird
Sombrero
Figure D3: MoCoy2 vs. Supervised Explanations. Similar to Figure DI, D2 the B-cos expla- nations for models trained via MoCov2 (cols. 2+3) exhibit significant improvements in localization when trained via BCE, Interestingly, compared to the explanations of supervised models (cols. 4+5), we find MoCov2 explanations exhibit less saturation.
23

---

Published as a conference paper at ICLR 2025
In Figure D4, additionally visualize the LRP (Bach al., 2015) explanations for conventional models. Similar to the B-cos models in the preceding figures, we find the explanations for BCE- trained models to exhibit significantly higher localization. Interestingly, despite the consistent im- provements localization, we again observe clear qualitative differences between the differently trained backhones: e.g., the explanations for the DINO model the explanations for the models trained via BYOL and MoCov2 are much sparser. See also Fig- D5 for LRP visualizations for the CLIP (Radford al., 2021) (visionllaguuage) model.
cover the full object, whereas
BYOL
MoCov2
DINO
CE
BCE
CE
BCE
BCE
Backpack
Candle
Sombrero
Pretzel
Space Shurtle
Figure D4: Layer-wise Relevance Propagation Explanations. Even for conventional models see a consistent improvement in attribution localization for probes trained with BCE loss. This is seen across all SSL methods (DINO, BYOL and MoCov2). We also observe significant qualitative differences between differently trained backbones: e.g., the explanations for DINO model (left) cover the full object, as compared explanations for models trained via BYOL (center) or MoCov2 (right), that are much sparser.
24

---

Published as a conference paper at ICLR 2025
CLIP
BCE
CE
BCE
CE
School Bus
Honey Bee
Black Swan
Cannon
Butterfly
Backpack
Figure D5: Layer-wise Relevance Propagation Explanations for CLIP. Similar to SSL and su- pervised pre-trained models, for CLIP (Radford et al., 2021) which is vision-language pre-trained model we see BCE-trained probes (cols. 2+5) to localize better as compared to CE-trained (cols. 3+6) probes.
25

---

Published as a conference paper at ICLR 2025
D.2 EFFECT OF COMPLEX PROBES
Figure D6 shows visual results depicting the impact of training with MLP probes. Notice, e.g., in cols. 3+4, two- and three-layer MLPs better localize the object class of interest with minimal reliance on background features for DINO model. Since the features learned by SSL models may not always be linearly separable with respect to the classes present in different downstream tasks, stronger MLP classifiers are able to distill class-specific features better, leading to an improvement in localization scores and also improved performance all SSL frameworks when we go from single probes to MLP probes. As seen previously, the B-cos explanations for BYOL and MoCov2 show lower saturation compared DINO.
the downstream task. This behaviour is consistent across
DINO
BYOL
MoCov2
Dregonily
Umbrelle
Plungerr
Nei
Single Probe
2-layer MLP 3-layer MLP
Single Probe 2laaer MLP 3-aayer MLP
Single Probe
2-layer MLP
3-layer MLP
Figure D6: Qualitative results of MLP probes on ImageNet. We find explanations for B-cos MLPs (cols. 3+4+6+749+10) trained on SSL pre-trained features to exhibit better localization than single linear probe (cols. 2+5+8). This behavior is seen more prominently for DINO (left) as compared BYL (center) and MoCov2 (right).
D.3 ADDITIONAL EXPLANATION METHODS
Comparison between explanation methods. Figures D7, D8, D9 show additional comparisons between different attribution methods for each SSL model (both B-cos and conventional) and how it compares to their fully-super counterparts; in particular we show results for B-cos (Böhle et al., 2022), Layer-wise Relevance Propagation (LRP) (Bach et al., 2015), GradCAM (Selvaraju et al., 2017), Integrated Gradients (IntGrad) (Sundararajan et al., 2017), and Input Gradient (IxG) (Shrikumar et al., 2017). Notice, every pair of consecutive rows illustrates this for a sin- gle SSL method and supervised training. We observe that visually the explanations for SSL models and supervised models are quite similar. Interestingly, this is more consistent for B-cos models as compared to conventional models. Note: For all SSL models, these are explanations for the setting when the frozen backbone features are trained using only single linear probe with BCE loss.
26

---

Published as a conference paper at ICLR 2025
Comparison DINO vs Supervised Models
Input
B-cos
GradCAM
IntGrad
IxG
LRP
GradCAM
IntGrad
lxG
DINO
Labrador
Supervised
DINO
Banarr
Supervised
DINO
Persian Cat
Suppriiied
DINO
Pay Phone
Suparised pevivee
(a) B-cos Models
Conventional Models
Figure D7: Qualitative comparison of different explanation methods for DINO and supervised training for (a) B-cos and (b) conventional models. Notice that B-cos explanations are able to highlight the object of interest quite well. Also LRP explanations for conventional models seem to localize well to object features however are quite sparse. GradCAM explanations although do focus the object but IntGrad and IxG explanations are scattered across the entire image and also highlight background regions.
spread out for DINO model compared to supervised model.
27

---

Published as a conference paper at ICLR 2025
Comparison BYOL vs Supervised Models
Input
B-cos
GradCAM
IntGrad
IGG
LRP
GradCAM
IntGrad
IG
BYOL
Labrador
BYOL
Banana
BYOL
Persian Cat
BYOL
Pay Phone
Supervized
(a) B-cos
(b)
Figure D8: Qualitative comparison of different explanation methods for BYOL and supervised training for (a) B-cos and (b) conventional models. Similar to Figure D7, we observe that B-cos explanations are able to highlight the object of interest quite well. The LRP explanations for con- ventional models seem to localize well to object features however are quite sparse. GradCAM ex- planations although do focus on the object but are more spread out for BYOL model as compared to supervised model (especially for conventional models). IntGrad and IxG explanations are scattered across the entire image and also highlight background regions.
28

---

Published as a conference paper at ICLR 2025
Comparison MoCov2 Supervised Models
Input
B-cos
GradCAM
IntGrad
IxG
LRP
GradCAM
IntGrad
IG
MoCo
Labrador
Supervited
MoCo
Banams
Superiised
MoCo
Persian Cat
Supervierd
McCo
Pay Phone
Super/isdd
(a) B-coe Models
(b) Conventional Models
Figure D9: Qualitative comparison of different explanation methods for MoCov2 and supervised training for (a) B-cos and (b) conventional models. Similar to Figures D7 (DINO), D8 (BYOL) observe that B-cos explanations for MoCov2 are able to highlight the object of interest quite well. The LRP explanations for conventional models seem localize well to object features however are quite sparse. GradCAM explanations although do focus on the object but are more spread out for MoCov2 model compared supervised model (especially for conventional models). IntGrad and I Ix G explanations are scattered across the entire image and also highlight background regions.
29

---

Published as a conference paper at ICLR 2025
D.4 ADDITIONAL QUALITATIVE RESULTS
In this sub-section we add more qualitative results for B-cos, ScoreCAM, and GradCAM explana- tions to demonstrate the impact of the training objective and probe complexity model explana- tions.
Probing: Samples Top Localization
DING
Ckew Yark
KG. Eac 305
DLC
Char Typder
CC Lac MA
Clea hearphal
CC Lac
DCC lec RLS.
BYOL
MoCav2
Class Mara
BCE vs. CE Probing: Randomly Sampled
DINO
Clase polca
CL Jac 61%
ICE 82%
Clan brack vagu
CC Lac NE5
Bos Lac
Clase Toldor web
Cl Lec 7ach
PCE Lec Mia.
BYON
M3. BD
CaHA Tome
Eec
Rss Lau M5
Clus Sarbentes
Uhe Lec
BtL Le
MoCov2
Figure D10: Qualitative results for BCE vs. CE probing on ImageNet. (a) Shows six examples each for DINO, BYOL, and MoCov2 sampled from the set top 10% of images to show greatest improvement of BCE probing over CE probing (Aper-ee GridPG scores) when explained with (b) Additionaly, shows a set of six images randomly sampled for each SSL. method; out of 18 samples find only I sample where the BCE probe localizes worse than the CE probe (highlighted with a red box). Please zoom-in to notice the finer differences the visual explanations.
B-cos explanations.
30

---

Published as a conference paper at ICLR 2025
BCE Probing: Randomly Sampled Examples ScoreCAM
Cham Ir
DINO
BYOL
MoCov2
Sariet
Figure DI1: Qualitative results for BCE vs. CE probing on ImageNet for ScoreCAM. The figure shows set of six examples sampled randomly for each SSL method, i.c. DINO. BYOL and MOCO when explained using ScoreCAM explanations. Overall BCE probes lead to more localized explanations over CE probes. We highlight examples where BCE probes perform worse than CE probes with a red box. Please zoom-in in to notice the finer differences in the visual explanations.
DCE vs. CE Probing: Randomly Sampled Examples for GradCAM T.T
DINO
Lan
Taw
BYOL
Clan Sao
Loc
BCL Nealeelo
CJ. Emo BMh
BCL. Tea
CE. 0
Bcs. Loc 14.5
MoCov2
Figure D12: Qualitative results for BCE vs. CE probing on ImageNet for GradCAM. The figure shows set of six examples sampled randomly for each SSL method, i.e. DINO, BYOL and MOCO when explained using GradCAM explanations. Overall BCE probes lead to more localized explanations over CE probes. We highlight examples where BCE probes perform worse than CE probes with a red box. Please zoom-in to notice the finer differences in the visual explanations.
31

---

Published as a conference paper at ICLR 2025
MLP Linear Probing: Random Samples
Clam
LB Let M8N
NIP LAU
Class Vance
Lec B8.
MEINA Lac BOOS
ERITENY
DINO
BYOL
workd
Claro Tmait
LB. Loc 93
MUP1 Lan 21%
Clrs bes Cesttoo
LP Lec 18%
MENG Las 95%
Clan 1pata
Li Loc By
NINS Iet H3
McCov2
Figure D13: More qualitative results for MIP Probing. The figure shows set of six examples sampled randomly for each SSL method, i.e. DINO, BYOL and MOCO when explained using B-cos explanations. MLP probes on average tend to get visually more localized samples as compared to linear probe. Please zoom-in notice the finer differences in the visual explanations.
32

---

Published as a conference paper at ICLR 2025
E ADDITIONAL QUANTITATIVE RESULTS
In this section, for completeness we present quantitative results for all explanation (or attribution) methods; specifically for B-cos (Böhle et al., 2022), Layer-wise Relevance Propagation (LRP) (Bach et al., 2015), GradCAM (Selvaraju et al., 2017), Integrated Gradients (IntGrad) (Sundararajan al., 2017), Input Gradient (IxG) (Shrikumar al., 2017), LIME (Ribeiro et al., 2016) and Guided- Backpropagation (Springenberg et al., 2015). In Sec. E.1 we first evaluate the impact of the opti- mization objective probing in terms of accuracy and explanation quality using the different met- rics (see Section 3.1 in main paper) on ImageNet. Then in Sec. E.2 we analyze the effect of probe complexity explanation localization. Finally in the second part of Sec. E.2, also present the results on the multi-label classification setting on COCO and VOC datasets, where the explanation localization metric used is EPG score
E.1
IMPACT OF BCE VS. CE
W(x) Input
Integrated Gradients
Input Gradlent
GridPG Score (3x3)
100
20-
CC
CE
02
66
6B
70
72
74
70
66
68
70
72
74
76
06
68
70
74
76
2008
Accuracy GradCAM
Accuracy
Accuracy
LIME
Guided Backpropagation
GridPG Score (3x3)
100
OND
20
20-
1528
68
70
72
74
76
06
88
70
72
66
70
74
Accuracy
Accuracy (a)
Accuracy
Figure E1: BCE vs. CE - Accuracy and GridPG scores on ImageNet. To study the impact of the optimization objective on probing we plot the accuracy vs localization (GridPG) score for B-cos models (B-RN50). For BCE trained probes (solid markers), steady improvement in the localization score is seen for all SSL. methods and across most explanation methods on ImageNet. For GradCAM (bottom left), DINO model does not show improvement in localization score for BCE trained probe, and Guided Backpropagation explanations (bottom right) are inconsistent for SSL models.
In fig. E1 we plot the linear probe accuracies vs GridPG score for MoCov2 (red). BYOL (green), DINO (blue), CLIP (yellow) and supervised (black) models for (a) B-cos models (B-RN50) and (b) conventional models (RNS0) on ImageNet. The linear probes trained with BCE loss are depticted with filled markers, and the hollow markers represent CE loss. We see consistent improvement in localization score for most explanation methods. Note, as discussed in the main paper IxG and GradCAM do not show consistent behaviour and only show consistent improvements for B-cos models (compare the bottom rows in fig. E1 (a) vs fig. E1 (b)). For IxG, this line with prior work, model gradients for conventional models known to suffer from 'shattered gradients' (cf. (Balduzzi et al., 2017)) and provide highly noisy attributions. Additionally, Guided Backpropagation also performs poorly overall to see any obvious trend (as it has been shown earlier produce noisy explanations (Rao et al., 2022)). Finally, for GradCAM it has been shown that especially for self- supervised models, it can perform poorly due focusing on background regions rather than object of interest (this can be induced due to the SSL specific training objectives) (Selvaraju et al., 2021).
Figures E2, E3 show the EPG localization scores the bounding boxes provided for the ImageNet validation set. For B-cos backbones we observe a consistent improvement for BCE trained probes for 11 out of 12 cases. The only odd case GradCAM explanations for the DINO model. For
*Rssult for ResNe150 backbones, unless specified otherwise.
33

---

Published as a conference paper at ICLR 2025
conventional backbones this is consistent for out of 12 cases (GradCAM explanations being the exception again).
Next figures E4, E5 show the pixel deletion plots on the ImageNet validation set. For B-cos back- bones we observe more consistent improvement terms stability when compared to conven- tional models. For B-cos backbones for GradCAM method, CE probes produce more stability, and for conventional backbones we only see consistent results for LRP attribution methods (where BCE probes are more stable than CE probes). For other attribution methods and pre-trained backbones we see mixed results.
Layervise Relevance Propagation
Integrated Gradients
Input Gradient
100
GziáPo Soore (3x3)
80
76
Accuracy
Accuracy
Accuracy
GradCAM
LIME
Guided Backpropagation
CHiPPC Score (3x3)
100
6:8
068
70
72
74
76
68
70
74
76
66
68
70
72
74
76
Accuracy
Accuracy
Accuracy
(b)
Figure E1: BCE vs. CE Accuracy and GridPG scores on ImageNet. To study the impact the optimization objective on probing we plot the accuracy vs localization (GridPG) score for conven- tional models (RN50). For BCE trained probes (solid markers), an improvement the localization score is seen for all SSL methods for LRP, IntGrad and LIME explanations on ImageNet. However, for IxG, and GradCAM this is not seen. For Guided Backpropagation there a slight improvement in localization score for BCE trained probes (except for DINO model).
34

---

Published as a conference paper at ICLR 2025
MoCov2 Boos: WOO Inpot
BYOL: Boos: W00 Input
DINO Boos: W0O Input
1.0
BCE
BCE
BCE
0.8
0.8
0.8
EPG Store
0.6
0.6
0.6
04
0.4
0.4
0.2
0.2
0.2
0.0
<25%
<550%
<75%%
<100%
0.0
+25%
-50%
<75%
<100%
0.0
255
+50%
+75%
<100%
BBOX %
BBOX
BBCX
MoCovv2 Integrated Gradients
BYOL: Integrated Gradients
DINO: Integrated Gradients
1.0
BCE
BCE
BCE
0.8
0.8
0.8
EPG Soora
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
<2.5
<50%
<75%
<100%
0.0
-25%
-50%
+70%
<100%
0.0
2255
<50%
<75%
<100%
BIUX
MoCov2 Inpot Gradient
BYOL: Input Gradient
DINO: Input Gradient
1.0
BCE
BCE
BCE
0.8
0.8
0.8
EPG Stora
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
-255%
<50%
<75%
x100%
0.0
+25%
-50%
+70%
<100%
0.0
225
+50%
+75%
<100%
BBOX %
BBOX %
BBOX
MoCov2 GradCAM
BYOL GrodCAM
DINO: GmdCCMM
1.0
1.0
BCE
BCE
BCE
0.8
0.8
0.8
EPG Storn
0.6
0.6
0.5
0.4
0.6
0.4
0.2
0.2
0.2
0.
-20%
<30%%
<75%
<100%
+20%
-50%
<70%
<100%
<25%
<50%
<75%
<100%
BROX %
BROX
BROX %
Figure E2: BCE vs. CE - EPG scores for B-cos models on ImageNet.
35

---

Published as a conference paper at ICLR 2025
McCov2 Layerwise Relevance Propegation
HYOL: Layervise Relevance Propogation
DINO: Layerwise Relevance Propagation
1.0
ERG S0000
0.4
0.4
0.4
BBOX
BBOX
MoCov2 Integrated
BYOL Integrated Gradients
DINO Integrated Gredients
ICC
EP0 Store
0.6
0.6
0.6
0.0
-20%
<500%
<70%%
<100%
0.0
+25%%
<500%
<100%
0.0
<20%%
<100%
BBOX
McCon2 Input Gradient
BYOL Input x Gradient
DINO: Input Gradient
1.0
1.0
BCB
BCE
BCE
0.8
Scone
04
022
0.2
022
-25%%
<SIPS
<75%
<100%
+25%%
-50%
-70%
<100%
-235%
-317%
<75%
<100%
BBOX
BBOX
BBOX
MoCov2: GeadCAM
BYOL: : GradCAM
DINO GraCCAM
EPU Storo
0.6
0.6
0.8
0.4
0.4
Figure E3: BCE vs. CE EPG scores for Conventional models on ImageNet.
36

---

Published as a conference paper at ICLR 2025
MoCov2: Hoos: W(X x Input
BYOL: : Hoos: W(X) x Input
DINO: : Hcos: W(X) x Input
Strqet clase priscatey
o
0.6
0.6
0.6
0.4
0.4
0.4
plans
posols removedd
plents
McCov2: Integrated Cradients
BYOL: Integraated Cradients
DINO: Integrrated Cradients
Turper clas prbsbiity
0.
0.8
0.0
0.6
10
20
0.2
10
20
0.2
10
1K of plass removed
75 pools remoned
of plotls renored
MoCov2: Input x Gradient
BYOL: : Input Gradient
DINO: Input x Gradient
Tteppt class probebttyy
30E
822
B0E
6k
CR
0.c
0.6
0.6
0.4
0.4
0.4
glonis
pools
pioele
MoCov2 GradCAM
BYOL: GradCAM
DINO: GradCAM
Tarpur clasa prosibllty
0.5
0.5
0 8
0.0
64-
0.6
DC2
C8
10
20
02
10
30
02-
10
9: of placs remowed
5r0f pionle remoned
N5 plonis removed
Figure E4: BCE vs. CE- Pixel deletion scores for B-cos models on ImageNet.
MoCov2 Layerwise Relevance Propegation
BYOL: Layerwise Relevance Propagation
DINO Layerwise Relevance Progaagation
Shapet claa pratectttt
8
0.4
0.4
0.2
0.2
ronoed romooedd
pfools ronowed
of pianis recooved
McCov2: Integrated Gradients
BYOL: Integrated Gredients
Integgrated Gredients
g
0.6
8
0.4
0.4
0.2
10
10
20
% plots recloedd
9: pfoos remoodd
of piasis recooed
MoCov2 Input Grudient
BYOL: Input Gradient
DINO Gradient
Chopetaan protectt
g
0.-
R
0.4
0.4
02
10
20
0.2
10
20
plod's rertoedd
To of menons
of piasis rerooed
MoCov2 GraCCAM
BYOL: GraNCAM
DINO GredCAM
Ttaget Clan groboattt
0.5
20
plods recsoeed
popod renovd
piunto recooved
Figure E5: BCE vs. CE Pixel deletion scores for Conventional models on ImageNet.
37

---

Published as a conference paper at ICLR 2025
Table E1: Compactness and Complexity of BCE CE Probing. A consistent improvement in compactness (Gini index pp. as in (Chalasani et al., 2018)) and reduction in complexity (entropy as (Tseng et al., 2020)) for BCE vs. CE, except for ImpuxGradient and GradCAM with conven- tional backhones.
Backbone
Pre-training Method
XAT Method
Accuracy
Aooplexxtty
ACompactness
CF
BCE
B-RN50
MoCoV2 BYOL
Beos: W(x) x Input
66.3
65.4
-2.72
+0.10
B-RN50
Bcos: W(X) x Input
68.3
68.1
-1.87
0.14
B-RN50
DINO
Bcos: W(X x Input
68.8
67.5
-2.35
+0.16
B- B-RN50
MoCoV2
Input x Gradient
66.3
65,4
-0.13
0.00
B-RN50
BYOL
Input Gradient
68.3
68.1
-0.08
0.00
B-RN5O
DINO
Input x Gradient
68.8
67.5
-0.08
0.00
B-RN50
MoCoV2
IntGrad
66.3
65.4
-0.09
0.00
B-RN50
BYOL
IntGrad
68.3
68.1
-0.04
0.00
B-RN5O
DINO
IntGrad
68.8
67.5
-0.05
0.00
B-RN50
MoCoV2
GradCAM
66.3
65.4
-1.07
+0.13
B-RN50
BYOL
GradCAM
68.3
68.1
-282
+0.20
B-RN50
DINO
GradCAM
68.8
67.5
-2.51
+0.19
B-RNSO
MoCoV2
GBP
66.3
65.4
-0,23
+0.02
B-RN50
BYOL
GBP
68.3
68.1
-0.14
+0.01
B-RN50
DINO
GBP
68.8
67.5
-0.13
+0.01
RN50
MoCoV2
Input Gradient
67.5
66.9
+0.01
0.00
RN50
BYOL
Input Gradient
70.1
69.3
-0.04
+0.01
RN50
DINO
Input x Gradient
71.2
70.6
+0.01
0.00
RN50
MoCoV2
IntGrad
67.5
66.9
-0.01
0.00
RN50
BYOL
IntGrad
70.1
69.3
-0.08
0.00
RN50
DINO
IntGrad
71.2
70.6
0.00
0.00
RN50
MoCoV2
IRP
67.5
66.9
-1.48
+0.07
RN50
BYOL.
IRPP
70.1
69.3
-0.85
+0.04
RN50
DINO
LRP
71.2
70.6
-2.01
+0.12
RN50
MoCoV2
GradCAM
67.5
66.9
+1.13
-0.25
RN50
BYOL
GradCAM
70.1
69.3
+1.61
-0.25
RN50
DINO
GradCAM
71.2
70.6
-2.4
+0.23
RN50
MoCoV2
GBP
67.5
66.9
-0.14
+0.01
RN50
BYOL
GBP
70.1
69.3
-0.24
+0.02
RN50
DINO
GBP
71.2
70.6
0.00
0.00
38

---

Published as a conference paper at ICLR 2025
Table E2: Supervised Probing. We present the effect of probing fully-supervised models (ref. Section 4.1 in main paper) for different attribution methods (B-cos Böhle et al. (2022), IntGrad Sun- dararajan et al. (2017). IxG Shrikumar et al. (2017), GradCAM Selvaraju al. (2017), and LRP Bach et al. (2015)), and demonstrate how this impacts the localization (GridPG) score. We observe similar accuracy for both (CE and BCE) probes. However, as seen earlier (cf. fig. Ela,b), the probes trained with BCE loss lead to consistently high improvements in GridPG scores.
Accuracy (%)
GridPG (%)
Att. Method
CE
BCE
CE
BCE
A
72.4
73.7
+1.3
B-RN50
B-cos Böhle (2022)
15.0
87.0
+72.0
IntGrad Sundararajan et al. (2017)
24.0
86.0
+62.0
TxG Shrikumar et al. (2017)
15.0
65.0
+50.0
GradCAM Selvaraju et al. (2017)
14.0
72.0
+58.0
75.5
75.6
+0.1
LRP Bach et al. (2015)
9.0
32.0
+23.0
RN50
IntGrad Sundararajan et al. (2017)
14.0
28.0
+14,0
[xG Shrikumar al. (2017)
10.0
18.0
+8.0
GradCAM Selvaraju et al. (2017)
16.0
42.0
+26.0
In table E1 we present the results for probing with BCE vs CE loss, when evaluated on the com- pactness and complexity metrics. A consistent improvement in compactness (Gini index pp. in (Chalasani et al., 2018)) and reduction in complexity (entropy as in (Tseng et al., 2020)) for BCE Gradients and GradCAM with conventional backbones.
vs. CE, except for InputxGradients
Probing Supervised Models. table E2, we demonstrate the effect of probing fully-supervised backbones with CE and BCE linear probes. This is to also increase comparability with the self- supervised and vision- langugge approaches that we present carlier (cf. fig. El (a) and (b)).
Additional Perturbation-base based Explanations In table E3, we show the improved localization abil- ity of BCE probes over CE probes for another perturbation-baat explanation method called Score- CAM (Wang et al., 2020). See Figure D11 for qualitative results.
Table E3: ScoreCAM BCE vs. CE Probing-GridPe GridPG scores on ImageNet. A consistent im- provement in localization score for BCE over CE probes is seen for both conventional and inher- ently interpretable B-cos backbones for supervised and self supervised pre-training.
Localization
Bakcbone
Pretraining
CE
BCE
Aloc
RNS0
MoCov2
52.6
54.4
+1.8
RNSO
BYOL
38.1
40.5
+2.4
RN50
DINO
44.2
55.9
+11.7
BRN50
MoCov2
30.2
43.2
+13.0
BRN50
BYOL.
45.3
50.2
+4.9
BRN50
DINO
51.0
67.7
+16.7
E.2 EFFECT OF MORE COMPLEX PROBES
In this subsection, we present the guantitative results for probing with more complex MLP probes, specifically instead of using single linear probe top of frozen backbone features, and three-layer MLPs for training on downstream tasks and study its effect on performance (in terms of classification accuracy f1-score) and explanation localization ability. For this setting, we train both B-cos and conventional MLPs, and show results for the same.
two-
Effect of MLP Probes on ImageNet In fig. E6 we show the results on ImageNet for B-cos MLPs for both (a) B-cos and (b) conventional SSL models. It is observed that for all SSL models there is both an increase in accuracy as well as in the localization (GridPG) score across most attribution methods.
Next, in fig. E7 we plot the results to see the effect of conventional MLPs. In contrast to B-cos MLPs, here the stronger MLP based probing does lead to an increase in accuracy but does not always lead to increase in GridPG localization score. This is observed independent of B-cos conventional
39

---

Published as a conference paper at ICLR 2025
model backbones. We attribute this to the alignment pressure introduced by B-cos layers (ef. (Böhle et al., 2022)) that helps distill object-class relevant features and rely less on background context. Thus, discussed in the main paper in order to get attribution maps with good localization downstream tasks, we suggest to probe pre-trained models with B-cos probes.
These are followed by presenting the bar plots for MLP probing experiments for EPG scores on ImageNet in figures E8, E9 for B-cos probes, and in figures E10, E11 for conventional probes.
Figures E12, E13, and E14 contain the pixel deletion plots on ImageNet for the MLP probing setup.
B-RN5O, W(x) Input
B-RN50, Integrated Gradients
B-RN5O, Input Gradient
100
100
G/MPG Score (3x3)
Score (3x2)
TOAL
Accuracy
B-RNSO,
Backpropagation
100
100
100
DNIG
G/MPG Score (3x3)
Score (3x3)
Accuracy
Accuracy
Accuracy
RN50, Integrated Gradlents
RN50, Input x Gradlent
RN50, GradCAM
GridPo Score (3x3)
GriPP Score (3x3)
Accuracy
Accuracy
ZTD CNE
RM50, Guided Backpropagation
100
100
GriPP Store E333
13
GriPP Score (5x3)
60
80
32
10
-0
28
70
72
64
Accuracy
Accuracy
(b)
Figure E6: Effect of complex B-cos probes on accuracy and GridPG scores on ImageNet on (a) B- cos backbones (B-RN50) and (b) conventional backbones (RN50). Notice that there is a consistent improvement in accuracy (X-axis) as well as GridPG localization score (y-axis) as we move from single probe to two- and three-layer MLP probes in most cases. Except only for B-cos models trained with MoCov2, for IntGrad (top-middle, subplot a) and GradCAM (bottom-left, subplot a) explanations a slight drop in localization scores. For the remaining cases (SSL frameworks and explanation methods), there a steady increase in both accuracy and localization.
40

---

Published as a conference paper at ICLR 2025
B. SRRDO, Boos Wini Input
B-RN50, Integrated Gradients
B-RN50, Input Gradient
100
130
Ooaen Seate (1411
ChIPP SCoTE (XC)
ZANORIC
Seare
TONS
Aecurery
Accenocy
Accutecy
B-RNS0, GradCAM
B-ANSO,
8-4N50, Gulded Backpropagation
DV81
(60)
OuArO Sivee CANO
2000L Sndii
Accuracy
Accuracy
Accurscy
(a)
RN50, Layerwise Relevance Propagation
Integrated Gradients
RN5S, Input Gradlent
Gr1P0 Soore 13001
60
Ascurney
Accurac
lcauncy
BN50, GredCAM
LIME
RN50, Backpropagation
CHAPN (343
8
8
2
Accursy
leaunyy
(b)
Figure E7: Effect of complex conventional probes on accuracy and GridPG scores on ImageNet on (a) B-cos backbones (B-RN50) and (b) conventional backbones (RN50). Interestingly, in contrast to B-cos MLP probes (cf. fig. E6). when conventional MLP probes are used, although there is an improvement in accuracy for all SSL models, there is consistent improvement in the GridPG localization score. In fact, in several cases there is a drop in localization score as we move from single probe to conventional MLP probes.
41

---

Published as a conference paper at ICLR 2025
McCov2 Boos: WoX Input
BYOL: Boos: Wix) Input
DINO Boos: Wik) x Input
1.0
MLP22
MLP22
0.8
MLP-2
EPG Scure
LP
LP
0.8
+25%%
<350%
<76%
<1000%
+2.5%
-550%
+775%
<100%
-25%
<500%
<7.5%
<1100%
IXXX %
BBOX
%
BISUX
McCov2 Inteegaaedd Gradients
BYOL: : Integrated Gradients
DINO: Integrated Gradients
1.0
1.0
1.0
MI.PS
MIP3
MIP-3
0.8
2
0.8
MLP
0.8
MLP 2
ERG Skore
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
-25%
+550%
-70%
<100%
<20%
<500%
<70%
<100%
0.0
-25%
-500%
<75%
-100%
BBOX
BBOX
BBOX
MoCov2: : Inpot Gradient
BYOL: Input Gradient
DINO Input Gradient
1.0
1.0
MLP-S
MLP33
MLP-3
0.8
MLP
0.8
MLP
0.8
EPG Scnro
0.6
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
0.0
<520%
<70%
<100%%
0.0
779%
<100%
0.0
<70%
<100%
BBOX
MoCov2 GradCAM
GrodCAM
CrodCAM
MLP-3
EPO Scnn
0.6
0.5
0.4
0.2
0.0
-100%
0.0
<100%
0.0
-70%
BBOX
BBOX
Figure F8: Bcos-MI.P EPG scores for Beos models on ImageNet.
42

---

Published as a conference paper at ICLR 2025
McCor2: Integrated Gradients
BYOL Integrated Gradients
DINO Integrated Gradients
1.0
MLP-3
MLP-3
MLP-3
0.8
MLP-2
0.8
MLP-2
LP
LP
BRG Stoce
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
+25%
<50%
<55%
<1 -100%
<5%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
BRXX 8
BRXX
THEX
MoCov2:
Gradient
BYOL:
Gradient
Gradient
MLP
MLP 3
MLP
MLP-2
MLP2
MLP-2
1P
1P
ERO Sours
0.4
0.2
0.2
0.2
0.0
0.0
0.0
<25%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
BBOX 8
BBOX
BBOX
MoCov2 GradCAM
GradCAM
GradCAM
1.0
MLP-3
MLP-3
MLP-3
ML.-2
MLP-2
ML.-2
EPO Scoee
0.0
<25%
<50%
<75%
<100%
<25%
<50%
<75%
<100%
<25%
<50%
<75%
<1 <100%
BIOX
BBOK'S
BBXX
Figure E9: Bcos-MLP EPG scores for Conventional models on ImageNet.
43

---

Published as a conference paper at ICLR 2025
McCov2 Boos: WoX Input
BYOL: Boos; Wix) Input
DINO Boos: Wik x Input
1.0
1.0
MLP22
MLP22
0.8
MLP-2
EPG Scure
LP
LP
0.8
+25%%
<350%
<76%
<1000%
+2.5%
-550%
+775%
<100%
-25%
<500%
<7.5%
<1100%
IXXX %
BBOO
%
BISUX
McCov2 Inteegaaedd Gradients
BYOL: : Integrated Gradients
DINO: Integrated Gradients
1.0
1.0
1.0
MI.PS
MIP3
MIP-3
0.8
2
0.8
MEP
0.8
MLP
ERG Skore
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
<25%
+550%
-70%
<100%
<25%
<500%
<75%
<100%
0.0
-25%
-50%
-7.%
~100%
BBOX
BBOX
BBOX
MoCov2: : Inpot Gradient
BYOL: Input Gradient
DINO Input Gradient
1.0
1.0
MLP-S
MLP33
MLP-3
0.8
2LP
0.8
MLP
0.8
EPG Scnro
0.6
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
0.0
<220%
<70%
<100%%
0.0
779%
<100%
0.0
<70%
<100%
BBOX
MoCov2 GradCAM
GrodCAM
CrodCAM
EPO Scnn
0.5
0.2
0.0
-100%
0.0
<100%
0.0
-70%%
BBOX
Figure E10: Conventional-MII.P EPG for B-cos models ImageNet.
44

---

Published as a conference paper at ICLR 2025
McCov2 Layeewise Relevance Progagation
BYOL; Layervise Relevance Propegalion
DINO: Layerwise Relevance Propagation
EDO State
0.6
0.6
0.8
0.4
0.4
0.4
<100%
<100%
BBOX
BBOX
BBOX
McCov2: inntagrated Gradients
BYOL: Integrated Cradients
DINO |Integrated Gradients
MLPS
MLP-3
MUPS
0.8
MI
0.8
ETO Store
0.6
0.6
0.6
0.2
0.0
<22075
770%%
<100%
0.0
-20%
<755%
<100%
0.0
+251%
-500%
<710%
<100%%
BBOX
BBOX
BBBXX
McCov2:
Input x Gradient
BYOL: Input x Gradient
DINO Ingut Gradient
1.0
1.0
1.0
MLP33
MLPS3
MLPS
0.8
MI
0.8
MI M-P
MIP2
RIV Sione
1P
1P
TP
0.1
0.2
0.2
02
0.0
-25%%
-310%
<73%
-100%
-.20%
-250%%
<7554
-110%%
-25%
<207%
<759
-1100%
JIBOX
ISBUX 75
McCov2: GradCAM
BYOL : GradCAM
DINO : GradCAM
1.0
1.0
EIG Scate
0.6
0.6
0.4
0.6
0.4
77%%
+100%
+25%%
<25%
+50%%
BBOX
BBOX
B80X
Figure E11: Conventional-MI.P EPG for Conventional models on ImageNet.
45

---

Published as a conference paper at ICLR 2025
MoCov2: Bops: W(x) x Input
BYOL: ; Boos: W(x) Input
DINO: Boos: W(K) x Input
Turget clas prolablity
0.8
0.6
0.6
0.-
04-
0.6
MLLS
0.4
NLP3
HLP2
NLP2
62
planta rampodd
posds rentred
plocla rettored
McCov2: Integrated Gradients
Integrated Gradients
DINO Integrated Gradients
class prebabiity
S0P3
08-
MEP
MUP
Teapet clapet
0.-
02
0.2
0.2
20
10
20
18 of plads remsved
of posls removed
Nu piosla estroved
McCov2 Inpul Gradient
BYOL: Inpulx Gradieul
DINO Iapul Gradient
Turge (lea pebsallity
50.3
MUSS
0.5
0.5
30.52
0.-
MUN
10
0.6
0.6
0.6
02
0.2
plads removed
posds remsvvdd
plocla rotsored
McCov2 GradCAM
BYOL: GradCAM
DINO: GradCAM
Target clus redaabiity
MIP-3
MLP3
MLP-3
MLP2
MLD2
MLPP2
02
02
1P
02
12
20
20
20
9t of plans removed
35 of posds removod
16 of piochs reccoved
Figure E12: Bcos-MLP- Pixel deletion scores for Bcos models on ImageNet.
MoCov2 Integrated Gredients
Integrated Gradients
Integrated Grudients
MLP-3
MLP33
MLP-3
Deys. Qww.ytakkty
MLP-2
ML.22
1P
LP
1P
15
10
McCow2 Ingut x Gradient
BYOL: Input x Gradient
DIND : Inpet x Gradient
MLP-3
0.8
MLP22
0.0
MLP2
0.0
MLP22
Tarpe Dww.pebatdry
0.5
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
10
15
20
0
10
15
20
5
10
15
20
porkeceeentt MoCov2: GreaCAM
BYOL:
ofppldds meoood
of glvell meyyend
GradCAM
Taepec Coo.prtatzry
MLP-3
MLP33
MLP3
ML.22
ML.-2
ML.
0.2
LP
0.2
LP
Figure E13: Bcos-MLP- - Pixel deletion scores for Conventional models on ImageNet.
46

---

Published as a conference paper at ICLR 2025
MoCov2 Layerwise Propagation
BYOL:
Propogation
Layerwise Propagation
Tareet che prosittty
S
0.-
2
0
0.2
MLP2
removed
peants removed
pinabs nenarved
MoClov2:
Integrated Gradients
BYOL: Integrated Gradients
DINO Integrated Gradients
Darget the proaattte
0.-
02
pinia
paata reseoved
piash nesoved
MoCov2 Input Cradient
BYOL: Cracieen
DINO Input Gradient
Target caam protenttt
S.
MLN2
0.0
NUN2
0.
giods aussed
pa
panshe
MoCov2 GradCAM
BYOL: : GradCAM
DINO GradCAM
Zagge dlan probucsity
NLD2
MLD
plads
pincls
Figure E14: Conventional-MI.P - -iiel deletion scores for Conventional models on ImageNet.
47

---

Published as a conference paper at ICLR 2025
Evaluation on Multi-label Classification We also evaluate all SSL models (both B-cos and con- ventional backbones) and explanation methods under the multi-label classification setting. We train single probes and MLP probes using the BCE loss (as is typical) COCO and VOC datasets. To measure model performance we report fl-score and explanation localization we report the EPG score. Figures E15, E16 show the fl f1-score EPG score plots for models probed with B-cos MIPs and conventional MLPs respectively on COCO, Here we notice, that stronger MLPs result in an improvement both classification performance (depicted by an increase in fl-score), and also im- proved localization EPG scores.
B-RNSO, Bcoost W(X) x Input
B-RN5O, Integrated Gradients
B-RN50, Inpet x Gradient
80000
E30 Score
55
03
70
F1 Score
F1 Scare
B-RN50, GradCAM
8-RN50, LIME
B-RN50, Guided Backpropagation
10000
(P0 Scorn
55
02
65
02
30
25
03
05
70
72
80
F1 S.coree
Integrated Gradients
RN50, Input Gradient
RN50, GradCAM
EVO Store
42
RN50, LIME
Backpropagation
32
30
P3
(b)
Figure E15: Effect of complex B-cos probes on accuracy and EPG scores on COCO on (a) B-cos backbones (B-RN50) and (b) conventional backbones (RN50). When we train with B-cos MLP probes as compared to a single probe, we observe an improvement in both fl-score (x-axis) and EPG localization score (y-axis) for all SSL models and explanation methods, except for GradCAM where improvement in localization is not seen consistently.
48

---

Published as a conference paper at ICLR 2025
B-RN50, Bcos: W(X) Input
B-RN5O, Integrated Gradlents
B-RN50, Input Gradlent
NDCIN
M000
EPG Score
0
F1 Score
F1 Score
F1 Score
B-RN50, GradCAM
B-RN50, LIME
B-RN50, Guided Backpropagstion
MOCONS
M000
DAD
EPG Score
Seuhe Pale
Hup our
50
35
00
55
80
50
35
60
70
75
00
70
35
50
70
75
80
Score
Score
Score
Layerwise Relevance Propagation
RN50, Integrated Gradlents
RN50, Gradlent
60
80
20
70
70
F1 Score
F1 Score
F1 Score
RN50, GradCAM
RN50, LIME
RN50, Guided Backpropagation
RC000)
MDCIO
50
080
DMD OLF
CLP
10
10
3-kper
70
Figure E16: Effect of complex conventional probes on accuracy and EPG scores on COCO on (a) B-cos backbones (B-RN50) and (b) conventional backbones (RN50). Similar to fig. E15, when training with conventional MLP probes as compared to a single probe, we observe an improvement in both fl-score (x-axis) and EPG localization score (y-axis) for all SSL models and explanation methods, except for GradCAM (bottom-leff, subplots a,b) where improvement localization is not seen consistently.
49

---

Published as a conference paper at ICLR 2025
Likewise, Figures E17, E18 show the corresponding plots for VOC for B-cos MLPs and conven- tional MLPs respectively. We see a similar trend as we saw on COCO, complex (two- and three-layer MLPs) probes lead to increased fl-score and EPG score. Interestingly, shows inconsistent behaviour, especially in the case of conventional MLPs (see Figures E16 and E18). This is well-known limitation for GradCAM, which has been shown to perform poorly (in terms of localization) at earlier layers (Jiang et al., 2021). Note: For B-cos MLPs we omit showing the results for LRP explanations as the relevance propagation rules for B-cos layers are not defined.
notice that GradCAM
nNse, Aftributtinn Bcos: WIX Impat
RMS0, Aftributten: Integrated Gradients
Isput
02
B: B-AN50, Attributtinn: GrASCAM
8-ANIo, Artribation: Guided Eactpropapaiinn
83
73
20
Score
F3 Store
Integrated
RNSO, Input
Actribattion GraNCAM
10
90
92
Attriaacio:: LIME
RN50, Attribution: Guided Backpropagetion
Sinple Prote
2aayer HUP Skaye
890
08
30
(b)
Figure E17: Effect of complex B-cos probes on accuracy and EPG scores on VOC on (a) B-cos backbones and (b) conventional backbones. Notice as seen for COCO (cf. fig. E15), stronger two- and three-layer MLPs lead to an improvement in both fl-score (x-axis) and EPG localization score (y-axis). Once again, for GradCAM this is not very consistent. Also note that for B-cos models trained with MoCov2, B-cos explanations (top-left, subplot a) for MLP probes perform worse in EPG-score compared to single probe.
50

---

Published as a conference paper at ICLR 2025
B-RN50, Attribution: Bcos: W(x) x Input
B-RN50, Attribution: Infegrated Gradients
B-RNSO, Attribution: Input x Gradient
M0OOV
1991
EPO Score
20
76
78
82
76
78
80
82
86
F1 Score
F1 Score
F1 Score
B-RN550 Attribution: GradCAM
B-RN50, Attribution: LIME
B-RN Attribution: Guided Backpropagation
M00M
70
DM0
1990
EP0 Score
50
20
Hoye HAP
76
F3 Score
F1 Score
04
55
76
78
80
F1 Score
86
(a)
RM50, Attributtion: Layerwise Relevance Propagation
RN50, Attribution: Integrated Gradients
RN50, Attributtion: Input Gradlent
00
60
60
Score
20-
MOOONG
20-
30
10
30
Score
P1
Attributtion: GradCAM
RN50, Attribution:
Attribution: Backpropagation
00
72
70
700
60
EPC Scare
EPG
15
30
20
20
Siggi
(b)
Figure E18: Effect of complex conventional probes on accuracy and EPG scores VOC (a) B-cos backbones and (b) conventional backbones. Even conventional two- and three-layer MLPs lead an improvement both fl-score (x-axis) and EPG localization score (y-axis), Yet again, for GradCAM this is not consistent.
51

---

Published as a conference paper at ICLR 2025
F IMPLEMENTATION DETAILS
In this section we describe in detail our experimental setting, i.c. the datasets we train and evaluate on, the training and implementation details, and finally the evaluation metrics used.
F.1 DATASETS
For our experiments we use three datasets namely ImageNet (Russakovsky et al., 2015), VOC 2007 (Everingham et al., 2009) and MS COCO 2014 (Lin et al., 2014).
ImageNet We use the ImageNet- IK dataset that is part of the ImageNet Large-Scale Visual Recog- nition Challenge (ILSVRC) (Russakovsky et al., 2015). This has 1000 classes, with roughly 1000 images belonging each category. In total, there are 1,281, 167 training images, and 50,000 valida- tion images. We perform all self-supervised pre-training on the training set, and evaluation on the validation set. For training, the images are resized an input resolution of 224 For measur- ing the performance of the model the top-1 accuracy (proportion of correctly classified samples) is reported, and evaluate model explanations the grid pointing game (GridPG) localization score is reported (discussed in more detail below).
VOC 2007 VOC 2007 (Everingham et al., 2009) is a popularly used multi-label image classification dataset. It comprises of 9,963 images in total and 20 object classes, and is split into the train-val set with 5,011 images and the test set with 4,952 images. We use the train-val set for training and test set for evaluation, For training, the images are resized to a fixed resolution of 224x224. As is typical (Rao al., 2023) for evaluating multi-label classification datasets, we report the fl-score and use the energy pointing game (EPG) score to evaluate the model explanations.
MS COCO 2014 Microsoft COCO (Lin et al., 2014) is another popular dataset generally used for image classification, segmentation, object detection and captioning tasks. We use COCO-2014 in our experiments, that has 82,081 training and 40,137 validation images and 80 object classes. For our training, we resize the images to fixed size of 448 Similar to VOC for evaluation we report the fl-score and use the EPG score to evaluate model explanations. Also note that the variation of objects' shapes and sizes are more complicated (Zhu & Wu, 2021) on COCO than those in VOC, and it is also substantially larger with more object classes.
In short, we evaluate the models on (1) single-tabel and (2) multi-label classification. For (1), we train probes on top of the frozen pre-trained features on ImageNet (Russakovsky et al., 2015), and report top-1 accuracy on the validation set. For (2), the probes are trained to predict all classes that are present in an image; for this, we use VOC (Everingham et al., 2009) and COCO (Lin et al., 2014) and report the F1 scores on the test set.
F.2 MODELS
We evaluate both conventional (ResNet-50 (He et al., 2016), VIT-B/16 (Kolesnikov et al., 2021)) and the recently proposed inherently interpretable B-cos models (B-cos ResNet-50, -ST,- ViT B/16) (Böhle et al., 2024). To adapt these to the various pre-training paradigms. we follow (Böhle et al., 2024) in converting the MLP heads of the respective SSL methods. Specifically, replace the linear layers in the MIP projection heads (MoCov2, BYOL, DINO) and the prediction head (BYOL) with corresponding B-cos layers (B=2), remove all ReL.U non-linearities, and replace all batch normalization layers with the corresponding uncentered version, see (Böhle et al., 2024).
F.3
FRAMEWORKS
As discussed previously, we aim to have a broad enough representative set that highlights how our evaluation generalizes across differently trained feature extractors, particularly (I) supervidpd (2) self-supervised, and (3) contrastive lasion-language learning.
(1) Suply-Super Learning We first, evaluate the explanation methods fully supervised backbones. Since, the backbones pre-trained in a supervised manner are still often used for transfer learning (Cheng et al., 2022; Xie et al., 2021; Chen et al., 2017). Additionally, evaluation of fully supervised models also provides a useful reference value, as most explanation methods have been developed in this context.
52

---

Published as a conference paper at ICLR 2025
In addition to evaluating the end-to-end trained classifiers, we also evaluate linear probes the frozen representations of these models, in order to increase the comparability to the sulf-supe approaches present in the following.
We consider three popular self-supervised pretraining (Chen et al., 2020c), BYOL (Grill et al., 2020), and DINO (Caron et al., 2021) regarding their interpretability. In the following, we briefty describe each of them.
(2) Self-Supervised Learning.
MoCov2 (He et al., 2019) employs a contrastive learning paradigm, which can be regarded as a form of instance classification. In particular, the backbone is trained to yield similar representations for different augmentations of the same image, whilst ensuring that representations different images (' ('negatives") are dissimilar.
BYOL (Grill et al., 2020) constructs self-supervised learning as form of Mean Teacher self- distillation (Tarvainen & Valpola, 2017) with no labels. Unlike MoCov2, it does not rely on negative pairs for training. The mean squared error between the normalized predictions (student) and target projections (teacher) is minimized during training.
DINO (Caron et al., 2021) uses similar setup BYOL, i.e., self-distillation approach. In- stead of employing an MSE loss, DINO optimizes for a low cross-entropy loss between the output representations of the teacher and the student models.
(3) Vision-Language Learning. CLIP (Radford al., 2021) employs contrastive learning on large-scale noisy dataset comprising of image-text pairs. It comprises of two encoders, an image encoder (ResNet (He et al., 2016) ViT (Kolesnikov al., 2021)) and text encoder (Trans- former (Vaswani et al., 2017)), and is optimized to align the embedding spaces of the two encoders.
Thus, to summarize, we evaluate across broad spectrum of pre-training mechanisms: a contrastive, two self- siff--istiilation-eeee based. and a multi-modal pre-training paradigm, which cover some of the most popular approaches self-supervised learning. This is contrasted with fully-supervised trained models.
F.4 TRAINING DETAILS
Here we mention the training details for the pre-training (self-supervised and sully-supervised), as well as probing experiments.
Self-supervised Pre-training. We pretrain all models on the ImageNet dataset (Russakovsky et al., 2015). For each self-supervised pre-training framework, we follow the standard recipes men- tioned their respective works To keep the configuration consistent we use a batch size of 256 for all models, distributed over 4 GPUs and train for 200 epochs. The learning rate for each SSL frame- work is updated following the linear scaling rule (Goyal et al., 2017): Ir = 0.0005 x batchsize/225 /256.
Supervised Training When training fully supervised models we train the model for 100 epochs. For probing the pre-trained SSL features, on ImageNet we train the probes for 100 epochs as is standard (Caron et al., 2021; Grill et al., 2020) and for 50 epochs when training on COCO and VOC datasets. Random resize crops and horizontal flips augmentations are applied during training, and we report accuracy separately for every dataset, and use the same training procedure for all settings (except the number of epochs for which the probe is trained).
central crop. Note that do not perform any hyperparameter tuning
Probing on Pre-trained SSL Features When probing on the pre-trained SSL (and CLIP) features, as is done for B-cos (Böhle et al., 2022) models we first apply the classifier as a 1x I convolution to the feature volume and then apply Global Average Pooling (GAP) to get the class-wise logits. To keep it consistent across all models, we apply this scheme to both B-cos as well as conventional models. Also it is important to note that when using conventional probes with B-cos models, we remove all biases.
Note. For fully supervised models and probing the pre-trained SSL features we use the following training configuration: the Adam (Kingma & Ba, 2015) optimizer, a batch size of 256, and a cosine learning rate schedule with warmup. Weight decay of 0.0001 is applied only for end-to-end training
$ 'For B-cos models with DINO, Adam optimizer and learning rate of 0.001 was used
Forr VOC and COCO when applying random resized cropping, the crop size is limited within (0.7, 1.0) fraction of the original image size.
53

---

Published as a conference paper at ICLR 2025
of standard models (i.e. non B-cos models). Random resize crops and horizontal flips augmentations are applied during training.
Finally, for the CLIP vision-language model we use the checkpoint for the conventional model provided by the authors' original implementation (Radford al., 2021).
F,5 EVALUATION DETAILS
Model Performance To evaluate the model performance on single-label datasets, i.e. ImageNet we report the top-1 accuracy: which is the proportion of correctly classified samples in the validation set. For multi-label datasets, i.e. COCO and VOC we report the fl-score as is typically done (Rao et al., 2023).
Grid Pointing Game As described in Section 3.1 of the main paper, use the grid pointing game (GridPG) (Böhle et al., ) to evaluate the model explanations on single-lahel image classification datasets (ImageNet). Originally, GridPG was used quantitatively evaluate explanation methods, instead in our work we use it to compare between diferently trained models. Figure 3 in the main paper shows an illustration of our evaluation setup for 2x2 grid image. Similar to the original setting (Böhle et al., 2021; 2022), we construct 500, 3x3 image grids from the most confidently and correctly classified images for each model independently and report the mean GridPG score. In every grid all images belong to distinct classes, and for each of the corresponding class logits we measure the fraction of positive attribution an explanation method assigns to the correct location in the grid. To put it mathematically the GridPG localization score L, for the ith cell in the grid is defined as:
L Epeeeti, A+()) ΣE= Epecen, A+ (p)
where A+(p) (p) denotes the amount of positive attributions given to the 4t pixel in the n x n grid image. Also note that in the case that image has no positive attributions (all attributions are negative), we ignore that sample from the evaluation set as L, is undefined in such cases since the denominator is O.
Energy Pointing Game For both single-label and multi-label classification datasets we use the energy pointing game (EPG) to evaluate model explanations. Introduced in (Wang et al., 2020) the EPG measures the concentration of positive attributions inside the bounding box for each object class k in the image. Finally report the mean EPG score on the entire validation (or test) set. We follow the formulation as in (Rao al., 2023) for given image with height, width Hx W and htth class the EPGs score is given by:
2k=1 E Mk. A
EPGg
Σ ENEE E. Ew=l Ak
where A& E TR H denotes the attribution map for class k. and A denotes the positive part of the attributions: Mj E (0, 1) denotes the binary mask for class k that is computed by taking the union of bounding boxes for all occurrences of class k in the given image.
HxW
This evaluation setting might be used for any dataset that has available annotations in the form of bounding boxes or segmentation masks for the object classes of interest.
Pixel Deletion This evaluation procedure has been used in prior works (Samek et al., 2017; Hed- ström et al., 2024) as a reliable metric to evaluate the faithfulness of explanations derived from given attribution method. In this setting, the least important pixels (or tokens) given by an expla- nation are incrementally removed from the input. This should not affect the model's prediction and result in slow decline of the model's prediction confidence.
Compactness In their work work Chalasani et al. (2018) used the Gini index p.p. measure of quality of explanations generated for neural networks during adversarial training. The higher the compactness, the better is the explanation quality.
54

---

Published as a conference paper at ICLR 2025
Complexity The authors in (Tseng et al., 2020) used entropy measure of quality for expla- nations generated by attribution methods of deep learning models for genomics. Lower entropy of explanations is considered to indicate higher quality.
Attribution Implementation The following attribution (or explanation) methods are used in our work: B-cos (Böhle et al., 2022), Layer-wise Relevance Propagation (LRP) (Bach et al., 2015), GradCAM (Selvaraju al., 2017), Integrated Gradients (IntGrad) (Sundararajan et al., 2017), InputxGradient (IxG) (Shrikumar et al., 2017), LIME (Ribeiro et al., 2016) and GuidedBack- al., 2015). For all methods except B-cos, LRP and LIME use the implementations provided by the captum library (github com /ttpppptt/cc///ttp captum), For IntGrad, similar to (Böhle et al., 2022) we set nisleps = 50 for integrating over the gradi- 16 to accommodate for limited compute. For computing LRP attribu- tions we rely on the zennit library (https: /igbtuu.... com/ chr5tphr'' / zennit) and use the epsilon gamma box composite. Many proppaacteopopattiionn rules exist for LRP, in this work focus on the in (Montavon et al., 2019; Rao et al., 2024). For LIME attributions we use the official implementa- tion available at https: : //gittub..bu com/ www.//ooootteaeeeeec http///aactctrrriie the author provided implementation given https: com/ 3-cos/5 /B cos-vo/....
propagation (GBP) (Springenberg
ents and batch_size
gww.gammaaaaao..... _bor composite because it has been shown to work particularly well
And for B-cos attributions we use
We also evaluate on explanation methods developed specifically for Vision Transform- ers (Kolesnikov et al., 2021). In particular we use CGWI (Chefer et al., 2020), Rollout (Ab- nar & Zuidema, 2020) and ViT-CX (CausalX, (Xie et al., 2022)). For CGWI and Rollout we the author provided implementation provided at https: //////g /ggttub....//.. hom/hila-chae Trans former- Expla a lity. And for ViT-CX we use the official implementation avail- able at https:://///o / http//ggttttttt/t/// thub. com homm/vvnnxie////eemmmmmam Causalx- X-ViT.
55


Reviews:
[{"review_id": "bLsN7SSiQR", "reviewer": "Reviewer_nDZf", "summary": "The paper challenges the tradition notation that model explanations are independent of training methods by demonstrating that the quality of attributions for pre-trained models depends significantly on how the classification head is trained. It shows that using binary cross-entropy (BCE) loss instead of conventional cross-entropy (CE) loss leads to marked improvements in interpretability metrics across several visual pre-training frameworks. Furthermore, it is found that the non-linear B-cos MLP probes boost the class-specific localization ability of attribution methods.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1.\tClarity and Organization: The paper is exceptionally well-written and structured, enhancing readability and accessibility of the key finding\n2.\tThe study reveals that training probes using binary cross-entropy (BCE) loss instead of the traditional cross-entropy (CE) loss consistently enhances interpretability metrics. The analysis of the Softmax Shift-Invariance Issue in interesting and insightful. This could have substantial implications for various DNN-based applications.\n3.\tThe improvements in interpretability metrics are shown to be consistent across various training methods for the visual encoder. The robustness of these findings was thoroughly validated using diverse learning paradigms, including supervised, self-supervised and CLIP.", "weaknesses": "1.\t(Major) Limited Model Diversity: The research exclusively utilizes the ResNet50 model backbone, which  canot adequately represent the behavior across various architectures. Testing additional backbones, especially Vision Transformers (ViTs), and incorporating explanation methods tailored for these models (referenced as [1][2][3]), would provide a more robust validation of the findings.\n2.\tInclusion of Additional Methods: The paper could be strengthened by including more population perturbation-based methods, such as RISE [4] and Score-CAM [5], to further substantiate the interpretability improvements.\n3.\tSelection of Examples: Concerns arise regarding whether the examples shown in Figures 1 and 6 are cherry-picked, especially since the GridPG Score in Figure 5 suggests that the BCE model does not always perform perfectly. Including a broader range of examples, particularly where the BCE model scores lower on the GridPG, would offer a more comprehensive understanding and enhance the paper's credibility. \n\nI would be happy to improve my rating of the paper if these issues are addressed thoroughly.\n\n[1] Transformer interpretability beyond attention visualization. \n\n[2] Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.\n\n[3] Vit-cx: Causal explanation of vision transformers.\n\n[4] RISE: Randomized Input Sampling for Explanation of Black-box Models.\n\n[5] Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JByuM62GuV", "reviewer": "Reviewer_7mwF", "summary": "This paper discovers and demonstrates the strong dependence of post-hoc importance attribution methods on the training details of the classification layer of the pre-trained model. Based on this findings, the paper also proposes a simple but effective adjustment to the classification layer to significantly improve the quality of model explanations.", "questions": "1. In the case of backbone freezing, increasing classifier parameters can improve performance. Is the design of B-cos MLP necessary? Is MLP not possible?\n2. Can you provide more loss function results to verify Softmax Shift-Invariance? How about the cross-entropy loss?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper reveals and demonstrates the strong dependence of post-hoc importance attribution methods on the training details of the classification layer in pre-trained models.", "weaknesses": "1. The experimental method is limited to ResNet50, and the results are not extensive enough. Thus, experimental results are not convincing enough to verify the effectiveness of their methods.\n\n2. The contribution of this article is not enough. The author discovered the impact of training details on post-processing methods, but the evaluation metrics used and the subsequent B-cos model are not the author's innovation.\n\n3. [minor] Figures in this paper have obvious flaws. It will be better that authors carefully revise their figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TrwFnlPWGn", "reviewer": "Reviewer_VgDa", "summary": "The main motivation behind this work is that two models using the same training regime and ending at the same loss can produce two extremely different attributions for the same image. The authors demonstrate that the training paradigm for the final classification layer of a network is the most important decider in generating more precise attributions, regardless of the attribution method. They specifically show that a binary cross entropy trained output layer produces better attributions than a cross entropy trained output layer. The increase in attribution quality does typically come at the cost of <10% accuracy reduction when using a linear layer, but the accuracy can be improved by using a more complex output layer.", "questions": "1. I am still confused as to why CE produces worse attribution than BCE. Could the authors explain this again?\n    2. Also, why is it that the last output layer is so important? Why is the rest of the model have such little importance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, it is an interesting read and demonstrates some interesting results. The writing is generally clear, the issue is well defined, and the experiments are impactful. It is difficult for me to say exactly what the authors did well, other than that it is a good read. \n    \n1. In-depth motivation section, outlining the issues around generating consistently clear attributions\n2. Plenty of qualitative results\n3. Experiments over a variety of pre-trained models and datasets\n4. The authors clearly show that this is an attribution-invariant issue.", "weaknesses": "There isn't any discussion of why there is an increase in accuracy and attribution quality with more complex output layers. Is it as simple as the layers being larger, or is there another reason? I assume proper train, test, and validation sets have been used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GsALJQUTZS", "reviewer": "Reviewer_Awie", "summary": "The authors find and support an interesting observation that the method of training the classifier layer of a model has a significant impact on the results of post-hoc attribution methods. Because many post-hoc attribution methods assume that model training does not have an impact, they find that this must be reconsidered, and in fact, simply modifying the method of training the last linear layer(s) can improve model accuracy and explainability.", "questions": "Would the authors suggest the development of future classification models take into consideration the information in this paper?  \n\nDo the authors think that a training loss could be created to further improve explainability as the minor differences in CE and BCE have a significant effect?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is very well written and planned. Not only are the approaches and findings very clear but the authors provide extensive support of their findings over numerous models, datasets, attribution methods, and metrics. \n\nThe choice to study multiple pre-training approaches adds significant strength to their arguments and findings.  \n\nThe overall findings are simple, but impactful for future considerations of interpretable model design, post-hoc explainability, and improving model interpretation.", "weaknesses": "There are not significant weaknesses to address. There are minor spelling mistakes, but it does not hurt the delivery of the information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)