B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable
Authors: Shreyash Arya
Sukrut Rao
Moritz Böhle
Bernt Schiele
Venue: NeurIPS 2024 poster (2024)
Keywords: B-cos networks
Explainability
Inherent Interpretability
CLIP
Primary area: interpretability_and_explainability
TL;DR: We propose B-cosification, a technique for transforming existing pre-trained models to become inherently interpretable at a fraction of the cost to train them from scratch.

Abstract: B-cos Networks have been shown to be effective for obtaining highly human interpretable explanations of model decisions by architecturally enforcing stronger alignment between inputs and weight. B-cos variants of convolutional networks (CNNs) and vision transformers (ViTs), which primarily replace linear layers with B-cos transformations, perform competitively to their respective standard variants while also yielding explanations that are faithful by design. However, it has so far been necessary to train these models from scratch, which is increasingly infeasible in the era of large, pre-trained foundation models. In this work, inspired by the architectural similarities in standard DNNs and B-cos networks, we propose ‘B-cosification’, a novel approach to transform existing pre-trained models to become inherently interpretable. We perform a thorough study of design choices to perform this conversion, both for convolutional neural networks and vision transformers. We find that B-cosification can yield models that are on par with B-cos models trained from scratch in terms of interpretability, while often outperforming them in terms of classification performance at a fraction of the training cost. Subsequently, we apply B-cosification to a pretrained CLIP model, and show that, even with limited data and compute cost, we obtain a B-cosified version that is highly interpretable and competitive on zero shot performance across a variety of datasets. We release our
code and pre-trained model weights at https://github.com/shrebox/B-cosification.

B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable
Shreyash Arya*:1.2, Sukrut R.. Moritz Böhle* Bernt Schiele'
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrticken, Germany RTG Neuroexplicit Models of Language, Vision, and Action, Saarbrücken, Germany
Kyutai, Paris, France
(sarya, Sukrut, sukrut. rao, schielel schiele))mpiiind @mpi-inf mpg. de morit boritz@kyutai.. org
"Equal contribution
Work done while at MPI Informatics
Abstract
B-cos Networks have been shown to be effective for obtaining highly human interpretable explanations of model decisions by architecturally enforcing stronger alignment between inputs and weight. B-cos variants of convolutional networks (CNNs) and vision transformers (ViTs), which primarily replace linear layers with B-cos transformations, perform competitively to their respective standard variants while also yielding explanations that are faithful by design. However, it has so far been necessary to train these models from scratch, which is increasingly infeasible in the era of large, pre-trained foundation models. this work, inspired by the architectural similarities in standard DNNs and B-cos networks, we propose 'B- cosification', a novel approach to transform existing pre-trained models to become inherently interpretable. We perform thorough study of design choices to perform this conversion, both for convolutional neural networks and vision transformers. We find that B-cosification can yield models that on par with B-cos models trained from scratch in terms of interpretability. while often outperforming them terms classification performance at fraction of the training cost. Subsequently, apply B-cosification to a pretrained CLIP model, and show that, even with limited data and compute cost, obtain B-cosified version that is highly interpretable and competitive on zero shot performance across a variety of datasets. We release our code and pre-trained model weights at http://gtthhb.coom/bbbbo//-//oss
Introduction
Despite their strong performance variety of tasks, understanding decisions of deep neural networks (DNNs) remains challenging. Explanation methods, such as feature attributions [46, 48, 53, 5]. have been proposed in attempt to explain such decisions post-hoc, but have often found to be unfaithful to the model being explained [2, 3, 40, 60].
Inherently interpretable Deep Neural Network (DNN) models have recently gained popularity. In contrast to the common approach of explaining existing DNNs in a post-hoc fashion, these models typically feature certain architectural constraints that allow for extracting human-interrie human-interpretablle model- faithful simplifications of the models' computations by design; examples of this include prototype- based [14, 17, 32], dynamic linear [9, 10], or concept-bottleneck models [28, 58, 33, 42]. However, given those architectural changes, this comes at a price: specifically, the models need to be trained from scratch, which-especie especially in the case of large foundation models, which are increasingly popular-es can cost millions of dollars.
To mitigate this, in this work, we explore a novel approach of fine-tuning DNNs for inherent interpretability and propose to 'B-cosify' existing DNNs. Specifically, we investigate whether pre-trained DNNs can simply be efficiently fine-tuned to obtain similar degree of interpretability as 38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

B cos
B cosffeed (Oun)
DonseNeet-22 121
VIT-5
70
Gerran
60
62
Accuracy (%)
Flaaale
40-
Speedup
9.0x Speeciup
20-
30-
cosified Accuracy
10
10-
B-con Aucuracy
Goldhock
Epoch
Epoch
Fig. 1: B-cosificction: Obtaining inherently interpretable models with competitive accuracy at low cost. Left: Accuracy progression over epochs for a DenseNet-122 and VIT-S, comparing B-cosified (blue) and B-cos (orange) training curves. B-cosified models achieve equivalent accuracy with substantial reduction training time, yiclding 4.7x speedup for DenseNct-12) and 9.0x speedup for ViT-S. Right: Qualitative comparison of explanations for various images for B-cos [11] and our B-cosified models at various stages of training. Specifically, we show the dynamic linear mappings W W(x) computed by the models in color as in [10, 11]; note that by formulating conventional models ('initial' in the plot) as a specific version of B-cos models, we are able to visualise the corresponding explanations in color too, see Sec. 3.2.1 for further details. We find that after only one epoch of training, the B B-cosified models exhibit similar explanations as B-cos models.
the recently proposed B-cos Networks [10, 11]. In contrast to the original B-cos Networks, which leverage existing architectures to obtain performant and interpretable models, we investigate whether we can additionally leverage the existing pre-trained weights, thus aiming to take advantage of the significant amount of resources that have been invested in training existing models. As result, hope to make inherently interpretable models more easily accessible to the community.
To do so, we first conduct detailed analysis of how B-cos DNNs differ from their conventional counterparts. Interestingly, we find that many existing models can converted into functionally equivalent B-cos models by small set of targeted implementational modifications (Tab. 1). To increase the interpretability of the models, we then increase the 'alignment pressure' [10] via the parameter B of the B-cos transformations and fine-tune the models their respective tasks, which leads to significantly more interpretable explanations (Fig. 2).
On supervised settings, we find that B-cosified models often outperform both conventional and B-cos DNNs fraction of the full training cost (Fig. 1, left), whilst exhibiting a similar degree of interpretability as the original B-cos DNNs (Fig. 1, right). We further apply B-cosification to a pre-trained CLIP model [39], large foundation vision- language model (VLM), and show that despite using comparatively limited data and compute cost, B-cosified CLIP models yield highly interpretable explanations whilst being competitive on zero-shot performance across a variety of downstream datasets.
Our work thus opens a new perspective on how to design inherently interpretable models in a cost- effective manner. Importantly, on the one hand it highlights that conventional models might be closer inherently interpretable models than previously understood. On the other hand, it highlights the benefits of designing inherently interpretable models via minor architectural modifications, such as e.g. the B-cos DNNs, as this can allow for leveraging the large array of existing, praited DNNs.
In summary, our contributions are:
We propose B-cosification, a novel technique to 'fine-tune for interpretability'' that ad- dresses the problem of high training cost associated with obtaining inherently intepretable models such as B-cos DNNs. Our B-cosified DNNs are highly interpretable while often outperforming both standard and B-cos DNNs.
We thoroughly study different design choices to find an optimal strategy for B-cosification.
We apply B-cosification to supervised image classifiers on ImageNet [16], including both CNNs and ViT', and show that the B-cosified variants perform on par on interpretability metrics while often outperforning in terms of accuracy. Overall, we find that B B-cosifying pre-trained black box DNN to be superior on both metrics as compared to training B-cos DNN from scratch, while being computationally significantly cheaper.

---

photo
binyle
bird
boat
bentle
hus
Image
CLIP
Fig. 2: B-cosified CLIP Models. After B-cosifying CLIP model and fine-tuning it according B B-cosification scheme, Sec. 3.2, find that is possible endow the model with the same level of inherent interpretability as the B-cos models proposed in [11]. whilst maintaining CLIP's zeroshot ability (see Fig. S). The resulting linear summaries of the models ((x)) can be visualised in color (row 3) and provide significantly more detail than GradCAM explanations (row 2), which are often used to explain conventional CLIP models.
proposed
We extend B-cosification to CLIP, foundation VLM, and show that B-cosified CLIP remains highly competitive on zero-shot performance across variety of downstream datasets, while also yielding similar interpretability benefits as B-cos models.
Related Work
Explanation Methods. Post-hoc attributions [46, 43, 53, 48, 5, 56] have popularly been used to understand the decisions of trained DNNs, but have often been shown to be unfaithful to the model being explained [2, 3, 60, 40J. Inherently interpretable models [14, 28, 10], contast, incorporate architectural changes to the model and can yield explanations that interpretable and faithful the model by design. However, such models need to be trained from scratch, which imposes a significant additional cost. In this work, explore fine-tuning for interpretability, and propose method to transform existing black-box DNNs to inherently interpretable B-cos DNNs, bringing together the best of both worlds.
Attribution Priors [38, 37, 27, 50, 4] have often been used to train or fine-tune models to have explanations with desirable properties, such as inducing smoother explanations [27], consistent explanations [37, 38), or to guide models to be 'right for the right reasons' [44, 20, 41, 34j. Similar to such work, we fine-tune black-box DNNs for interpretability. but contrast, we only make architectural modifications to transform the DNNs to B-cos DNNs, and do not use any additional constraints on the explanations themselves while training.
CLIP Interpretability and Localization. Post-hoc attribution methods [46, 36, 13, 7] have also been used to explain VLMs such as CLIP [39], however, as with supervised DNNs, their faithfulness to the model is not guaranteed and the explanations often coarse graiine and not very human interpretable. While inherently interpretable architectures could address this, the high costs of training such large models from scratch makes their unappealing. In this work, bridge the gap by instead min-ttning from pre-trained black-box box CLIP models to inherently interpretable B-cosified CLIP variants, and find that the B-cosification process is effective in yielding performant and interpretable models. A separate line work [8] involves improving localizability of VIMMs, and orthogonal work since goal is to obtain explanations that faithful to the model.
Learning mappings between model features. Recent work [30, 33] has explored using simple linear transforms to map features between models, and in particular also mapping features from arbitrary models to CLIP's representation space. In the context of our work, such methods can be used to map supervised B-cos feature extractor to CLIP using linear transform, to obtain an inherently interpretable DNN that can mimic CLIP. In our evaluation, we compare with such an approach, and find that our approach of architecturally transforming the full model and fine-tuning for interpretability yields improved zero shot performance.

---

3 From conventional to B-cos models
In the following, we describe the process of fine-tuning standard black box DNNs into inherently interpretable B-cos DNNs. In Sec. 3.1, we first introduce the B-cos models and enumerate the key ways in which they differ from standard models. In Sec. 3.2, we then perform detailed study on strategies to bridge each of these differences for effective B-cosification.
3.1 B-cos Models: Background
Many common DNNs consist of a series of blocks of linear layers followed by non-linear ReLU activations [31], and are thus piece-wise linear functions': i.e., for every imput x. they effectively compute a linear transformation of that input: yxx) (xx)x + b(x). cf. [52]. In [9, 11], models of this kind have been called 'dynamic linear', a naming convention that we adopt in this paper.
Interestingly, for piece-wise linear models, (xx) is given by the models' gradient with respect to x [52]- for the input-dependent bias (xx), the gradient thus constitutes an exact summary of the models' computations. This linear mapping W (x) is unfortunately typically not easily interpretable, and many techniques have been proposed to derive qualitatively more convincing explanations [46, 563. These, however, have been shown to often not faithfully reflect the underlying model [1, 40J.
Further, if the models employ bias terms, (xx) does not yield complete explanation [52], i.c. y(x) # W(x)x. Integrating bias terms as proposed by [52] yields a set of importance attribution maps, summarizing which requires carefully selecting a post-processe pest-poccssing function with inherent tradeoffs. Even when not using bias terms [23], however, the resulting matrices W(x) are often not easily human interpretable, and the resulting models suffer from significant drops in performance.
To address this, [10, 11] propose to architecturally modify the DNNs to introduce additional aligument pressure during model optimisation. For this, they replace the ubiquitously used linear transformation by the B-cos transformation, which dynamically scales the output of the linear transformations:
B-cos transformation:
fB-cos (x; w) (Icos(x, w)1 w)
w (x)xx
(I)
with B a hyperparameter, cos the cosine similarity between x and the weights w, and w w-w/ /[w]]
Like piece-wiss linear models, B-cos models are dynamic linear and thus accurately summarised by a single linear transformation W (x) s.t. yxx = Wx)x;; as B-cos models do not employ bias terms, this model summary is complete. Crucially, it has been shown that with B> 1, the matrix W(x) aligns with rele-relevant input patterns, making it more easily human interpretable (e.g. Fig. 1, right).
Importantly, as the B-cos transformation can serve as drop-in in replacement for linear transformations at every layer of a DNN, it is possible [10, 11] to leverage existing DNN architectures and the resulting B-cos models obtain similar classification accuracies as their conventional counterparts (Tab. 4, cols. 'pretrained' and 'B-cos").
Extending this, we investigate if it is possible to leverage existing DNN weights-i.c i.c., our goal is to fine-tune existing models to be similarly interpretable B-cos models, whilst not requiring train them from scratch. However, despite the architectural similarities between B-cos and conventional models, there are multiple key differences that make transforming pre-trained models into B-cos models non triviaiii e.g., apart from replacing linear transformations with the B-cos transformation and not employing biases, B-cos models are trained on image representations with color channels as [r, b, 1 1-r; 1-1- g. 1 1- bj to be able to visualise the model-inherent linear summaries W(x) in color, whereas conventional models use channels (see also Tab. 1). In the next section, we show how to overcome these differences and convert existing models into functionally equivalent B-cos models.
3.2 B-cosification of Deep Neural Networks
We analysed the differences between B-cos models and their conventional counterparts in detail and compiled the results in Tab. 1. In this section, we discuss one by one how to bridge these differences. In particular, we show that conventional model can be framed
finctionally equivalent B-cos model as in [11] with B B-1, 1, which additionally employs bias terms. Only upon modifying these two aspects, i.e. biases and B, does the model need to be fine-tuned adapt the weights those changes.
As noted by [52]. "piece-wise linear' is actually misnomer. As the models additionally employ biases, the resulting DNNs are in fact wise affine. For simplicity, we maintain the common naming convention.

---

Table 1: Overview. To allow for comparing the models, we compiled the identified differences between the conventional models (Standard), their B-cosified version (B-cosified) and the original B-cos models (B-cos). For each design choice in the B-cosified models, we summarise the respective discussion in Sec. 3.2 (reason)).
Property
Standard
B-cos
B-cosified
reason
Image Encoding
3 channels
6 channels
6 channels
+ colored explanations
Normalized Inputs
yes
no
yes unnormalised ReLU
in-distribution (ID)
Weights
unnormalised
normalised none
equivalent and ID
Activations
ReLU
compatible and ID
Biases
yes no
no 2
+ complete explanations
B in B-cos
2
weight-input alignment
3.2.1
Functionally Equivalent B-cos Models
Input Encoding and Normalisation. As mentioned in Sec. 3.1, B-cos models use input represen- tations with six color channels [r. 1 1-r, 1-1. b] to be able to visualise the explanations in color, cf. [11]. However, most conventional DNNs (e.g. models from Torchvision [54]. CLIP [39]) are applied 3-channel inputs in which images are encoded via (r. g. b]. As result, visualising the dynamic matrices W (x) of piece-wise linear models (cf. Sec. 3.1) in color would not seem possible.
However, note that in combination with the commonly used input normalisation, we can convert the first linear transformation in conventional models (e.g., a convolutional layer) into an equivalent transformation that accepts 6-channel inputs. Specifically, for input normalisation, the channel-wise means 14a are subtracted from the individual channels, followed by a division by the standard devia- tions g. yielding s' (s /a, for S (r, g, b). Conversely, mean-normalising the 3 additional color channels yields ss' Leveraging this, we use the models' weights learnt for 3-channel inputs, W, [wy.r, Wj.g: W5 for every feature j. to construct an equivalent 6-channel transformation:
W Wig
5
W.r
Wig
W
w
(2)
2
2
Note that applying w to the mean-normalise 6-channel inputs yields the same results as applying Wj to the original mean-normalised inputs that the pre-trained models have seen during training.
Owing to the non-linearity inherent to the B-cos transform, explicit activation functions are not necessary in between B-cos layers. However, the authors of [10, 11] showed that the model-inherent explanations are compatible with MaxOut [21]. Note that the very commonly used ReLU non-linearity applied to v x for any weight vector is just special case of MaxOut:
Activation Functions.
MaxOut(x; v, 0) = max x, 0mxx) x) = ReLUUv x).
o
(3)
As the pre-trained models' weights have been optimised for the ReLU non-linearity and given its compatibility with the B-cos explanations, we leave them untouched in the B-cosification process.
Weight normalization. B-cos transformations employ unit norm weights, see also Eq, (1), which the authors motivated by the fact that the only way any given neuron can achieve its maximal output is by increasing the weight-input alignment, which turns leads to the improvements of the explanations.
However, conventional models have been trained with unconstrained weights and using unit norm weights would thus lead unpredictable model behaviour. Interestingly, we note that the weight normalisation in the latest version of the B-cos models can actually not impact the explanation quality, as the authors of [11] re-introduce normalisation layers into the B-cos models. To better understand this, let us consider the compound function of a batch normalisation layer and a B-cos layer:
f(x) = BatchNorm B-cos( xx)
with BatchNorm(y)) = a x y mean(y) + 1.
(5)
vva(y)
with a and 3 trainable parameters of the BatchNorm layer. Note that /var is scaled by any factor
by which the output of B-cos layer might be scaled, which cancels in the fraction Eq. (5) and thus makes of fx) invariant to scaling the B-cos transformation: i.c. BatchNorm(y)) = BatchNorm(7> x y).
5

---

Table 2: Increasing B for B-cosification.
R8
Baselines
Discrete B=1.5
Linear
Learnt
Metric
Standard
B-cos
B=1
B=2
5 cpo.
45 epo.
90 epo.
Accuracy
69.6202
68.5+02
70.6
71.6-01
71.5+0
71.6-02
71.1=a1
70.2=00 88.8203
71.8=a1
Localisation
21.4
87,4205
33.9202
84.3=02
87.6202
18.1
88.8202
89.4
In particular, the output of f(x) is thus invariant to weight normalisation, as the output of B-cos (x) scales linearly with the weight norm, cf. Eq. (1).
This is of course only true if every B-cos layer were always followed by a normalisation layer, which is not necessarily the case. Nonetheless, we find that not using normalised weights yields consistently good results across all models. Therefore, we use B-cos transformations without weight normalisation throughout our experiments; for an ablation, see Tab. B2 in the appendix.
In summary, we showed that it is possible to adapt the implementation of existing models in a way that allows us to integrate certain aspects of B-cos models without functionally changing the pre-trained models. Notably, we can now visualise color explanations similar to B-cos models (Fig. I, right, col. 3); unsurprisingly, however, these explanations have poor interpretability due to the absence of the alignment pressure imposed during B-cos training. In the next section, we discuss the necessary functional changes for B B-cosification to obtain interpretable explanations.
3.2.2 Fine-tuning for Interpretability
The changes introduced in the preceding section have not functionally changed the pre-trained models, but rather allow us to interpret the existing models as special case of B-cos models. Now we introduce the necessary changes to increase the interpretability of the dynamic matrices W(x). As these functionally change the models, they need to be fine-tuned to recover their original performance.
In particular, the remaining differences between conventional and B-cos models are (1) the value of B, and (2) the use of biases, Tab. 1. We will now discuss how we bridge these differences individually.
Ablation Setup. We evaluate various fine-tuning strategies using a ResNet-18 [22] model supervised on ImageNet [16] from Torchvision [54] for B-cosification, and compare with B-cos ResNet-18 from [1]]. We optimize using Adam W [26] with cosine scheduling and train for 90 epochs, and evaluate both classification accuracy well as interpretability using the GridPG metric [9].
(1) Increasing B. As shown in [10], using B >>1 is critical to obtain easily interpretable explanations. To increase B for the praitee models, we investigate three strategies: (1) immediately setting B to higher value and then fine-tume fine-tuning, (2) linearly interpolating from B = to B = 2 throughout fine-tuning, and (3) setting B as a learnable parameter. (2) has the advantage of changing the model in small steps, making it more likely that it maintains performance while fine-tuning, but requires using the full number of epochs to reach the target value of B. (1) on the other hand is likely to adversely affect the utility of the weights, but offers the opportunity to stop fine-tuning early if performance and interpretability metrics are sufficiently high. (3) offers the most flexibility, but also adds a new set of parameters that need to be optimized. We show the results of this evaluation in Tab. 2. Interestingly, we find that using (I), i.c. setting B = 2 and then fine-tuning, yields performance that is on par with learnable B parameters, whilst being significantly simpler to implement. To easily test the generality of the B- B-cosification scheme, we therefore opt for this approach in Sec. 4.1.
(2) Decreasing biases. As discussed in Sec. 3.1, dynamic linear models with bias terms are not exactly summarised by the matrix W(x), cf. [52]. To obtain the same level of faithfulness of the explanations as B-cos models (in particular w.r.t. explanation completeness, cf. [52, 53]), we need to remove the biases from the model. To do so, we investigate two approaches: (1) removing all biases first and then fine-tuning, and (2) fine-tuning while decaying biases using weight decay. Similar to the setup with B, (2) has the advantage of avoiding drastic changes to the model, but requires potentially fine-tuning for longer. Further, the weight given to the bias decay in the loss constitutes a tradeoff between maintaining classification performance and pushing the biases to be close to zero. We report the results of this evaluation in Tab. 3. Similarly to the experiments for B, we find that immediately setting the biases to zero constitutes simple yet performant approach to achieve both good localisation and accuracy. To assess the generality of the B-cosification scheme across wide range of models, we thus choose this the simpler approach of setting biases to zero in Sec. 4.1.

---

Table 3: Decreasing biases for B-cosification.
ResNet-18
Baselines
Fixed bias
Bias decay
Metrie
Standard
B-cos
With bias
No bias
decay=0.2
decay=0.5
dccay=0.9
Accuracy
69.6102
68.5
71.2
71.5
71.2.02
71.4
716612
Localisation
21.4202
87.4205
47.2
87.6202
81.420a
90.2202
91.2201
In short, we find that a very simple approach, i.e., setting the bias and the B values to the target values immediately, constitutes simple and casy-to-use, but nonetheless performant strategy to B-cosify models. In the following sections, we test whether these findings generalise well to other models.
B-cosification Results
In the following, evaluate the effectiveness of the B-cosification strategy developed in Sec. 3. In Sec. 4. we first apply B-cosification to supervised models across various architectures, and evaluate for classification performance and interpretability. In Sec. 4.2, we B-cosify CLIP [39]. large foundation vision-lang lasion-language model, and show that despite fine- mee-tuning at fraction of the training cost, the B-cosified CLIP shows strong zero shot generalization whilst being highly interpretable.
4.1 Supervised Classification Models
Table Classification Accuracy. We report the top-1 classification accuracy the ImageNet validation of the pre-trained models (pretrained) the B-cosified models (B-cosified) after fine-tuning them. Additionally, we report the accuracy of the corresponding B-cos models trained from scratch (B-cos) as well as difference to them (Axc), and how much faster and at which epoch (t) the same accuracy as in [11] was achieved (speedup). Results for B-cosified models averaged over three runs; full results including standard deviation in appendix.
Top-1 Accuracy (%)
Efficiency Gains
Model
pretrained
B-cos [11]
B B-cosified
A acc
t
speedup
ResNet-18
69.8
68.7
71.5
+2.8
29
x3.1
ResNet-50-vl
76.1
75,9
76.5
+0.6
46
x2.0
ResNet-50-v2
80.9
75.9
77.3
+1.4
10
x9.0
DenseNet-121
74.4
73.6
76.3
+2.7
18
x5.0
ViT-Ti
70.3
60.0
69.3
+9.3
10
x9.0
VIT-S
74.4
69.2
75.2
+6.0
10
x9.0
ViT-B
75.3
74.4
75.3
+0.9
x1.6
VIT-L
75.8
75.1
75.5
+0.4
66
x1.4
VTT.-Ti
72.6
67.3
72.3
+5.0
10
x9.0
-
75.7
74.5
76.0
+1.5
32
X2.8
VBT.-B
76.8
77.1
76.7
-0.4
VIT.-L
77.9
77.8
77.1
-0.7
Setup. We B-cosify models from Torchvision [54] supervised on ImageNet [16]. We use diverse set architectures, including both CNNs (ResNet-18 [22], ResNet-50 [22], and DenseNet-121 [24]), and ViT's [18, 6, 57] with -Ti -ST -B, VTT.-L) and without (ViT-Ti, ViT-S, ViT-B, ViT-L) convolutional stems. For ResNet-50, we use both the weights originally released by Torchvision and the updated V2 weights, which constitute models trained for longer and with more augmentations [55]. As Sec. 3.2, we evaluate both for classification accuracy and for interpretability using the GridPG [9] metric. We compare both accuracy and interpretability of the B-cosified models with B-cos models trained from scratch from [1]]. For interpretability. we also compare with several post-hoc attribution methods baselines, namely Guided Backprop [51], Gradient [49], DeepL.IFT [48], IxG [48], IntGrad [53], and GradCAM [46]. For full details, see Appendix C.I.
Classification performance. Tab. 4 reports the classification accuracy of the B-cosified models across architectures, and compares them with their conventional counterparts from Torchvision and B-cos models trained from scratch. We find that across architectures (col. 1), B-cosified models perform competitively with conventional DNNs (cols. 2-4) and interestingly, in contrast to the findings reported by [11], often outperform them, i.e. for five out twelve architectures. Notably, find (col. 5) that our B B-cosified models significantly outperform B-cos models trained from
7

---

Standard Model Wis's
B-cosifed Model Wisis
B-cos Model W(sjs
Localisation (%)
+4.2
++6.
+43.7
+00L3
-43.0
+224
14
-689
1874
+9.4
-PL.
60
Reshatt18 18
Reghitt
Rastie v0
Densaiter- 121
VI-II
Wi-S
Vi-BB
VIFL
VIT-Ti
VIT
WIT-B
VITAL
Fig. 3: Localisation Performance of (x))x. x. We compute the contribution maps according to the dynamic linear summaries W(x) of the pre-trained models ('Standard'), their B-cosified versions, and the original pre-trained B-cos models and evaluate their localisation performance the Grid Pointing Game in [11]. We find localisation significantly improve for B-cosified models, achieving results on par with models of [1 []]
ResNet-50.v1
Cracks attriattons
Uniterm attribation
DenseNet-12
Localisation (%)
DeepLIFT
Intrrad
Gra
B-cooified
Fig. 4: Comparison to Post-hoe Methods, For two of the models in Fig. (ResNet-50-v1, DenseNet- 121) compare the localisation performance of the dynamic matrices W(x to post-hoc explanations for the pre- trained models. Similar to the original B-cos models [11], the model-inherent explanations perform favourably.
scratch across all but the two largest ViT, architectures. Further, B-cosified models often achieve the same performance as their corresponding B-cos models at fraction of the training cost (col. 6). Specifically, we find that averaged across architectures, B-cosified models outperform B-cos models trained from scratch by 2.5 pp, with an average training speedup (to match performance) of 5. 5.2x. These results strongly advocate for B-cosification as a superior alternative training from scratch for obtaining performant inherently interpretable models at low compute cost.
Interpretability. To evaluate the interpretability of our B-cosified models, we report the GridPG localization scores in Fig. 3, and compare with conventional and B-cos models; following [11], report the results 3x3 image grids for convolutional models, and of 2x2 grids for the ViTs. For a fair comparison, for all models, W (x)x (see Sec. 3. 1). We find that across architectures, B-cosified models significantly outperform conventional DNNs in terms of localization (32. 222.7pp-71.0pp
evaluate the localization of the dynamic lincar summary of the model
and perform on par with B-cos models. Since post-hoc attribution methods (e.g. [46, 48, 53)) are often used to interpret conventional DNNs, similar to [10], in Fig.
compare the localization of the model inherent explanations from two of our B-cosified models with post-hoc explanations applied to the corresponding conventional models. Similar the results reported by [10], find our B-cosified models strongly outperform all post-hoc methods, including GradCAM [46], with near perfect localization score, showing that B-cosification is effective in yielding highly interpretable yet model-faithful explanations.
Impact of pre-trained weights. Since our aim is to fine-tune for interpretability, we investigate how crucial the quality the weights of the conventional model are for effective B-cosification. Specifically, we expect weights from stronger models to be a better starting point for B B-cosification. We evalaute this by performing B-cosification both with vl and v2 variants of ResNet-50 [22] from Torchvision [54], where the latter is trained for longer and with stronger augmentations [55]. From Tab. 4, we find that using strong initialization highly useful for effective B-cosification,
"Note that ViTs differ from CNNs discussed in Tab.
the attention mechanism and the GELU
(i((z//z). activation with GELU(X) = x (0.5 + 0.5 x erf((z) /τ). As attention is also dynamic linear, cf. [11], it can seamlessly be integrated into the model summary W (x). Similarly, we interpret the second factor in GELU as dynamic weight wxr). thus allowing to integrate it in similar fashion.

---

Zero-shot Evaluation
Linear Probe Evaluation
Texe2Cancept
B-casified (IMN)
B-cosfied (CC3M)
Standard
100
50.9 #2.5
$5.5
Accuracy (%)
80
68.5
76.7
77.8
60
47.2
44.7
$1.5
45.2
48.0 48.5
39.0
34.6
34.5
20
9.0
10.5
10.3
Natural Data
Specialized Data Structured Data
Natural Data
Specialized Data Structured Data
Fig. 5: Classification performance the CLIP Benchmark [15] of various CLIP models for the zero-shot setting (leff) and linear probing (right). Specifically, we compare two B-cosified CLIPs trained on ImageNet (IMN) and CC3M respectively-tte to the Text2Concept approach by [30] and the original pre-trained CL.IP model. We find B-cosified versions of CI.IP to consistently outperform Text2Concept on natural and specialised data.
specifiiallly, fine-tuning starting from v2 weights outperforms B-cos models by 0.8pp as compared to vl, and achieves equal accuracy with 9x speedup compared to 2x with vl; similar results are observed for initialising the weights with models pretrained via the self-supervised DINO paradigm [12] (final accuracy: 77.0, speedup: 3.2x), for further discussion see Tab. B3 in the appendix.
4.2 B-cosifying CLIP Towards Inherently Interpretable Foundation Models
In this section, we evaluate our B-cosification paradigm CLIP [39], a powerful pre-trained vision-language model, and evaluate its interpretability and zero shot performance.
Setup. We B-cosify CLIP [39] with ResNet-50 [22] backbone using the procedure described in Sec. 3.2. We use the recently proposed SigLIP loss [59] between the image embeddings of the pre-trained CLIP and the B-cosified CLIP's and train the models on either the ImageNet [16] the CC3M datasets [47]. For evaluation, we rely on the CLIP Benchmark [15] and report zeroshot and linear probing results for accuracy. To assess the models' interpretability, we explain the similarity between the image embeddings and the text embedding of the pre-trained CLIP model via the dynamic linear summaries, see Sec. 3.1 or GradCAM, and report the EPG scores [56, 41] on the VOC dataset [19]. For full details, see Appendix C.2.
Evaluating Model Performance. Fig. 5, we report the zeroshot and linear probing accuracies of the two B-cosified CLIP models (trained on ImageNet or CC3M) and compare it to the original CLIP (Standard) and the recently proposed Text2Concept technique [30]; for the latter, we train linear layer on top of a frozen, pre-trained B-cos ResNet-50 from [11] to mimic the embeddings of CLIP [30]. We find that the B-cosified models significantly outperform the Text2Concept approach and achieve accuracies that are more similar to the original CLIPP's zeroshot and linear probing accuracies.
Evaluating Model Interpretability. We evaluate the B-cosified CLIP's ability to localise classes in the VOC dataset in two ways. On the one hand, directly explain the similarity of the models' embedding to the text embedding of a given prompt such as "A photo of a cow.". On the other hand, note that the final attention pooling layer in the CLIP model only computes a weighted sum of the last layer's value vectors. Therefore, we additionally evaluate whether we can also explain the similarity between the text embeddings and these value vectors to improve the localisation ability.
In this context, we notice that explaining the average similarity to the text embedding yields highly distributed attribution maps, see Fig, 6b, col. 2. On the other hand, explaining only the most similar embedding localises very well, see Fig. 6b, col. 5. To better understand this phenomenon, we additionally interpolate between these two approaches and compute weighted means Σ, W,V, of those value vectors Vi, in which the weights are determined by the cosine similarity between the value vectors v, and the text embedding t, i.e. with weights w; cos" v:) for various p.
We find that this not only significantly improves the explanations qualitatively, see Figs. and 6b, but also quantitatively: in Fig. 6a we report results for explaining the final image embedding (B- cosified CLIP), the dynamic linear summary for the CLIP ResNet-50 (CLIP W (x)x), its GradCAM explanations (CLIP GradCAM), and the weighted mean of the value vectors, which we call

---

04
EPG Score (%
B cooked FeatureLIP
Sutelne ON
GodCaM
CUP Wix+s
cootiee CUP
15
Cosine Power
(b)
(a)
Fig. 6: CLIP Localisation. (a), we compare GradCAM and dynamic linear explanations for the pre-trained CLIP model to the inherent explanations of the B-cosified CLIP, as well as to our proposed B-cos FeatureCLIP approach. We find that good localisation with highly detailed explanations (b) is possible via the B B-cosified CLIP models, especially for high cosine powers, despite only explaining the similarity text prompts.
B-cosified FeaPureCLIP. While B-cos CLIP already yields noticeable improvement over the pre-trained CLIP explained via GradCAM and significantly improves the localisation ability of the linear summary Wxx),, the weighted means yield even stronger performance for sufficiently high p.
5
Discussion
The B-cosification approach presented in this work addresses a common issue with developing inher- ently interpretable models: achieving model interpretability without compromising performance or incurring high training costs. By leveraging pre-trained models, B-cosification opens a new path towards developing interpretable yet performant models, which can be of particular interest in the context of foundation models such as CLIP [39], which might otherwise be prohibitively expensive train on a limited budget. Our results suggest that B-cosification not only maintains but, several cases, even enhances model accuracy, whilst yielding significant improvements on interpretability metries, providing viable and resource-efficient alternative to training B-cos models from scratch.
Specifically, we find B-cosified models to much faster reach the same levels of interpretability and accuracy than their counterparts trained from scratch, with training speedups of up 9x in some models. The approach appears to be general, being applicable for both CNNs and ViT models. We hope that this increase in efficiency will make interpretable models much more accessible settings with constrained computational resources and could thus facilitate their adoption. In particular, when applying our proposed B-cosification scheme a foundation model-CLIP-ww find that the B B-cosified CLIP model is able to maintain competitive zero-shot performance while at the same time providing interpretable and fodel-faithful explanations.
Despite these advancements, certain aspects remain open for further exploration. Specifically, while some models quickly recover original performance after B-cosification, others exhibit slower convergence rates, suggesting potential for optimisations in the fine-tuning process. Additionally, for the larger B-cosified ViT, models, while yielding results that scratch, the B-cosification process did not succeed in achieving speed-ups, indicating that the interplay between model architecture and the proposed B-cosification might require further exploration.
par with those trained from
summary, our results establish B-cosification as an effective method for enhancing interpretability in pre-trained models with low computational cost. The method consistently enables high interpretabilitty without compromising performance, even achieving substantial training speedups in many cases.
Acknowledgements
Funded in part by the Deutsche Forschungsgemeinschaaft (DFG, German Research Foundation) GRK 2853/1 "Neuroexplicit Models of Language, Vision, and Action" - project number 471607914.
10

---

References
[1] Adebayo, J. Gilmer, I. Goodfellow, and B. Kim. Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values. In ICLRW, 2018.
[2] Adehayo, J. Gilmer, M. Muelly, Goodfellow, M. Hardt, and B. Kim. Sanity Checks for Saliency Maps. In NeurlPS, 2018.
[3] Adebayo, M. Muelly, H. Abelson, and B. Kim. Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation. In ICLR, 2022.
[4] S. Asgari, A. Khani, F. Khani, A. Gholami, L. Tran, A. Mahdavi-Amiri, and G. Hamarneh. MaskTune: Mitigating Spurious Correlations by Forcing to Explore. In NeurIPS, 2022.
[5] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation. PloS one, 10(7):e013010, 2015.
[6] L. Beyer, X. Zhai, and A. Kolesnikov. Better plain vit baselines for imagenet- 1k, 2022. URL https://arxiv org/ 2225 211.//205.0.00..0.2
[7] W. Bousselham, A. Boggust, S. Chaybouti, H. Strobelt, and H. Kuehne. LeGrad: An Ex- plainability Method for Vision Transformers via Feature Formation Sensitivity, arXiv preprint arXiv:2404.0322 2024.
[8] W. Bousselham, F. Petersen, V. Ferrari, and H. Kuehne. Grounding Everything: Emerging Localization Properties in Vision-Language Transformers. In CVPR, 2024.
[9] M. Böhle, M. Fritz, and B. Schiele. Convolutional Dynamic Alignment Networks for Inter- pretable Classifications. In CVPR, 2021.
[10] M. Böhle, M. Fritz, and B. Schiele. B-cos Networks: Alignment is All We Need for Inter- pretability. In CVPR, 2022.
[11] M. Böhle, N. Singh, M. Fritz, and B. Schiele. B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers. IEEE TPAMI, 2024.
[12] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties in Sele-Supervieed Vision Transformers. In ICCV, 2021.
[13] H. Chefer, S. Gur, and L. Wolf. Generic Attention-Model Explainability for Interpreting Bi-Modal and Encoder- Decoder Transformers. In ICCV, 2021.
[14] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. In NeurlPS, 2019.
[15] CLIP Benchmark maintainers and contributors. CLIP Benchmark. https://githut com/ LAION-AI CIIN-AAA/LLP..encharrk. benchmark, 2024.
[16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
[17] Donnelly, A. J. Barnett, and C. Chen. Deformable ProtoPNet: An Interpretable Image Classifier using Deformable Prototypes. In CVPR, 2022.
[18] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In ICLR, 2021.
[19] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. IJCV, 88:303-308. 2009.
[20] Y. Gao, S. Sun, G. Bai, S. Gu, S. R. Hong, and Z. Liang. RES: Robust Framework for Guiding Visual Explanation. In KDD, 2022.

---

[21] I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout Networks, In ICML, 2013.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[23] R. Hesse, S. Schaub- MMyer,, and S. Roth. Fast Axiomatic Attribution for Neural Networks, In NeurIPS, pages 19513-19524, 2021.
[24] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely Connected Convolutional Networks, In CVPR, pages 4700-4708, 2017.
[25] A. Khaddaj, H. Salman, A. Ilyas, G. Leclerc, and A. Madry. Extra Training Provides Strong Baseline for CLIP. In Fouuuteuussss of Few-shot and Zero-shot Learning in Large Foundation Models, 2023, URL
[26] D. Kingma and J. Ba. Adam: Method for Stochastic Optimization. In ICLR, 2015.
[27] K. Kiritoshi, R. Tanno, and T, Izumitani. L1-Norm Gradient Penalty for Noise Reduction of Attribution Maps. In CVPRW, pages 118-121, 2019.
[28] W. Koh, Nguyen, Y. Tang, S. Mussmann, E. Pierson, B. Kim, and Liang. Concept Bottleneck Models. In ICML, pages 5338-5348, 2020.
[29] N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Reblitz Richardson. Captum: A Unified and Generic Model Interpretability Library for PyTorch. arXiv preprint 1xXiv:2009.07896, 2020.
[30] M. Moayeri, K. Rezaei, M. Sanjabi, and Feizi. Text-to-Concept (and Back) via Cross-Model Alignment. In ICML, pages 25037-25060, 2023.
[31] V. Nair and G. E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML, pages 807-814, 2010.
[32] M. Nauta, R. Van Bree, and C. Seifert. Neural Prototype Trees for Interpretable Fine-grained Image Recognition. In CVPR, pages 14933-14943, 2021.
[33] T. Oikarinen, S. Das, L. M. Nguyen, and T.-W. Weng. Label-Free Concept Bottleneck Models. In ICLR, 2023.
[34] A. Parchami-Araghi, M. Böhle, Rao, and B. Schiele. Good Teachers Explain: Explanation- Enhanced Knowledge Distillation. In ECCV, 2024.
[35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An Imperative Style, High- Perf-Peerfomance Deep Learning Library. NeurlPS, 2019.
[36] S. Petryk, L. Dunlap, K. Nasseri, J. Gonzalez, T. Darrell, and A. Rohrbach, On Guiding Visual Attention with Language Specification. In CVPR, pages 18092-181000
19102, 18002 2022.
[37] V. Pillai and H. Pirsiavash. Explainable Models with Consistent Interpretations. In AAAI, 2021.
[38] V. Pillai, S. A. Koohpayegani, A. Ouligian, D. Fong, and H. Pirsiavash. Consistent Explanations
by Contrastive Learning. In CVPR, pages 10213- 2022.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML, pages 8748-8763, 2021.
[40] S. Rao, M. Böhle, and B. Schiele. Towards Better Understanding Attribution Methods. In CVPR, pages 10223-10232 2022.
[41] S. Rao, M. Böhle, A. Parchami-Araghi, and Schiele. Studying How to Efficiently and Effectively Guide Models with Explanations. In ICCV, pages 1922-1933, 2023.
12

---

[42] S. RRa, S. Mahajan, M. Böhle, and B. Schiele. Discover-then-Name: Agnooiti Concept Bottlenecks via Automated Concept Discovery. In ECCV, 2024.
[43] M. T. Ribeiro, S. Singh, and C. Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier. In KDD, pages 1135-1144, 2016.
[44] A. S. Ross, M. C. Hughes, and F. Doshi-Velez. Right for the Right Reasons: Training Differen- tiable Models by Constraining their Explanations. In ICAI, pages 2662-2670, 2017.
[45] O. Russakovsky. J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 15(33)21112222 2015.
[46] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. ICCV, pages 618-626, 2017.
[47] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual Captions: A Cleaned, Hyper- nymed, Image Alt-text Dataset for Automatic Image Captioning. In ACL, pages 2566-2565, 2018.
[48] A. Shrikumar, P. Greenside, and A. Kundaje. Learning Important Features Through Propagating Activation Differences. In ICML, pages 3 3145-3153, 2017.
[49] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In ICLRW, 2014.
[50] K. K. Singh, D. Mahajan, Grauman, Y. J. Lee, M. Feiszli, and D. Ghadiyaram. Don't Judge an Object by its Context: Learning to Overcome Contextual Bias. In CVPR, pages 11070-11078, 2020.
[51] J. Т. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for Simplicity: The All Convolutional Net. In ICLRW, 2015.
[52] S. Srinivas and F. Fleuret. Full-Gradient Representation for Neural Network Visualization. In NeurlPS, 2019.
[53] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic Attribution for Deep Networks. In ICML, pages 3319-3328, 2017.
[54] Vision maintainers and contributors. Torch PyTorch's Computer Vision Library. https://githut: cww/ hamp/yytocch/viiie wwtcch//viiin,.. 2016.
[55] V. Vryniotis.
How to Train Visioons Latest Primitives.
Models Using https://pytore: httpttorrc..rggbloggge ofeo-of-hhe thee art-modeessur uuin--using-tttn torchhvision-latet----iii. 2021. Accessed: 2023-05-23.
State-of-the-Ar
how-to-
[56] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu. Score-CAM: Score-Weer Weighted Visual Explanations for Convolutional Neural Networks. In CVPRW, pages 111-119, 2020.
[57] T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Girshick. Early convolutions help transformers see better, 2021. URL https::/t////ttt //apxii./////..////e//as org/ h/tg//b//// abs.///6. 14881.
[58] M. Yuksekgonul, M. Wang, and J. Zou. Post-hoc Concept Bottleneck Models. ICLR, 2023.
[59] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid Loss for Language Image Pre- Training. In ICCV, pages 11975-11986 2023.
[60] Y. Zhou, S. Booth, M. T. Ribeiro, and J. Shah. Do Feature Attribution Methods Correctly Attribute Features? In AAAI, pages 9623-9633, 2022.
13

---

B-cosification:
Transforming Deep Neural Networks to be Inherently Interpretable
Appendix
Table of Contents
In this supplement to our work on B-cosification of black box models, we provide:
(A) Additional Qualitative Results
15
(B) Additional Quantitative Results
17
Implementation Details
23
14

---

A Additional Qualitative Results
In Fig. Al, we provide additional qualitative examples illustrate the interpretability gains achieved by B-cosifying CLIP model. Specifically, we show explanations generated by the original CLIP model using GradCAM [46] (row 2) for diverse set of input images (row 1). for which explanations are generally coarse and lack clear localization. In contrast, the third row displays explanations produced by B-cosified CLIP, which yields finer-grained, more visually interpretable explanations.
bottle
chair
dog
dog
bird
Image
CLIP
B-cosified CLIP
Fig. A1: Additional, randomly sampled examples for comparing GradCAM explanations of the original CLIP model the inherent explanations of the B-cosified CLIP. The top row shows input images from various classes. The middle row provides explanations generated by the original CLIP model, which tend to coarse and lack precise localization. The bottom row shows explanations from B-cosified CLIP, which produce more focused, detailed visualizations, highlighting class-relevant features with greater clarity.
In Fig. A2, we show further comparisons on specific object classes with using different cosine powers (cos, cos-7, cos-19, and cos-inf) to qualitatively demonstrate the effect of increasing the exponent p in gathering the value vectors, see also Sec. 4.2. Higher cosine thresholds result in increasingly focused and interpretable representations, capturing fine details that are often absent in the original CLIP explanations.
In Fig. A3, we show additional qualitative examples for prompting the B-cosified CLIP model with different prompts for the same image, thus highlighting the class-specificity the explanations as well as the potential that inherently interpretable CLIP models might yield. Specifically, B-cosified CLIP models allow to explain the similarity of a given image with a free-form textual prompt, which shows that the zero-shot performance of CLIP with respect to classification also transfers well to the corresponding explanations.
aeroplane
cos
COS-7
cos-19
cos-inf
pottedplant
COS
COS-7
cos-19
cos-inf
bird
dog
dog
dog
Fig. A2: Additional randomly chosen examples highlighting effect of increasing cosine power p on specificity of the explanations in the B-cosified CLIP model. Each row corresponds to a specific object class, with explanations generated at different cosine power levels: cos, cos-7, cos- 19, and cos-inf. Higher cosine power values result in increasingly precise and interpretable representations, capturing finer details and producing sharper focus on class-relevant features; further examples in Fig. 6h in the main paper.
15

---

ice Cream
Ice Cream
Wattr
wottr
heart
hand
Schconor
skip
Pop Bottle
grhen
orang
while
Fig. A3: B-cosified CLIP text-based localisation. We find different objects can be localised in high-quality color using B-cosified CL.P. For example, we compute B-cos explanations using text-prompts: a picture of ('x') enended using standard CLIP [39] text-encoder for image taken from ImageNet [45]. The first column shows the original image with the class caption, the second column shows the colored B-cos explanations with the text caption used at the top, the third column shows the raw attributions with denoting the positive contribution towards the text input and blue denoting the negative contribution.
16

---

B Additional Quantitative Results
In this section, we provide a series of additional quantitative results on the performance and in- terpretability of B-cosified models. These tables cover various ablation studies, comparisons with standard and B-cos models, and performance across different configurations. Specifically, in Tab. BI, we extend our evaluation to compare models that are trained for the same effective number of epochs'. Further, in Tab. B2, we evaluate the impact of using normalized weights the B-cos layers. In Tab. B3, we show additional results for B-c B-cosified ResNet-50 models that are initialised from different pre-trained checkpoints. Specifically, we find that apart from initialising the ResNer-50 from CLIP weights, observe consistent improvements through B-cosification. with stronger pre-training paradigms that are based on the ImageNet dataset (V2, DINO [12]) leading to larger improvements. Finally, report full ablation results the impact of the individual changes that we perform on the pre-trained models Tab. B4, B-cosifying models with different strategies for setting the parameter in the B-cos transformation Tab. B5, decaying the bias term Tab. B6, as well as full zero-shot and linear probing results for the CLIP benchmark, see Tabs. B7 and B8.
Table Classification accuracy comparison with further trained standard and B-cos models. Extending Tah. 4, we report the comparison of top-1 classification accuracy on the ImageNet validation set between the B-cosified models and comparison with standard (block 3) and B-cos (block 4) pre-trained models fine-tuned using the same process B-cosification. All models are thus effectively trained for 180 epochs. Results for B-cosified models averaged over three
Training for 180 Epochs
B-cosified acc
Standard models
B-cos models
Model
acc
A
speedup
acc
A
speedup
ResNet-18
71.5
71.0
+0.3
x x1.410.1
68.9
+2.4
X2,810.0
ResNet-50-vl
76.5
76.8
-0.3
75.6
+0.9
X2,410.1
ResNet-50-v2
77.
77.7
-0.4
75.6
+1.7
x x10.440.7
DenseNet-121
76.3
76.1
+0.2
x1.4
73.8
+2.5
x5.2404
ViT-Ti
69.3
72.1
-2.8
63.7
+5.6
x3.440.1
ViT-S
75.2.1..1
76.4
-122
72.5
+2.7
x2.7100
ViT-B
75.3
75.8
-0.5
75.3
ViT-L
75.5
75.7
-0.2
75.6
-0.1
ViT.-Ti
72.3+0.1
74.8
-2.5
70.1
+2.2
x1.8+00
ViT.-S
76.0
76.9
-0.9
75.9
+0.1
x1.3
ViT,
76.9
77.2
-0.3
77.2
-0.3
VLT
77.1
77.6
-0.5
77.7
-0.6
'B-cosified models have effectively been trained for 180 epochs, i.e., 90 epochs of pre-training and 90 epochs fine-tuning To fairly compare impact of additional training, we show how this would impact both pre-trained models themselves, as well as the B-cos models from [11]. Note that however the primary goal to leverage pre-trained weights to more efficiently train B-cos models, and as shown in Tab. 4, B B-cosification helps obtaining similarly accurate and interpretable models at much lower training cost.
17

---

Table B2: Ablation normed weights for the VIT models. Extending Tab. 4 for subset of models, report the top-1 classification accuracy on the ImageNet validation of the pre-trained models (pretrained) and the B-cosified models (B-cosified) Additionally, we report the accuracy of the corresponding B-cos models trained from scratch (B-cos) as well as the difference to them (Acc)). and how much fäster and at which epoch (t) the same accuracy as in [11] was achieved (speedup). Results for B-cosified models are averaged over three
fine-tuning them along with difference between them (A
Top-1 Accuracy (%)
Gains over B-cas [11]
Model
pretrained B-cosified
A icc
B-cos
A
speedup
Un-norm wt.
ViT-Ti
70.3
69.3.0.1
-1.1
60.0
+9.3
10
x9.0
ViT-S
74.4
75.2+0.1
+0.8
69.2
+6.0
10
X9.0
ViT-B
75.3
75.3+0.1
-0.1
74.4
+0,9
57
x1.6
ViT,-Ti
72.6
72.3+0.1
-0.3
67.3
+5.0
10
X9.0
V
75.7
76.010.1
+0.3
74.5
+1.5
32
x2.8
V
76.8
76.7102
+0.1
77.1
-0.4
Norm wt.
ViT-Ti
70.3
68.5+0.2
-1.8
60.0
+8.5
26
x3.5
ViT-S
74.4
76.010.2
+1.6
69.2
+6.8
24
x3.8
ViT-B
75.3
76.9
+1.6
74.4
+2.5
30
X3.0
ViT,-Ti
72.6
73,2+0.2
+0.6
67.3
+5.9
28
X3.2
VIT--S
75.7
78.010.1
+2.3
74.5
+3.5
27
x3.3
ViT,-B
76.8
78.310.11
+1.5
77.1
+1.2
32
x3.8
Table B3: Impact of Pre-trained Weights. We report the top-1 classification accuracy the ImageNet validation set the B-cosified ResNet-50 model with different weight initialisations (acc), the difference to the results reported in [11] when training the pre-trained B-cos ResNet-50 50 model (Awe) under the same B-cosification recipe, as well as how much faster the same accuracy was achieved (speedup). Additionally, report the GridPG localisation scores (loe) similar those reported in [I []]] and the difference the B-cos ResNet-50 model's localisation trained (Aww) under same B-cosification training recipe. The pre-trained accuracy (in %6 for: B-cos = 75.9, CLIP = 73.3, and DINO 75.3. Random Init is the baseline with randomly initialized B-cosified model training. Results are for random initialized single run.
Top-1 Accuracy (%)
Localisation [11]
Weights
acc
Age
speedup
loc
A Ajec
Random Init
72.7
-2,9
88.8
-1.6
VI
76.6
+1.0
x2.4
92.4
+2.0
V2
77.3
+1.7
x11.3
91.7
+1.3
CLIP
75.3
-0.3
91.2
+0.8
DINO
77.0
+1.4
x3.2
90.9
+0.5
18

---

Table B4: Experimental ablations results for the B-cosification design choices. The table shows the ablation results for the B-cosified ResNet-18 model (with B = 2 and no Bias) shown in row 1. Further in rows 2-6, the ablated components shown in col. I, the accuracy of the ablated model (ace) col. 2, and the change accuracy (A compared to the non-ablated model (row 1) is shown in col. 3. Further, col. shows the GridPG localisation scores (loc) and the difference to the non-ablated B-cosified model (Atx). Similar to components used in standard models, we replace the BatchNorm Uncentered with BatchNorm Centered, change the order of the Global Average Pool where features first averaged and then passed the last linear layer, and replace average pooling with max pooling at stem. Further, we also replaced ReLU activation with Identity; and removed the Logit Bias layer. Results are for a random initialized single run.
Top-1 Accuracy (%)
Localisation loc
Ablated Component
Aace
A
B-cosified
71.5
86.8
BatchNorm Centered
71.0
-0.5
28.8
-58.0
Global Average Pool Order
71.5
0.0
63.8
-23.0
Identity Activation
63.4
-8.1
86.0
-1.4
Logit Bias removal
70.8
-0.7
92.6
+5.2
Max Pool (Stem)
71.4
-0.1
87.7
+0.3
Table B5: Increasing for B-cosification Extended. Extending Tah. 2 for different convolutional models, we compare various strategies to increase the parameter. We evaluate these strategies using accuracy and localization scores to measure interpretability performance. Columns 2-3 represent the baseline models: Standard ResNet-18 18 [54] and B-cos ResNet-18 [11]. Columns 4-8 set the B value directly E {1, 1.25, 1.5, 1.75, 2}. Columns 9-13 apply
(5, 10, 20, 45, 90). Column 14 shows results for increasing B as learned parameter. Further, row blocks denote different models, each denoting accuracy, followed by localisation scores. Results are for a random initialized single run.
over 'n' epochs with
Buselines
Discrete B=1.5
Lincar B
Learnt B
Metric
Standard
B-cos
B=1
B=1.25
B=1.75
B=2
5 cpo.
10 epo.
20 epo.
45 epo.
90 epo.
ResNet-18
Accumacy
Localisation
21.2
88.0
33.8
68.1
84.2
86.6
87.4
88.0
87.9
88.6
88.8
89.0
89.3
ResNet-50 V1
Accuracy
76.1
75.9
76.49
76.8
76.8
76.4
76.56
76.5
76.4
76.5
76.3
76.2
76.5
Localisation
24.8
90.4
45.8
86.4
91.4
92.0
92.4
92.9
92.8
92.9
92.9
92.7
93.6
ResNet-50 V2
Accuracy
80.9
75.9
77.7
77.5
77.6
77.5
77.3
77.3
77.5
77.4
77.1
77.2
77.6
Localisation
24.2
90.4
86.9
91.2
91.9
91.7
91.8
92.1
92.3
92.7
92.8
92.8
DemseNet-121
Accuracy
74.4
73.6
75.83
76.3
76.4
76.6
76.4
76.4
76.4
76.4
76.3
75.8
76.5
Localisation
19

---

Table B6: Decreasing blases for B-cosification extended. Extending Tab, for different convolutional models, compare various stralegies decrease the bias parameter. We evaluate these strategies using accuracy and localization scores to measure interpretability performance. Columns 2-3 represent baseline models: Standard ResNet-18 [54] and B-cos ResNet-18 [11]. Columns 4-5 show the setting with bias (col, 4) and without bias (col. 5). Columns 6-8 show the bias decay setup using the weight decay with different values E {0.2, 0.5, 0.9}. Further, row blocks denote different models, each denoting accuracy, followed by localisation scores. Results are random initialized single run.
Model
Baselines
Fixed bias
Bias decay
Metric
Standard B-cos
With bias
No bias
decay=0.2
decay=0.5
decay=0.9
ResNet-18
Accuracy
69.8
68.7
71.3
71.5
71.4
71.8
71.9
Localisation
21.2
88.0
46.8
87.4
81.6
90.3
91.3
ResNet-50 V1
Accuracy 76.1
75.9
76.6
76.6
76.6
76.8
76.7
Localisation
24.8
90.4
92.6
92.7
93.4
93.1
92.8
ResNet-50
V2
Accuracy 80.9 24.2
75.9
77.2
77.3
77.5
77.4
77.5
Localisation
90.4
88.5
91.7
93.5
93.1
93.0
DenseNet-122
Accuracy
74.4 20.2
73.6
76.3
76.4
76.8
76.9
76.9
Localisation
92.3
86.9
91.2
90.0
90.3
90.7
20

---

Table B7: Zero-shot performance of various CLIP-based models over 38 datasets using CLIP Benchmark [15], Scores within the 99.5% Clopper- Clopper-Pearson Pearson confidence interval of cach dataset's top score are shown in bold. Basclines contain results for the Standard CLIP [39] and Text2Concept (T2C) [30] models; ImageNet and CC3M column sections contain the B-cosified RN-50 50 CLIP models trained with cosine cyclic learning schedulers trained with ImageNet [16] CC3M [47] datasets, respectively. The cyclic learning training inspired from
[25]. Dataset type is taken from [15].
Model
Baselines
ImageNet [16]
CC3M [47]
Dataset
Standard [39]
T2C [30]
Cosine
Cyclic
Cosine
Cyclic
Natural
cars
0.54
0.02
0.37
0.38
0.35
0.36
country211
0.15
0,03
0.12
0.12
0.12
0.13
fer2013
0.35
0.18
0.21
0.19
0.20
0.23
fgve_aircraft
0.17
0.02
0.11
0.12
0.10
0.11
glsrb
0.35
0.05
0.21
0.20
0.32
0.31
imagenct-a
0.23
0.04
0.16
0.16
0.14
0.13
imagenet-o
0.57
0.68
0.65
0.64
0.57
0.57
imagenet-r
0.61
0,28
0.52
0.52
0.53
0.53
imagenetlk
0.60
0.52
0.59
0.59
0.52
0.52
imagenet_ mnatc sketch
0.35
0.15
0.28
0.28
0.29
0.30
imagenetv2
0.53
0.42
0.48
0.48
0.44
0.44
objectnet
0.41
0.23
0.33
0.32
0.32
0.31
stl10
0.94
0.91
0.94
0.94
0.93
0.93
sun397
0.60
0.29
0.62
0.60
0.55
0.54
voc2007
0.65
0.65
0.63
0.62
0.62
0.61
vtab/caltech101
0.77
0.74
0.74
0.74
0.72
0.72
vtab/cifar10
0.71
0.35
0.71
0.71
0.71
0.72
vtab/cifar100
0.40
0.09
0.40
0.41
0.37
0.36
vtab/dtd
0.41
0,29
0,41
0.42
0.37
0.38
vtab/flowers
0.66
0.07
0.58
0.58
0.56
0.58
vtab/pets
0.86
0.69
0.83
0.85
0.80
0.81
vtab/svhn
0.30
0.08
0.11
0.15
0.13
0.14
Specialized
bangenet_sketchh
0.35
0.15
0.28
0.28
0.29
0.30
mnist
0.58
0.17
0.38
0.37
0.38
0.37
renderedsst2
0.56
0.50
0.50
0.50
0.50
0.50
coandhaaheti_retinoppall
0.17
0.69
0.38
0.43
0.29
0.35
viab/eurosat
0.41
0.27
0.36
0.33
0.41
0.42
vlab/pcam
0.64
0.50
0.52
0.69
0.52
0.50
vtab/resisc45
0.45
0.15
0.28
0.29
0.35
0.37
Structured
0.15
0.15
0.14
0.15
0.25
0.24
fub/clevr_ccont__.
0.22
0.15
0.22
0.21
0.27
0.29
vtab/dmlab
0.15
0.19
0.16
0.19
0.16
0.17
vtab/dsprites label_ orientation
0.01
0.02
0.05
0.06
0.06
0.05
vtab/dsprites label x x_posstiio
0.03
0.03
0.06
0.06
0.06
0.06
vtab/dsprites label soee____p_r y_positionnnnn
0.03
0.03
0.11
0.12
0.12
0.13
vtab/kitti closest vehicle distance
0.18
0.12
0.17
0.17
hatpraallnnoolllllalaaaiiie vtab/smallnorb. Label_clevatte
0.06
0.06
0.05
0.06
0.05
0.06
0.11
0.12
0.12
0.12
0.12
0.13
21

---

Table Linear-Proid performance various CLIP-based models over 29 datasets using Benchmark [15]. Scores within the 99.5% Clopper-Pearson confidence interval of each dataset's ' top score are shown in bold. Baselines contain results for the Standard CLIP [39] and Tex12Concept (T2C) [30] models; ImageNet and CC3M column sections contain the B- B-cosified RN-50 CLIP models trained with cosine and cyclic learning
schedulers trained with ImageNet [16] and CC3M [47] datasets, respectively. [47] datasets, respectively. The cyclic learning training inspired from [25]. Dataset type is taken from [15].
Model
Baselines
ImageNet [16]
CC3M [47]
Dataset
Standard [39]
T2C [30]
Cosine
Cyclic
Cosine
Cyclic
Natural
cars
0.80
0.33
0.71
0.71
0.67
0.69
fer2013
0.63
0.48
0.60
0.60
0.61
0.61
fgve_aircraft
0.42
0.23
0.36
0.36
0.33
0.34
gtsrb
0.84
0.69
0.82
0.83
0.81
0.82
imagenetlk
0.71
0.73
0.72
0.72
0.67
0.68
stl10
0.97
0.96
0.98
0.98
0.96
0.97
voc2007
0.82
0.82
0.83
0.83
0.81
0.81
vtab/caltech101
0.92
0.88
0.92
0.92
0.86
0.86
vtab/cifar100
0.70
0.70
0.74
0.74
0.71
0.72
vtab/cifar10
0.89
0.89
0.91
0.91
0.88
0.89
vtab/dtd
0.74
0.66
0.73
0.73
0.69
0.70
vtab/flowers
0.92
0.73
0.91
0.91
0.89
0.89
vtab/pets
0.88
0.89
0,86
0.88
0.84
0.85
vtab/svhn
0.65
0.60
0.66
0.66
0.63
0.65
Specialized
mnist
0.98
0.97
0.97
0.97
0.98
0.97
renderedsst2
0.72
0,51
0.56
0.56
0.62
0.60
wtabdlibbetic_reiiopatty
vtab/diabetic
0.76
0.75
0.76
0.76
0.75
0.76
viab/eurosat
0.94
0.95
0.95
0.95
0.95
0.95
viab/pcam
0.82
0.84
0.82
0.84
0.82
0.81
vlab/resisc45
0.91
0.84
0.89
0.89
0.87
0.88
Structured
wtal/ceevrclooesst object distance
0.53
0.53
0.52
0.53
0.54
0.55
vaabcleev_coomt__
0.62
0.53
0.68
0.67
0.65
0.64
oabiionnt Maab/lppries__lbee__orte
0.49
0.57
0.58
0.63
Tubbllspries_llbbe___ppsss
0.52
0.43
0.55
0.53
0.53
0.54
vtab/dsprites label y_position
0.56
0.50
0.58
0.58
0.57
0.59
vtab/smallnorb label azimuth
0.14
0.14
0.14
0.13
0.14
0.14
vtab/smallnorb label elevation
0.37
0.33
0.39
0.39
0.40
0.40
vtab/dmlab
0.48
0.44
0.48
0.49
0.47
0.47
vtab/kiuti vtab/kiti_closest_eee ccciiicoolossssseeeeee cerss___chcccc_llceeeee
0.52
0.47
0.48
0.50
0.47
0.47
22

---

C
Implementation Details
We implement our code in Pytorch [35] for all the experiments and use Captum [29] for visualisations.
C.1 Standard Models
C.1.1 Models
We B-cosify models from Torchvision [54] supervised on ImageNet [16]. We use diverse set architectures, including both CNNs (ResNet-18 [22], ResNet-50 [22], and DenseNet-12) [24]), and ViTs [18, 6, 57] with -Vi -S, ViT -LT and without (ViT-Ti, ViT-S, ViT-B, ViT-L) convolutional stems. For ResNet-50,, we use both the weights originally released by Torchvision and the updated V2 weights, which constitute models trained for longer and with more augmentations [55].
C.1.2 Datasets
We use ImageNet [16] to fine-tune all the B-cosified standard models and evaluate them on ImageNet's validation set. For training, probability, random resized crop of 224 with bilinear interpolation, Add Inverse transform [10] and modified mean-std normalisation (to accommodate for 6 channel input from the AddInverse). For evaluation, instead random resized crop, we do a center crop with a crop size of 224.
train transforms crop size of 224, horizontal flip with 0.5
C.1.3 Optimization
For each architecture, we use the B-cosification stategy derived in Sec. 3, and fine-tune for 90 epochs using the AdamW optimizer [26] and cosine scheduling for the learning rate learning rate of 10 for the convolutional models (since the standard pre-trained models end with a learning rate of 10 the 90th epoch, from which we want fine-tune further). For ViTs, as the learning rate decays to very small value, we tested with different learning rates (10-3, 10-4, 10 10-b) and found 10 worked best for all the models, Also, we only use linear learning rate warmup of 10,000 steps with decay of 0.01 for the base and large ViT models.
C.1.4 Experiments
Increasing B: We tested three different setups for increasing B. 1) Discrete B setting to 1, 1.2.2 1.5, 1.75, 2, 2.5, 3, 5, 7; 2) Linear increase of B in n epochs from B=1 to B=2. We used 5, 10, 20, 45, 90; 3) Learning B parameter to increase to B=2 using weight decay with coeffi- cients 0.2, 0.5 and 0.9. See Tab. 2 for results.
Removing biases: We test two setups for removing the biases from the network: 1) Removing all the bias parameters; 2) Decay the bias parameter using the weight decay with coefficients 0.5 and 0.9. See Tab. 3 for results.
Impact of pre-trained weights: To check the impact of pre-trained weights fine-tuning, fine-tuned weights from CLIP [39] ResNet-50 [22], DINO ResNet-50 [12], and Torchvision [54] ResNet-50 weights vl and v2 (long trained recipe) [55].
C.1.5 Evaluation
As in Sec. 3.2, we evaluate both for classification accuracy and for interpretability using the GridPG [9] metric. We compare both accuracy and interpretability of the B-cosified models with B-cos models trained from scratch from [11]. For interpretability, we also compare with several post-hoc attribution methods as baselines, namely Guided Backprop [51], Gradient [49], DeepL.ITT [48], IxG [48], IntGrad [53], and GradCAM [46]. Qualitatively, we visualize the colored B-cos explanations and the attribution maps [1]].
C.2 CLIP Models
We CLIP [39] ResNet-50 [22] model for B-cosification.
23

---

C.2.1 Datasets
We use ImageNet [16] and CC3M [47] to fine-tune all the B-cosified CLIP models and test them on multiple datasets from CLIP benchmark [15]. For training, we use the same transform setup as the standard B-cosified models. We use train transforms crop size of 224, horizontal flip with 0.5 probability, random resized crop 224 with bilinear interpolation, and Add Inverse transform [10] and modified mean-std normalisation (to accommodate for channel input from the AddInverse) as discussed in the paper. For evaluation, instead of a random resized crop, we center crop with crop size of 224.
C.2.2 Evaluation
We use the CLIP benchmark [15] for zeroshot and linear probing experiments with the default parameters provided in the official benchmarking code. For text-based localisations, we the text-based templates from the CLIP for the ImageNet dataset and use them to encode the text features. As text encoder, CLIP's image encoding and the pre-trained text encoder are used to do B-cos style localisations and calculate the GridPG scores. We use the unpooled features technique at inference increase the localisation focus.
the CLIP ResNet-50 text encoder. The cosine scores between the B-cosified
We use B-cosified CLIP ResNet-50 fine-tuned on ImageNet using SigLIP loss [59] and cosine scheduling for visualisation.
C.2.3 Optimization
We use the Adam optimizer [26] and fine-tuned models till 90 epochs, while the CC3M models are fine-tuned for 30 epochs. The size of CC3M is approximately three times that of ImageNet, so the trained models are comparable. Keeping consistent with the Standard B-cosification recipe, we train with learning rate le-4 using cosine scheduling. SigLIP contrastive loss [59] is used to train the models.
24

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract and intro we claim to fine-tune models for inherent interpretabil- ity (by converting them to the recently proposed B-cos models) and that we find that our proposed scheme (I) outperforms training from scratch, (2) often recovers the original per- formance, (3) incurs lower training cost than training from scratch, (4) improves the inherent interpretability of the model, and that (5) it lends itself even to fine-tuning foundation models like CLIP for an increase in inherent interpretability. This exactly what we do in our paper: e.g., Tab, 4 shows (1), (2), and (3), Fig. 3 and Fig. 4 show improvements in inter- pretability (4), and Sec. 4.2 dedicated to analysing the performance and interpretability of our B B-cosified CLIP (ef. Fig. 5 and Fig. 6a).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the paragraph on limitations in Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
25

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Our theoretical arguments contained in Sec. 3.2.1, namely that (1) it is possible to convert the first layer of DNNs such that they accept channel inputs as required by B-cos models, that (2) ReLU is a special case of MaxOut, and that (3) weight normalisation is irrelevant a layer is followed by a batch normalisation layer. These claims are shown to be true in the context of Eq. (2) (1), Eq. (3) (2) and Eq. (5) (3).
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: To the best of our knowledge, all necessary details to reproduce our results are contained in the submission. We mainly build on publicly avail- able code (for B-cos models, see https://github.com waatbaccoo/ccoo/oooo//oe see https//gtihub..oo/ppeaa////CL com/openai/CLIPP/ for CLIP Benchmark, see github. coorb/o.O/LAA AI/CLIP beiilp_beehharkkkkk and httpa///w////ggtooooooooootooooo wow////ooonnaiioosoopopeeeoe __cpeen___pppeeeee and will make all our modifications available for ensuring full reproducibility of the reported results.
for CLIP,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
26

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We mainly rely on publicly available code and established training paradigms and clearly describe any changes we introduce. That said, we make our code available to ensure full reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. wwwp//pwwc/gguccccoeeeeooocoecle guw.e//eeee/eeee/ httpppiic//u.eeee/eeeeetaaiiiiia hwtp///oooeeeiiiineeee www.snolicccyio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We rely on standard datasets (CC3M, ImageNet- 1k) or publicly available benchmarks (github.com wwt..cooo///oo//CIIIIC _eench_re)nh)akk and describe our training and eval- uation procedure in detail; see the setup sections in Sec. 3.2.2, Sec. 4.1, Sec. 4.2, and Sec. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
27

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Given the scope of ablations (Sec. 3.2.2), datasets and models (Sec. 4), as well as computational cost (see question in the checklist), each experimental result is currently based on three training runs. That said, observe consistent behaviour across all our experiments, which corroborates the validity of our claims (see, e.g., Tab. 2 and Tab. 3).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We use NVIDIA A100-SXM4-4000 and Quadro RTX 8000 GPUs from internal cluster. We require GPUs per run (except ViTs with 8 A100GPUs for ViT models). Runtime per run: 1-2 days depending model size for standard models and ImageNet fine-tuned CLIP; for CC3M fine-tuned CLIP 28 days for 90 epochs runs on Quadro RTX 8000 GPUs. We have 500 experiment runs in total approximately (CC3M 5 runs, ViTs - 150 runs, 345 rest of the models). For storing all the experimental results, we utilize approximately 2.5 TB of storage.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
28

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. http://wurrppp.cpppuliiiuiee hwwpppblicc/cchhccccuiieeeeeeeee
Question:
Answer: [Yes]
Justification: With our work, make contribution towards making Deep Neural Networks more explainable, which we believe has an overwhelmingly positive societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We believe the interpretability of models to be important for society to trust models and validate their predictions, and for researchers and developers to able to understand failure cases and debug models. As we discuss in Sec. 5, our work makes step in this direction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We propose technique towards making existing models more interpretable, which-to the best of our understanding does not necessitate putting in place any safe- guards regarding the release of our code and models.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow standard practice and cite every author of any existing asset that we OpenAI CLIP MIT License, CLIP Benchmark MIT License, B-cos Apache-2.0 license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not provide any new assets alongside this submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
30

---

Justification:
This question does not apply to our submission.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This question does not apply to our submission.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "iITH3T9aMY", "reviewer": "Reviewer_Z7HS", "summary": "The authors discuss the B-cosification of a pre-trained model.\nThe B-cosification involves changing the operations performed in the linear layers to one involving a cosine. Not all properties of B-cos models are in the end satisfied, yet performance in terms of accuracy and localization are on par with a fully trained B-cos model.", "questions": "* P1 L33: which are increasingly popular—can cost millions of dollars.\n    What if I am not using dollars? Maybe time would be a better metric? Or energy?\n\n* P2 L38: the recently proposed B-cos Networks.\n    Until here, B-cos was never defined. Since it is not a contribution of this paper, would you have a citation to guide the reader?\n\n* P2 L42: What does \"functionally equivalent\" mean? What do you mean with \"alignment pressure\"?\nMaybe instead of introducing these vague notions, why not use the opportunity given by this paragraph to explicitly say what B-cos consists of, namely (up to my understanding) replacing the linear operations of the hidden layers by a cosine-based function parametered with B? You can go further and say that for B=1, the operation is (\"is\" and not \"is equivalent to\") the classic linear matrix multiplication. \"Linear\" implies the absence of a bias term. Otherwise, that would be the \"classic affine matrix multiplication\". Nevertheless, one can stress this further.\n\n* P2 L45: \"significantly more interpretable explanations (Fig 1)\"\n    What do you mean by \"more interpretable explanations\"? What I see in the third row of Fig1 is kind of a mix between the second row and an edge detector or an Integrated Gradient saliency map. Is capturing the shape or details of the \"important part of the image\" making an explanation more interpretable?\n    Following this discussion, the result of GradCam+CLIP on the first example is preferable from my point of view because one can see that the model uses the help of the grass to actually predict a sheep. The same applies to the \"[...] photo of a boat.\"\n\n* P2 L47: \"On supervised settings, we find that B-cosified models often outperform\"\n    How is performance measured here? Classification accuracy or caption prediction?\n\n* P2 L53-57: I am a bit confused when you say \"design[ing] inherently interpretable models\". I do not understand your contribution as a new design but a conversion. This is a valuable and efficient contribution that aligns with current concerns, be it ecological, time efficient, or simply democratizing ML for the majority who do not have access to huge computational resources.\n    This work fits into the recent line of research investigating how to alter the architecture of trained models to make them \"interpretable.\" Why not stress this aspect more?\n\n* P2 L65-: I am unsure if the claimed results are that important to count as contributions.\n    Again, what you mean by \"significantly interpretable\" is not clear. Could you please clarify?\n\n* Sec2: If you choose to attach your contribution to the line of \"converting trained models\" (you don't have to) to \"interpretable ones\", I suggest some works:\n    * Stalder, S., Perraudin, N., Achanta, R., Perez-Cruz, F., & Volpi, M. (2022). What you see is what you classify: Black box attributions. Advances in Neural Information Processing Systems, 35, 84-94.\n    * Aytekin, C. (2022). Neural networks are decision trees. arXiv preprint arXiv:2210.05189.\n    * Gautam, S., Boubekki, A., Höhne, M. M., & Kampffmeyer, M. (2024) Prototypical Self-Explainable Models Without Re-training. Transactions on Machine Learning Research.\n\n* P3 L111: \"Many common DNNs consist of a series of blocks of linear layers followed by non-linear ReLU activations [29], and are thus piece-wise linear functions1 : i.e., for every input x, they effectively compute a linear transformation of that input: y(x) = W(x)x + b(x)\"\n    What are W and b? I assume the model's input and output are x and y, respetively.\n    If it is \"effectively [...] a linear transformation\", why are W and b functions of x?\n\n* P3 L113: \"dynamic linear\"\n    Why do you need to introduce this notion? What is dynamic in an affine matrix multiplication?\n\n* P3 L116: \"This linear mapping W(x)\". The same goes here: W cannot be a linear mapping/function if we are talking about \"a piece-wise linear model\".\n\n* P3 L120: Why does a \"complete explanation\" requires \"y(x)=W(x)x\"? Two sentences later, you actually contradict this claim.\n    I fail to grasp why having a bias is so problematic if most of the backward operations are based on the gradient. Could you clarify?\n\n* P4 L123: Can you clearly define \"alignment pressure\" and why it is necessary to be introduced?\n    From this paragraph, I understand that the conversion from line (B=1) to (B>1)-cos model has already been investigated in [6] and [7]. Has it?\n    If it has been, this jeopardizes the relevance of your main contribution.\n\n* P4 Eq1: How do you compute $cos(x,w)$?\n    I understand that $x$ is a vector, $W$ is a matrix, and $\\times$ is the element-wise multiplication.\n\n* P4 L127: Why do we need the notion of \"dynamic linear\"?\n    If it has to do with W being a function of x: if B=1, W does not depend on x.\n\n* P4 L141: Missing reference with Fix XX.\n\n* P4 L146: \"In particular, we show that a conventional model can be framed as a functionally equivalent B-cos model as in [7] with B =1, which additionally employs bias terms. Only upon modifying these two aspects, i.e., biases and B, does the model need to be fine-tuned to adapt the weights to those changes.\"\n    These two sentences are not clear.\n\n* P4 L150: \"As mentioned in Sec. 3.1, B-cos models use input representations with six color channels\"\n    This was not mentioned in Sec 3.1. This seems to be an atavism from an older version of the paper. Please fix.\n\n* P4 \"Input Encoding and Normalisation\": The operation is straightforward: B-cos models requires 6 channels: so inputs need to be transformed according to Eq2 and the first layer of the B-cosified models is discarded and replaced by one with 6 channels (implying a full training thereof).\n\n* P5 \"Activation Functions.\" You start the section by saying that activations are unnecessary, yet you introduce them. Why?\n\n* P5 \"...unit norm weights, ..., which the authors motivated by the fact that the only way any given neuron can achieve its maximal output is by increasing the weight-input alignment, which in turns[typo] leads to the improvements of the explanations.\"\n    What is \"weight-input alignment\"? I fail to see why normalized weights improves the explanations.\n\nYou could streamline this section by leaving the justification of the B-cos meaningfulness to an introductory paragraph and focusing on the changes required by your approach. There is no need to justify or make claims about the benefit of B-cos since it is not one of your contributions here.\n\n* P5 Section 3.2.2\n    None of the tables present statistical tests, which makes it difficult to draw any conclusion, especially given that the results are quite similar from one setting to another.\n    * How many times the experiments were repeated? What are the standard deviation? Which result is significantly better?\n    * What is the localization score?\n\n\n* P6 L204: \"setting B = 2 and then fine-tuning, yields performance that is on par with\nlearnable B parameters,\"\n    If this holds, it is also on par with Linear B 90 epochs.\n\n* P6 Table 4: The increments are quite small. Have you considered running a t-test to identify significant differences?\n    Is the \"speedup\" with respect to the fully trained B-cos model? What about the \"speedup\" with respect to the vanilla models with classic matrix multiplication?\n\n* P7 L245: \"Specifically, we find that averaged across architectures, B-cosified models outperform B-cos models trained from scratch by 2.31 pp, with an average training speedup (to match performance) of 2.96x.\"\n    This is an important result. Why not highlight it in a separate section/ablation study?\n\n* P8 Interpretability: In the text you refer to paper [6] while in the caption of Fig2 you refer to [7] (These papers share essentially the same figures). Fig2 would be better positioned closer to its text.\n    Finally, interpretability is defined in paper [6] (or rather in paper [6] of paper [6]) as the performance at the \"grid pointing game\". If I understand it, it is related to the sum of the GCAM attribution to each class of the grid cell. My understanding of gradient-based explainable methods is that the scale of the output saliency maps is inconsistent. Is it valid? If yes, how does it affect the score returned by the Pointing Game? How robust is this score against possible spurious relationships between grid cells? How does performing well in this game relate to the interpretability of the model's explanations?\n\n* P8 Fig4: What are Natural/Specialized/Structured Data?\n\n* P9 L284: \"We find that the B-cosified models significantly outperform the Text2Concept approach and achieve accuracies that are more similar to the original CLIP’s zeroshot and linear probing accuracies.\"\n    Text2Concept is a quite different approach. In the original paper thereof, they report performance close to CLIP's. How do you explain this difference? What is different in your setting? Do you train supervised or self-supervised?\n\n* Evaluating Model Interpretability.\n    In the image classification case, you compute the explanation based on a single label; the equivalent case here is thus to back-propagate using the\n    Isn't the average similarity equivalent to the similarity with an out-of-distribution point in the text embedding?\n    Why did you choose p=7 and p=19?\n\n    Overall I did not quite understand this section. I am not very familiar with CLIP, so it is not easy for me to follow what is happening.", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method contributes to the current effort to turn pre-trained models into explainable ones without full retraining.\n- The performance dropped are limited both against the original model and the equivalent B-cos model", "weaknesses": "- The method still requires a full fine-tuning of the weights.\n- The experimental setting is not very clear.\n- The presentation is not always clear and sometimes verbose.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback and the suggestions to improve clarity; we will incorporate them as well as the following answers in our revision.\n- **Full fine-tuning**: The research question we examine is how to leverage existing pre-trained uninterpretable models to make training more interpretable ones more cost-effective. To endow pre-trained models with _inherent interpretability_, it is indeed inherently necessary to fine-tune the full models, as we aim to fundamentally change how they operate - however, we find that this approach is significantly cheaper than training the corresponding interpretable models from scratch (up to 9 times speed-up, Tab. 4).\n- **Multiple runs**: See global response.\n- **Tab 4 small increments**: The increments in accuracy are indeed small but also not our core focus. Instead, we show that it is possible to _maintain the original models’ accuracy_ whilst significantly increasing their interpretability, at a significantly lower cost than when training from scratch as in [6]. \n- **B-cos more interpretable**: As reported in [6,7], B-cos explanations **explain the full model from input to output** and are significantly more localized, and allow to visualize the important features in color (Figs. 1,7 in [6]), while also allowing to understand misclassifications (Fig 9, [6]), making it easier for humans to see understand the model's predictions. Please note that in the examples in Fig. 1, the rows correspond to different models, and so are not directly comparable. I.e., the CLIP model might indeed use the grass to predict the sheep, while the B-cosified CLIP model might not. \n- **Comparing to Text2Concept**: [28] shows that a linear transform can map an arbitrary model's features to CLIP features. Hence, training such a transform on an existing B-cos model serves as a good baseline for B-cosification. We follow [28] and use ImageNet to learn the linear map. However, other parameters are different: we use a B-cos encoder instead of a standard model, a CLIP ResNet-50 instead of ViT-B/16, and evaluate across a broader range of datasets (Fig. 4). Understanding the performance difference requires further investigation, but is orthogonal to our current work.\n- **Dynamic linearity**: As discussed in prior work (cf. [18]),  a linear layer followed by ReLU or MaxOut is  ‘piece-wise linear’: for every input x, the prediction is effectively computed via a linear transformation of x, with the choice of the linear matrix W depending on x. Dynamic linearity (cf. [6, 7]) is a generalization of this concept, making the dependence of the linear matrices on the input more explicit. In particular, ‘dynamic linear models’ compute their output via an input-dependent linear transformation of the input y(x) = W(x) x. Finally, we agree with the reviewer that when bias terms are employed, “piecewise affine” is more accurate in contrast to commonly used nomenclature; given that the bias can be modeled via an additional input dimension, however, we opted to stick to standard nomenclature. \n- **Completeness/bias**: The importance of biases has been discussed in prior work: [48] discusses how gradient-based explanations tend to neglect important contributions from biases (cf Eq. 1, [48]), which have been shown to play a significant role in the models’ outputs (Fig. 13 in [7]); completeness further constitutes one of the foundational axioms of IntGrad [49]. We will revise to better reflect these connections.\n- **Novelty of conversion**: No, while [6] discussed and emphasized the differences between B-cos and conventional networks, we leverage the similarities and show that the existing weights of pre-trained DNNs can be used to efficiently obtain performant and interpretable DNNs, despite e.g. using a different number of input channels (3 vs. 6) and activation functions.\n- **Input encoding**: No, we do not discard the first convolutional layer and train a new one from scratch; instead, as part of our contributions, we show how to construct a first layer and compute the corresponding weight that takes six channels as input and is nonetheless functionally equivalent to the original layer **with no additional training**.\n- **Activation functions**: We keep ReLU since (i) we want to at first maintain functional equivalence with the original model, (ii) using ReLU does not affect the dynamic linearity of B-cos, (iii) the pre-trained weights were trained with them, so using ReLU keeps us closer to the original models.\n- **Normalized weights**: Using normalized weights ensures that the output can be maximized only if the input and weight vectors align, i.e. if the cosine term in the dot product between them is high, see [6]. This helps interpretability, since weights being aligned to inputs better highlights the most important regions. \n- **Localization score**: It refers to the GridPG metric used in [6,7].\n- **Robustness of GridPG**: GridPG [6, 7] computes the ratio of attributions in a grid cell to the total attributions, and so is invariant to scaling To reduce the risk of spurious relationships, the metric uses grids where each image is of a distinct class and is classified correctly with high confidence by the original model, implying that it is unambiguous to the model.\n- **CLIP Interpretability**: We show p=7 and p=19 for illustrative purposes, the full trend across p is shown in Fig 5a. We also provide a more detailed discussion on the CLIP experiments in our response to reviewer JC7z.\n- **Natural/Specialized/Structured data**: We follow the CLIP benchmark [26] for zero-shot and linear probing results (Fig. 4) and categorize datasets based on the images they contain.\n- Sec 3.1: We do mention it, see L139.\n- **\"Outperform\" performance measure**: We report accuracy for image classification, and zero-shot performance for CLIP models.\n- **Citations**: Thank you, we will add them.\n- **Eq1**: We use row-wise dot products, following prior work (cf. Eq. 9 in [6]). We will revise for clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZJgpqjVey", "reviewer": "Reviewer_71N9", "summary": "This work targets the interpretability of modern architectures via a process that the authors call B-cosification. Contrary to the original B-cos networks that are trained from scratch by architecturally enforcing alignment between inputs and weights, B-cosification constitutes a post-hoc method, aiming to convert/finetune exististing pre-trained models towards interpretability.", "questions": "Please see the Weaknesses section.", "limitations": "The authors have a limitation paragraph in the main text.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work aims to transform existing pre-trained models to \"inherently interpretable\" ones via a post-hoc b-cosification method. Contrary to other post-hoc approaches, this finetuning allows for essentially altering the properties of the network, bypassing some of the criticisms of post-hoc approaches. \n\nOverall, this work constitutes an interesting investigation of how to transform standard architectures to B-cos like ones.", "weaknesses": "Even though this is an interesting approach, its novelty is limited. The authors explore how to finetune the model to match the b-cos alignment pressure, by exploring some architectural changes like normalisation and bias decrease.\n\nOne of the main issues the proposed process is the complexity of the finetuning. Some of the ResNet architectures considered in the main text are trained for around 90 epochs, e.g., on ImageNet. That is a common number of epochs to train these models from scratch. Can the authors provide a summary/visualisation of the acc/localization with respect to the number of epochs?\n\nWhat is the behaviour of the base models when finetuned the same way as in the proposed b-cosification process (without b-cos)?\n\nCan the authors add another column to Table 4, denoting the accuracy of the standard trained from scratch b-cos networks?\n\nThe text needs to be proofread. There are several sentences that lack clarity, typos and other issues, e.g., line 141 \"see also Fig. XX, whereas conventional models use 3 channels\".\n\nThe authors are encouraged to remove the spacing altercations that do not follow the format of the conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback on our work. We have carefully considered your comments and would like to address each point you raised.\n- **Limited novelty**: To the best of our knowledge, we are the first to investigate how to transform existing uninterpretable models into inherently interpretable B-cos models. In particular, we find that we can effectively B-cosify existing models at a fraction of the cost of training them from scratch, which has the potential to make interpretable models far more accessible than they currently are. In order to be able to do this, we discuss in detail how the models differ and devise specific solutions to address those differences. In particular, we first show that it is possible to cast conventional models in the same framework as B-cos models without changing the function that they represent (i.e., we adapt the model $f_{org}$ to a B-cos version $f_{Bcos}$ with B=1 whilst maintaining functional equivalence, such that $f_{org}(x) = f_{Bcos}(x) $ $\\forall$ $x$ ). Specifically,  in this context we address the input encoding and normalization, activation functions, weight normalization, and incorporate the pre-trained weights. Thereafter, to increase interpretability, we increase the parameter B and decrease the models’ biases. This, of course, functionally changes the model and hence requires fine-tuning as a final step to adapt and recover the model’s performance to the new changes. We are unaware of previous work that explored this and would be grateful if the reviewer could point us to relevant related work that would limit the novelty of our contribution.\n- **Complexity of fine-tuning**: We indeed train the models for a commonly used number of epochs (90) - however, to understand the efficiency gains by using pre-trained models, apart from accuracy and localisation scores, we also report the speed-up for reaching the same accuracy as randomly initialized B-cos models in the results (Table 4). We are very grateful for the reviewer’s suggestion to provide a visualization of accuracy and localisation scores over the training epochs, which we added in the attached PDF  in Figure 6, as this helps clearly see how quickly the models’ explanations improve in their localisation (**localisation scores > 80% after the first epoch**). While the accuracy generally improves over the course of 90 epochs, we find that they tend to reach the performance of B-cos models trained from scratch very quickly (e.g. **ResNet-18 reaches the performance of the original B-cos models within 29 epochs**). This is our primary contribution.\n- **Fine-tuning base models**: In Table 8 (column blocks 3-4) in the PDF attached, we report the results for the standard pre-trained and B-cos pre-trained models fine-tuned further for 90 epochs in the same way as proposed in the B-cosification process for two CNNs and two ViTs models. We find that even after fine-tuning the pre-trained models (standard and B-cos) further, B-cosified models perform competitively with conventional DNNs (column block 3) and outperform the B-cos DNNs (column block 4) consistent with the original findings mentioned in the main submission. We will provide full results in our revision.\n- **Accuracy of B-cos trained from scratch**: In Table 8 (column 5) in the global response PDF, we now report this additional column for two CNNs and two ViTs models. We will provide full results in our revision.\n- **Writing fixes**: Thank you for pointing these out; we will make the changes in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vctrWBrj1p", "reviewer": "Reviewer_DdYE", "summary": "This work builds on the recent line of works aiming to design inherently interpretable models. In particular, it considers the recently proposed B-cos networks and shows how pre-trained conventional CNN/VIT models can be converted into a B-cos network through fine-tuning. The authors first discuss which parts of conventional DNNs can be converted into an equivalent B-cos model and then propose two simple mechanisms that address the two missing components (increasing B and removing the biases) through fine-tuning. Results illustrate that the proposed B-cosified models are able to obtain equivalent accuracy to both the original model as well as the standard B-cos network, while obtaining similar localization results to the B-cos network in a shorter time due to the re-use of the original weights.", "questions": "For the ViT results, there are some more cases where the B-cosified networks performance degrades slightly compared to the black-box and the localization is slightly lower than for the B-cos model. While these differences are small, do the authors have a hypothesis on their reason? Are these induced by the interpretation of the GeLU?", "limitations": "The limitations of the proposed method have been briefly but sufficiently discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The work addresses an important problem of converting pre-trained black-box models into inherently interpretable models. Instead of requiring complete retraining, which often restricts the applicability of these models, only requiring fine-tuning is a first step into the right direction.\n\nThe paper is well structured and the different modifications required are presented in sufficient detail. \n\nThe experimental evaluation on both CNN and Transformers as well the case study on CLIP shows the potential of the proposed approach.", "weaknesses": "It would be beneficial to include a plot on how accuracy and localization score vary as fine-tuning progresses for one/some of the standard models (CNN/VIT).\n\nIn Sec. 3.2.2, it is unclear how the bias is handled when performing the ablation study in Table 2. Also, the B-cosified model consistently outperforms the baseline in Table 2 and 3, do the authors have an hypothesis on why this fine-tuning process increases performance?\n\nMinor:\n\nIt would be beneficial to also provide the number of fine-tuning epochs that were required in Table 4.\n\nFigure reference missing in Line 141.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your recognition of the significance of our work and your constructive suggestions. We have carefully considered your comments and would like to address each of the points you raised.\n- **How do localization scores vary with epochs?:** As we write in our global response, we now report these results in Figure 6 in the attached PDF. We find both quantitatively (left) and qualitatively (right) that the localization improves very quickly. For example, for ResNet-18, the localization improves from 21.5 to 82.0 after just one epoch of fine-tuning, which is close to 86.9 achieved by the B-cos model trained from scratch. On the other hand, modifications to the model architecture at the beginning of the B-cosification process lead to an initial drop in accuracy, from 69.8 for the standard model to 59.9 after one epoch. However, we recover the accuracy much quicker than training from scratch, with the B-cosified model needing just 29 epochs to reach the accuracy of a B-cos model trained from scratch for 90 epochs.\n- **How is bias handled?** Apologies if this was not sufficiently clear. In the ablation studies, we remove the bias when varying B since that is the setup in B-cos as well as our B-cosification process (Table 3); for an additional discussion, also see ‘Completeness/bias’ in answer to reviewer Z7HS. \n- **Performance of B-cosified models in Tables 2-3:** The baseline in Tables 2-3 refers to the standard and B-cos models trained for 90 epochs. During B-cosification, we start from the 90 epoch trained standard model and further fine-tune, which leads to performance gains. For a fairer comparison, we also report results when fine-tuning the baselines for an additional 90 epochs in Table 8 (blocks 3-4) in the global response PDF, and find that the B-cosified models perform similarly or better, while also being interpretable.\n- **Speedup epochs:** In Table 4, we always fine tune for 90 epochs, however, the speedup column shows the point at which the B-cosified model reaches the performance of the corresponding B-cos model when trained from scratch. We now also report the epoch number at which this ‘overtake’ happens at in Table 8 ($t_\\text{ovt}$) in global response PDF, e.g. for ResNet-18, we find that the B-cosified model reaches B-cos performance with just 29 epochs of fine-tuning. \n- **ViT performance:** Please note that in comparison to many other post-hoc explanation methods, the localisation scores of both B-cos and B-cosified ViTs are relatively high, differing only by a few percentage points. We will analyse and discuss (e.g., impact of GELU)  potential reasons for the differences between the ViT models in more detail in the final version.\n- **Missing reference:** Thank you for pointing this out, we will correct the missing reference for the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNombcMig1", "reviewer": "Reviewer_JC7z", "summary": "This work proposes B-cosification, a method to transform a pre-trained network to a B-cos network [1,2]. Consequently, the transformed model can be finetuned for better explanations of the model behavior while retaining predictive performance. Experiments are done on various CNN and Transformer architectures, as well as a case study on CLIP.\n\n\n[1] Böhle, Moritz, Mario Fritz, and Bernt Schiele. \"B-cos networks: Alignment is all we need for interpretability.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Böhle, Moritz, et al. \"B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).", "questions": "* In line 295, is \"interpolate\" really the right word (it confused me slightly in my read). That is, if it was interpolating, then shouldn't Fig. 5a B-cos FeatureCLIP for p=1 be equal to B-cos CLIP? (Also, there are some minor reference naming issues with respect to the figure in line 292-297, and the legend of Fig. 5a needs to be updated)\n* While the Grid Pointing Game appears to be a good metric to measure how well the explanations are localized, it is not telling enough on a fine-grained level how informative the explanations are. While the qualitative figures 1 & 5b give some insights, into this, they're obviously cherry-picked (which I don't condone). What I like in papers with qualitative examples is that, in the Appendix, there are also some randomly chosen examples so that an interested reader can get a feeling for how the method actually performs. Thus, I would be interested in seeing some more qualitative examples that are (truly) randomly drawn.\n* How well would the proposed method be transferrable to different input modalities such as text?", "limitations": "One could maybe write one more sentence with respect to which models are *not* B-cosifiable. Apart from that, limitations are adequately discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* B-cos networks are an upcoming, interesting research direction. The need to train a network from scratch is commonly considered a limitation (e.g., see the Concept Bottleneck Model Literature [3,4]); therefore, alleviating this need is a valuable contribution to the research landscape.\n* The paper is clearly structured and well-written. Only towards the end of experiments, it becomes a bit cramped with information, which is minor. \n* Experiments are performed over a broad range of backbones. Also, I appreciate that the authors make the effort to actually measure the interpretability rather than just stating that it is interpretable.\n\n[3] Yuksekgonul, Mert, Maggie Wang, and James Zou. \"Post-hoc Concept Bottleneck Models.\" The Eleventh International Conference on Learning Representations.\n\n[4] Marcinkevičs, Ričards, et al. \"Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?.\" arXiv preprint arXiv:2401.13544 (2024).", "weaknesses": "* My biggest uncertainty is about the stated (inherent) interpretability. Firstly, to me, I see B-cos(ified) networks more as a post-hoc gradient-based explanation method, which additionally regularizes during training for more faithful explanations. However, there is no certainty upon seeing an explanation that the model actually acted upon these highlighted pixels, which to me makes B-cos networks not inherently interpretable. In other words, a regularizer does not enforce that the model truly depends on the given explanations, and therefore, the explanation given is not necessarily faithful to the underlying model. Thus, I also disagree with the Abstract that states that the explanations are \"model-faithful by design\".\n* Followingly, such a method should be evaluated with respect to faithfulness to the underlying model. As such, I would be very interested in how B-cosified networks perform in some faithfulness sanity checks such as the Model Parameter Randomization Test of [5].\n* Without any standard deviation, it is hard to assess the (statistical) significance of performance improvements. Also, multiple seeds (up to some degree) prevent the all-too-common hyperparameter optimization on the test set.\n* The CLIP explanations when B-cosifying CLIP are on par with GradCAM on the pre-trained weights. To be better requires the introduction of a new, arbitrary aggregation of value vectors. \n* If I understand it correctly, this new aggregation would also affect CLIP's performance. It is not explored how the model performance on CLIP Benchmark would be affected by this aggregation that appears to make CLIP more explainable.\n* While this work has high significance in today's world of foundation models, the methodological novelty is limited, as most B-cosification tricks have already been introduced in [2].\n* In my opinion. speedups of 2x are not necessarily big enough to warrant that it's a huge improvement over training from scratch.\n* Page 1 appears to be missing some vspace after line 37.\n\n[5] Adebayo, Julius, et al. \"Sanity checks for saliency maps.\" Advances in neural information processing systems 31 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable feedback. We appreciate your positive remarks on our work's significance and your constructive suggestions for improvement. We have carefully considered your comments and would like to address each point you raised.\n- **Inherent Interpretability of B-cos:** Please note that B-cos explanations are not mere post-hoc explanations. Instead, the linear summaries W(x) of the models are in fact exact summaries (i.e., yielding the same outputs when applied to x) of the models’ computations and are optimized during training to align with task-relevant features, as described in [6], and have further been found to compare favorably to post-hoc explanations (Figure 6 in [6]). While we agree that the B-cos explanations of course constitute a simplification of the full network, previously reported results suggest that they indeed capture important aspects of the models’ computations, allowing them to e.g. localise class-specific features at a fine granularity. Our work builds on those prior findings and examines how to bring these previously reported benefits to pretrained conventional models. \n- **Model Randomization test performance:** Note that the localisation scores give clear evidence that the explanations are highly model-dependent: if they were not, they would not allow for localizing specific classes with such high specificity (>90%, see Figure 6 in [6]). Further note that conversely, explanation methods that failed the randomisation tests, as proposed in Adebayo et al., notably score very low in the grid pointing game. We are happy to include results regarding the model randomisation tests in the final version of the paper. Lastly, as can be seen in the new figure we provide in the attached global response PDF (Figure 6, right), before fine-tuning the models for inherent interpretability, they give highly uninterpretable explanations (cf. ‘at start’).\n- **Multiple seeds:** As we write in our global response, we now provide results over three runs corresponding to Tables 2, 3, and 4 in Tables 5, 6, and 7, respectively, with standard deviations. From tables 5-6, we find that the conclusions in Section 3.2.2 remain unchanged — the simplest approach of directly setting B=2 and removing the bias at the beginning performs as well as more involved strategies such as learning B or decaying bias. Also, from Table 7, we find that the results regarding classification and interpretability performance, as in Section 4.1, are supported by repeated experiments.\n- **Aggregation of CLIP vectors:** We agree with the reviewer that the aggregation mechanism changes the model’s behavior and thus does not faithfully reflect the full model’s predictions. We introduce this aggregation mechanism to show that our B-cosification approach allows us to interpret intermediate features of the CLIP model at a high level of detail (clearly outperforming the localisation ability of both full models), as the features still maintain a higher level of spatial resolution. This highlights an additional aspect in which B-cosified models could help better understand the inner workings of foundational models.\n- **Speedups in fine-tuning:** We find that the speedups are often much better than 2x for convolutional models (Table 4). We agree that the speedups are lower for ViTs, and believe this could be because our method could further be optimized for ViTs, e.g. see the discussion on ViT performance in response to reviewer DdYE.\n- **Interpolation:** For aggregating and explaining the intermediate features of the CLIP model, many options exist - e.g., attention pooling as in the original (and B-cosified) CLIP model, average pooling, or max pooling. To better understand the implications of the feature aggregation, we propose a mechanism based on the cosine similarities that allow us to interpolate between average (p=1) and max pooling (p=inf); note that since both of them differ from attention pool (denoted by B-cos CLIP in the figure), neither of them give the same results as attention pool. These results also indicate that the attention pooling mechanism selects the embedding features in a more specific fashion than mere average pooling, but aggregates the features in a coarser fashion than selecting only the features with high cosine similarities (p$\\gg$1).\n- **Qualitative examples:** We provide additional randomly selected examples corresponding to both Figures 1 and 5b in Figure 7 of the attached global rebuttal PDF and will provide an extended set as a supplement to the final version of the paper. \n- **Performance on text modalities:** We agree this would be a very interesting investigation. Since B-cos models were primarily used for image classification, we also focused on the same setting in this work. However, exploring how to extend to other modalities, such as text, would be a fruitful direction for future research.\n- **Which models are not B-cosifiable?** Based on our experimental results, we believe this question does not have a yes or no answer - different models might benefit from a B-cosification to different degrees. For example, while the convolutional models we tested adhere closely to the original formulation of the B-cos models and thus lend themselves well to B-cosification with a high degree of localisation, our results regarding the feature aggregation mechanism (attention pooling vs. our proposed approach) suggest that some models might lack the necessary alignment pressure to exhibit equally strong localisation abilities. We appreciate the reviewer’s suggestion and will update our discussion accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed comments and constructive feedback. We are encouraged to find that the reviewers appreciate that our proposed approach allows us to convert pre-trained models to be interpretable whilst maintaining performance (Z7HS, DdYE). We are further encouraged to find that the reviewers find our work to constitute an interesting research direction (71N9, HC7z), our submission to be well structured (DdYE, JC7z), and to have a broad experimental evaluation (JC7z, DdYE). 

While we are happy to find our submission to be generally very positively received, we of course also highly appreciate the various suggestions by the reviewers to improve our work. In the following, we address some of the concerns shared by multiple reviewers, with more detailed responses to be found in the respective rebuttal sections:
- **Repeat experiments multiple times (Z7HS, JC7z):** We now provide results over three runs corresponding to Tables 2, 3, and 4 in Tables 5, 6, and 7, respectively, in the attached PDF. From Tables 5-6, we find that the conclusions in Section 3.2.2 remain unchanged — the simplest approach of directly setting B=2 and removing the bias at the beginning performs as well as more involved strategies such as learning B or decaying bias. Also, from Table 7, we find that the results regarding classification and interpretability performance, as in Section 4.1, are supported by repeated experiments.
- **Localization scores vs. epochs (71N9, DdYE):** Thank you for the suggestion. We now report these results in Figure 6 in the attached PDF. We find both quantitatively (left) and qualitatively (right) that the localization improves very quickly. For example, for ResNet-18, the localization improves from 21.5 to 82.0 after just one epoch of fine-tuning, which is close to 86.9 achieved by the B-cos model trained from scratch. On the other hand, modifications to the model architecture at the beginning of the B-cosification process lead to an initial drop in accuracy, from 69.8 for the standard model to 59.9 after one epoch. However, we recover the accuracy much quicker than training from scratch, with the B-cosified model needing just 29 epochs to reach the accuracy of a B-cos model trained from scratch for 90 epochs.
- Finally, we will carefully proofread the final version of the manuscript and incorporate the valuable suggestions regarding the clarity of the writing and experimental evaluation given by the reviewers.

Meta-review: The paper proposes a method to convert a regular network into a more interpretable network (a B-cos network). This addresses a major limitation in B-cos networks that previously had to be trained from scratch. Interpretability is a challenging and important area, and the results in this paper are compelling. The reviewers unanimously vote for acceptance, and I agree. Congratulations to the authors!
Decision: Accept (poster)