Title: B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable
Venue: NeurIPS 2024
Accepted: yes
Keywords: B-cos networks

Explainability

Inherent Interpretability

CLIP
Abstract: B-cos Networks have been shown to be effective for obtaining highly human interpretable explanations of model decisions by architecturally enforcing stronger alignment between inputs and weight. B-cos variants of convolutional networks (CNNs) and vision transformers (ViTs), which primarily replace linear layers with B-cos transformations, perform competitively to their respective standard variants while also yielding explanations that are faithful by design. However, it has so far been necessary to train these models from scratch, which is increasingly infeasible in the era of large, pre-trained foundation models. In this work, inspired by the architectural similarities in standard DNNs and B-cos networks, we propose ‘B-cosification’, a novel approach to transform existing pre-trained models to become inherently interpretable. We perform a thorough study of design choices to perform this conversion, both for convolutional neural networks and vision transformers. We find that B-cosification can yield models that are on par with B-cos models trained from scratch in terms of interpretability, while often outperforming them in terms of classification performance at a fraction of the training cost. Subsequently, we apply B-cosification to a pretrained CLIP model, and show that, even with limited data and compute cost, we obtain a B-cosified version that is highly interpretable and competitive on zero shot performance across a variety of datasets. We release our
code and pre-trained model weights at https://github.com/shrebox/B-cosification.

B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable

Abstract

B-cos Networks have been shown to be effective for obtaining highly human interpretable explanations of model decisions by architecturally enforcing stronger alignment between inputs and weight. B-cos variants of convolutional networks (CNNs) and vision transformers (ViTs), which primarily replace linear layers with B-cos transformations, perform competitively to their respective standard variants while also yielding explanations that are faithful by design. However, it has so far been necessary to train these models from scratch, which is increasingly infeasible in the era of large, pre-trained foundation models. In this work, inspired by the architectural similarities in standard DNNs and B-cos networks, we propose 'Bcosification', a novel approach to transform existing pre-trained models to become inherently interpretable. We perform a thorough study of design choices to perform this conversion, both for convolutional neural networks and vision transformers. We find that B-cosification can yield models that are on par with B-cos models trained from scratch in terms of interpretability, while often outperforming them in terms of classification performance at a fraction of the training cost. Subsequently, we apply B-cosification to a pretrained CLIP model, and show that, even with limited data and compute cost, we obtain a B-cosified version that is highly interpretable and competitive on zero shot performance across a variety of datasets. We release our code and pre-trained model weights at https://github.com/shrebox/B-cosification.

1 Introduction

Despite their strong performance on a variety of tasks, understanding decisions of deep neural networks (DNNs) remains challenging. Explanation methods, such as feature attributions [46, 48, 53, 5], have been proposed in an attempt to explain such decisions post-hoc , but have often found to be unfaithful to the model being explained [2, 3, 40, 60].

Inherently interpretable Deep Neural Network (DNN) models have recently gained popularity. In contrast to the common approach of explaining existing DNNs in a post-hoc fashion, these models typically feature certain architectural constraints that allow for extracting human-interpretable, modelfaithful simplifications of the models' computations by design ; examples of this include prototypebased [14, 17, 32], dynamic linear [9, 10], or concept-bottleneck models [28, 58, 33, 42]. However, given those architectural changes, this comes at a price: specifically, the models need to be trained from scratch, which-especially in the case of large foundation models, which are increasingly popular-can cost millions of dollars.

To mitigate this, in this work, we explore a novel approach of fine-tuning DNNs for inherent interpretability and propose to 'B-cosify' existing DNNs. Specifically, we investigate whether pre-trained DNNs can simply be efficiently fine-tuned to obtain a similar degree of interpretability as 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

0

45

90

19

Epoch

0

10

20

30

40

50

60

70

80

DenseNet-121

0

45

90

10

Epoch

0

10

20

30

40

50

60

70

80

ViT-S

Accuracy (%)

B-cos Accuracy

German

Shepherd

Initial

After 1 Epoch

Final

Flagpole

Limousine

B-cosified (Ours)

B-cos

the recently proposed B-cos Networks [10, 11]. In contrast to the original B-cos Networks, which leverage existing architectures to obtain performant and interpretable models, we investigate whether we can additionally leverage the existing pre-trained weights , thus aiming to take advantage of the significant amount of resources that have been invested in training existing models. As a result, we hope to make inherently interpretable models more easily accessible to the community.

To do so, we first conduct a detailed analysis of how B-cos DNNs differ from their conventional counterparts. Interestingly, we find that many existing models can be converted into functionally equivalent B-cos models by a small set of targeted implementational modifications (Tab. 1). To increase the interpretability of the models, we then increase the 'alignment pressure' [10] via the parameter B of the B-cos transformations and fine-tune the models on their respective tasks, which leads to significantly more interpretable explanations (Fig. 2).

On supervised settings, we find that B-cosified models often outperform both conventional and B-cos DNNs at a fraction of the full training cost (Fig. 1, left), whilst exhibiting a similar degree of interpretability as the original B-cos DNNs (Fig. 1, right). We further apply B-cosification to a pre-trained CLIP model [39], a large foundation vision-language model (VLM), and show that despite using comparatively limited data and compute cost, B-cosified CLIP models yield highly interpretable explanations whilst being competitive on zero-shot performance across a variety of downstream datasets.

Our work thus opens a new perspective on how to design inherently interpretable models in a costeffective manner. Importantly, on the one hand it highlights that conventional models might be closer to inherently interpretable models than previously understood. On the other hand, it highlights the benefits of designing inherently interpretable models via minor architectural modifications, such as e.g. the B-cos DNNs, as this can allow for leveraging the large array of existing, pre-trained DNNs.

In summary, our contributions are:

We propose B-cosification , a novel technique to 'fine-tune for interpretability', that addresses the problem of high training cost associated with obtaining inherently intepretable models such as B-cos DNNs. Our B-cosified DNNs are highly interpretable while often outperforming both standard and B-cos DNNs.

We thoroughly study different design choices to find an optimal strategy for B-cosification.

We apply B-cosification to supervised image classifiers on ImageNet [16], including both CNNs and ViTs, and show that the B-cosified variants perform on par on interpretability metrics while often outperforming in terms of accuracy. Overall, we find that B-cosifying a pre-trained black box DNN to be superior on both metrics as compared to training a B-cos DNN from scratch, while being computationally significantly cheaper.

Image

'A photo of a

{}

'

sheep

bicycle

bird

boat

bottle

bus

car

horse

cat

CLIP

B-cosified CLIP

Weextend B-cosification to CLIP, a foundation VLM, and show that B-cosified CLIP remains highly competitive on zero-shot performance across a variety of downstream datasets, while also yielding similar interpretability benefits as B-cos models.

2 Related Work

Explanation Methods. Post-hoc attributions [46, 43, 53, 48, 5, 56] have popularly been used to understand the decisions of trained DNNs, but have often been shown to be unfaithful to the model being explained [2, 3, 60, 40]. Inherently interpretable models [14, 28, 10], in contast, incorporate architectural changes to the model and can yield explanations that are interpretable and faithful to the model by design. However, such models need to be trained from scratch, which imposes a significant additional cost. In this work, we explore fine-tuning for interpretability, and propose a method to transform existing black-box DNNs to inherently interpretable B-cos DNNs, bringing together the best of both worlds.

Attribution Priors [38, 37, 27, 50, 4] have often been used to train or fine-tune models to have explanations with desirable properties, such as inducing smoother explanations [27], consistent explanations [37, 38], or to guide models to be 'right for the right reasons' [44, 20, 41, 34]. Similar to such work, we fine-tune black-box DNNs for interpretability, but in contrast, we only make architectural modifications to transform the DNNs to B-cos DNNs, and do not use any additional constraints on the explanations themselves while training.

CLIP Interpretability and Localization. Post-hoc attribution methods [46, 36, 13, 7] have also been used to explain VLMs such as CLIP [39], however, as with supervised DNNs, their faithfulness to the model is not guaranteed and the explanations are often coarse-grained and not very human interpretable. While inherently interpretable architectures could address this, the high costs of training such large models from scratch makes their use unappealing. In this work, we bridge the gap by instead fine-tuning from pre-trained black-box CLIP models to inherently interpretable B-cosified CLIP variants, and find that the B-cosification process is effective in yielding performant and interpretable models. A separate line of work [8] involves improving localizability of VLMs, and is orthogonal to our work since our goal is to obtain explanations that are faithful to the model.

Learning mappings between model features. Recent work [30, 33] has explored using simple linear transforms to map features between models, and in particular also mapping features from arbitrary models to CLIP's representation space. In the context of our work, such methods can be used to map a supervised B-cos feature extractor to CLIP using a linear transform, to obtain an inherently interpretable DNN that can mimic CLIP. In our evaluation, we compare with such an approach, and find that our approach of architecturally transforming the full model and fine-tuning for interpretability yields improved zero shot performance.

3 From conventional to B-cos models

In the following, we describe the process of fine-tuning standard black box DNNs into inherently interpretable B-cos DNNs. In Sec. 3.1, we first introduce the B-cos models and enumerate the key ways in which they differ from standard models. In Sec. 3.2, we then perform a detailed study on strategies to bridge each of these differences for effective B-cosification.

3.1 B-cos Models: Background

Many common DNNs consist of a series of blocks of linear layers followed by non-linear ReLU activations [31], and are thus piece-wise linear functions 1 : i.e., for every input x , they effectively compute a linear transformation of that input: y ( x ) = W ( x ) x + b ( x ) , cf. [52]. In [9, 11], models of this kind have been called 'dynamic linear', a naming convention that we adopt in this paper.

Interestingly, for piece-wise linear models, W ( x ) is given by the models' gradient with respect to x [52]-except for the input-dependent bias b ( x ) , the gradient thus constitutes an exact summary of the models' computations. This linear mapping W ( x ) is unfortunately typically not easily interpretable, and many techniques have been proposed to derive qualitatively more convincing explanations [46, 56]. These, however, have been shown to often not faithfully reflect the underlying model [1, 40].

̸

Further, if the models employ bias terms, W ( x ) does not yield a complete explanation [52], i.e. y ( x ) = W ( x ) x . Integrating bias terms as proposed by [52] yields a set of importance attribution maps, summarizing which requires carefully selecting a post-processing function with inherent tradeoffs. Even when not using bias terms [23], however, the resulting matrices W ( x ) are often not easily human interpretable, and the resulting models can suffer from significant drops in performance.

To address this, [10, 11] propose to architecturally modify the DNNs to introduce additional alignment pressure during model optimisation. For this, they replace the ubiquitously used linear transformation by the B-cos transformation, which dynamically scales the output of the linear transformations:

with B a hyperparameter, cos the cosine similarity between x and the weights w , and w = w / w .

| | × ̂ ̂ ∥ ∥

Like piece-wise linear models, B-cos models are dynamic linear and thus accurately summarised by a single linear transformation W ( x ) s.t. y ( x ) = W ( x ) x ; as B-cos models do not employ bias terms, this model summary is complete. Crucially, it has been shown that with B> 1 , the matrix W ( x ) aligns with task-relevant input patterns, making it more easily human interpretable (e.g. Fig. 1, right).

Importantly, as the B-cos transformation can serve as a drop-in replacement for linear transformations at every layer of a DNN, it is possible [10, 11] to leverage existing DNN architectures and the resulting B-cos models obtain similar classification accuracies as their conventional counterparts (Tab. 4, cols. 'pretrained' and 'B-cos').

Extending this, we investigate if it is possible to leverage existing DNN weights -i.e., our goal is to fine-tune existing models to be similarly interpretable as B-cos models, whilst not requiring to train them from scratch. However, despite the architectural similarities between B-cos and conventional models, there are multiple key differences that make transforming pre-trained models into B-cos models non-trivial: e.g., apart from replacing linear transformations with the B-cos transformation and not employing biases, B-cos models are trained on image representations with 6 color channels as [ r, g, b, 1 -r, 1 -g, 1 -b ] to be able to visualise the model-inherent linear summaries W ( x ) in color, whereas conventional models use 3 channels (see also Tab. 1). In the next section, we show how to overcome these differences and convert existing models into functionally equivalent B-cos models.

3.2 B-cosification of Deep Neural Networks

We analysed the differences between B-cos models and their conventional counterparts in detail and compiled the results in Tab. 1. In this section, we discuss one by one how to bridge these differences. In particular, we show that a conventional model can be framed as a functionally equivalent B-cos model as in [11] with B =1 , which additionally employs bias terms. Only upon modifying these two aspects, i.e. biases and B , does the model need to be fine-tuned to adapt the weights to those changes.

3.2.1 Functionally Equivalent B-cos Models

Input Encoding and Normalisation. As mentioned in Sec. 3.1, B-cos models use input representations with six color channels [ r, g, b, 1 -r, 1 -g, 1 -b ] to be able to visualise the explanations in color, cf. [11]. However, most conventional DNNs (e.g. models from Torchvision [54], CLIP [39]) are applied to 3-channel inputs in which images are encoded via [ r, g, b ] . As a result, visualising the dynamic matrices W ( x ) of piece-wise linear models (cf. Sec. 3.1) in color would not seem possible.

However, we note that in combination with the commonly used input normalisation , we can convert the first linear transformation in conventional models (e.g., a convolutional layer) into an equivalent transformation that accepts 6-channel inputs. Specifically, for input normalisation, the channel-wise means µ s are subtracted from the individual channels, followed by a division by the standard deviations σ s , yielding s ′ =( s -µ s ) /σ s for s ∈{ r, g, b } . Conversely, mean-normalising the 3 additional color channels yields -s ′ . Leveraging this, we use the models' weights learnt for 3-channel inputs, w j = [ w j,r , w j,g , w j,b ] for every feature j , to construct an equivalent 6-channel transformation:

Note that applying w ′ j to the mean-normalised, 6-channel inputs yields the same results as applying w j to the original mean-normalised inputs that the pre-trained models have seen during training.

Activation Functions. Owing to the non-linearity inherent to the B-cos transform, explicit activation functions are not necessary in between B-cos layers. However, the authors of [10, 11] showed that the model-inherent explanations are compatible with MaxOut [21]. Note that the very commonly used ReLU non-linearity applied to v T x for any weight vector v , is just a special case of MaxOut:

As the pre-trained models' weights have been optimised for the ReLU non-linearity and given its compatibility with the B-cos explanations, we leave them untouched in the B-cosification process.

Weight normalization. B-cos transformations employ unit norm weights, see also Eq. (1), which the authors motivated by the fact that the only way any given neuron can achieve its maximal output is by increasing the weight-input alignment, which in turns leads to the improvements of the explanations.

However, conventional models have been trained with unconstrained weights and using unit norm weights would thus lead to unpredictable model behaviour. Interestingly, we note that the weight normalisation in the latest version of the B-cos models can actually not impact the explanation quality, as the authors of [11] re-introduce normalisation layers into the B-cos models. To better understand this, let us consider the compound function of a batch normalisation layer and a B-cos layer:

with α and β trainable parameters of the BatchNorm layer. Note that √ var is scaled by any factor γ by which the output of a B-cos layer might be scaled, which cancels in the fraction in Eq. (5) and thus makes f ( x ) invariant to scaling the B-cos transformation: i.e. BatchNorm ( y ) = BatchNorm ( γ × y ) .

In particular, the output of f ( x ) is thus invariant to weight normalisation, as the output of B-cos ( x ) scales linearly with the weight norm, cf. Eq. (1).

This is of course only true if every B-cos layer were always followed by a normalisation layer, which is not necessarily the case. Nonetheless, we find that not using normalised weights yields consistently good results across all models. Therefore, we use B-cos transformations without weight normalisation throughout our experiments; for an ablation, see Tab. B2 in the appendix.

In summary, we showed that it is possible to adapt the implementation of existing models in a way that allows us to integrate certain aspects of B-cos models without functionally changing the pre-trained models. Notably, we can now visualise color explanations similar to B-cos models (Fig. 1, right, col. 3); unsurprisingly, however, these explanations have poor interpretability due to the absence of the alignment pressure imposed during B-cos training. In the next section, we discuss the necessary functional changes for B-cosification to obtain interpretable explanations.

3.2.2 Fine-tuning for Interpretability

The changes introduced in the preceding section have not functionally changed the pre-trained models, but rather allow us to interpret the existing models as a special case of B-cos models. Now we introduce the necessary changes to increase the interpretability of the dynamic matrices W ( x ) . As these functionally change the models, they need to be fine-tuned to recover their original performance.

In particular, the remaining differences between conventional and B-cos models are (1) the value of B , and (2) the use of biases, Tab. 1. We will now discuss how we bridge these differences individually.

Ablation Setup. We evaluate various fine-tuning strategies using a ResNet-18 [22] model supervised on ImageNet [16] from Torchvision [54] for B-cosification, and compare with a B-cos ResNet-18 from [11]. We optimize using AdamW [26] with cosine scheduling and train for 90 epochs, and evaluate both classification accuracy as well as interpretability using the GridPG metric [9].

(1) Increasing B . As shown in [10], using B> 1 is critical to obtain easily interpretable explanations. To increase B for the pre-trained models, we investigate three strategies: (1) immediately setting B to a higher value and then fine-tuning, (2) linearly interpolating from B = 1 to B = 2 throughout fine-tuning, and (3) setting B as a learnable parameter. (2) has the advantage of changing the model in small steps, making it more likely that it maintains performance while fine-tuning, but requires using the full number of epochs to reach the target value of B . (1) on the other hand is likely to adversely affect the utility of the weights, but offers the opportunity to stop fine-tuning early if performance and interpretability metrics are sufficiently high. (3) offers the most flexibility, but also adds a new set of parameters that need to be optimized. We show the results of this evaluation in Tab. 2. Interestingly, we find that using (1), i.e. setting B = 2 and then fine-tuning, yields performance that is on par with learnable B parameters, whilst being significantly simpler to implement. To easily test the generality of the B-cosification scheme, we therefore opt for this approach in Sec. 4.1.

(2) Decreasing biases. As discussed in Sec. 3.1, dynamic linear models with bias terms are not exactly summarised by the matrix W ( x ) , cf. [52]. To obtain the same level of faithfulness of the explanations as B-cos models (in particular w.r.t. explanation completeness, cf. [52, 53]), we need to remove the biases from the model. To do so, we investigate two approaches: (1) removing all biases first and then fine-tuning, and (2) fine-tuning while decaying biases using weight decay. Similar to the setup with B , (2) has the advantage of avoiding drastic changes to the model, but requires potentially fine-tuning for longer. Further, the weight given to the bias decay in the loss constitutes a tradeoff between maintaining classification performance and pushing the biases to be close to zero. We report the results of this evaluation in Tab. 3. Similarly to the experiments for B , we find that immediately setting the biases to zero constitutes a simple yet performant approach to achieve both good localisation and accuracy. To assess the generality of the B-cosification scheme across a wide range of models, we thus choose this the simpler approach of setting biases to zero in Sec. 4.1.

In short, we find that a very simple approach, i.e., setting the bias and the B values to the target values immediately, constitutes a simple and easy-to-use, but nonetheless performant strategy to B-cosify models. In the following sections, we test whether these findings generalise well to other models.

4 B-cosification Results

In the following, we evaluate the effectiveness of the B-cosification strategy we developed in Sec. 3. In Sec. 4.1, we first apply B-cosification to supervised models across various architectures, and evaluate for classification performance and interpretability. In Sec. 4.2, we B-cosify CLIP [39], a large foundation vision-language model, and show that despite fine-tuning at a fraction of the training cost, the B-cosified CLIP shows strong zero shot generalization whilst being highly interpretable.

4.1 Supervised Classification Models

Setup. We B-cosify models from Torchvision [54] supervised on ImageNet [16]. We use a diverse set architectures, including both CNNs (ResNet-18 [22], ResNet-50 [22], and DenseNet-121 [24]), and ViTs [18, 6, 57] with (ViT c -Ti, ViT c -S, ViT c -B, ViT c -L) and without (ViT-Ti, ViT-S, ViT-B, ViT-L) convolutional stems. For ResNet-50, we use both the weights originally released by Torchvision and the updated V2 weights, which constitute models trained for longer and with more augmentations [55]. As in Sec. 3.2, we evaluate both for classification accuracy and for interpretability using the GridPG [9] metric. We compare both accuracy and interpretability of the B-cosified models with B-cos models trained from scratch from [11]. For interpretability, we also compare with several post-hoc attribution methods as baselines, namely Guided Backprop [51], Gradient [49], DeepLIFT [48], IxG [48], IntGrad [53], and GradCAM [46]. For full details, see Appendix C.1.

Classification performance. Tab. 4 reports the classification accuracy of the B-cosified models across architectures, and compares them with their conventional counterparts from Torchvision and B-cos models trained from scratch. We find that across architectures (col. 1), B-cosified models perform competitively with conventional DNNs (cols. 2-4) and interestingly, in contrast to the findings reported by [11], often outperform them, i.e. for five out of twelve architectures. Notably, we find (col. 5) that our B-cosified models significantly outperform B-cos models trained from

ResNet-18

ResNet-50-v1

ResNet-50-v2

DenseNet-121

ViT-Ti

ViT-S

ViT-B

ViT-L

ViT

c

-Ti

ViT

c

-S

ViT

c

-B

ViT

c

-L

0

20

40

60

80

100

Localisation (%)

+65.9

+67.6

+67.4

+71.0

+42.9

+46.9

+43.7

+41.2

+38.9

+43.6

+33.4

+32.7

W

(

x

)

x

B-cosified Model

W

(

x

)

x

B-cos Model

W

(

x

)

x

0

20

40

60

80

100

Localisation (%)

ResNet-50-v1

DenseNet-121

Oracle attributions

Uniform attributions

GB

Grad

DeepLIFT

IxG

IntGrad

GradCAM

W

(

x

)

x

B-cosified

scratch across all but the two largest ViT c architectures. Further, B-cosified models often achieve the same performance as their corresponding B-cos models at a fraction of the training cost (col. 6). Specifically, we find that averaged across architectures, B-cosified models outperform B-cos models trained from scratch by 2.5 pp, with an average training speedup (to match performance) of 5.2x. These results strongly advocate for B-cosification as a superior alternative to training from scratch for obtaining performant inherently interpretable models at a low compute cost.

Interpretability. To evaluate the interpretability of our B-cosified models, we report the GridPG localization scores in Fig. 3, and compare with conventional and B-cos models; following [11], we report the results of 3x3 image grids for convolutional models, and of 2x2 grids for the ViTs. For a fair comparison, for all models, we evaluate the localization of the dynamic linear summary of the model 2 W ( x ) x (see Sec. 3.1). We find that across architectures, B-cosified models significantly outperform conventional DNNs in terms of localization (32.7pp-71.0pp) and perform on par with B-cos models. Since post-hoc attribution methods (e.g. [46, 48, 53]) are often used to interpret conventional DNNs, similar to [10], in Fig. 4, we compare the localization of the model inherent explanations from two of our B-cosified models with post-hoc explanations applied to the corresponding conventional models. Similar to the results reported by [10], we find our B-cosified models to strongly outperform all post-hoc methods, including GradCAM [46], with a near perfect localization score, showing that B-cosification is effective in yielding highly interpretable yet model-faithful explanations.

Impact of pre-trained weights. Since our aim is to fine-tune for interpretability, we investigate how crucial the quality of the weights of the conventional model are for effective B-cosification. Specifically, we expect weights from stronger models to be a better starting point for B-cosification. We evalaute this by performing B-cosification both with v1 and v2 variants of ResNet-50 [22] from Torchvision [54], where the latter is trained for longer and with stronger augmentations [55]. From Tab. 4, we find that using a strong initialization is highly useful for effective B-cosification,

Natural Data

Specialized Data

Structured Data

0

20

40

60

80

100

Accuracy (%)

34.6

47.2

44.7

51.5

34.9

34.3

39.0

45.2

9.9

10.8

11.4

10.3

Zero-shot Evaluation

Natural Data

Specialized Data

Structured Data

68.5

76.7

74.1

77.8

80.9

82.5

83.3

85.5

43.0

48.6

48.8

48.5

Linear Probe Evaluation

Text2Concept

B-cosified (IMN)

B-cosified (CC3M)

Standard

specifically, fine-tuning starting from v2 weights outperforms B-cos models by 0.8pp as compared to v1, and achieves equal accuracy with a 9x speedup as compared to 2x with v1; similar results are observed for initialising the weights with models pretrained via the self-supervised DINO paradigm [12] (final accuracy: 77.0, speedup: 3.2x), for further discussion see Tab. B3 in the appendix.

4.2 B-cosifying CLIP - Towards Inherently Interpretable Foundation Models

In this section, we evaluate our B-cosification paradigm on CLIP [39], a powerful pre-trained vision-language model, and evaluate its interpretability and zero shot performance.

Setup. We B-cosify a CLIP [39] with a ResNet-50 [22] backbone using the procedure described in Sec. 3.2. We use the recently proposed SigLIP loss [59] between the image embeddings of the pre-trained CLIP and the B-cosified CLIP's and train the models on either the ImageNet [16] or the CC3M datasets [47]. For evaluation, we rely on the CLIP Benchmark [15] and report zeroshot and linear probing results for accuracy. To assess the models' interpretability, we explain the similarity between the image embeddings and the text embedding of the pre-trained CLIP model via the dynamic linear summaries, see Sec. 3.1 or GradCAM, and report the EPG scores [56, 41] on the VOC dataset [19]. For full details, see Appendix C.2.

Evaluating Model Performance. In Fig. 5, we report the zeroshot and linear probing accuracies of the two B-cosified CLIP models (trained on ImageNet or CC3M) and compare it to the original CLIP (Standard) and the recently proposed Text2Concept technique [30]; for the latter, we train a linear layer on top of a frozen, pre-trained B-cos ResNet-50 from [11] to mimic the embeddings of CLIP [30]. We find that the B-cosified models significantly outperform the Text2Concept approach and achieve accuracies that are more similar to the original CLIP's zeroshot and linear probing accuracies.

Evaluating Model Interpretability. We evaluate the B-cosified CLIP's ability to localise classes in the VOC dataset in two ways. On the one hand, we directly explain the similarity of the models' embedding to the text embedding of a given prompt such as 'A photo of a cow.'. On the other hand, we note that the final attention pooling layer in the CLIP model only computes a weighted sum of the last layer's value vectors. Therefore, we additionally evaluate whether we can also explain the similarity between the text embeddings and these value vectors to improve the localisation ability.

In this context, we notice that explaining the average similarity to the text embedding yields highly distributed attribution maps, see Fig. 6b, col. 2. On the other hand, explaining only the most similar embedding localises very well, see Fig. 6b, col. 5. To better understand this phenomenon, we additionally interpolate between these two approaches and compute weighted means ∑ i w i v i of those value vectors v i , in which the weights are determined by the cosine similarity between the value vectors v i and the text embedding t , i.e. with weights w i = cos p ( t , v i ) for various p .

We find that this not only significantly improves the explanations qualitatively, see Figs. 2 and 6b, but also quantitatively: in Fig. 6a we report results for explaining the final image embedding ( Bcosified CLIP ), the dynamic linear summary for the CLIP ResNet-50 ( CLIP W ( x ) x ), its GradCAM explanations ( CLIP GradCAM ), and the weighted mean of the value vectors, which we call

1

3

5

7

9

11

13

15

17

19

inf

Cosine Power

35

40

45

50

55

60

EPG Score (%)

B-cosified FeatureCLIP

CLIP GradCAM

CLIP W(x)x

B-cosified CLIP

(a)

(b)

‘A photo of a aeroplane"

COS

COS-7

COS-19

cos-inf

"A photo of a bicycle"

"A photo of a cow"

B-cosified FeatureCLIP . While B-cos CLIP already yields a noticeable improvement over the pre-trained CLIP explained via GradCAM and significantly improves the localisation ability of the linear summary W ( x ) x , the weighted means yield even stronger performance for sufficiently high p .

5 Discussion

The B-cosification approach presented in this work addresses a common issue with developing inherently interpretable models: achieving model interpretability without compromising on performance or incurring high training costs. By leveraging pre-trained models, B-cosification opens a new path towards developing interpretable yet performant models, which can be of particular interest in the context of foundation models such as CLIP [39], which might otherwise be prohibitively expensive to train on a limited budget. Our results suggest that B-cosification not only maintains but, in several cases, even enhances model accuracy, whilst yielding significant improvements on interpretability metrics, providing a viable and resource-efficient alternative to training B-cos models from scratch.

Specifically, we find B-cosified models to much faster reach the same levels of interpretability and accuracy than their counterparts trained from scratch, with training speedups of up to 9x in some models. The approach appears to be general, being applicable for both CNNs and ViT models. We hope that this increase in efficiency will make interpretable models much more accessible in settings with constrained computational resources and could thus facilitate their adoption. In particular, when applying our proposed B-cosification scheme to a foundation model-CLIP-we find that the B-cosified CLIP model is able to maintain competitive zero-shot performance while at the same time providing interpretable and model-faithful explanations.

Despite these advancements, certain aspects remain open for further exploration. Specifically, while some models quickly recover original performance after B-cosification, others exhibit slower convergence rates, suggesting potential for optimisations in the fine-tuning process. Additionally, for the larger B-cosified ViT c models, while yielding results that are on par with those trained from scratch, the B-cosification process did not succeed in achieving speed-ups, indicating that the interplay between model architecture and the proposed B-cosification might require further exploration.

In summary, our results establish B-cosification as an effective method for enhancing interpretability in pre-trained models with low computational cost. The method consistently enables high interpretability without compromising performance, even achieving substantial training speedups in many cases.

References

J. Adebayo, J. Gilmer, I. Goodfellow, and B. Kim. Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values. In ICLRW , 2018.

J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity Checks for Saliency Maps. In NeurIPS , 2018.

J. Adebayo, M. Muelly, H. Abelson, and B. Kim. Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation. In ICLR , 2022.

S. Asgari, A. Khani, F. Khani, A. Gholami, L. Tran, A. Mahdavi-Amiri, and G. Hamarneh. MaskTune: Mitigating Spurious Correlations by Forcing to Explore. In NeurIPS , 2022.

S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation. PloS one , 10(7):e0130140, 2015.

L. Beyer, X. Zhai, and A. Kolesnikov. Better plain vit baselines for imagenet-1k, 2022. URL https://arxiv.org/abs/2205.01580 .

W. Bousselham, A. Boggust, S. Chaybouti, H. Strobelt, and H. Kuehne. LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity. arXiv preprint arXiv:2404.03214 , 2024.

W. Bousselham, F. Petersen, V. Ferrari, and H. Kuehne. Grounding Everything: Emerging Localization Properties in Vision-Language Transformers. In CVPR , 2024.

M. Böhle, M. Fritz, and B. Schiele. Convolutional Dynamic Alignment Networks for Interpretable Classifications. In CVPR , 2021.

M. Böhle, M. Fritz, and B. Schiele. B-cos Networks: Alignment is All We Need for Interpretability. In CVPR , 2022.

M. Böhle, N. Singh, M. Fritz, and B. Schiele. B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers. IEEE TPAMI , 2024.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV , 2021.

H. Chefer, S. Gur, and L. Wolf. Generic Attention-Model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers. In ICCV , 2021.

C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. In NeurIPS , 2019.

CLIP Benchmark maintainers and contributors. CLIP Benchmark. https://github.com/ LAION-AI/CLIP_benchmark , 2024.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR , 2009.

J. Donnelly, A. J. Barnett, and C. Chen. Deformable ProtoPNet: An Interpretable Image Classifier using Deformable Prototypes. In CVPR , 2022.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR , 2021.

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. IJCV , 88:303-308, 2009.

Y. Gao, T. S. Sun, G. Bai, S. Gu, S. R. Hong, and Z. Liang. RES: A Robust Framework for Guiding Visual Explanation. In KDD , 2022.

I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout Networks. In ICML , 2013.

K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR , 2016.

R. Hesse, S. Schaub-Meyer, and S. Roth. Fast Axiomatic Attribution for Neural Networks. In NeurIPS , pages 19513-19524, 2021.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely Connected Convolutional Networks. In CVPR , pages 4700-4708, 2017.

A. Khaddaj, H. Salman, A. Ilyas, G. Leclerc, and A. Madry. Extra Training Provides a Strong Baseline for CLIP. In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models , 2023. URL https://openreview.net/forum?id=v3JJmLYk12 .

D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In ICLR , 2015.

K. Kiritoshi, R. Tanno, and T. Izumitani. L1-Norm Gradient Penalty for Noise Reduction of Attribution Maps. In CVPRW , pages 118-121, 2019.

P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept Bottleneck Models. In ICML , pages 5338-5348, 2020.

N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Reblitz-Richardson. Captum: A Unified and Generic Model Interpretability Library for PyTorch. arXiv preprint arXiv:2009.07896 , 2020.

M. Moayeri, K. Rezaei, M. Sanjabi, and S. Feizi. Text-to-Concept (and Back) via Cross-Model Alignment. In ICML , pages 25037-25060, 2023.

V. Nair and G. E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML , pages 807-814, 2010.

M. Nauta, R. Van Bree, and C. Seifert. Neural Prototype Trees for Interpretable Fine-grained Image Recognition. In CVPR , pages 14933-14943, 2021.

T. Oikarinen, S. Das, L. M. Nguyen, and T.-W. Weng. Label-Free Concept Bottleneck Models. In ICLR , 2023.

A. Parchami-Araghi, M. Böhle, S. Rao, and B. Schiele. Good Teachers Explain: ExplanationEnhanced Knowledge Distillation. In ECCV , 2024.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS , 2019.

S. Petryk, L. Dunlap, K. Nasseri, J. Gonzalez, T. Darrell, and A. Rohrbach. On Guiding Visual Attention with Language Specification. In CVPR , pages 18092-18102, 2022.

V. Pillai and H. Pirsiavash. Explainable Models with Consistent Interpretations. In AAAI , 2021.

V. Pillai, S. A. Koohpayegani, A. Ouligian, D. Fong, and H. Pirsiavash. Consistent Explanations by Contrastive Learning. In CVPR , pages 10213-10222, 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML , pages 8748-8763, 2021.

S. Rao, M. Böhle, and B. Schiele. Towards Better Understanding Attribution Methods. In CVPR , pages 10223-10232, 2022.

S. Rao, M. Böhle, A. Parchami-Araghi, and B. Schiele. Studying How to Efficiently and Effectively Guide Models with Explanations. In ICCV , pages 1922-1933, 2023.

S. Rao, S. Mahajan, M. Böhle, and B. Schiele. Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery. In ECCV , 2024.

M. T. Ribeiro, S. Singh, and C. Guestrin. 'Why Should I Trust You?': Explaining the Predictions of Any Classifier. In KDD , pages 1135-1144, 2016.

A. S. Ross, M. C. Hughes, and F. Doshi-Velez. Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. In IJCAI , pages 2662-2670, 2017.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV , 115(3):211-252, 2015.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In ICCV , pages 618-626, 2017.

P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset for Automatic Image Captioning. In ACL , pages 2556-2565, 2018.

A. Shrikumar, P. Greenside, and A. Kundaje. Learning Important Features Through Propagating Activation Differences. In ICML , pages 3145-3153, 2017.

K. Simonyan, A. Vedaldi, and A. Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In ICLRW , 2014.

K. K. Singh, D. Mahajan, K. Grauman, Y. J. Lee, M. Feiszli, and D. Ghadiyaram. Don't Judge an Object by its Context: Learning to Overcome Contextual Bias. In CVPR , pages 11070-11078, 2020.

J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for Simplicity: The All Convolutional Net. In ICLRW , 2015.

S. Srinivas and F. Fleuret. Full-Gradient Representation for Neural Network Visualization. In NeurIPS , 2019.

M. Sundararajan, A. Taly, and Q. Yan. Axiomatic Attribution for Deep Networks. In ICML , pages 3319-3328, 2017.

TorchVision maintainers and contributors. TorchVision: PyTorch's Computer Vision Library. https://github.com/pytorch/vision , 2016.

V. Vryniotis. How to Train State-of-the-Art Models Using TorchVision's Latest Primitives. https://pytorch.org/blog/ how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/ , 2021. Accessed: 2023-05-23.

H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu. Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks. In CVPRW , pages 111-119, 2020.

T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Girshick. Early convolutions help transformers see better, 2021. URL https://arxiv.org/abs/2106.14881 .

M. Yuksekgonul, M. Wang, and J. Zou. Post-hoc Concept Bottleneck Models. In ICLR , 2023.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid Loss for Language Image PreTraining. In ICCV , pages 11975-11986, 2023.

Y. Zhou, S. Booth, M. T. Ribeiro, and J. Shah. Do Feature Attribution Methods Correctly Attribute Features? In AAAI , pages 9623-9633, 2022.

B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable

Appendix

Table of Contents

A Additional Qualitative Results

In Fig. A1, we provide additional qualitative examples to illustrate the interpretability gains achieved by B-cosifying a CLIP model. Specifically, we show explanations generated by the original CLIP model using GradCAM [46] (row 2) for a diverse set of input images (row 1), for which explanations are generally coarse and lack clear localization. In contrast, the third row displays explanations produced by B-cosified CLIP, which yields finer-grained, more visually interpretable explanations.

person

bottle

car

car

chair

car

6op

bop

bird

Image

CLIP

B-cosified

CLIP

In Fig. A2, we show further comparisons on specific object classes with using different cosine powers p (cos, cos-7, cos-19, and cos-inf) to qualitatively demonstrate the effect of increasing the exponent p in gathering the value vectors, see also Sec. 4.2. Higher cosine thresholds result in increasingly focused and interpretable representations, capturing fine details that are often absent in the original CLIP explanations.

In Fig. A3, we show additional qualitative examples for prompting the B-cosified CLIP model with different prompts for the same image, thus highlighting the class-specificity of the explanations as well as the potential that inherently interpretable CLIP models might yield. Specifically, B-cosified CLIP models allow to explain the similarity of a given image with a free-form textual prompt, which shows that the zero-shot performance of CLIP with respect to classification also transfers well to the corresponding explanations.

aeroplane

cOS

COS-7

COS-19

cos-inf

pottedplant

COS

COS-7

COS-19

cos-inf

bird

dog

dog

dog

cup

heart

cherry

ice cream

Ice Cream

girl

boy

hand

water jug

Water Jug

ship

water

fortress

mountain

Schooner

blue

white

green

orange

Pop Bottle

B Additional Quantitative Results

In this section, we provide a series of additional quantitative results on the performance and interpretability of B-cosified models. These tables cover various ablation studies, comparisons with standard and B-cos models, and performance across different configurations. Specifically, in Tab. B1, we extend our evaluation to compare models that are trained for the same effective number of epochs 3 . Further, in Tab. B2, we evaluate the impact of using normalized weights in the B-cos layers. In Tab. B3, we show additional results for B-cosified ResNet-50 models that are initialised from different pre-trained checkpoints. Specifically, we find that apart from initialising the ResNet-50 from CLIP weights, we observe consistent improvements through B-cosification, with stronger pre-training paradigms that are based on the ImageNet dataset (V2, DINO [12]) leading to larger improvements. Finally, we report full ablation results on the impact of the individual changes that we perform on the pre-trained models Tab. B4, B-cosifying models with different strategies for setting the parameter B in the B-cos transformation Tab. B5, decaying the bias term Tab. B6, as well as full zero-shot and linear probing results for the CLIP benchmark, see Tabs. B7 and B8.

Training for 180 Epochs

×

×

C Implementation Details

We implement our code in Pytorch [35] for all the experiments and use Captum [29] for visualisations.

C.1 Standard Models

C.1.1 Models

We B-cosify models from Torchvision [54] supervised on ImageNet [16]. We use a diverse set architectures, including both CNNs (ResNet-18 [22], ResNet-50 [22], and DenseNet-121 [24]), and ViTs [18, 6, 57] with (ViT c -Ti, ViT c -S, ViT c -B, ViT c -L) and without (ViT-Ti, ViT-S, ViT-B, ViT-L) convolutional stems. For ResNet-50, we use both the weights originally released by Torchvision and the updated V2 weights, which constitute models trained for longer and with more augmentations [55].

C.1.2 Datasets

Weuse ImageNet [16] to fine-tune all the B-cosified standard models and evaluate them on ImageNet's validation set. For training, we use train transforms - crop size of 224, horizontal flip with 0.5 probability, random resized crop of 224 with bilinear interpolation, Add Inverse transform [10] and modified mean-std normalisation (to accommodate for 6 channel input from the AddInverse). For evaluation, instead of a random resized crop, we do a center crop with a crop size of 224.

C.1.3 Optimization

For each architecture, we use the B-cosification stategy derived in Sec. 3, and fine-tune for 90 epochs using the AdamW optimizer [26] and cosine scheduling for the learning rate learning rate of 10 -4 for the convolutional models (since the standard pre-trained models end with a learning rate of 10 -4 at the 90 th epoch, from which we want to fine-tune further). For ViTs, as the learning rate decays to a very small value, we tested with different learning rates ( 10 -3 , 10 -4 , 10 -5 ) and found 10 -3 worked best for all the models. Also, we only use a linear learning rate warmup of 10,000 steps with a decay of 0.01 for the base and large ViT models.

C.1.4 Experiments

Increasing B: We tested three different setups for increasing B. 1) Discrete B setting to 1 , 1 . 25 , 1 . 5 , 1 . 75 , 2 , 2 . 5 , 3 , 5 , 7 ; 2) Linear increase of B in n epochs from B=1 to B=2. We used n = 5 , 10 , 20 , 45 , 90 ; 3) Learning B parameter to increase to B=2 using weight decay with coefficients 0 . 2 , 0 . 5 and 0 . 9 . See Tab. 2 for results.

Removing biases: We test two setups for removing the biases from the network: 1) Removing all the bias parameters; 2) Decay the bias parameter using the weight decay with coefficients 0 . 5 and 0 . 9 . See Tab. 3 for results.

Impact of pre-trained weights: To check the impact of pre-trained weights on fine-tuning, we fine-tuned weights from CLIP [39] ResNet-50 [22] , DINO ResNet-50 [12], and Torchvision [54] ResNet-50 weights v1 and v2 (long trained recipe) [55].

C.1.5 Evaluation

As in Sec. 3.2, we evaluate both for classification accuracy and for interpretability using the GridPG [9] metric. We compare both accuracy and interpretability of the B-cosified models with B-cos models trained from scratch from [11]. For interpretability, we also compare with several post-hoc attribution methods as baselines, namely Guided Backprop [51], Gradient [49], DeepLIFT [48], IxG [48], IntGrad [53], and GradCAM [46]. Qualitatively, we visualize the colored B-cos explanations and the attribution maps [11].

C.2 CLIP Models

We use a CLIP [39] ResNet-50 [22] model for B-cosification.

C.2.1 Datasets

We use ImageNet [16] and CC3M [47] to fine-tune all the B-cosified CLIP models and test them on multiple datasets from CLIP benchmark [15]. For training, we use the same transform setup as the standard B-cosified models. We use train transforms - crop size of 224, horizontal flip with 0.5 probability, random resized crop of 224 with bilinear interpolation, and Add Inverse transform [10] and modified mean-std normalisation (to accommodate for 6 channel input from the AddInverse) as discussed in the paper. For evaluation, instead of a random resized crop, we do a center crop with a crop size of 224.

C.2.2 Evaluation

We use the CLIP benchmark [15] for zeroshot and linear probing experiments with the default parameters provided in the official benchmarking code. For text-based localisations, we use the text-based templates from the CLIP for the ImageNet dataset and use them to encode the text features. As text encoder, we use the CLIP ResNet-50 text encoder. The cosine scores between the B-cosified CLIP's image encoding and the pre-trained text encoder are used to do B-cos style localisations and calculate the GridPG scores. We use the unpooled features technique at inference to increase the localisation focus.

We use B-cosified CLIP ResNet-50 fine-tuned on ImageNet using SigLIP loss [59] and cosine scheduling for visualisation.

C.2.3 Optimization

We use the Adam optimizer [26] and fine-tuned models till 90 epochs, while the CC3M models are fine-tuned for 30 epochs. The size of CC3M is approximately three times that of ImageNet, so the trained models are comparable. Keeping consistent with the Standard B-cosification recipe, we train with a learning rate of 1e-4 using cosine scheduling. SigLIP contrastive loss [59] is used to train the models.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and intro we claim to fine-tune models for inherent interpretability (by converting them to the recently proposed B-cos models) and that we find that our proposed scheme (1) outperforms training from scratch, (2) often recovers the original performance, (3) incurs lower training cost than training from scratch, (4) improves the inherent interpretability of the model, and that (5) it lends itself even to fine-tuning foundation models like CLIP for an increase in inherent interpretability. This is exactly what we do in our paper: e.g., Tab. 4 shows (1), (2), and (3), Fig. 3 and Fig. 4 show improvements in interpretability (4), and Sec. 4.2 is dedicated to analysing the performance and interpretability of our B-cosified CLIP (cf. Fig. 5 and Fig. 6a).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see the paragraph on limitations in Sec. 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our theoretical arguments are contained in Sec. 3.2.1, namely that (1) it is possible to convert the first layer of DNNs such that they accept 6 channel inputs as required by B-cos models, that (2) ReLU is a special case of MaxOut, and that (3) weight normalisation is irrelevant if a layer is followed by a batch normalisation layer. These claims are shown to be true in the context of Eq. (2) (1), Eq. (3) (2) and Eq. (5) (3).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: To the best of our knowledge, all necessary details to reproduce our results are contained in the submission. We mainly build on publicly available code (for B-cos models, see https://github.com/B-cos/B-cos-v2/; for CLIP, see https://github.com/openai/CLIP/; for CLIP Benchmark, see github.com/LAIONAI/CLIP_benchmark and https://github.com/mlfoundations/open_clip) and will make all our modifications available for ensuring full reproducibility of the reported results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We mainly rely on publicly available code and established training paradigms and clearly describe any changes we introduce. That said, we make our code available to ensure full reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We rely on standard datasets (CC3M, ImageNet-1k) or publicly available benchmarks (github.com/LAION-AI/CLIP_benchmark) and describe our training and evaluation procedure in detail; see the setup sections in Sec. 3.2.2, Sec. 4.1, Sec. 4.2, and Sec. C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Given the scope of ablations (Sec. 3.2.2), datasets and models (Sec. 4), as well as computational cost (see question 8 in the checklist), each experimental result is currently based on three training runs. That said, we observe consistent behaviour across all our experiments, which corroborates the validity of our claims (see, e.g., Tab. 2 and Tab. 3).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use NVIDIA A100-SXM4-40GB and Quadro RTX 8000 GPUs from internal cluster. We require 4 GPUs per run (except ViTs with 8 A100GPUs for ViT models). Runtime per run: 1-2 days depending on model size for standard models and ImageNet fine-tuned CLIP; for CC3M fine-tuned CLIP - 28 days for 90 epochs runs on Quadro RTX 8000 GPUs. We have 500 experiment runs in total approximately (CC3M - 5 runs, ViTs - 150 runs, 345 rest of the models). For storing all the experimental results, we utilize approximately 2.5 TB of storage.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: With our work, we make a contribution towards making Deep Neural Networks more explainable, which we believe has an overwhelmingly positive societal impact.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We believe the interpretability of models to be important for society to trust models and validate their predictions, and for researchers and developers to be able to understand failure cases and debug models. As we discuss in Sec. 5, our work makes a step in this direction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We propose a technique towards making existing models more interpretable, which-to the best of our understanding-does not necessitate putting in place any safeguards regarding the release of our code and models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We follow standard practice and cite every author of any existing asset that we use. OpenAI CLIP - MIT License, CLIP Benchmark - MIT License, B-cosV2 - Apache-2.0 license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not provide any new assets alongside this submission.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This question does not apply to our submission.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This question does not apply to our submission.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes B-cosification, a method to transform a pre-trained network to a B-cos network [1,2]. Consequently, the transformed model can be finetuned for better explanations of the model behavior while retaining predictive performance. Experiments are done on various CNN and Transformer architectures, as well as a case study on CLIP.\n\n\n[1] Böhle, Moritz, Mario Fritz, and Bernt Schiele. \"B-cos networks: Alignment is all we need for interpretability.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Böhle, Moritz, et al. \"B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).\n\nSTRENGTHS:\n* B-cos networks are an upcoming, interesting research direction. The need to train a network from scratch is commonly considered a limitation (e.g., see the Concept Bottleneck Model Literature [3,4]); therefore, alleviating this need is a valuable contribution to the research landscape.\n* The paper is clearly structured and well-written. Only towards the end of experiments, it becomes a bit cramped with information, which is minor. \n* Experiments are performed over a broad range of backbones. Also, I appreciate that the authors make the effort to actually measure the interpretability rather than just stating that it is interpretable.\n\n[3] Yuksekgonul, Mert, Maggie Wang, and James Zou. \"Post-hoc Concept Bottleneck Models.\" The Eleventh International Conference on Learning Representations.\n\n[4] Marcinkevičs, Ričards, et al. \"Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?.\" arXiv preprint arXiv:2401.13544 (2024).\n\nWEAKNESSES:\n* My biggest uncertainty is about the stated (inherent) interpretability. Firstly, to me, I see B-cos(ified) networks more as a post-hoc gradient-based explanation method, which additionally regularizes during training for more faithful explanations. However, there is no certainty upon seeing an explanation that the model actually acted upon these highlighted pixels, which to me makes B-cos networks not inherently interpretable. In other words, a regularizer does not enforce that the model truly depends on the given explanations, and therefore, the explanation given is not necessarily faithful to the underlying model. Thus, I also disagree with the Abstract that states that the explanations are \"model-faithful by design\".\n* Followingly, such a method should be evaluated with respect to faithfulness to the underlying model. As such, I would be very interested in how B-cosified networks perform in some faithfulness sanity checks such as the Model Parameter Randomization Test of [5].\n* Without any standard deviation, it is hard to assess the (statistical) significance of performance improvements. Also, multiple seeds (up to some degree) prevent the all-too-common hyperparameter optimization on the test set.\n* The CLIP explanations when B-cosifying CLIP are on par with GradCAM on the pre-trained weights. To be better requires the introduction of a new, arbitrary aggregation of value vectors. \n* If I understand it correctly, this new aggregation would also affect CLIP's performance. It is not explored how the model performance on CLIP Benchmark would be affected by this aggregation that appears to make CLIP more explainable.\n* While this work has high significance in today's world of foundation models, the methodological novelty is limited, as most B-cosification tricks have already been introduced in [2].\n* In my opinion. speedups of 2x are not necessarily big enough to warrant that it's a huge improvement over training from scratch.\n* Page 1 appears to be missing some vspace after line 37.\n\n[5] Adebayo, Julius, et al. \"Sanity checks for saliency maps.\" Advances in neural information processing systems 31 (2018).\n\nQUESTIONS:\n* In line 295, is \"interpolate\" really the right word (it confused me slightly in my read). That is, if it was interpolating, then shouldn't Fig. 5a B-cos FeatureCLIP for p=1 be equal to B-cos CLIP? (Also, there are some minor reference naming issues with respect to the figure in line 292-297, and the legend of Fig. 5a needs to be updated)\n* While the Grid Pointing Game appears to be a good metric to measure how well the explanations are localized, it is not telling enough on a fine-grained level how informative the explanations are. While the qualitative figures 1 & 5b give some insights, into this, they're obviously cherry-picked (which I don't condone). What I like in papers with qualitative examples is that, in the Appendix, there are also some randomly chosen examples so that an interested reader can get a feeling for how the method actually performs. Thus, I would be interested in seeing some more qualitative examples that are (truly) randomly drawn.\n* How well would the proposed method be transferrable to different input modalities such as text?\n\nLIMITATIONS:\nOne could maybe write one more sentence with respect to which models are *not* B-cosifiable. Apart from that, limitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work builds on the recent line of works aiming to design inherently interpretable models. In particular, it considers the recently proposed B-cos networks and shows how pre-trained conventional CNN/VIT models can be converted into a B-cos network through fine-tuning. The authors first discuss which parts of conventional DNNs can be converted into an equivalent B-cos model and then propose two simple mechanisms that address the two missing components (increasing B and removing the biases) through fine-tuning. Results illustrate that the proposed B-cosified models are able to obtain equivalent accuracy to both the original model as well as the standard B-cos network, while obtaining similar localization results to the B-cos network in a shorter time due to the re-use of the original weights.\n\nSTRENGTHS:\nThe work addresses an important problem of converting pre-trained black-box models into inherently interpretable models. Instead of requiring complete retraining, which often restricts the applicability of these models, only requiring fine-tuning is a first step into the right direction.\n\nThe paper is well structured and the different modifications required are presented in sufficient detail. \n\nThe experimental evaluation on both CNN and Transformers as well the case study on CLIP shows the potential of the proposed approach.\n\nWEAKNESSES:\nIt would be beneficial to include a plot on how accuracy and localization score vary as fine-tuning progresses for one/some of the standard models (CNN/VIT).\n\nIn Sec. 3.2.2, it is unclear how the bias is handled when performing the ablation study in Table 2. Also, the B-cosified model consistently outperforms the baseline in Table 2 and 3, do the authors have an hypothesis on why this fine-tuning process increases performance?\n\nMinor:\n\nIt would be beneficial to also provide the number of fine-tuning epochs that were required in Table 4.\n\nFigure reference missing in Line 141.\n\nQUESTIONS:\nFor the ViT results, there are some more cases where the B-cosified networks performance degrades slightly compared to the black-box and the localization is slightly lower than for the B-cos model. While these differences are small, do the authors have a hypothesis on their reason? Are these induced by the interpretation of the GeLU?\n\nLIMITATIONS:\nThe limitations of the proposed method have been briefly but sufficiently discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work targets the interpretability of modern architectures via a process that the authors call B-cosification. Contrary to the original B-cos networks that are trained from scratch by architecturally enforcing alignment between inputs and weights, B-cosification constitutes a post-hoc method, aiming to convert/finetune exististing pre-trained models towards interpretability.\n\nSTRENGTHS:\nThis work aims to transform existing pre-trained models to \"inherently interpretable\" ones via a post-hoc b-cosification method. Contrary to other post-hoc approaches, this finetuning allows for essentially altering the properties of the network, bypassing some of the criticisms of post-hoc approaches. \n\nOverall, this work constitutes an interesting investigation of how to transform standard architectures to B-cos like ones.\n\nWEAKNESSES:\nEven though this is an interesting approach, its novelty is limited. The authors explore how to finetune the model to match the b-cos alignment pressure, by exploring some architectural changes like normalisation and bias decrease.\n\nOne of the main issues the proposed process is the complexity of the finetuning. Some of the ResNet architectures considered in the main text are trained for around 90 epochs, e.g., on ImageNet. That is a common number of epochs to train these models from scratch. Can the authors provide a summary/visualisation of the acc/localization with respect to the number of epochs?\n\nWhat is the behaviour of the base models when finetuned the same way as in the proposed b-cosification process (without b-cos)?\n\nCan the authors add another column to Table 4, denoting the accuracy of the standard trained from scratch b-cos networks?\n\nThe text needs to be proofread. There are several sentences that lack clarity, typos and other issues, e.g., line 141 \"see also Fig. XX, whereas conventional models use 3 channels\".\n\nThe authors are encouraged to remove the spacing altercations that do not follow the format of the conference.\n\nQUESTIONS:\nPlease see the Weaknesses section.\n\nLIMITATIONS:\nThe authors have a limitation paragraph in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss the B-cosification of a pre-trained model.\nThe B-cosification involves changing the operations performed in the linear layers to one involving a cosine. Not all properties of B-cos models are in the end satisfied, yet performance in terms of accuracy and localization are on par with a fully trained B-cos model.\n\nSTRENGTHS:\n- The method contributes to the current effort to turn pre-trained models into explainable ones without full retraining.\n- The performance dropped are limited both against the original model and the equivalent B-cos model\n\nWEAKNESSES:\n- The method still requires a full fine-tuning of the weights.\n- The experimental setting is not very clear.\n- The presentation is not always clear and sometimes verbose.\n\nQUESTIONS:\n* P1 L33: which are increasingly popular—can cost millions of dollars.\n    What if I am not using dollars? Maybe time would be a better metric? Or energy?\n\n* P2 L38: the recently proposed B-cos Networks.\n    Until here, B-cos was never defined. Since it is not a contribution of this paper, would you have a citation to guide the reader?\n\n* P2 L42: What does \"functionally equivalent\" mean? What do you mean with \"alignment pressure\"?\nMaybe instead of introducing these vague notions, why not use the opportunity given by this paragraph to explicitly say what B-cos consists of, namely (up to my understanding) replacing the linear operations of the hidden layers by a cosine-based function parametered with B? You can go further and say that for B=1, the operation is (\"is\" and not \"is equivalent to\") the classic linear matrix multiplication. \"Linear\" implies the absence of a bias term. Otherwise, that would be the \"classic affine matrix multiplication\". Nevertheless, one can stress this further.\n\n* P2 L45: \"significantly more interpretable explanations (Fig 1)\"\n    What do you mean by \"more interpretable explanations\"? What I see in the third row of Fig1 is kind of a mix between the second row and an edge detector or an Integrated Gradient saliency map. Is capturing the shape or details of the \"important part of the image\" making an explanation more interpretable?\n    Following this discussion, the result of GradCam+CLIP on the first example is preferable from my point of view because one can see that the model uses the help of the grass to actually predict a sheep. The same applies to the \"[...] photo of a boat.\"\n\n* P2 L47: \"On supervised settings, we find that B-cosified models often outperform\"\n    How is performance measured here? Classification accuracy or caption prediction?\n\n* P2 L53-57: I am a bit confused when you say \"design[ing] inherently interpretable models\". I do not understand your contribution as a new design but a conversion. This is a valuable and efficient contribution that aligns with current concerns, be it ecological, time efficient, or simply democratizing ML for the majority who do not have access to huge computational resources.\n    This work fits into the recent line of research investigating how to alter the architecture of trained models to make them \"interpretable.\" Why not stress this aspect more?\n\n* P2 L65-: I am unsure if the claimed results are that important to count as contributions.\n    Again, what you mean by \"significantly interpretable\" is not clear. Could you please clarify?\n\n* Sec2: If you choose to attach your contribution to the line of \"converting trained models\" (you don't have to) to \"interpretable ones\", I suggest some works:\n    * Stalder, S., Perraudin, N., Achanta, R., Perez-Cruz, F., & Volpi, M. (2022). What you see is what you classify: Black box attributions. Advances in Neural Information Processing Systems, 35, 84-94.\n    * Aytekin, C. (2022). Neural networks are decision trees. arXiv preprint arXiv:2210.05189.\n    * Gautam, S., Boubekki, A., Höhne, M. M., & Kampffmeyer, M. (2024) Prototypical Self-Explainable Models Without Re-training. Transactions on Machine Learning Research.\n\n* P3 L111: \"Many common DNNs consist of a series of blocks of linear layers followed by non-linear ReLU activations [29], and are thus piece-wise linear functions1 : i.e., for every input x, they effectively compute a linear transformation of that input: y(x) = W(x)x + b(x)\"\n    What are W and b? I assume the model's input and output are x and y, respetively.\n    If it is \"effectively [...] a linear transformation\", why are W and b functions of x?\n\n* P3 L113: \"dynamic linear\"\n    Why do you need to introduce this notion? What is dynamic in an affine matrix multiplication?\n\n* P3 L116: \"This linear mapping W(x)\". The same goes here: W cannot be a linear mapping/function if we are talking about \"a piece-wise linear model\".\n\n* P3 L120: Why does a \"complete explanation\" requires \"y(x)=W(x)x\"? Two sentences later, you actually contradict this claim.\n    I fail to grasp why having a bias is so problematic if most of the backward operations are based on the gradient. Could you clarify?\n\n* P4 L123: Can you clearly define \"alignment pressure\" and why it is necessary to be introduced?\n    From this paragraph, I understand that the conversion from line (B=1) to (B>1)-cos model has already been investigated in [6] and [7]. Has it?\n    If it has been, this jeopardizes the relevance of your main contribution.\n\n* P4 Eq1: How do you compute $cos(x,w)$?\n    I understand that $x$ is a vector, $W$ is a matrix, and $\\times$ is the element-wise multiplication.\n\n* P4 L127: Why do we need the notion of \"dynamic linear\"?\n    If it has to do with W being a function of x: if B=1, W does not depend on x.\n\n* P4 L141: Missing reference with Fix XX.\n\n* P4 L146: \"In particular, we show that a conventional model can be framed as a functionally equivalent B-cos model as in [7] with B =1, which additionally employs bias terms. Only upon modifying these two aspects, i.e., biases and B, does the model need to be fine-tuned to adapt the weights to those changes.\"\n    These two sentences are not clear.\n\n* P4 L150: \"As mentioned in Sec. 3.1, B-cos models use input representations with six color channels\"\n    This was not mentioned in Sec 3.1. This seems to be an atavism from an older version of the paper. Please fix.\n\n* P4 \"Input Encoding and Normalisation\": The operation is straightforward: B-cos models requires 6 channels: so inputs need to be transformed according to Eq2 and the first layer of the B-cosified models is discarded and replaced by one with 6 channels (implying a full training thereof).\n\n* P5 \"Activation Functions.\" You start the section by saying that activations are unnecessary, yet you introduce them. Why?\n\n* P5 \"...unit norm weights, ..., which the authors motivated by the fact that the only way any given neuron can achieve its maximal output is by increasing the weight-input alignment, which in turns[typo] leads to the improvements of the explanations.\"\n    What is \"weight-input alignment\"? I fail to see why normalized weights improves the explanations.\n\nYou could streamline this section by leaving the justification of the B-cos meaningfulness to an introductory paragraph and focusing on the changes required by your approach. There is no need to justify or make claims about the benefit of B-cos since it is not one of your contributions here.\n\n* P5 Section 3.2.2\n    None of the tables present statistical tests, which makes it difficult to draw any conclusion, especially given that the results are quite similar from one setting to another.\n    * How many times the experiments were repeated? What are the standard deviation? Which result is significantly better?\n    * What is the localization score?\n\n\n* P6 L204: \"setting B = 2 and then fine-tuning, yields performance that is on par with\nlearnable B parameters,\"\n    If this holds, it is also on par with Linear B 90 epochs.\n\n* P6 Table 4: The increments are quite small. Have you considered running a t-test to identify significant differences?\n    Is the \"speedup\" with respect to the fully trained B-cos model? What about the \"speedup\" with respect to the vanilla models with classic matrix multiplication?\n\n* P7 L245: \"Specifically, we find that averaged across architectures, B-cosified models outperform B-cos models trained from scratch by 2.31 pp, with an average training speedup (to match performance) of 2.96x.\"\n    This is an important result. Why not highlight it in a separate section/ablation study?\n\n* P8 Interpretability: In the text you refer to paper [6] while in the caption of Fig2 you refer to [7] (These papers share essentially the same figures). Fig2 would be better positioned closer to its text.\n    Finally, interpretability is defined in paper [6] (or rather in paper [6] of paper [6]) as the performance at the \"grid pointing game\". If I understand it, it is related to the sum of the GCAM attribution to each class of the grid cell. My understanding of gradient-based explainable methods is that the scale of the output saliency maps is inconsistent. Is it valid? If yes, how does it affect the score returned by the Pointing Game? How robust is this score against possible spurious relationships between grid cells? How does performing well in this game relate to the interpretability of the model's explanations?\n\n* P8 Fig4: What are Natural/Specialized/Structured Data?\n\n* P9 L284: \"We find that the B-cosified models significantly outperform the Text2Concept approach and achieve accuracies that are more similar to the original CLIP’s zeroshot and linear probing accuracies.\"\n    Text2Concept is a quite different approach. In the original paper thereof, they report performance close to CLIP's. How do you explain this difference? What is different in your setting? Do you train supervised or self-supervised?\n\n* Evaluating Model Interpretability.\n    In the image classification case, you compute the explanation based on a single label; the equivalent case here is thus to back-propagate using the\n    Isn't the average similarity equivalent to the similarity with an out-of-distribution point in the text embedding?\n    Why did you choose p=7 and p=19?\n\n    Overall I did not quite understand this section. I am not very familiar with CLIP, so it is not easy for me to follow what is happening.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}