Holistically Explainable Vision Transformers
Authors: Moritz Böhle
Mario Fritz
Bernt Schiele
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainable Deep Neural Networks
Vision Transformers
XAI

Abstract: Transformers increasingly dominate the machine learning landscape across many tasks and domains, which increases the importance for understanding their outputs. While their attention modules provide partial insight into their inner workings, the attention scores have been shown to be insufficient for explaining the models as a whole. To address this, we propose B-cos transformers, which inherently provide holistic explanations for their decisions. Specifically, we formulate each model component—such as the multi-layer perceptrons, attention layers, and the  tokenisation module—to be dynamic linear, which allows us to faithfully summarise the entire transformer via a single linear transform. We apply our proposed design to Vision Transformers (ViTs) and show that the resulting models, dubbed Bcos-ViTs, are highly interpretable and perform competitively to baseline ViTs on ImageNet. Code will be available at: github.com/anonymous/authors.

Under review as a conference paper at ICLR 2023
HOLISTICALLY EXPLAINABLE VISION TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Transformers increasingly dominate the machine learning landscape across many tasks and domains, which increases the importance for understanding their out- puts. While their attention modules provide partial insight into their inner work- ings, the attention scores have been shown to be insufficient for explaining the models whole. To address this, propose B-cos transformers, which inher- ently provide holistic explanations for their decisions. Specifically, we formulate each model component-sucl the tokenisation modulc-to be dynamic linear, which allows us to faithfully sum- marise the entire transformer via a single linear transform. We apply our proposed design to Vision Transformers (ViTs) and show that the resulting models, dubbed Bcos-Vill ViT', are highly interpretable and perform competitively baseline ViTs on ImageNet. Code will be available: gi thub. com/ anonymous/ authors.
the multi-layer perceptrons, attention layers, and
INTRODUCTION
Convolutional neural networks (CNNs) have dominated the last decade of computer vision. However, recently they often surpassed by transformers (Vaswani et al., 2017), which- if the current development any indication- will replace CNNs for ever more tasks and domains. Transformers are thus bound to im- pact many aspects of our lives: from health- care, over judicial decisions, autonomous driving. Given the sensitive nature of such ar- cas, it is of utmost importance to ensure that can explain the underlying models, which still remains a challenge for transformers.
Input
[W
Ours
Eq: (7)
Ours
Roilowt
FinAtt
Ibies hound
co--
magge
To explain transformers, prior work often fo- cused on the models' attention layers (Jain & Wallace, 2019; Serrano & Smith, 2019; Ab-
blue bied
nar Zuidema, 2020; Barkan al., 2021), as they inherently compute their output in an in- terpretable manner. However, as transformers consist of many additional components, ex- planations derived from attention alone have
rocking chair
been found insufficient explain the full
models (Bastings & Filippova, 2020; Chefer al., 2021). To address this, our goal is to develop transformers that inherently pro- vide holistic explanations for their decisions, i.c explanations that reflect all model com- ponents. These model components are given by: tokenisation module, mechanism for providing positional information to the model, multi-layer perceptrons (MLPs), as well as normalisation and attention layers, see Fig. 2a. By addressing the interpretability of each component individually, we obtain transformers that inherently explain their decisions, see, for example Fig. I and Fig. 2b.
Fig. 1: Inherent explanations (cols. 2+3) of B-cos ViTs vs. attention explanations (cols. 4+5) for the same model. Note that W(x) faithfully reflects the whole model and yields more detailed and class-specific explanations than attention alone. detailed discussion, see supplement.
In detail, our approach is based on the idea of designing each component to be dynamic linear, such that it computes an input-dependent linear transform. This renders the entire model dynamic linear, cf. Böhle et al. (2021; 2022), s.t. it can be summarised by single linear transform for each input.

---

Under review as a conference paper at ICLR 2023
B-cos VIT
X L
Tokember::
Input
[W()]]
xx(X)
Chasollter
(x)
(x)
W (N)
Dynamic Linear Transformation
Wix = W (x) I W W "00)
W"
Probability(Lesserr Panda) a (Sum(cc = 100%
(b)
(a)
Fig. 2: (a) B-cos ViTs. We design each ViT component to be dynamic linear, allowing us to summarise the entire model by single linear transform (x). as shown in the bottom. (b) Computation is Explanation. The model output exactly computed by the linear transform W (xx). As result, we visualise this effective linear transform either by corresponding matrix row (center) or the contributions cx(x) (right), cf. Eq. (7).
In short, we make the following contributions. (I) We present a novel approach for designing in- herently interpretable transformers. For this, (II) carefully design each model component to be dynamic linear and ensure that their combination remains dynamic linear and interpretable. Specifi- cally, we address (IIa) the tokenisation module, (IIb) the attention layers. (IIc) the MLPs, and (IId) the classification head. (III) Additionally, we introduce a novel mechanism for allowing the model learn attention priors, which breaks the permutation invariance of transformers and thus allows the model to easily leverage positional information. In our experiments, we find that B-cos ViTs with such a learnt 'attention prior' achieve significantly higher classification accuracies. (IV) Finally, we evaluate wide range of model configurations and show that the proposed B-cos ViTs are not only highly interpretable, but also constitute powerful image classifiers.
2 RELATED WORK
Attention as Explanation. As the name exemplifies, attention is often thought to give insight into what a model 'pays attention to' for prediction. As such, various methods for using attention to understand the model output have been proposed, such as visualising the attention of single attention heads, cf. Vaswani et al. (2017). However, especially in deeper layers the information becomes in- creasingly distributed and it is thus unclear whether given token still represents its original position in the input (Serrano & Smith, 2019; Abnar Zuidema, 2020), thus complicating the interpretation of high attention values deep in the network (Serrano & Smith, 2019; Bastings & Filippova, 2020).
Therefore, Abnar & Zuidema (2020) proposed 'attention rollout', which summarises the various attention maps throughout the layers. However, this summary still only includes the attention layers and neglects all other network components (Bastings & Filippova, 2020). In response, various improvements over attention rollout have been proposed, such as GradSAM (Barkan et al., 2021) 1) or an L.RP-based explanation method (Chefer et al., 2021), that were designed to more accurately reflect the computations of all model components. The significant gains in quantitative interpretability metrics reported by Chefer et al. (2021) highlight the importance of such holistic explanations.
Similarly, we also aim to derive holistic explanations for transformers. However, instead of deriving an explanation 'post-hoc' as in Chefer al. (2021), we we explicitly design our models be holistically explainable. For this, we formulate each component-and and thus the full model-to be dynamic linear.
Dynamic Linearity. Plain linear models, i.e. y(x)= - Wx, usually considered interpretable, as y(x) can be decomposed into individual contributions c= I2 from any dimension i: y= Σ; % (Alvarez-Melis & Jaakkola, 2018). However, linear models have limited capacity, which has lead to various works aimed at extending their capacity without losing their interpretability. see Alvarez- Melis & Jaakkola (2018); Brendel Bethge (2019); Böhle et al. (2021; 2022). An appealing strategy for this is formulating dynamic linear models (Alvarez-Melis & Jaakkola, 2018; Böhle et al., 2021; 2022), i.e. models that transform the input with a data-dependent matrix W(x): yx)= (ok)=W(<] Wx)x.
In this work, we rely on the B-cos framework (Böhle et al., 2022), but instead of focusing on CNNs as in Böhle et al. (2022), we investigate the applicability of this framework to transformers.
Interpretability in DNNs. The question of interpretability extends, of course, beyond transformers and many methods for explaining DNNs have been proposed. While other approaches exist, cf. Kim
2

---

Under review as a conference paper at ICLR 2023
et al. (2018), these methods typically estimate the importance of individual input features, which can be visualised as a heatmap, cf. Lundberg & Lee (2017); Petsiuk et al. (2018); Ribeiro et al. (2016); Simonyan et al. (2014); Springenberg et al. (2015); Zhou al. (2016); Bach et al. (2015); Selvaraju et al. (2017); Shrikumar et al. (2017); Srinivas & Fleuret (2019); Sundararajan et al. (2017).
Similarly, our models yield explanations in form contribution heatmaps. However, in contrast to the above-referenced post-hoc explanation methods, the contribution maps of our B-cos ViTs model-inherent. Further, as the interpretability of the B-cos ViT' relies on aligning the weights with the inputs, the weights can be visualised in colour in Böhle et al. (2022), see Figs. and 7.
DESIGNING HOLISTICALLY EXPLAINABLE TRANSFORMERS
In the following, we present the overarching goal that we pursue and how we structure this section around it. First, however, we introduce the necessary background and the notation used in our work.
Preliminaries. Vision Transformers (ViT) (Dosovitskiy al., 2021) with I. blocks are given by: (x) = Classifier o I (MLPBlock) o AutBlock)) o Tokens(x).
(1)
Here, the input x ((CHW) denotes a vectorised image of H W height and width and with C color channels; the functions, concatenated by o, are defined in Eqs. (2) (5) (left), with P, E E RDxN, N the number of tokens and their dimensionality. Further, convolutional neural network's T 'tokenises" the CNN output (see Sec. 3.2), Linear is a learnable linear transform p' (p) = Wp- b with parameters W and b that is applied to each token p (columns of P) independently. MSA denotes Multi-head Self-Attention, and E is learnable embedding. Following Graham et al. (2021), Pool performs average pooling over the tokens, and the model output is given by yx) E RM with M classes. Last, we omit indices for blocks and layers whenever unambiguous and it may be assumed that each layer has its own set of learnable parameters.
Eqs. (2) - (5), CNN is a
Our goal in this work is to reformulate the VITs such that they compute their output in a more interpretable manner. Specifically, we aim to make them dynamic linear such that they compute (x) = W(x) Instead of using the ViTs to predict ((x), cf. Alvarez-Melis & Jaakkola (2018), we achieve this by rendering each of the ViT components dynamic linear on their own as follows:
Tokens (x) = T(CNN(x)) + E
B-cos Tokens
(x) Tokens (x) x
AttBlock (P) = MSA(P) +
B-cos AttBlock (P) W
(P) P
(3)
MLPBlock (P) MLP(P) + P
B-cos ML.PBlock (P) WMP
(P) P
(4)
Classifier (P) Linear Pool(P)
B-cos Classifier
(P) WCClas
(P) P.
(5)
Crucially, we define each component such that it can be expressed dynamic linear function, see the right-hand side of Eqs. (2) (5). As result, the entire model will become dynamic linear:
y(x) Clas (x) IL (WMLP(x) wxt(x)) WTokens (x) x W(x)x.
(6)
Specifically, we develop the B-cos ViT's in accordance with the B-cos framework (Böhle et al., 2022) to render (x) interpretable by aligning it with relevant input patterns.
Outline. In the following, we shortly summarise the most relevant aspects of B-cos networks and how to explain them (Sec. 3. Then, we discuss Eqs. (2) - (5) in detail and how we ensure that the resulting linear transform W (x) will be interpretable. In particular, we discuss the tokenisation (Sec. 3.2), attention (Sec. 3.3), and the multi-layer perceptrons (Sec. 3.4). Finally, in Sec. 3.5, we discuss how encode positional information in B-cos ViT',, introducing "position-aware" attention.
3.1 B-COS NETWORKS: INTERPRETABLE
INHERETT EXPLANATIONS
As shown in Eq. (6), a dynamic linear transformer is summarised exactly by single matrix W(x) for every x. These linear explanations lend themselves well for understanding the model decisions: as in plain linear models, one can calculate linear contributions from individual features (e.g., pixels) to each output unit. detail, the effective linear contributions Cę to the k-th class logit are given by
Dynamic Linear Contribution Maps: c₁ (x) = W (x)
C x,
'To simplify later equations, we take advantage of the fact that conv. layers are equivalent to linear layers with weight constraints (weight sharing and local connectivity) and assume the CNN process vectorised images.
3

---

Under review as a conference paper at ICLR 2023
with @ denoting element-wiss multiplication. Crucially, these contribution maps faithfflly sum- marise the entire model, as this linear summary is inherent to the model formulation, see also Fig. 2. Thus, use these contribution maps to explain the B-cos ViTs, see, e.g., Figs. 1, and 7.
Note, however, that while the contribution maps in Eq. (7) accurately summarise any given dynamic linear model, this summary need not be interpretable. E.g., for piece-wise linear models, which also dynamic linear, this amounts 'Inputx Grad", cf. Adebayo et al. (2018). For such models, however, the contributions с are very noisy and not easily interpretable for humans. Hence, we de- sign the transformers accordance with the 'B-cos' framework (Böhle al., 2022), which ensures that W (x) aligns with relevant input features and thus becomes easily interpretable. In detail, the B-cos transform induces weight alignment by suppressing outputs for badly aligned weights:
(acos(a; W) (7)=(om))) -(cos8- W)
W a == W(a)aa
(8)
here, cos is applied row-wise, W denotes that the matrix rows are of unit norm, and @ represents row-wise scaling. As can be seen on the RHS of Eq. (8), the B-cos transform is dynamic linear.
As result, the matrix rows [W(x]] align with relevant patterns of class By encoding the image such that the color is uniquely determined by the angle of the pixel encodings, it is possible to directly visualise those matrix rows in color, see Figs. and For details, see Böhle al. (2022).
Requirements. To ensure that a B-cos network aligns W (x) with its inputs, each of its layers needs (a) be bounded, (b) yield its maximum output if and only if its weight vectors align with its input, and (c) directly scale the overall model output by its own output norm, see Böhle et al. (2022). In the following, we ensure that each of the model components (Eqs. (2)-(5)), fulfills these requirements.
3.2 INTERPRETABLE TOKENISATION MODULES: B-COS CNNS
While the original Vision Transformer only applied single-layered CNN, it has been shown that deeper CNN backbones yield better results and exhibit more stable training behaviour (Xiao et al., 2021). Hence, address the general case, and take advantage of the increased training stability and performance, we take the tokenisation module to be given by a general CNN backbone. Being able to explain the fill ViT models consequently requires using an explainable CNN for this.
Tokenisation. We use B-cos CNNs (Böhle et al., 2022) as feature extractors; thus, the requirements (a-c), see Sec. 3.1, of B-cos networks are, of course, fulfilled. The input tokens are computed
B-cos Token (x) = T o B-cos CNN(x) = W WNN (x) x
W Tokens (x) x.
Here, P; R.D corresponds to the ith column in the token matrix P, Ti extracts the respective features from the CNNs' output with WT. denoting the corresponding linear matrix. WCNN is the dynamic linear mapping computed by the B-cos CNN, and as color-indicated, WTTokems (x) W WCNN (x).
Finally, note that we did not include the additive positional embedding E, cf. Eq. (2) (left). For detailed discussion on how to provide positional information the B-cos ViTs, please see Sec. 3.5.
3.3 B-COS ATTENTION
Interestingly, the attention operation itself is already dynamic linear and, as such, attention lends itself well to be integrated into the linear model summary according to Eq. (6). To ensure that the resulting linear transformation maintains the desired interpretability, changes to make the attention layers compatible with the B-cos formulation as discussed in Sec. 3.1.
discuss the necessary
B-cos Attention. Note that conventional attention indeed computes a dynamic linear transform:
Attention P:Q, K. V)
softmax QP'QQPP)) VP
A W(P)P. (10)
Attention matrix A(P)
Value(P)
W(P)
Here, Q K and V denote the respective query, key, and value transformation matrices and P denotes the input tokens to the attention layer, further, softmax computed column-wise
In multi- head self-attention (MSA), see Eq. (3) (left), H distinct attention heads are used in parallel after normalising the input. Their concatenated outputs are then linearly projected by matrix U:
MSA(P) U (P)PP), W2)P), ..., (P))) with P LayerNorm(P) (11)
4

---

Under review as a conference paper at ICLR 2023
While this still" be expressed dynamic linear transform of P. observe the following issues with respect to the requirements (a-c), see Sec. 3.1. First, (a) while the attention matrix ( is bounded, the value computation VP and the projection by U are not. Therefore, can arbitrarily increased by scaling and U, (b) a high weight alignment is not necessary to obtain large outputs. Finally, by normalising the outputs of the previous layer, the scale of those outputs does not affect the scale of the overall model output anymore, which violates requirement (c).
the output
To address (a+b), we propose to formulate B-cos Attention Block as follows. First, we replace the value computation and the linear projection by U by corresponding B-cos transforms. As Böhle et al. (2022), we employ MaxOut and for a given input P the resulting projections are computed as
B-cos Linear (P; S) = MaxOut B-cos(P: S) = (P)P with S VU.
(12)
To fulfill (c), whilst not foregoing the benefits of LayerNorm', we propose to exclusively apply LayerNorm before the computation of the attention matrix. i.e. we compute AP)), see Eq. (10)
(P; Q K)
softmax
P P"'Q"KPP KPP
with P = LayerNorm(P) (P).
(13)
In total, a 'B-cos AttBlock' thus computes the following linear transformation:
B-cos Autlloc(P)) = (ANP)))) + +1) (P)
(14)
A=1' WU and W in Eq. (12), and (P(P) as in Eq. (13); the identity matrix I reflects the skip connection around the MSA computation, see Fig. 2 and Eq. (3). For an ablation study regarding the proposed changes, we kindly refer the reader to the supplement.
Here, P' = (P) |
3.4 INTERPRETABLE MLPs AND CLASSIFIERS
To obtain dynamic linear and interpretable MLPs, we convert them 'B-cos' MLPs, such that they are compatible with the B-cos formulation (Sec. 3.1) and align their weights with relevant inputs.
B-cos ML.ss. Typically, MI.P block in VIT computes the following:
MLPBlock (P) = Lineary GELU Linear, o LayerNorm(P) P
(15)
Here, Linear is as in Eq. (5), the GELU activation function is as in Hendrycks & Gimpel (2016), and LayerNorm as in Ba et al. (2016).
To obtain our 'B-cos MLPBlock', we follow Böhle et al. (2022) and replace the linear layers by B- transforms, remove the nineriinasssies and the normalisation (cf. Sec. 3.3); further, each 'neuron' modelled by two units, to which we apply MaxOut (Goodfellow et al., 2013). As a result, each ML. block becomes dynamic linear:
B-cos MLPBlock(P)
LPP) (P) (P) + I)P PMP.PP).. (16)
Here, M.PP) and L;(P) correspond to the effective linear transforms performed by the MaxOut and B-cos operation respectively, and I denotes the identity matrix stemming from the skip connection.
Classifier. Similar to the B-cos MLPs, we also replace the linear layer in the classifier, cf. Eq. (5) (left), by corresponding B-cos transform and the B-cos Classifier is thus defined as
B-cos Classifier(P): B-cos o Pool(P) L(P") Avggo Wliss (P)P,
(17)
with L(P") the dynamic linear matrix corresponding the B-cos transform and P'=Pool(P).
3.5 POSITIONAL INFORMATION IN B-COS TRANSFORMERS
In contrast to CNNs, which possess strong inductive bias w.r.t. spatial relations (local connectivity), transformers (Vaswani et al., 2017) are invariant w.r.t. the token order and thus lack such a 'locality
"To he exact, it can be represented as dynamic affine transform, since LayerNorm adds a hias term. We noticed normalised inputs to be crucial for the computation of the attention matrix A(P) in Eq. (10): for unconstrained inputs, softmax easily saturates and suffers from the vanishing gradient problem.
5

---

Under review as a conference paper at ICLR 2023
bias'. To nevertheless leverage spatial information, it is common practice to break the symmetry between tokens by adding a (learnt) embedding E to the input tokens P, see Eq. (2) (left).
However, within the B-cos framework, this strategy is not optimal: in particular, note that each B-cos transformation needs to align its weights with its inputs forward large output to the next layer, see Eq. (8) and Böhle et al. (2022). As result, a B-cos ViT would need to associate contents (inputs) with specific positions, which could negatively impact the model's generalisation capabilities.
Therefore, we investigate two alternative strategies for providing positional information to the B- cos ViTs: additive and multiplicative attention priors, see Eqs. (18) and (19) respectively. Specifi- cally, we propose to add a learnable bias matrix B) to each attention head h in every layer in the model. This pair-wise (between tokens) bias is then either added before the softmax operation or multiplied to the output of the softmax operation in the following way (omitting sub/superscripts):
ada P) = SM (R(P) + B) (18)
and mail PP) = SM (R(P)) SM (B) (19)
Here, QPP" K" and SM denotes softmax. The bias B thus allows the model to learn an attention prior, and the attention operation is no longer invariant to the token order. As such, the model can learn spatial relations between tokens and encode them explicitly in the bias matrix B. In our experiments, this significantly improved the performance of the B-cos ViTs, see Sec. 5.1.
EXPERIMENTAL SETTING
Dataset. In this work, we focus on Vision Transformers (ViT', Dosovitskiy et al. (2021)) for image classification. For this, we evaluate the B-cos and conventional ViTs and their explanations on the ImageNet dataset (Deng et al., 2009). We use images of size 224x224. For B-cos models, we encode the images as in Böhle al. (2022).
Models. We follow prior work and evaluate ViTs of different sizes in common configurations: Tiny (Ti), Small (S), and Base (B), cf. Steiner et al. (2021). We train these models on the frozen features of publicly available (Böhle et al., 2022; Marcel & Rodriguez) (B-cos) DenseNet-121 models and extract those features at different depths of the models: after 13, 38, or 87 layers. Model names are thus as follows: (B-cos) ViT (iTeesize)-(L) with size (Ti, S, B) and LE (13, 38, 87). We opted for (B-cos) DenseNet- 121 backbones, as the conventional and the B-cos version achieve the same top- accuracy on the ImageNet validation set. In particular, we compare B-cos ViTs on B-cos backbones to normal ViTs on normal backbones.
Training. We employ simple training paradigm that is common across models for comparability. All models are trained with Rand Rugnent (Cubuk et al., 2020) for 100 epochs with learning rate of 2..5 which is decreased by factor of 10 after 60 epochs; for details, see supplement.
Evaluation Metrics. We evaluate all models with respect to their accuracy on the ImageNet valida- tion set. Further, we employ two common metries to assess the quality of the model explanations.
First, we evaluate the grid pointing game (Böhle et al., 2021). For this, we evaluate the explanations (see below) on 250 synthetic image grids of size 448x containing 4 images of distinct classes, see Fig. 4; the individual images are ordered by confidence and we measure the fraction of positive attribution an explanation method assigns to the correct sub-image when explaining given class.
Note that, in contrast to fully convolutional networks, transformers with positional embeddings ex- pect a fixed-size input. To nevertheless evaluate the models on such synthetic image grids, we scale down the image grid the required input size of 224x224 allow for applying the ViTs seamlessly.
Second, we evaluate two pixel perturbation metrics, ef. Chefer et al. (2021). For this, the pixels are ranked according to the importance assigned by given explanation method. Then, we increasingly zero out up to 25% of the pixels in increasing (decreasing) order, whilst measuring the model confi- dence in the ground truth class; a good explanation should obtain a high area under (over) the curve, iee. the model should be insensitive to unimportant pixels and sensitive to important ones.
We evaluate the perturbation metrics on the 250 most confidently and correctly classified images enable a fair comparison between models, as the confidence affects the metrics; more details in supplement. Last, to succinctly summarise the two metrics, we evaluate the area between the curves.
"Note that additive positional bias in attention layers has heen proposed hefore (Graham ct al., 2021).

---

Under review as a conference paper at ICLR 2023
Encoding of positional information
Tigh
Poetion Enitaddnng
Adi Arr Box Eq (16)
Mut Air Bixe Ee (19)
Example
hoxt
Top accuracy (N)
75.6
7h-6
71.0
70
Tiy
Small
Base
Size of the B-cos VIT-size-87
Fig. 3: ImageNet accuracy of differently sized B-cos VITs (Tiny, Small, Base) depending the positional encoding. We find B- ViT's with A muls sec Eq. (19), perform significantly better.
Fig. 4: In the localisation metric, we mea- the fraction of pos. evidence assigned to the correct grid cell for each occurring class.
13 backbone layers
backbone layers
87 backbone layers
Conventional
B-cos
Conventional
B-cos
Conventional
B-cos
Top-1 accuracy (%)
80
74.5
75.6
74
75.7
74.9
76.6
76.7
72.1
71.3
73,0
73.1
70
Tiny
Small
Bose
Transformer size
Fig. 5: ImageNet accuracies of B-cos ViTs with multiplicative attention bias (Eq. (19)) compared standard ViTs and backbones. both for differently sized ViTs (Tin,. Small, Base) and backbones (13, 38, or layers). find that the B-cos ViT's perform least as well as the baseline ViTs over almost all tested configurations.
Explanation Methods. Apart from the model-inherent explanations (Eq. (7)), evaluate sets of cxplanation methods. First, we follow Chefer et al. (2021) and evaluate common transformer- specific explanations such as the attention in the final layer (FinAtt), attention rollout (Rollout) (Ab- nar & Zuidema, 2020), transformer-specti al. (2021), 'partial LRP' (pLRP) (Voita al., 2019), and 'GradSAM' (Barkan et al., 2021). methods such Integrated Gradients (IntGrad) (Sun-
LRP implementation (CheferLRPP proposed by Chefer
Further, evaluate architecture-agnoss dararajan et al., 2017), adapted GradCAM (Selvaraju et al., 2017) as Chefer et (2021), and 'Inputx Gradient' (IxG), ef. Adebayo et al. (2018). As no LRP rules are defined for B-cos ViT' we only apply it to baseline models. For method details, we kindly refer the reader to the supplement.
We evaluate all of those methods (if applicable) to the proposed B-cos ViTs, as well as the baselines consisting of conventional ViTs and backbones and compare them on the metries described above.
5 RESULTS
In the following, present our experimental results. Specifically, in Sec. 5.1 we analyse the classi- fication performance the B-cos ViTs: we investigate how the encoding positional information affects model accuracy (see Sec. 3.5) and compare the classification performance of B-cos and con- ventional ViTs. Further, in Sec. 5.2, we evaluate the model-inherent explanations of the B-cos ViTs against common post-hoc explanation methods evaluated on the same models. To highlight the gain in interpretability over conventional ViT models, we also compare the inherent explanations of the B-cos ViT' to the best post-hoe explanations evaluated conventional ViTs, see supplement.

---

Under review as a conference paper at ICLR 2023
Explanation Method
Localisation Metric
Perturbation Metric
Ours
Rollout
GradSAM
IntGrad
lxG
FinAtt
GradCAM
1.00
1.25
Mormalisse ABC
0.75
1.00
Localisation
x2.47
x1.99
0.75
0.50
0..0
0.25
0.25
0.00
0..0
71
72
74
75
76
72
73
74
75
76
77
Model accuracy
Model accuracy
Fig. 6: Quantitative comparison of explanation methods according to two metrics: turbation (right): for a description of metrics and methods, see Sec. 4. We evaluated the methods for all B-cos ViTs shown in Fig. 5 and plot the corresponding scores (markers). We also plot the mean score over all models (dashed lines) per method and the average improvement the model-inherent over the best post-hoc explana- tion (localisation: x2.47. perturbation: x 1.99).. Note that for the perturbation metric, we normalised the area between curves (ABC) by the scores of the model-inherent explanations for better cross-model comparison.
localisation (left) and per-
5.1 CLASSIFICATION PERFORMANCE OF B-COS VITS
In Fig. 3, we compare the top-1 ImageNet accuracy of various B-cos VITs trained on the feature embeddings of the 87th layer of frozen B-cos DenseNet-121 (Böhle et al., 2022). Specifically, we compare ViTs of different sizes (Tiny, Small, Base) and with different ways of allowing the models to use positional information, see Eqs. (2), (18) and (19). We find that the multiplicative attention bias, see Eq. (19), consistently yields significant gains in performance. As discussed in Sec. 3.5, we believe this could be due to the higher disentanglement between content and positional information. However, in preliminary experiments with conventional ViTs, we did not observe signifieant benefits from such a multiplicative prior and this seems to be particularly advantageous for B-cos ViTs.
Interestingly, once trained with such multiplicative attention prior, we find the B-cos ViTs to perform at least as good as their conventional counterparts over a wide range of configurations, see Fig. 5; we find consistent results even without MaxOut in the Transformer layers (cf. Sec. 3), show in Appendix B.3. However, these results have to be interpreted with caution: ViT' are known to be highly sensitive to, e-g., the amount of data augmentation, the number of training iterations, and model regularisation, see Steiner et al. (2021). Moreover, our goal in this work is to develop interpretable ViTs and our focus thus lies on evaluating the quality of the explanations (Sec. 5.2).
5.2 INTERPRETABILITE OF B-COS VITS
Here, we assess how well the inherent explanations (Eq. (7)) of B-cos ViTs explain their output and compare to common post-hoc explanations; for comparisons to baseline ViTs, see supplement.
Localisation Metric. In Fig. 6 (left), we plot the mean localisation score per model confi ourftiuration (B-cos ViT=(size)- H-(L)) and explanation method, see Sec. 4. We find that across all configurations, the model-inherent explanations according to Eq. (7) yield by far the best results under this metric and outperform the best post-hoc explanation for the B-cos ViTs (Rollout) by factor of 2.47.
Pixel Perturbation. As for the localisation, in Fig. 6 (right), we plot the normalised mean area between the curves (ABC) per model configuration and explanation method of the B-cos ViTs. Specifically, the mean ABC computed as the mean area between the curves when first removing the most/ / least important pixels from the images; normalise the mean ABC for each explanation by the mean ABC of the model-inherent explanation (Ours) per model configuration to facilitate cross-model comparisons. Again, the model-inherent explanations perform best and, on average, they outperform the second best post-hoc method (Rollout) on B-cos ViTs by factor of 1.99.
Qualitative Examples. In Figs. and we qualitatively compare the inherent explanations (size: B. 38 backbone layers, see Fig. 5) to post-hoc explanations evaluated on the same model. As becomes apparent, the model-inherent summaries not only perform well quantitatively (cf. Fig. 6). but are also qualitatively convincing. Colour visualisations as in Böhle et al. (2022); more results in supplement. In contrast to attention explanations, which are not class-specific (Chefer et al., 2021), we find the
We chose to freeze the hackhones to reduce the computational cost and compare the architectures across wide range of settings. We observed comparable results when training the full models for indívidual architectures.
8

---

Under review as a conference paper at ICLR 2023
Input image
Ours [Woxele
Ours Eq. (7)
GradSAM
GradCAM
Rollout
FinAtt
IntGrad
black sooe
Forkiit
00
0
digital clock
lion
Fig. 7: Comparison of the model-inherent explanations (Ours) of a B-cos ViT-B-38, and several post-hoc expla- nations (GradSAM, GradCAM, Rollout, FinAtt, InIGrad) for class & (left). In particular, we show explanations for the classes 'black swan', 'forklift', 'digital clock', and 'lion' used the localisation metrie, Fig. As B-cos VITs follow the B-cos formulation, we can visualise rows of W (x) in colour (Böhle et al., 2022). Additionally, we show contribution maps according to Eq. (7).
synthetic image containing these classes,
model-inherent explanations B-cos ViTs to be highly detailed and class-specific. E.g., in Fig. 1, we compare model-inherent explanations to attention based explanations for single images from the ImageNet dataset which are inherently ambiguous. used in the localisation metric, see Sec. 4, i.e. synthetic images with multiple classes. In both cases we find the model-inherent explanations to accurately highlight the respective features for the class logit that aim to explain, whereas other methods are much less sensitive to the class logit; in fact, attention-baan based explanations are inherently agnostic to the choice logit and thus the same for all classes. For comparisons to explanations for conventional ViTs, the supplement.
Fig. 7, we evaluate the model on images as
CONCLUSION
We present a novel approach for designing ViTs that are holistically explainable. For this, design every component of the ViTs with the explicit goal of being able to summarise the entire model by a single linear transform. By integrating recent advances designing interpretable dynamic linear models (Böhle et al., 2022), these summaries become interpretable, as they are implicitly optimised align with relevant input patterns. The resulting B-cos ViTs constitute competitive classifiers and their inherent linear summaries outperform any post-hoc explanation method on common metrics.
Compared to attention- based explanations, method can be understood 'fill the blanks' in at- tention rollout (Abnar & Zuidema, 2020). Specifically, attention rollout computes a linear summary of the attention layers only. By integrating explanations for the remaining components (tokenisation, attention, MLPs), we are able to obtain holistic explanations of high detail, see Figs. and 7.
As transformers are highly modality-agnostic we believe that our work has the potential to positively impact model interpretability across wide range of domains. Evaluating B-cos transformers on different tasks and modalities is thus exciting direction that we aim explore in future work.
Limitations. While the B-cos ViTs allow us to extract model-faithful explanations for single images, note that these explanations are always local in nature, i.e. for single data points. The explanations thus help understanding an individual classification, but do not directly give insights into which features the models most focus over the entire dataset. It would thus be interesting to combine B-cos ViTs with global explanation methods, such as in Bau et al. (2017); Kim et al. (2018).
Further, we focused primarily on the designing of interpretable transformers, and, to test across wide range of models, limited experiments to the ImageNet-Ik dataset. As transformers are known to significantly benefit from additional data and training (Dosovitskiy et al., 2021), it would be interesting to test the limits of capacity of the B-cos ViTs and scale to more complex tasks.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
The growing adoption of deep neural network models in many different settings is accompanied by an increasingly louder call for more transparency in the model predictions; especially in high-stake opaque decision process can have severe consequences (Rudin, 2019). With this work, we make a step towards developing inherently more transparent neural network models that explain their decisions without incurring losses in model performance.
situations, relying
However, we would like to emphasise that our contribution can only be seen step in this di- rection; while the explanations might seem meaningful on a per sample basis. they could lead false sense of security terms of "understanding" model behaviour. Currently, we can give no formal guarantees for model behaviour under unseen input data and more research on explainable machine learning necessary. Lastly, any research that holds the potential for ac- celerating the adoption of machine learning systems could have unpredictable societal impacts.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020. 1, 2. 7, 9, 14, 15, 23
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 4, 23
David Alvarez-Melis and Tommi S. Jaakkola. Towards Robust Interpretabilityy with Self-Explaining Neural Networks. In Advances in Neural Information Processing (NeurIPS), 2018. 2,
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. CoRR,
abs/ abs//100700645,, 2016. URL http:// arwivv.ariivvvv/va//ii/ org/ abs / / 1607. 06450. 5
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojciech Samek. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer- Wise Relevance Propagation. PLtS ONE, 2015.
Oren Barkan, Edan Hauon, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, and Noam Koenigstein. Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps. In Pro- ceedingy of the International Conference Information and Knowledge Management (CIKM), pp. 2882-2887, 2021. 2, 2,
Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLPP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020. 1,2
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings the IEEE conference computer vision and pattern recognition, pp. 6541-6549 1649, 2017. 9
Wieland Brendel and Matthias Bethge. Approximating CNNs with Bag-of-local-Feature models
works surprisingly well ImageNet. In International Conference on Learning Representations (ICLR), 2019.
Moritz Böhle, Mario Fritz, and Bernt Schiele. Convolutional Dynamic Alignment Networks for Interpretable Classifications. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2021. 1, 2, 6, 23,
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos Networks: Attention is All We Need for Inter- pretability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2, 3, 4, 5, 6, 8, 14, 21, 22, 23
10

---

Under review as a conference paper at ICLR 2023
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1,2, 6, 8, 22, 23
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Workshops, 2020. 6, 23
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 6
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale. In International Conference on Learning Representations, 2021. URL https: / //ooenreview... net. fortt/forumm/ d=Y cbF cbw.N 3, 6,
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International Conference on Machine Learning (ICML), 2013. 5, 22
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. LeViT: Vision Transformer in ConvNet's Clothing for Faster Inference. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 3,
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606: 084 2016. 5, 22
Sarthak Jain and Byron C Wallace. Attention is not Explanation. In Proceedings the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL7), pp. 5543-3556, 2019. 1
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International Conference on Machine Learning (ICML), 2018. 2,
Scott M. Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 3
Sébastien Marcel and Yann Rodriguez. Torchvision library, pretrained models. sytorch.org/
vision/ wwwloon//sablleeeee /oooele//ooel...mm html. Accessed: 2021-11-11.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems (NeurIPS), 2019. 22, 23
Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models. In British Machine Vision Conference (BMVC), 2018. URL http:// bwwc2018.commmmmm org/ contentetttteccoo wttpeeets///ppps//1/e.
pdf.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": the Predictions of Any Classifier. In International Conference on Knowledge Discovery and Data Mining (SIGKDD), 2016.
Explaining
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019. 10

---

Under review as a conference paper at ICLR 2023
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Bass Localization. In International Conference Computer Vision (ICCV), 2017. doi: 10.1109/ ICCV.2017.74. URL https: / //oii org/ 10w.11199//CCC 2017 74. 3, 3,7.23
Sofia Serrano and Noah A Smith. Is Attention Interpretable? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019. 1,2
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning Important Features Through Propagating Activation Differences. In International Conference on Machine Learning (ICML), 2017. 3
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In International Conference on Learning Representations (ICLR), Workshop, 2014. URL
6034. 3
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for Simplicity: The All Convolutional Net. In International Conference on Learning Representations (ICLR), Workshop, 2015. URL http: //axivv. org/ abs 1412. 6806. 3
Suraj Srinivas and François Fleuret. Full-Gradient Representation for Neural Network Visualization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. arXiv e-prints, art. arXiv:2106.10270, June 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Doina Precup and Yee Whye Teh (eds.), International Conference on Machine Learning (ICML), 2017. 3, 7, 23
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems (NeurIPS), 2017. 1, 2,
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi-Heaa Self-Attentio:: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL.), 2019. 7, 23
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 30392 30400, 2021. 4
Bolei Zhou, Aditya Khosla, Ágata Lapedriza, Aude Oliva, and Antonio Torralba. Learning Deep Features for Discriminative Localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
12

---

Under review as a conference paper at ICLR 2023
Supplementary Material
Table of Contents
In this supplement to work designing holistically explainable transformers, provide:
(A) Additional Qualitative Results
14
this section, we show additional qualitattive results and discuss the quali- tative differences between the various explanations methods in more detail. For this, we include explanations for B-cos well for conventional ViTs.
(B) Additional Quantitative Results
15
In this section, we show additional quantitutive results. In particular, we compare the model-inherent explanations of the B-cos ViTs to explanations for conventional ViTs, both for the localisation and the perturbation metrics. Further, we present the results of an ablation study which we investigate the impact of the design choices within the attention layer in more detail.
(C) Implementation Details
22
In this section, we describe the model architectures, the training procedure, the explanation methods, well the evaluation metries in more detail.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL QUALITATIVE RESULTS
A.1 COMPARISON TO ATTENTION EXPLANATIONS
Qurs [Wix
For the reader's convenience, in Fig. A1 we repeat the qualitative results presented in Fig. I, such as to facilitate the following dis- cussion of the qualitative differences between the holistic and the purely attention-ban -baee ex- planations of B-cos ViTs.
Input
Eq. (7)
FinAst
Ibicam hound
cock
In particular, we would like point sev- eral key differences between our holistic ex- planations as per Eq. (7) and the attention- based explanations according to Attention Rollout (Abnar Zuidema, 2020) and the last layer's attention.
First, only the linear mapping W(x) used for the contribution maps in Eq. (7) is able capture 'negative evidence" for the respec- tive classes, see col. 2 in Fig. Al. Note that this does not depend on the particular choice of images shown here. Instead, as the at- tention matrices consist only of non-negative values, the attention-based explanations can-
blue bird
rocking chair
not distinguish between positively and neg-
atively contributing features. Therefore, to improve the attention visualisation and more clearly highlight details in the attention maps, we plot the attention- based explanations on colour scale from 0 the maximal atten- tion value. The model-inherent explanation,
Fig. Al: Inherent explanations (cols. 2+3) of B-cos ViTs attention explanations (cols, 4+5) for the same model. Note that W(x) faithfully reflects the whole model and yields more detailed and class-spocific explanations than attention alone. For a detailed discussion, see Sec. A.I.
in contrast, use colour scale from [-p.p] with p the maximum absolute pixel contribution.
Importantly, as attention-basee explanations do not distinguish between positively and negatively contributing neurons / pixels, they are inherently not class-specific. For example, especially in images in which various classes are present (rows I, 2, 3, 5), attention focuses on all occurring class instances: all birds in rows 2, 3, and 4, as well both dogs in row 1. The model-inherent explanations, on the other hand, clearly distinguish between positive and negative contributions and thus able to resolve class-speciff details.
Secondly, the attention- based explanation are of much lower resolution than the model-inherent ones. Again, this cannot be attributed to the choice of images, but reflects an intrinsic difference between the explanations: whereas the linear mapping W(x) reflects the entire model including the tokenisation module and thus attributes on the level of pixels, the attention explanations can only yield attributions at the level of tokens, which highlights a key difference between the methods: while the attention explanations only include a few layers in their attributions, the model-inherent linear map (x) constitutes an exact summary of the entire model.
Third, as shown in the first column, the rows of the linear mapping W(x) can directly be visualised in colour space, as the B-cos ViTs are designed according to the B-cos framework proposed by Böhle et al. (2022). Note that this is not a masked version of the original image, but instead a direct reflection of the dynamically computed weight matrix Wxx), for details see Sec. C.3. Such visualisations are not possible with iitention-basee explanations.
Finally, we would like to highlight the relation between attention rollout and the model-inherent linear mapping. Specifically, as shown in Eq. (6), note that the entire model can be summarised by
W(x) = Classs (x) I (WMLP(x) wx"(x)) W Tokens (x).
(A.1)
Interestingly, attention rollout in fact computes the overall attention attributions in a similar manner:
A (x)
Class I1 (IMLP A((x))
Tokens
(A.2)
14

---

Under review as a conference paper at ICLR 2023
with Ilaer replacing the actual linear transformation of specific layer by identity matrix and A, denoting the average attention distribution of layer 1, see Abnar & Zuidema (2020). Comparing Eqs. (A.1) and (A.2) succinctly shows how solely attention-base explanations leave out a large part of the model computations, which are seamlessly integrated in the complete linear mapping given by Wxx).
A.2 NORMALISING THE VISUALISATIONS ACROSS MULTIPLE EXPLANATIONS
As we discuss in Appendix C.3, the individual explanations are normalised independently of each other; i.e., all explanations shown on the blue-white-red colour map (Ours (Eq. (7)), GradSAM, GradCAM, IxG, IntGrad, and pLRP) are plotted on scale from - to v with v the 99. ,thh percentile of the absolute value of the given attribution map. As such, the resulting attribution maps are not directly comparable across different explanations. To show the effect of normalising across multiple explanations, in Fig. A2 we repeat Fig. 7 from the main paper, once normalised across contribution maps, and once normalised independently. To normalise across contribution maps, is computed as the 99.9th percentile of the absolute value in all of the four class explanations per method in the figure.
A.3 ADDITIONAL EXPLANATIONS AND COMPARISONS
In Fig. A3, we compare the model-inherent explanations of a B-cos ViT-B-38 model to additional explanation methods apart from purely preentio-bbase ones. Moreover, we show explanations ex- tracted for a conventional ViT-B-38 model in Fig. A4 for comparison.
We would like to highlight the following. First, we find that the model-inherent explanations of the B-cos ViT' provide more detailed and convincing explanations than any of the post-hoc explanations when evaluated on the same model. Crucially, these explanations do not only look convincing, but in fact accurately reflect the model computations of the B-cos ViTs-ice.. they are model-faithiul.
Further, the model-inherent explanations do not only compare favourably to other explanations eval- uated on our newly proposed B-cos ViT's. Instead, they also provide much more detail and highlight more class-specific features than any of the post-hoc explanation methods yield on conventional ViTs, see Fig. A4. As such, we find that there is a clear gain in interpretability when using B-cos ViTs instead of conventional ones.
B ADDITIONAL QUANTITATIVE RESULTS
In this section, we quantitatively compare the interpretability of the B-cos ViTs to that of conven- tional ViTs (Appendix B. Specifically, as in Sec. 5.2, we discuss the localisation and the pertur- bation metrics. Additionally, in Appendix B.2, we present results of an ablation study in which we investigate the design choices of the B-cos Attention module in more detail.
B.1 INTERPRETABILITY COMPARISON: B-COS VS. CONVENTIONAL VIT MODELS
Localisation. In Fig. B1 (right) we present the localisation results of post-hoe explanation methods evaluated on conventional ViTs; for comparison. repeat the results of the B-cos ViTs (see Fig. 6) on the left. As becomes apparent, no post-hoc explanation method evaluated on conventional ViTs allows for localising the correct grid images (cf. Fig. 4) in the localisation metric to the same degree as is possible with the modeel-nnherent explanations. Specifically, we find that the model-inherent explanations yield on average 2.32 times higher localisation scores across the various model config- urations than the best post-hoc explanation method on conventional ViTs (IntGrad).
Perturbation. In Fig. B2 (right) we present the perturbation metric results of post-hoe explana- tion methods evaluated on conventional ViTs; for comparison, we repeat the results of the B-cos VITs (see Fig. 6) on the left. Specifically, as discussed in the main paper, on the left we show the normalised mean area between the curces (ABC) for each model configuration (differently sized backbones and transformers), in which the ABC of each configuration is normalised by the ABC of the model-inherent explanations (Ours). To enable a comparison between the conventional and the B-cos ViTs, on the right normalise by the best post-hoc method (FinAtt) and further multiply the resulting score by the ratio between the mean scores across configurations of FinAtt on conven- tional ViTs and the mean scores of Ours B-cos ViTs (corresponding the respective dashed lines before normalisation).
As discussed in the main paper, we find that the model inherent explanations of B-cos ViTs con- sistently yield the best pixel ranking for each of the configurations of the B-cos ViTs. Further, the
15

---

Under review as a conference paper at ICLR 2023
Input image
((x))s
Ours Eq. (7)
Grad5AM
GradCAM
Rollout
FinAlt
IntGrad
Blok swann
p
foriift
0
digttal clock
0
1::22
Von
(a) Independently normalised attribution maps.
Input image
Ours [W(x))
Ours Eq.
(7)
GradSAM
GradCAM
Rollout
FinArt
IntGrad
back saon
forkiit
23:22
dggaall clock
(b) Jointly normalised attribution maps.
Fig. A2: (a) Repetition of Fig. from the main paper, i.e., with each attribution map normalised independently. (b) The same figure shown again, but this time the normalisation done volumn-wise, i.c., all explanations of a given method are shown on the same scale (except for the coloured explanations in the second column, which are unchanged). Note that the attention based explanations thus not affected. The IntGrad explanations well the forklift explanation according to Ours (7) change most notably-in the case of the model-inherent explanations, this directly reflects the fact that the model has found the least evidence for forklift: the sum of positive contributions according to the model-inherent contribution maps are 13.6 (black swan). 3.3 (forklift), 7.0 (digital clock). and 5.2 (lion).
the same across all classes to begin with and
ABC is on average much higher for the model-inherent explanations for B-cos ViTs than the ABC resulting from the rankings of post-hoc explanations on conventional ViTs. Note, however, that this could also reflect a difference model stability and the comparisons across models have thus to be interpreted with care.
B.2 ABLATION STUDY: ANALYSING THE DESIGN CHOICES MODULE
B-COS ATTENTION
In this subsection, we analyse the impact of the proposed changes for the attention module in more detail. Specifically, investigate the effect of changing the position of the LayerNorm module within the attention layer. Further, we discuss the effect of changing the model's 's value computation and projection layers B-cos layers.
Position of the LayerNorm module. In Fig. B3, we present the classification performance results of various B-cos VIT-S models trained embeddings extracted at different depths of the back-
16

---

Under review as a conference paper at ICLR 2023
[W(x)e
Ours Eq. (7)
Grad5AM
GrodCAM
Rollout
FindAc
IxG
IntGrad
Implin
[W(e]
Ours Eq. (7)
Grad5AM
GrodCAM
Rollout
FinAkt
IG
IntGrad
redshank
Ours [W()]
Ours
GradSAM
GradCAM
Rollout
Findett
IrG
IntGrad
Ours [Wix)
GradSAM
IntGrad
harp
[ix)
Our (7)
Grad5AM
GradCAM
Rellout
FinAet
IkG
IntGrad
asoloti
Ours [W(x]
Ours Eq. (?]
GradSAM
GradCAM
Rollout
Findlt
IGG
IntGrad
traffic light
Ours [W(x]
Ours
Grad5AM
GradCAM
Rollout
FinvAt
lxG
IntGrad
Ours [W(x]
(7)
IntGrad
Fig. A3: Comparison of inherent explanations (Ours) of B-cos ViT-B-38, and several post-hoc explanations for class (left). Ours, we show a colour visualisation (see Sec. C.3) weight matrix W(x) as well as the corresponding contribution maps as defined in Eq. (7). Note that compared methods, the model-inherent explanations yield by far the most detail and class-specificity, as also discussed in Sec. A. above. For comparison explanations generated for conventional ViT-B-38 model the same of images, the following Fig. A4.
corresponding row
bone B-cos 121 model. Specifically, evaluate three different model configurations for each backbone depth: 'Standard Attention', 'Shifted LayerNorm (B=l)', and 'Shifted LayerNorm (B=2)'. Here, 'Standard Attention' refers to the unchanged attention module as it is used in con- ventional ViT models. On the other hand, the 'Shifted LayerNorm' models implement the change described in eq. (13); i.e., for these models, the normalisation layer is moved inside the softmax computation instead of being applied before the attention module as a whole. We observe that the models with the shifted LayerNorm perform significantly better than those using standard attention, especially for shallower backbone models.
We attribute this to the fact that using LayerNorm only within the softmax computation leaves the norm of the value vectors unchanged, such that they are inherently on similar scale as the token embeddings they are added to in the skip connection- -his allows the model to compute significant and well-scaled updates of the token embeddings via the attention module. Moreover. the model can learn bias and scale parameters that are specifically tailored to the softmax layer, instead of affecting both the softmax computation as well as the value vectors at the same time.
In the standard attention module, on the other hand, the norm of the value vectors can differ signif- icantly from the norm of the token embeddings in the residual connection, which make it more
17

---

Under review as a conference paper at ICLR 2023
CheferLip
GradSAM
GradCAM
IntGrad
Impkin
CheferLRP
GradSAM
IntGrad
ambulance
CheferLRP
PLRP
Grad5AM
GradCAM
Rollout
FindAt
IxG
IntGrad
harp
CheferLRP
PLRP
Grad5AM
GradCAM
Rollout
FindAct
IxG
IntGrad
asoleni
Chefael RP
pl RP
GradSAM
GradCAM
Rollout
FinAet
lcG
IntGrad
tradfic light
Chefael
pl RP
GradSAM
GracCAM
Rollout
FinArt
kc
IntGrad
indigo bunting
CheferLIRP
PLRP
Grad5AM
GradCAM
Rollout
FinAAt
IG
IntGrad
Fig. A4: Importance attributions given by common post-hoc explanation methods applied to a conventional ViT-B-38 model (see Sec. 4 for the model specifications) on the same set of images as in Fig. A3. We find that none of the common post-hoc explanations for conventional ViTs give similarly detailed results as the model- inherent explanations do for B-cos ViT', see Fig. A3. As such, observe clear gain interpretability when using B-cos ViTs instead conventional ones.
difficult for the model to take advantage of the attention module. This hypothesis is corroborated by an analysis of the norms of the embeddings. In particular, we find the embeddings coming from the skip connections to be average several orders of magnitude larger than those returned by the attention layer in the models trained with standard attention: e.g., by factor of 10 in the model trained 13-layer backbone model. In contrast, in the Shifted LayerNorm model with B=2, this factor is on the order of 10" i.e both embeddings are in fact on similar scale. While models with standard attention could in principle learn large scale values in the LayerNorm modules, this would effectively result in one-hot encodings in the softmax computation, which could hamper learning.
As the attention modules thus have very limited impact on the model output, we further observe that the model does not seem learn useful attention maps in the first place. F.g., Fig. B4 we compare the attention maps of the models with Standard Attention and the Shifted LayerNorm models various images and find those of the Standard Attention model to be much less structured; note that for the Standard Attention model, the model output no longer dynamic linear transformation of the input, due to the LayerNorm module.
B-cos transforms for value and projection layers. In the following, discuss the impact of replacing the linear layers typically used in the value computation and the projection layers by cos layers (cf. eq. (14). In particular, would first like highlight that B-cos transform with
18

---

Under review as a conference paper at ICLR 2023
Localisation Metric
0
Ours
Rollout
FinAtt
IntGrad
lxG
GradSAM
GradCAM
pLRP
RReferLRRP
1.00
Conventional ViTs
0.75
Localistiin
B-Cos ViTs
0.50
0.25
0.30
72
73
71
75
76
77
71
72
Model accuracy
Model accuracy
Fig. B1: Left: Localisation metric results for the B-cos VITs various sizes, same as shown the main paper in Fig. (left). Right: For comparison, we show the results of common post-hoe explanations evaluated on the corresponding conventional ViTs. As can be seen, the model-inherent explanations of the B-cos ViTs not only constitute the best localising explanation for any given B-cos ViT, but also achieve much higher localisation scores than the best post-hoc hoc explanations evaluated on conventional ViTs.
Perturbation Metric
Ours
Rallout
FinAAt
IntGrad
lxG
GradSAM
GradCAM
pLRP
CheferLRP
Normalised alssed ABC
1.0
ViTs
x1.9
1.05
Conventional ViTs
0.5
0.0
Model accuracy
Model accuracy
Fig. B2: Left: Perturbation metric results for the B-cos ViTs of various sizes, same as the main paper in Fig. 6 (left). Right: For comparison, we show the results of common post-hoc explanations evaluated on the corresponding conventional ViTs. Note that, to allow for comparison between the B-cos ViTs and the conventional ViTs, we normalised mean ABCs of conventional ViTs by the mean ABC of the best post- method (FinAtt) and multiplied results by the ratio between mean ABCs FinAtr on conventional ViTs and the mean ABCs of Ours on B-cos ViTs. For a detailed discussion, see Sec. B.
Standard Attention
Shifted LayerNoorm (B-1)
Shifted LayerNorm (B-2)
0.90
Top-1 accuracy
0.70
0.65
0.90
0.55
backbone layers
38 backbone layers
87 backbone layers
Fig. B3: Ablation results for different version of the attention module. Specifically, show the top-1 accuracy on the ImageNet validation set for B-cos ViTs with the conventional attention module ( ('Standard Attention the proposed B-cos Atlention ('Shifted LayerNorm', see eq. (13)). Specifically, results of models trained with different values for B (B=l and B=2) in the value computation and the projection head, see eq. (14). Note that for B=1, the B-cos transform is equivalent to a linear transformation. As such, 'Standard Attention' and 'Shifted LayerNorm (B=1)' differ only in the positioning of the LayerNorm module.
latter we show
19

---

Under review as a conference paper at ICLR 2023
Standard Attention
Inpuc
Rallout
FinAt:
(a)
Shifted LayerNorm (B-1)
Shifted LayerNorm (B-2)
Ous WixiL
Ous |Wixil
Rellout
Fiehtt
(b)
Fig. B4: Visualisation of attention-basee explanations as well as the model-inherent explanations for models trained with (a) the standard attention module, (b) Shifted LayerNorm (B=1), and (c) Shifted LayerNorm (B=2); for details, see Sec. B.2. As we describe in that section, the models with standard attention seem unable take advantage of the attention module and thus do not leam structured attention maps (a). In contrast, the LayerNorm is shifted inside the sofımax computation.
models are not only inherently explainable by single linear transformation, but also learn to use the attention layers in a much more structured manner (b+c).
20

---

Under review as a conference paper at ICLR 2023
B-cos ViT-(Ti/S/BB--1
B-cos VIT-(TT/S/B)-38
B-cos ViT- ViT-(Ti/S/BB--8
No MaxOut
No MaxOut
No MaxOut
80
Top-1 accuracy
73.7
75.6
72.7
74.6
75.7
76.1
76.6
74.9
76.4
76.6
76.8
76.7
71.1
70.4
70
66.9
60
Tiny
Small
Base
Transformer size
(a) Comparison between B-cos VITs with (solid) and without (striped) MaxOut in the Transformer layers.
VTT-TT3/SBBJ-13
VIT-/Ti/S/BB-33
(T-(Ti//S/B--B
B-cosify (No MaxOut)
B-cosify (No MaxOut)
B-cosify (No MaxOut)
Top-1 accuracy (%)
80
74.6
74.5
74.5
75.3
76.4
74.3
76.8
74.5
72.7
73.1
71.4
70.4
70
66.9
60
Tiny
Small
Base
Transformer size
(b) Comparison between B-cos VIT' (no MaxOut) (striped) in the Transformer layers and conventional ViTs.
Fig. B5: Understanding the impact of MaxOut model performance. (a) For the Tiny transformers, find that MaxOut indeed significantly improves model performance. However, this performance gap between models with and without MaxOut closes with increasing model size (Small and Base). (b) As such, when comparing to conventional ViT' (same numbers as in Fig. 5), we find that B-cos ViTs without MaxOut can achieve similar performance without adding additional parameters in the Transformer layers, as long the initial model size is sufficiently large.
B=1 is in fact equivalent to a linear layer and the Shifted LayerNorm (B=1) model thus only differs from the conventional attention by the placement of the LayerNorm module.
When comparing the classification performance of the Shited LayerNorm models with different val- ues for B, we find the models to perform very similarly, see Fig. B3. Given that the main difference between those models a slightly higher value of B in one them, this is not surprising. Moreover, given that both models still adhere to the B-cos formulation, both models are accurately summarised by a global linear transformation and allow for a model-faithful decomposition into individual in- put contributions, already induce significant alignment, irrespective of the value and projection layers'. In contrast to the model with Standard Attention, find both models with Shifted LayerNorm learn highly structured attention maps, see Fig. B4.
Fig. B4; note that all other modules still use B=2 in both models and thus
B.3 ABLATION STUDY: ANALYSING THE IMPACT OF MAXOUT ON PERFORMANCE
As discussed in Sec. 3, when converting the baseline ViTs to B-cos ViTs, we follow Böhle et al. (2022) and add MaxOut unit to every B-cos transformation. This, of course, doubles the number of parameters, which can skew the comparison to the baseline models. In the following, we assess how Max impacts the model performance.
In particular, in Fig. B5 (a), we compare the performance of B-cos ViTs as presented in the main paper to version that does not use MaxOut when converting the baseline Transformer layers. We
As was shown in Böhle et al. (2022), higher values of B can lead to higher degree of alignment, but this transition is smooth and value of B=1 in the attention layers seems be sufficient.
21

---

Under review as a conference paper at ICLR 2023
observe that for smaller models (see, e.g.. Tiny), MaxOut indeed significantly improves the perfor- mance of the B-cos ViTs. However, with increasing model size (Small and Base), this performance gap closes and the B-cos ViTs without MaxOut perform par with those that have twice the number of parameters in the Transformer layers.
As result, we find that for sufficiently large Transformers (Small and Basc), the B-cos ViTs without MaxOut are able achieve similar performance as the baseline models, without increasing the parameter count, as we show in Fig. B5 (b).
C IMPLEMENTATION DETAILS In the following, we provide further implementation details regarding the models (C. I), the training and evaluation procedure (C.2), the explanation methods (C.3), and the evaluation metrics (C.4).
C.I MODELS
For all models, we rely on the implementation by Chefer al. (2021), which we use unchanged for the conventional ViTs and modify as we describe below for the B-cos ViTs (C.1.1). The configu- rations of the ViTs follow the conventional specifications for ViTs of size Ti, S, and B, ef. Chefer et al. (2021), As described in Sec. 3, we use average pooling over the tokens and pass the result to the classifier head for all models. The conventional ViTs use DenseNet- 121 backbone as available in the torchvision library (Paszke et al., 2019).
C.I.1 B-COS VITs
Tokenisation. For all B-cos ViT', et al. (2022) as a tokenisation module, cf. Fig. 2a; specifically, we use the DenseNet-121 with the training* + schedule, as this one achieves the same accuracy on the ImageNet validation set as the conventional DenseNet-121 contained in the pytorch library (Paszke et al., 2019). As described in the main paper, we freeze this backbone and extract features after either 13, 38, or 87 convolutional layers.
pretrained B-cos DenseNet-121 as provided by Böhle
We then apply a single B-cos convolutional layer with kernel size k= 1, 2, 4 and stride 1= 2, with no padding on the feature maps after 87, 38, or 13 convolutional layers respectively, such that the number of tokens is the same for all B-cos ViTs. We found it advantageous to scale the features of the backbones by 10², as this improved signal propagation and lead to better results Depending on the size of the transformer (Ti, S, B, see main paper), this B-cos convolution produced activations with c= 192, 384, 768 channels after MaxOut (Goodfellow et al., 2013) over every two output units. then reshaped n with the number of input The resulting activation map of size ex h tokens.
Attention. As described in the main paper, we replace the value computation as well as the linear projection of the attention heads by linear B-cos transformations. Further, we apply layer normalisa- tion the inputs before the query and key computations; the value computations use the raw input, cf. Fig, 2a. Finally, when additionally learning 'attention priors', see Sec. 3.5, we add a learnable pa- rameter E " to each attention layer, with m the number of attention heads. The parameter B thus contains separate pair- wair priors between any two tokens for every attention head.
MLPs. As discussed Sec. 3.4, we convert the MLPs to B-cos MLPs by replacing the linear transformations by B-cos transformations with two units and MaxOut (Goodfellow et al., 2013). Further, we remove the normalisation layer before the MLP block, and the GELU (Hendrycks & Gimpel, 2016) non-linearities within the MLPs.
Classifier. We use a single B-cos transformation as classification head, without MaxOut and C-1000 output features, i.c., one output for each class.
General remarks. Similar to Böhle et al. (2022), we scale the output of every B-cos layer in the network by scaling factor f= /c //to to improve signal propagation. In particular, as more channels lead to a stronger decay, we used f= 15, 20, 25 for ViTs of size Ti, S. B respectively; further, since the multiplicative prior computes the product of two attention values < 1, the activations in these networks decay even more quickly and we scale each f by an additional factor of 10. Moreover, as
"The pretrained models were downloaded from hatthuu.com/mcbochee/BBB--e Note that in contrast to standard ViTs, in which normalisation layers ensure that the input to each layer is well-hehaved, in B-cos Networks the activations can decay very quickly since no normalisation is used.
22

---

Under review as a conference paper at ICLR 2023
in Böhle et al. (2022), we scale down the model output by 10 after which we add logit bias RE for each of the C' output logits. Lastly, for the
to the model output which is set to (0.00.01100 B-cos ViTs, we encode the imput images in Böhle et al. (2021), i.e. such that each pixel uses 6 color channels [r, g. b, - r. 1- g. - bj with r, g, (o 1]. As we discuss in Sec. C, this allows for visualising the matrices W in color.
C.2 TRAINING AND EVALUATION PROCEDURE
Training. We trained the B-cos ViTs with batch size of 256. For the conventional ViTs, we found larger batch sizes to yield better results and thus trained those with a batch size of 1024. Further, we trained all our models with RandAugment (Cubuk et al., 2020) (n=2 and m=9) and used images of size 224 x 224. While the conventional models were, as is common, trained with SoftMax and cross entropy loss, for the B-cos ViTs we followed Böhle al. (2022) and trained with binary cross entropy and sigmoid applied to the output logits. Note that, as discussed in Böhle et al. (2022), binary cross entropy induces the necessary logit maximisation for every input, which in turn leads to weight alignment with class-relevant patterns.
Evaluation. We evaluated all networks on the ImageNet validation set after resizing the images such that the smaller dimension measured 256 pixels and then center-croppes images of size 224 x 224.
C.3 ATTRIBUTION METHODS
In the following, we describe the explanation methods that we evaluate on the B-cos as well as the conventional ViTs in more detail.
Model-inherent explanations.
The model-inherent explanations that we quantitatively evaluated are given by the contribution maps defined in Eq. (7). Specifically, for given class logit, we extract the effective linear contribution as performed by the model and multiply it with the input in an element-wise manner and sum all values per pixel location, i.e., across the colour channels; note that this is conceptually equivalent to 'Input x Grad' for piece-wise linear models. For a visualisation of contribution maps, see Figs. 1 and 7; here, we use blue-white-red colormap with blue colors denoting negative, and red colors denoting positive contributions. This visualisation method is the same for all methods except for those that use the jet colormap (CheferLRP, Rollout, and FinAtt). Additionally, for better visibility, we clamp the contribution values to the interval [-v, z). with v the 99. 9th percentile of the absolute values of the given spatial attribution map (i.e., after summing over the colour channels). Note that as a result, the explanations are normalised independently; for a discussion of the effect of normalising across multiple explanations, see Appendix A.2.
Further, as also shown in those figures, the B-cos formulation allows to directly visualise the trans- formation matrix (x) in color. Specifically, note that the input to B-cos networks is encoded as p = [r, 9, 6, 1-r, 1 1-1- g, 1 1] with r, g. be [0, 1] the color channels, cf. Sec. C.I.I and Böhle et al. (2022). As such, the color of each pixel is unambiguously encoded by the angle of the pixel vector and it is thus possible to reconstruct the image colors from the angles alone. Crucially, as discussed in Sec. 3.1 ((x) is implicitly optimised to align with relevant patterns in the input, i.e., to have similar angle as the input, such that the weights for any given pixel can be mapped to a specific color in RGB. We further follow Böhle et al. (2022) and use the norm of the pixel vectors to compute the opacity a in an RGB-a encoding, for details see Böhle et al. (2022), and only show pixels that positively contribute the respective logit.
Transformer-specfffi explanations. As described in the main paper, we evaluate against common transformer-speeiffcc explanations such as the average attention distribution in the final layer Atin,tt), Attention Rollout as proposed by Abnar & Zuidema (2020), and GradSAM (Barkan et al., 2021). On the conventional ViTs, we further evaluate LRP-based explanations: partial LRP (pLRP) Voita et al. (2019) and the mansformer-speeifee LRP adaptation by Chefer et al. (2021) (Cheferl RP). For all these transformer-speciffe explanations, we rely on the implementation provided by Chefer et al. (2021).
Architecture-agnosti explanations. Further, as shown in Figs. 6 and 7, we also evaluate other commonly used explanation methods. Specifically, we use IntGrad (Sundararajan et al., 2017) with n=32 steps, 'Input x Gradient' (IxG), cf. Adebayo et al. (2018), as well as an adapted Grad- CAM (Selvaraju et al., 2017) as in Chefer et al. (2021). For the last, we rely on the implementation provided by Chefer et al. (2021). For the remaining methods, we use the implementations contained in the captum library of pytorch (Paszke et al., 2019).
23

---

Under review as a conference paper at ICLR 2023
C.4 EVALUATION METRICS
C.4.1 LOCALISATION METRIC
For the localisation metrie, evaluated all attribution methods the grid pointing game (Böhle et al., 2021). For this, we constructed 250 2 2 grid images, sec, c.g., Fig. 4. As was done Böhle al. (2021), sorted the images according the models' classification confidence for each class and then sampled random set of classes for each multi-image. For each of the sampled classes, we then included the most confidently classified image in the grid that had not already been used in previous grid image.
Further, transformers with positional information expect fixed size input, see Sec. 4. To neverthe- less evaluate the attribution methods on the localisation metric, which uses images of size 448 448, we scale the synthetic images down by factor of two such that they are of size 224 x 224 and thus of size that compatible with the transformers.
C.4.2 PERTURBATION METRIC
As for the pixel perturbation metrics, we proceed as follows.
First, we sort the images the validation set by their classification confidence and evaluate on the first 250 images for each model. We do this to reduce the computational cost of evaluating this metric whilst nevertheless allowing for a fair comparison between models. Specifically, we observed the model stability correlate with the model confidence and by choosing the most confidently predict images, each model is evaluated in favourable setting, which ensures comparability.
Second, we remove up to 25% of the pixels from the images in increasing / decreasing order as ranked by a given attribution method. Specifically, we sample the resulting confidence curves equidistant points r in the interval [0, 25%] and 'remove" the pixels by zeroing out the respective pixel encoding.
Finally, we record the mean model confidence at the sampled points, which we normalise by the initial mean confidence; as such, each curve starts at (0, 1) with the percentage pixels removed and the normalised model confidence. To assess the quality of the ranking, we then measure the area between the two confidence curves corresponding the order in which remove the pixels, i.e., least / most important first, see, e.g., Fig. 6 (right).
24


Reviews:
[{"review_id": "gvvMNQk1p-", "reviewer": "Reviewer_oHsW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel approach towards improving the explainability of vision transformers. Specifically, the paper proposes updating each component in the model using the B-cos transform, which is designed to increase the alignment of the inputs and the weights. The paper builds on a previous state-of-the-art work that introduced the B-cos transform on DNN/CNNs and extends it to vision transformers. Comprehensive experiments are conducted comparing the proposed approach with several state-of-the-art explainability methods.", "summary_of_the_review": "I don't think it's a strong accept due to the slightly derivative nature of the work, but overall, the paper extends a very novel idea from DNN/CNNs to vision transformers and supports this with strong benchmark results. Due to this, I think its marginally above the acceptance threshold. ", "strength_and_weaknesses": "Pros\n\t- The proposed method is simple and seems easy to implement with the provided details.\n\t- Strong benchmark performance improvements across multiple explainability metrics. Further, the attribution maps of the model are highly detailed and have very low noise.\n\nCons\n\t- This paper is a direct extension of \"B-cos Networks: Alignment is All We Need for Interpretability\" by Bohle et al, which proposed the B-cos transform and implemented it on MLPs and CNNs. In this paper, that transform is applied to the vision transformer. However, the main contribution of this paper is incorporating the B-cos transform in the attention layers and the positional encodings. It's incorporation in the MLP/CNN layers is directly taken from the original paper. This makes the architecture seem a bit derivative.\n\t- The paper proposes using max-out network in equation 12. However, that would double the number of parameters in the model. It would be helpful if some comments are added regarding the increased model size. It would also be helpful to know if removing the max-out layer will significantly impact the final results.\nMost of the results compared the model with the state-of-the-art visualization models. Given this model is a significant architectural modification of the original vision transformer, it would be really helpful to compare the results of the model with those of the vision transformer on other benchmark tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "d3hPqHeS9j", "reviewer": "Reviewer_NoVr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to replace the modules of transformer with dynamic linear operation so that it is more explainable. It follows the guideline of BCos Networks but adopts transformers on the top of DenseNet features. The method is evaluated on ImageNet with localization and perturbation metric. It claims to have to be more interpretable and perform competitively to ViT.", "summary_of_the_review": "Overall, I feel that the contribution does not meet the bar of ICLR and some claims are very misleading to readers. I would recommend weak reject for the current version.", "strength_and_weaknesses": "Strength:\n+ The idea of improving interpretation of transformer is interesting.\n+ The paper is well organized and easy to follow.\n+ The method is well evaluated and the results are well presented.\n\nWeakness:\n- My first concern is the claim of Bcos-ViTs. The authors are using transformers on the top of DenseNet features which is not ViT. A ViT is trained directly on images by considering each patch as a token. The method is more like Bcos Transformer, not Bcos-ViT. \n- The whole explanation principle is exactly the same as the Bcos Networks (CVPR2022) which makes this paper more like an extension from deep network to transformer. What is the new challenge here for explaining DenseNet+transformer using Bcos idea as compared to explaining a DenseNet?\n- The method itself cannot be adopted to explain existing ViT models like [Chefer et al. (2021)], but has to be re-trained to gain interpretation ability. But this paper does not show how the ViT would performance if all the transformers are replaced with the proposed block. It is very misleading to claim DenseNet+Transformer as ViT in Fig 5. How would a standard ViT perform if the proposed Bcos method is used?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cZ1-GRur-F", "reviewer": "Reviewer_rHTt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new Transformer architecture for better holistic explainability. The architecture generates explanations that reflect all the model components instead of only the attention layers. To do so, the authors take explicit designs for each of the Transformer modules so that they can be summarized by a single linear transformation. The authors perform the main and ablative experiments on ImageNet to evaluate the effectiveness of the proposed methods.", "summary_of_the_review": "There are many confusing points that need to be addressed.", "strength_and_weaknesses": "Strength:\n\n•\tThe qualitative results look promising. The proposed method generates visualizations with richer details than the baseline methods shown in Fig. 7 and Fig. A1. \n•\tThe quantitative results look impressive. The proposed B-cos Transformer have higher localisation and perturbation scores than the baseline methods under the same model accuracy. Compared to the conventional ViT models of the same depth, B-cos Transformer has better overall performance across different depths. \n•\tThe paper is well-written and easy to follow.\n\nWeakness:\n\n•\tThe authors argue that B-cos Transformer performs at least as well as the baseline ViTs under the same number of backbone layers in Section 5.1. However, the experimental setting is unfair without comparing the computational complexity to the baseline methods, especially when the authors replace the conventional tokenization module with a pre-trained B-cos DenseNet-121. Though deriving more explainable models, the proposed model should be toned down if they are too heavy to deploy. This is the same for Fig. 6, which also lacks comparisons of model complexity vs. Localisation/perturbation. \n\n•\tI am also confused about where the good performance of B-cos Transformer comes from. The authors claim that the positional information design largely improves performance. However, the effectiveness of the positional information cannot be concluded from Fig. 5, with many other components modified. To fully explain the performance gain, comparisons have to be done by attaching the B-cos positional information to the conventional ViTs.\n\n•\tI am also curious about how the third column of Fig. 7 is generated, as the visualizations do not seem to be normalized.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject