Attention-based Interpretability with Concept Transformers
Authors: Mattia Rigotti
Christoph Miksovic
Ioana Giurgiu
Thomas Gschwind
Paolo Scotton
Venue: ICLR 2022 Poster (2022)
Keywords: attention
transformer
concepts
interpretability

Abstract: Attention is a mechanism that has been instrumental in driving remarkable performance gains of deep neural network models in a host of visual, NLP and multimodal tasks.
One additional notable aspect of attention is that it conveniently exposes the ``reasoning'' behind each particular output generated by the model.
Specifically, attention scores over input regions or intermediate features have been interpreted as a measure of the contribution of the attended element to the model inference.
While the debate in regard to the interpretability of attention is still not settled, researchers have pointed out the existence of architectures and scenarios that afford a meaningful interpretation of the attention mechanism.

Here we propose the generalization of attention from low-level input features to high-level concepts as a mechanism to ensure the interpretability of attention scores within a given application domain.
In particular, we design the ConceptTransformer, a deep learning module that exposes explanations of the output of a model in which it is embedded in terms of attention over user-defined high-level concepts.
Such explanations are \emph{plausible} (i.e.\ convincing to the human user) and \emph{faithful} (i.e.\ truly reflective of the reasoning process of the model).
Plausibility of such explanations is obtained by construction by training the attention heads to conform with known relations between inputs, concepts and outputs dictated by domain knowledge.
Faithfulness is achieved by design by enforcing a linear relation between the transformer value vectors that represent the concepts and their contribution to the classification log-probabilities.

We validate our ConceptTransformer module on established explainability benchmarks and show how it can be used to infuse domain knowledge into classifiers to improve accuracy, and conversely to extract concept-based explanations of classification outputs. Code to reproduce our results is available at: \url{https://github.com/ibm/concept_transformer}.

Published as a conference paper at ICLR 2022
ATTENTION-BASED INTERPRETABILITY WITH CONCEPT TRANSFORMERS
Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind & Paolo Scotton IBM Research
Zurich, Switzerland
{mrg, cmi, ig t,g, psc) @eereiuiiii ch. b.mm com
ABSTRACT
Attention is mechanism that has been instrumental in driving remarkable per- formance gains of deep neural network models in a host of visual, NLP and mul- timodal tasks. One additional notable aspect of attention is that it conveniently exposes the "reasoning" behind each particular output generated by the model. Specifically, attention scores over input regions intermediate features have been interpreted as a measure of the contribution of the attended element to the model inference. While the debate in regard to the interpretability of attention is still not settled, researchers have pointed out the existence of architectures and scenarios that afford a meaningful interpretation of the attention mechanism.
Here we propose the generalization of attention from low-level input features high-level concepts as mechanism to ensure the interpretability of attention scores within a given application domain. In particular, we design the Concept- Transformer, a deep learning module that exposes explanations of the output of model in which it is embedded in terms of attention over user-defined high-level concepts. Such explanations are plausible (i.c. convincing the human user) and faithful (i.e. truly reflective of the reasoning process of the model). Plausibility of such explanations is obtained by construction by training the attention heads conform with known relations between inputs, concepts and outputs dictated by domain knowledge. Faithfulness is achieved by design by enforcing a linear re- lation between the transformer value vectors that represent the concepts and their contribution to the classification log-prou pog-prbbabilities.
We validate our CrnsepeTrr module on established explainability bench- marks and show how it can be used to infuse domain knowledge into clas- sifiers to improve accuracy, and conversely to extract concept-based explana- tions of classification outputs. Code to reproduce our results is available at: httpps://qithui com ibm./ concept. transformer
INTRODUCTION
The spectacular gains in accuracy of recent large-scale machine learning models like deep neural networks have generally come at the cost of a loss of transparency into their functioning. This "black box" aspect severely limits their applicability in safety-critical domains, such as medical diagnostics, healthcare, public infrastructure safety, visual inspection for civil engineering, to name just few, where it essential for decisions be corroborated by robust Nomain-relevan knowledge. In recent years, approaches focusing on explaining black box models have emerged, mostly with the goal of providing post-hoc explanations terms of a set of relevant features used by the underlying model to make predictions (Ribeiro et al. 2016 Sclvaraju al. 2017 Lundberg & Lee [017 Smilkov al.) 2017). While widely used, such post-hoc explainability methods have been criticized for operating on low-level features such pixel values, sensory signals that are combined in unintelligible ways and do not correspond to high-level concepts that humans easily understand (Kim et al. 2018
Mevarez--Meis & Jaakkola 2018 Kindermans et al. 2019 Suu et al. (019)).
To overcome these limitations and sidestep the potential perils resulting from a misuse of post- hoc explainability of black box models, some researchers have been vocally advocating for the use

---

Published as a conference paper at ICLR 2022
of inherently interpretable models (Rudin 2019) that in particular would generate decisions based
on human- uumaannnudderttdddable categories (i.e., concepts) grounded in domain expertise rather than raw
2019: Kim et al. 2018 Ych et al.) 2020 Koh et al. 2020 Goyal et al., 2019; Kazhdan et al. 2020; Chen et al. 2020 Alvarez-Melis & Jaakkola 2018 Li al. 2018: Chen et al. 2019]. For example, identify bird species, model should focus on morphologically meaningful gul concepts, such as the shape, size and colors of beak, feathers or wings, rather than focusing on raw pixels, and combine them in ways that a domain expert (in this case an ornithologist) would reckon as intelligible to produce a classification. In addition, using high-level concepts emulates a human's thinking process (i.e., structured into familiar concepts) and provides insights into the model's 'ss reasoning in a human-understandabll way.
features (Barbiero et al., [202]] Ghorbani et al.
The chasm between post-hoc explainability vs. inherently interpretable models closely reflects a related ongoing discussion in the NLP community on the interpretation of attention mechanisms (Bahdanau et al., 2014), and in particular on the interpretability of attention weights over input tokens, with researchers while others claim that "Attention is not explanation" (Jain & Wallace 2019). While the debate over what degree of interpretability that can be ascribed to attention weights is still not settled (Wiegreffe & Pinter 2019). it is arguable that in many situations attention is not a -fail-safe indicator" (Serrano & Smith 2019), particularly when decisions rely the interaction of multiple interacting tokens as is typically the case in deep architectures. Conversely then, a way to guarantee the interpretability of attention weights would be to make sure that they are not being processed by downstream operations that renders their relation to the decision outputs uninterpretable. This is indeed something that had been proposed in the past, in particular in architectures that preserve the interpretability of "relevance scores" (akin to attention weights) by acting on them only through a restricted class of intelligible "aggregation functions" such as additive models (Alvarez-Melis & Jaakkola 2018). which are a common functional elements in interpretable and white-box models (Caruana et al. 2015).
one end of the debate claiming that attention provides interpretability,
In this paper, we propose the Concep(rransormer (CT), a transformer-based module (Vaswani et al., 2017) for classification tasks, that can be used to enhance an arbitrary deep learning classifier with domain knowledge in the form of plausible cross- cooss-attention weights between input features and high- level interpretable concepts. The CT can be used as a drop-in replacement for the classifier head of any deep learning architecture. The resulting model can then be trained end-to-end without any additional overhead on the training pipeline, except a modification of the loss function that enforces plausibility of the explanation. Importantly, the CT was specifically conceived to provide explana- tions that guarantee faithfulness by design and plausibility by construction. Faithfulness is defined as the degree to which the explanation reflects the decision and aims therefore to ensure that the expla- nations are indeed explaining the model's operation (Lakkaraju et al. 2019 Guidotti et al. 2018). In our model this is achieved by enforeing a linear relation between the transformer value vectors that represent the concepts and their contribution to the classification log- pog-pbbbabilities. Plausibility refers to how convincing the interpretation is to humans (Guidotti et al. | In the CT architecture, plausibility is achieved by construction by supervising the attention heads of the cross-attention mechanism to conform with inputs-conceeps--otppus relations derived by domain knowledge.
Carvalho et al., (211))
We validate our approach on three image benchmark datasets, MNIST Even/Odd (Barbiero et al.) 202I]. CUB-200-2011 (Welinder et al. 2010)), and aPY (Farhadi et al. 2009). On these datasets we will examine how the faithfulness and plausibility of the CT explanations into domain-relevant explanations behind particular output decisions or diagnostic insights about ensuing wrong classifications. We will also quantify the benefit of domain-expert knowledge in terms of statistical efficiency by showing that providing domain-relevant explanations to our CT model tends to improve the performance of the downstream classification, in particular in low- regime. This for instance translates in an 8-9% improvement in accuracy on the bird classification CUB-200-2011 dataset when CT is trained in conjunction with part location annotations.
practically translated
We note in addition that one of the strengths of our CT model is its versatility that allows it to be effortlessly applied to other data modalities as well by combining it with deep learning classi- fiers which are then rendered interpretable with no appreciable overhead or change in their training pipeline. This is in stark contrast to other inherently interpretable models that are often specifically designed for the domain at hand, and require adhoc multi-stage training procedures. CT on the other hand, differentiable and can be flexibly included in any end-to-end training pipeline that uses backpropagation, as will showcase by combining it with a host of different deep learn
2

---

Published as a conference paper at ICLR 2022
ing backbones ranging from convolutional architectures like Residual Networks (He al. 2016) to more modern Vision Transformer (Dosovitskiy et al. 2020) and hybrid Compact Convolutional Transformer models (Hassani al. 2021).
RELATED WORK
In recent years, there have been significant advancements towards designing interpretable models that quantify the importance of individual features with respect to the prediction output. One general approach is post-hoc analysis, in which interprets trained model by fitting explanations the classification outputs (Alvarez-Melis & Jaakkola 2018 Ribeiro et al. [2016] Lundberg & Lee 2017).
In particular for CNNs. popular techniques are activation maximization (van den Oord et 2016
Nguyen et al. 2016; Yosinski et al. (2155) and saliency visualization (Selvaraju et al. 201 Smilkov
al. 2017 7 Sundararajan al. (2017)
However, these post-hoc methods do not actually explain how the underlying model reached a par- ticular classification outcome. In contrast, attention-base interpretable techniques aim to expose which parts of a given input a network focuses and therefore deems to be most relevant, when making a decision. Examples of attention models are Zhang et (2014); Zhou et (2016) 2018); Zheng et al. (2017): Fu et al. (2017); Girshick (2015); Girshick et al. 2014); Huang et al. 2016). The problem with these models is that they focus on low-level individual features when providing an explanation. Such features are often not intuitive for humans, are typically noisy and non-robust, or can be misleading when interpreted afterwards (Kim et al. 2018).
One of the recent advancements in the field of interpretability was to design methods that explain predictions with high-level human understandable concepts (Ghorbani et al. 2019
Kim et al. 2018
Yeh et al.] 2020: Koh et al.
2020; Goyal et al.
2019 Kazhdan et al. 2020 Chen al.
2020
Barbiero et al. [221]] Li et al.
2018 Chen et al. 2019) either by identifying common activation
patterns the last nodes of the neural network corresponding human understandable categories or
constraining the network to learn such concepts. For instance, TCAV (Kim et al. 2018) proposes to
define concepts from user-annotated examples in which concepts appear. Others propose prototypes- based explanation models (Li et al. 2018 Chen et al.) 2019). but they typically require specialized particular, ProtoPNet (Li al. 2018) uses previously learned prototypes to focus attention on various parts of an image. This architectural design implies that object-level (global) concepts cannot be easily incorporated, and since prototypes are not learned together with the attention model, explanations based on these prototypes may lack faithfulness. SENN Melaarez-Melis & Jaakkola 2018) proposes a network that transforms inputs into interpretable basic features, generates concept relevance scores and then aggregates concepts with relevance scores to explain predictions. While it is out-of-the-bor interpretable, it lacks concept localization. Barbiero et al. (2021) proposed differentiable approach that allows the extraction of logic explanations from neural network models using First-Order Logic. The approach relies on an catropy-based criterion automatically identify the most relevant concepts that have contributed to a particular classification output.
convolutional architectures to ensure feature extraction.
In our approach, high-level concepts are defined with set of related dimensions, and can be part- specific or global. Such concepts are typically readily available in many domains and can be used to enhance the performance of the learning task while offering explainability at no additional cost for the network. Obtained explanations are plausible and guaranteed to be faithful, since concepts participate in the model computation. Finally, CT also allows in some cases to discover the presence concepts that were not annotated.
3 APPROACH
The ConceptTransformer module. The CrnsepeTranssormer (CT) is transformer-based mod- ule designed to be used as classifier head in deep learning architecture that generates classification outputs using cross-attention (Vaswani et al. 2017) between input features and set of embeddings representing domain-relevant concepts. Fig. shows the case where the inputs to the CT are em- beddings of P visual patches of input image that are linearly projected and concatenated into the query matrix Q E Iº Pxd,.. of a query-key-veee vayue cross-attention mechanism whose corresponding key matrix K Cxdm is the linearly projected concatenation of the embeddings representing
3

---

Published as a conference paper at ICLR 2022
the C' concepts. In addition, the concepts are linearly projected with value projection matrix and concatenated to result in the value matrix V E RCxdm. Cross Cross-attentine attention then outputs an attention weight
(V @KT
1 (ain
Ppc = softmax
with p p=1,..., c= c=
between each patch-concept pair, which are combined into an attention map matrix A = [Apel RPCC The final output of the CT is the product obtained by multiplying the attention map A, the value matrix V and an output matrix E Rdmxxn, that projects onto (unnormalized) nt, logits over the output classes, and averaging over patches:
P
logit; D[AVO] with = ,,..,,
(I)
p=1
Notice that here for simplicity we described single-head head attention model, but in our experiments will be using multi-head version (Vaswani al. 2017).
Equation says that, given an input æ to the network, the conditional probability of output class i is
Pr(ilæ) softmax, (Een) Be (c(r)
with Be with components (Be); [VO]
and (x()) positive relevance scores that depend on through the averaged attention weights: 7c(z) } Ep= Oppc The output of the CT is therefore essentially a simple multinomial logistic regression model over positive variables (e(x) that measures the contribution of each concept.
Notice that this result follows from the linear relation between the value vectors and the classification logits, which itself comes from the design choices of computing outputs from the value matrix V through the linear projection VO, and aggregating patch contributions by averaging.
Cross Attention
classification
Averaa
outputs
Input Image
patch embeddings
logits
V
concept
embeddings
ConceptTransformer
Figure 1: The ConcepTTransformer (CT) is a transformer module that provides concept-based inter- pretability by design. a drop-in replacement for the classifier head of an arbitrary deep learning classifier and uses cross-attention to generate classification rog-pobbabilities as additive contribu- tions from user-defined concepts that are plausible the domain under consideration. In the figure it is used as classifier head of patch-based architecture like ViT a patch-level CNN.
Faithful concept-based explanations by design. The CT was conceived drop-in replacement for the classifier head of an arbitrary deep learning classifier to provide concept-based explanations of the outputs that guaranteed to be faithful by design. We formalize this statement as follows:
Proposition 1 Each concept relevance score (x(x) in Equation 2 is a faithful explanation of the output. More specifically, the probability of choosing the preferred output to arg max, (Bc); of concept (assuming it's unique) is guaranteed to decrease (e(x) is externally set to zero. Moreover, the correlation between (x( and Pr(j9))) is strictly positive.
Proof Proof of Proposition is provided Appendix A
Note that the last statement in the Proposition above is corollary of the first one. and it specifically shows that CT is guaranteed to satisfy the technical definitions faithfulness given for instance by Alvarez-Melis & Jaakkola (208)).
4

---

Published as a conference paper at ICLR 2022
Training and plausibility by construction. As mentioned, the CT is a differentiable transformer- based module that can be embedded in a deep learning architecture trained end-to-end with back- propagation. In addition, the fact that it exposes attention weights over concept tokens that can be user defined gives us the freedom to shape these attention weights according domain-expert knowl- edge relevant for the problem under consideration. This can be done by explicitly guiding the atten- tion heads to attend to concepts in the input that are priori known to be informative to correctly classify the input. In practice this can be achieved by supervising the attention weights at training as for instance proposed by Deshpande & Narasimhan (222)) as a self-supervised technique for bi- directional language models. In particular, given desired distribution of attention provided by domain knowledge (e.g., we know which patches in the input image contain which concepts that are relevant to classify the input) we can force the CT attention weights A by adding an "explanation cost" term to the loss function that is proportional to Leepi = A| HIB. where || II is the Frobenius norm. The final loss used to train the architecture then becomes L = Lets + Lezpls where Lote denotes the original classification loss, Leupt the additional explanation loss, and the constant À 2 O controls the relative contribution of the explanation loss to the total loss. Notice that setting À 0 essentially amounts to just minimizing the classification loss and disregarding the prior domain knowledge as imparted into CT by guiding the attention heads.
4 RESULTS
We have evaluated the proposed approach on three datasets: MNIST Even/Odd (mni). CUB-200- 2011 (Welinder et al., 2010) and aPY (Attribute Pascal and Yahoo) (Farhadi et al.j 2009). Each dataset illustrates a slightly different use case and different visual backbone in combination with the CT. This variety is meant to showcase the flexibility and versatility of the CT module.
With MNIST Even/Odd we consider a case where the correspondence between concepts and output classes is many-to-one and deterministic. In addition, concepts are global as opposed to being spatially localized. That means that we will not use patch-based representation of the input, and instead the concepts refer to the whole input, which is essentially equivalent to having only one patch. For this task, CT is combined with small Compact Convolutional Transformer architecture (Hassani et al. 2021).
In the case of CUB-200-2011 1 we consider is instead many-to- tomyy-to-mand many and non-deterministic, and there exists a mixture of global and spatially localized concepts. In order to handle both types of concepts we instantiate two CTs, one for spatial concepts and one for global concepts and we then average the logits provided by both in order preserve interpretability (as explained in the previous section). For this dataset, we use Vision Transformer as a backbone. As inputs to the CT handling the spatial concepts we use the embeddings of the tokenized image patches, while as input to the CT in charge of the global concepts we use the embedding of the CLS token.
where the relation between concepts and outputs
Finally, with the aPY (Attribute Pascal and Yahoo) dataset we consider another situation where con- cepts are exclusively global but this time are many-to-mant The architecture of the visual backbone used on this dataset a larger version the Compact Convolutional Transformer architecture used on MNIST Even/Odd, but with larger tokenizer consisting of ResNet50 model pre-trained on ImageNet.
4.1 EVALUATION ON MNIST EVEN/ODD
The objective of this evaluation is simply to illustrate how our architecture works. We perform simple binary classification task, based on the MNIST Even/Odd dataset, aiming at classifying 28x28 pixel images of hand-written digits ranging from to 9 as either 'even' or 'odd'. this case, exploit the fact that we know the identity of each digit and use that as an explanation for the binary classification prediction. In other words, for instance a '7' should be classified as 'odd", and plausible explanation to support this prediction is that is 'odd' because it is '7'.
Figure 2 shows the accuracy on the test set (left) and explanation loss during validation (right), relative to the number of samples used at training. which varies from 100 to 7000. With the largest number of training samples, the CT achieves 99% accuracy. irrespective of whether concepts are leveraged (A = 2.0) not (A 0.0). However, when the model is trained with significantly
5

---

Published as a conference paper at ICLR 2022
fewer samples (e.g., 100 500), using concepts provides performance boost, in the range of 5%.
Accordingly, we observe a more graceful decay of the explanation loss when À = 2.0.
0.975
vell explamatioo losss
0.555
Aeup 0.0
0.-
Aewp 0.0
Acup 2.0
0.6
Aow 2.0
0.75
0.825
0.100
sco 10000000000000000000000000000000
500 róo sdo receooosooecooooooooooooo training samples
training samples
Figure 2: Sample-efficiency gain of using explanations at training on the MNIST Even/Odd dataset.
Figure 3 (left) shows a test sample, '7' whose correct binary label is 'odd", which is itself a prediction that should be supported by the correct ground-truth explanation '7' (i.e." "hee sample is 'odd', because it is a "). CT in this case outputs the correct prediction, and looking at the concept attention weights we can see that indeed this prediction supported by the correct concept.
prediction: (correct)
prediction: (wrong)
ground Sruth explanation
ground-truth explanation
7
9
attention
concept attention scores
Figure Examples of correct and incorrect predictions on MNIST Even/Odd.
Figure 3 (right) on the other hand shows the example of a test sample which is being misclassified by CT: the sample (an '8') is supposed to be classified an even digit, but instead classified as an odd number. If we look at the concept attention weights for this sample we see that the reason for the incorrect prediction can traced back to the fact that the CT strongly associated the sample to the '9' concept and hence predicted the sample to be 'odd' because it "thought" that it was '9'. Notice that also the correct concept '8' being attended to by the architecture, but the wrong concept received in this case higher attention score. Interestingly, visually inspecting the sample it is not far from being a '9'.
4.2 EVALUATION ON CUB-200-2011
The CUB-200 CUB-200-200 200-2011 (Welinder et al. 2010) dataset contains 11788 images of birds, classified in 200 species. Each image is annotated with a given number of concepts (e.g., shape of the beak, color of the back, etc.) explaining the visual characteristies of the bird in the image. It uses 312 concepts, but their distribution varies across images. We therefore pre-process the dataset and retain concepts that sufficiently representative in class-level annotations. Specifically, for a given class, we retain only concepts that are present in at least 45% of the images of that class and subsequently present in least 9 classes. This led us retain 108 concepts.
Table 1 compares our CT against other deep models (top models in bold), classified based on their training procedure Multi-Stage (i.e.. complex training) and End-to-end (i.e., training with back- propagation). When concepts are leveraged to train the CT, we notice significant boost in accuracy of 8.2% (i.e., CT [w/o] vs. CT). Therefore, not only does the CT provide explainability by design, but it also boosts the overall performance of the classification task. In future, plan to examine ways modeling and taking advantage the relation between extracted concepts. This has been shown to further boost performance by Barbiero et al. (2021) by extracting First Order Logic clauses that relate concepts.
The test accuracy of model is par with the baseline, non-interpretable model (B-CNN) as well as with ProtoPNet, PN-CNN, SPDA-CNN, MA-CNN and RA-CNN, even though some of these techniques require significantly more complex training than the CT. In terms of the type of

---

Published as a conference paper at ICLR 2022
Table 1: Performance on CUB-200-2011 when concepts are leveraged at training (CT) or without concepts (CT [w/o]). Comparison against state-of-the-art approaches, classified by their training complexity: B-CNN (Lin et al. 2015b). Part R-CNN (Zhang et al. 2014p. PS- PS-CNN (Huang et al.
20161, PN-CNN (Branson et al. 2014), SPDA-CNN (Zhang et al. 2016), PA-CNN (Krause et al.
2015 MG-CNN (Wang et al. 2015). 2-1evel attn. (Xiao et al. 2015), FCAN (Liu et al. 2016).
Neural const. (Simon & Rodner (013), ProtoPNet (Li et al. 2018), CAM (Zhou et al. 2016
DeepLAC (Lin et al. 2015a), ST-CNN (aderberg et al., 2015). MA-CNN (Zhegg et al.
2017 ;
RA- (Fu et al.) 2017). We report their best performance from (Li et al. 2018). irrespective of whether they are trained on full images or bounding boxes.
Training
Accuracy [%]
Multi-stage
Part R-CNN: 76.4
PS-CNN: 76,2
PN-CNN: 85.4 2-level atm.:: 77.9
SPDA-CNN: 85.1 FCAN: 82.0
PA-CNN: 82.8
MG-CNN: 83.0
Neural const.: 81.0
ProtoPNet: 84.8
End-to-end
B-CNN: 85.1
CAM: 70.5
DeepLAC: 80.3
ST-CNN: 84.1
MA-CNN: 86.5
RA-CNN: 85.3
CT [w/o]: 76.9++3
CT: 88.0
explanations provided, at the coarsest level, CAM highlights the entire object the reason behind a classification decision. At a finer level, most models offer part-level attention, however they dif- fer in how attention is generated. For example, Part R-CNN, SPDA-CNN, PS-CNN, PN-CNN use additional part-localization models previously trained with part annotations. RA-CNN uses an ad- ditional neural network to decide where to focus attention, whereas MA-CNN uses convolutional feature maps to direct attention to various parts of the image. ProtoPNet generates attention based on previously learned prototypes, namely it focuses on a specific region of the input image because the region is similar to a learned prototypical example. This leads to two main drawbacks. First, object- level concepts cannot be naturally incorporated, since the approach is based specifically on part-level concepts. Second, finding learned prototypes for other models requires post-hoe analysis, specifi- cally inspecting which region mostly activates model's convolutional filter. Since such prototypes are not computed together with the model, resulting explanations cannot be always guaranteed to be faithful to the classifier's s decisions.
In contrast, the CT makes use of both part-level (spatial) and object-level (global) concepts to gen- erate explanations, as shown Fig. 4 We highlight concepts with the highest attention scores and hence most relevant for the classifier for four correctly predicted bird species. In addition, the model guarantees faithful explanations by learning concepts together with the classification model. Specif- ically, this is achieved by enforcing an additive relation between the transformer value vectors that represent the concepts and their contribution to the classification bog-probabilities. We show that explanations mimic the reasoning of the classifier in Fig. for one misdiagnosed bird species (left image). The CT predicts the species to be Summer Tanager, when in fact the bird is a Scarlet Tan- ager. If we look at the concept attention scores for this sample, we notice that there are two reasons for the incorrect prediction. First, relevant parts of the bird such as upper-parts parts_color :: black and has primary color :: black are not fully visible in the left most image. Both concepts are ex- tremely important to ensure a correct classification as Scarlet Tanager, as seen in the middle image. Second, attention is not attended to the has wwing_cooo :: black concept (missing), but instead to the has has_shape:: perching like concept, which is typical for the Summer Tanager species.
4.3 EVALUATION ON APY (ATTRIBUTE PASCAL. AND YAHOO)
The aPY (Attribute Pascal and Yahoo) dataset introduced by Farbadi et al. (0099 consists of two subsets. The aPascal dataset was created for the Visual Object Classes Challenge 2008 by |Eerr ingham et al.) and consists of 6340 training samples and 6355 test samples. Yahoo is dataset developed by Farhadi et al. (2009) using Yahoo image search and comprises 2644 samples, which are typically used for testing only. Each sample image in either dataset contains one or more objects which are enclosed by a bounding box and labeled with a single class. In total there are 32 classes describing everyday objects (in the form of nouns) such as animals, vehicle types, common items found in apartments. In addition, for most of the objects there is a set of associated attributes. de- noting present object parts characteristics, e.g., "tail", "hair", "door", or "screen". The number
7

---

Published as a conference paper at ICLR 2022
Prediction: Back Jowtedddobeeeee Abstioes (conect)
Prediction: Groove bllled (conect
Prediction: Indigo, Burting (conect)
Prediction: Back biled, Cusioo (corect
Spatial explanations:
Spatial expl lolantaionn:
Spatial explanations:
Spatial explanations:
bas. eye calor::black
has eye color::blcck
has breast sottern:soold
has color:bblack
Global explanations:
has forehead cblack
has eye color::black
has forehead cblor::brown
has, prisary color:
has color::grey
color: :hler
has leg color:grey
:blakk
belly isolid
has leg colori:
Global explanations:
color::black
exp ixplanations
has pwwsprrrrr.................rp..p :olar::black
Global. inplanstioos:
has sos_sice::see :ssle::ssllll Ll (9.- 9__i))
has ptl.pprrarrr..cm color::blue
Figure 4: Attention scores for spatial levell and global (object-level) concepts for four correctly predicted bird species from -UB-200-2011.
Prediction: Summer Tanagor (wrong)
Prediction: Scaret Tanager (conret)
Prediction: Summer, Tanager (coneet)
Soariet Tanager)
Spatial expl asplanations:
Spatial explanations:
Spatial explanations:
has wing co color::black
has breast pattern::solie
isolid
has upperparts color::black
has color::blace
eye :black
has breast pattern:solie
has forehead color: red
has patppaaattsoll....
blac
back, pattern:ssolid
has, mellyypatterm::solee
forehead color:
belly pattten::ssllt.
has Leg color:grey
belly patterrr:soolid
color::gry
Global explanations:
color::grey
Global explanations:
has shape::pecching peerching-chkeee Like
Global explanstions:
shape: pwwppperrhhnggtteeee
has size:snall, (5 in)
has p.ppprrrrrry..m color::black
has size::ineeele [ 9n
has sos_size::salll (5. 9_1n)
Figure 5: Diagnosing classification mistakes in CUB-200-2011.
of attributes is 64, with an average of around 20 present attributes per object while the minimum number of attributes 2 and the maximum is 43. In contrast to the CUB dataset, there is notion of object parts and associated attributes, i.c. the attributes are referring to the entire object ("global attributes"). Furthermore, the combined dataset aPascal and a Yahoo contains 989 objects without any attribute associated. We pruned these cases for our experiments so that 14350 objects remain. A cropped image was created for each bounding box in the samples the dataset, yielding one image object. At training time, the following augmentations were applied to the individual object sam- ples: resizing to a standardized format (H x with probability p 0.5, random rotations in the range of +55 based on uniform probability distribution and normalization For validation and testing, only resizing and normalization were applied.
320 x 320 pixels), random horizontal flipping
We evaluated the aPascal as well as the combined aPY dataset with varying regularization parameter values applied to the explanation loss. Table 2 shows the negative log likelihood loss for the classi- fication, the mean squared error (MSE) loss for the concepts (explanation loss) and accuracy figures for the aPascal and the aPY datasets. The accuracy numbers are in narrow range. implying that the concept regularization does not seem to significantly affect the classification accuracy. On the other hand, the decrease of the concept-related MSE loss for increasing À values by approximately one order of magnitude suggests, that the labeled concepts are indeed learned by the model. The experiments were conducted with six different data-seeds and it was confirmed that the results are generalizable.
We the Albumentations library by Buslaev al. 2020
8

---

Published as a conference paper at ICLR 2022
Table Classification loss, explanation loss, accuracy on the PY test and accuracy on the aPascal test set for selected concept regularization parameter values À after 500 epochs.
NLL class
MSE concepts
Accuracy aPY [%]
Accuracy aPascal [%]
0.0
0.1475
0.2463
61,2
84.7
0.5
0.1526
0.1152
60.1
85.1
1.0
0.1530
0.0855
61.3
85.6
2.0
0.1596
0.0737
61.5
85.8
5.0
0.1734
0.0596
61.2
85,4
Evaluations on the combined aPY dataset appear to be focused on specific applications like zero-shot learning and classifying attributes in the literature, which renders this dataset difficult to baseline. Still, [Wang & Jil (2013) report 63.02% overall class prediction accuracy the aPascal dataset, while (Mahajan et al. 2011 reached an accuracy of 67 67.33% on the same dataset.
Prediction: (correct)
Prediction: (corredt) Prodiction: bird (comect)
Prediction: (wrong, monkey)
Prediction: (wrong, gt bettle)
Prediction: diningtable (wrong, is person)
3D Boxy
Occluded
Tail
Head
Occluded
2D Boxy
Window
Head
Torse
Head
Occluded
Head
Leg
Furn.
Door
Snout
Eye
Skin
Plastic
Headlight
Eye
Cloth
Metal
Furry
Loog
Shiny
Figure 6: Correctly and incorrectly classified samples and activated global (object level) concepts above the given threshold from the aPY dataset. Red labels denote attributes that are not annotated the ground truth.
Ssows samples of correctly and incorrectly classified objects and the activated concepts (i.e., above given threshold) for the aPY dataset. Notably, our model able discover meaningful and obviously correct concepts that were not annotated in the ground truth, e.g., the "wheel", "door" and "shiny" attributes that pertain the first image with the car, or the "snout" the image with the dog. The incorrectly classified objects were associated with concepts that seem, to certain extent, understandable. For example, the monkey (predicted as dog) or the bottle (predicted as person). In the latter case, we find that the classification is consistent with the concepts that were discovered, i.e., if "head", "torso", "arm", etc. are detected, it coherent that the corresponding classification "person" [ also shows that the concepts allow person to better understand how the model arrived at the misffassiiioation as well as more easily to identify it as such. We believe that the CT approach can be extended for applications in concept discovery and zero-shot learning.
5 CONCLUSIONS
In this work, generalize the notion of attention from low-level features to high-level concepts to ensure an interpretability of attention scores that agrees with a human's 's reasoning for classification tasks. We propose the Concep"Transformer, form of interpretability and show on three benchmark image datasets that the explanations obtained plausible and faithful. Our architecture achieves performance that matches the state-of-the-art techniques, while being more versatile, significantly less complex and easier to train.
novel deep learning module that accommodates this
TThis resembles the role of attention weights in the NL.P literature where they can be associated with syn- tactic elements the input sentence.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY
We provide code for the model implementation in the Appendix. Upon acceptance of the paper, we plan to release the full code for the experiments that we reported in a public repository. The datasets used for evaluation of our approach are public benchmarks, therefore already available in community.
REFERENCES
David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks. Proceedings of 32nd Conference on Neural Information Processing Systems, 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:/409.000m 2014.
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-base based logic explanations of neural networks. arXiv preprint arXiv:2106. 06804, 2021.
Branson, G. Van Horn, S. Belongie, and P. Perona. Bird species categorization using pose nor- malized deep convolutional nets. In In Proceedings of the British Machine Vision Conference, 2014.
Alexander Buslaev, Vladimir Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin. Albumentations: Fast and flexible image augmentations. Information, 11(2), 2020. URL pnttps://www.nt..
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intel- ligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1721-1730, 2015.
Diogo V. Carvalho, Eduardo M. Pereira, and Jaime S. Cardoso. Machine learning interpretability: A survey methods and metrics. In Electronics, 2019.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems, volume 32, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Ameet Deshpande and Karthik Narasimhan. Guiding attention for self-supervised learning with transformers. arXiv preprint arXlv:2010.02339, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
M. Everingham. Van Gool, C. K. Williams. J. Winn, and A. Zisserman. pascal visual object classes challenge 2008 (voc2008) results. http://www.paseht hww.wkkc//aallggsss///oo/////enn
The
Ali Farhadi, Ian Endres. Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778 1785, 2009. doi: 109/CVPR.2009
10

---

Published as a conference paper at ICLR 2022
J. Fu, H. Zheng, and T. Mei. Look closer to better: Recurrent attention convolutional neural network for fine-gre ome-grained image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4438 08884444,,00 2017.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019.
R. Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1440-1444 1448, 2015.
R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 580 -57, 2014.
Yash Goyal, Amir Feder, Uni Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:1907.07/60 2019.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. In ACM Computing Survey, volume 51(5), pp. 1-42, 2018.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
S. Huang, Z. Xu, D. Tao, and Y. Zhang. Part-stacked cnn for fineggrined visual categorization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 1173-1182, 2016.
M. Jaderberg. K. Simonyan, and A. Zisserman. Spatial transformer networks. In Advances in Neural Information Processing Systems (NIPS), pp. 2017-2025, 2015.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint arXiv: arXiv:1902.1 1902. 2019.
Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pietro Liò, and Adrian Weller. Now you see me (cme): concept-based model extraction. arXiv preprint arXiv:2010.1323:: 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 1668-2677 PMLR, 2018.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. 2019.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machine Learning, pp- 5338-5348 PMLR, 2020.
J. Krause, H. Jin, J. Yang, and L. Fei-Fei. Fine-grained recognition without part annotations. In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5546-5555, 2015.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In AIES, 2019.
Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAA! Conference Artificial Intelligence, volume 32, 2018.

---

Published as a conference paper at ICLR 2022
D. Lin, X. Shen, C. Lu, and J. Jia. Deep lac: Deep localization, alignment and classification for finegrained recognition. In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1666-1674, 2015a.
T.-Y. Lin, A. RoyChowdhur,, and S. Maji. Bilinear enn models for fine- fine-graine grained visual recognition. In In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1449- 1457, 2015b.
X. Liu, T. Xia, J. Wang. Y. Yang, F, Zhou, and Y. Lin. Fully convolutional attention networks for fine-grained recognition. In arXiv preprint 1603vv6000007660 2016.
Scout M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference on neural information processing systems, pp. 4768- 4777, 2017.
Dhruv Mahajan, Sundararajan Sellamanickam, and Vinod Nair. A joint learning framework for attribute models and object descriptions. In 2011 International Conference Computer Vision, pp. 1227-1234, 2011. 1. doi: 10.1 1..100//CCV22111112266633
A. Nguyen, Dosovitskiy, J. Yosinski, Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, In In Advances in Neural Information Processing Systems 29 (NIPS), pp. 3387-3395, 2016.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. BMVC, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-22) 215, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618 618-626, 2017.
Sofia Serrano and Noah Smith. Is attention interpretable? arXiv preprint arXiv:1906.0373:. 2019.
M. Simon and E. Rodner. Neural activation constellations: Unsupervised part model discovery with convolutional networks. In In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 114--1151, 2015.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: arXiv:1706.03825, 2017.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, volume 23(5), pp. 828- 841. 2019.
M. Sundararajan, A. Taly, and Q Yan. Axiomatic attribution for deep networks. Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference Machine Learning (ICML), pp. 1747 1756, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of 31st Conference Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2022
D. Wang, Z. Shen, J. Shao, W. Zhang, X. Xue, and Z. Zhang. Multiple granularity descriptors for finegrained categorization. In In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2399-2406, 2015.
Xiaoyang Wang and Qiang Ji. A unified probabilistic approach modeling relationships between attributes and objects. In 2013 IEEE International Conference on Computer Vision, pp. 2120- 2127, 2013.
Welinder, S. Branson, Mita, C. Wah, Schroff, S. Belongie, and Perona. Caltech-uesd birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint
1e0p:,998,.222.. 2019.
T. Xiao, Y. Xu, K. Yang. J. Zhang, Y. Peng, and Z. Zhang. The application of two-level atten- tion models deep convolutional neural network for fine-grained image classification. In II Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 842-850, 2015.
Chih-Kuan Yeh, Been Kim, Serean Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awatt concept-based explanations in deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp.
Curran Associates, Inc., 2020.
Yosinski, Clune, T. Fuchs, and H. Lipson. Understanding neural networks through deep vi- sualization. In In Deep Learning Workshop at the 32nd International Conference on Machine Learning (ICML), 2015.
H. Zhang, T. Xu, M. Elhoseiny, X. Huang, Zhang, A. Elgammal, and D. Metaxas. Spda-cnn: Unifying semantic part detection and abstraction for fine recognition. In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1143-1152, 2016.
N. Zhang, J. Donahue, R. Girshick, and T. Darrell. Part-based r-enns for fine-grained category detection, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 834- 849, 2014.
H. Zheng, J. Fu, T. Mei, and J. Luo. Learning multi-attention convolutional neural network for fine grained image recognition. In Proceedings of the IEFF International Conference on Computer Vision (ICCV), pp. 5207, 2017.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discrim- inative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921-2929, 2016.
B. Zhou, Y. Sun, D. Bau, and A. Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119-134, 2018.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX: ADDITIONAL DETAILS
PROOF OF PROPOSITION
Let us fix specific and let us denote Pr()")a)) in Equation after we externally set 7e(x) > 0 to zero with Pr(
We want calculate the difference:
exp (Be) -e L
exp
E c' (Be))))
Pr(ff)a)
Σ, exp (Be) + Ee
E, exp
Ed' (
(o + #c (Be (Be) k0)6)
exp
exp
Σ, exp (((e))r) exp (Eeto(Be)ine))
exe exp
(Eer (Bc)
0,
where the inequality comes from the assumption that the preferred output arg max, (Bc); of concept e is unique, i.e. exp((A.)== > exp((3.)), Vi * 1.
This proves that setting e
non-zero drop in Pr(i'1x).
The same type of inequality more in general proves that increasing/decreasingg 0 < 7e(x) < increases/decreases Pr()*)æ), which proves the second part of Proposition [ that (o(2) is strictly positively correlated with Pr(1*(x).
B APPENDIX: ADDITIONAL FIGURES
Original Sample
Altered Sample
Cardinal 0002000222 17189jjg
Cardinal Sendnal_0072_11111 bld_buejjog
Prediction: Cardinal
Prediction: Indigo_Buriie Bunting
Spatial explanations:
Spatial explanations:
breast partem:solid
has breast pattem:solld
identical
hass threat color:black
eye color:black
identical
eye color:black
has forchead, color:blue
changed
forcheed, color:red
has belly. patter:solid
identical
has, belly pamtm:soold
has cog_ooorryrr
identical
has, leg. color:grey
colur:bulf
identical
leg. solor:buff"
has throst color::blcck
removed
Global explanations:
Global explanations:
has shope:.perching-lke
identical
has shape:perching.llie
has sizc:ssmll [5 9 in)
idcatical
has size:small (5 in)
sizenvery small in)
changed
Figure 7:
Counterfactual intervention on an exemplary CUB sample. Artificially chang- ing the color of Cardinal from red blue, causes attention shifts from the concept red to has for.foorhed..oloo ingly, the bird that was classified as Cardigan is then being classified as Indigo Bunting.
bbuee (among others). Correspond-
14

---

Published as a conference paper at ICLR 2022
insertions (attention rank):
insertions (random):
insertions (attention rank): 42
insertions (random): 95
Figure 8: Illustration of the patch-based version of the insertion metric in Petsiuk et al. (2018). On the left patches corresponding to sample in CUB inserted one by one to the image according to the rank given by the maximal attention weight that they give to a concept until the classifier outputs the correct classification. On the right, the patches inserted in random order. Inserting according to the rank of the attention weights allows the model to recognize the bird with less insertions, indicating that the attention weights meaningfully related the decision. This can be used measure of faithfulness captured by an insertion metric.
Fraction of images with final prediction
0.8
0.6
0.4
0.2
attention weights sorting
random sorting
0
25
50
75
100
125
150
175
200
Number of inserted patches
Figure 9: Patch-based insertion metric on CUB: inserting patches following the rank given by the at- tention scores learned by ConceprTrrnsformmr image has the probability of being recognized that reaches 50% of its maximum after 34 insertion. Random insertions on the other hand require on average 60 insertions.
15

---

Published as a conference paper at ICLR 2022
C
APPENDIX: CONCEPTTRANSFORMER PYTORCH CODE
import math import
import no
import torch. nn. functional as
class Concept (nn M)
*en
aon-spatisn spatial concepts in parallel and aggregates the pro at the end
Processes spatial and non
10
11
def
init__
13
self,
embedding_diim,
15
sum_classes,
16
suiLheads,
17
sttention_dropouss
18
projection_dropout
r_conoepts==00,
20
i_spatiai__onooptts-
*azge,
22
k>kaargs
23
1:
24
init_()
25
Non-spacial concepts
27
self.n_concepu n__ennepps
concepts an.Parameter (torch.zeroc a_consepts, embedding_diam requizes_grad
29
nn. (steee__ooooolsslf...ooo
std=1.0 math. (enbedding_ddim)
30
n_concepts > 0:
self. waal.cconaet_.ranffmome Crossättention)
12
dlm-embadding_diin
n_outppte-minncleeeee
34
nun_heeds-num__eeet
attentiio__doooott
36
37
19
$ Sequence pooling for non-spatial
n_concepts celf. na, Linear (enbeciding_dinn 1)
41
43
Spatfal Concepts
spsl.ins ponccpts Leaaatial_coneepe
self.spatizl,, caattiilloonoottss Pn.
46
torch.seros 11, "_apatiallcommee enbedding_dim), requiree_grad-True
48
na. in.init tranttttruc__ormallee saee....ptiel_coocpttt. stdel.0 math. agrt (embedding_diz))
n_spattil
50
self cateiial__oooep__raaaaa
CrossAttention/
dimrenbeddi:g__iim
__outpita-nun_claen
53
num num_heads--in heads--muu_heddes
55
57
58
def (self, x):
concept_attn, apatial__conept__tte
None
60
out
62
if solf.n..n.mmmo token_attn F.softmax (self wtt.rttoooetttttee ceiieee_atttttiiieee pactronntorrooolliia 1X), dim=l) .transpos -2) x_pooled tereh maluul (Loken_atin,
sellfin_cncoptt 0:
64
65
self.n_concepts
eut_n, concept_attn spaliall stream concopt_attn concept_astn.meet out out + out_n.equiezze(") aqueeze tokan dimenston
67
con.rept_ttannooomeeeee (x_poled, self.concepts)
average over
69
70
71
if self relfin_spttiellcooeepts out_s, spacial_concept___
Spatial stream self, wel...sattaaaaacc ceatiaalconcept_nie maar.mmaanine cut out + cut_s.mean # pool tokens sequence
72
(x, self, stt...spaiel__coccppe heada
apatial_connept...
74
75
76
return out, concept_attn, spat clooaallo___tttttt
16


Reviews:
[{"review_id": "wAa3fBd_g5", "reviewer": "Reviewer_H9Z1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a fairly straightforward yet novel modification to a vanilla Transformer that introduces concepts, which the user has to provide during training time. These concepts are then attended to via cross-attention against queries, which as usual come from the input. This allows an auxiliary, explanation loss to be used, which the authors show to be useful to increase classification accuracy. For most datasets presented, the authors' architecture is relatively competitive but does not achieve state of the art results, as I would expect from a model that baked in explainability. One cool bit is that classification accuracy does not necessarily fall with the use of the auxiliary loss (table 2), as one would expect it to.\n\nA weakness of the work is that the concepts provide an explanation of perhaps why the model is wrong but what I expected was the ability to do some sort of fix of the model's attention so that it can be forced to output the right answer. As it is, I am not sure I am convinced that the concepts are causally linked to classification / misclassification performance. The authors provide a few examples where a misclassification happens and they show how the model attended to the wrong concepts, but is it possible this is a coincidence? The authors do not investigate in detail. It would be nice to see if the authors can change the attention weights in a natural way to explore this, such as for example altering the input image (ie, changing the color of certain pixels) so that certain concepts are / are no longer attended to and see how this affects classification performance.", "summary_of_the_paper": "This work presents a new, Transformer-like architecture that has explainability built into it in the form of concepts which are then cross-attended against the input. The authors present their work in a variety of domains and show how it aids model decision interpretability. ", "summary_of_the_review": "A reasonable first draft of a promising line of work but more analysis is needed to make it interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uaSbHN4Fj3", "reviewer": "Reviewer_5kGA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper explores an important problem and propose an interesting model that goes in a good direction in terms of encouraging models to not just achieve stronger numerical performance but also perform explainable reasoning along the way to justify their predictions. \n* **Writing quality**: The writing clarity of the paper is good, the idea and content are easy to follow, the motivation is clearly explained, and there are good visualizations that help understanding. \n* **Related work**: the paper misses some closely related prior works that explored the idea of reasoning over concepts for reasoning over images: one paper is “The consciousness prior” by Bengio and the other is “Learning by Abstraction: the Neural State Machine” by Hudson and Manning. The former discussed the idea mainly in the conceptual level and the latter studied it also empirically. At the same time, overall the related work section gives a good coverage of prior works about interpretability and attention.\n* **Model structure**: the model proposed is simple yet interesting and general and so can be easily incorporated into transformer-based models for variety tasks and in different domains.\n* **Attention supervision**: I really like the idea of supervising the concept attention directly when such information exists, as in the case of CUB. I think that’s a nice advantage of the proposed approach.\n* **Experiments**: The experimental section is overall extensive enough and provides good balance both over multiple datasets, some with extra supervision info (like CUBs) and some more diagnostic (like MNIST), experiments include both quantitative and qualitative results. To demonstrate the potential generality of the approach, it could be nice to explore over different domains (e.g. textual only) or tasks (e.g. VQA, or other multimodal tasks) but I feel in the context of the proposed model exploring just image classification is nevertheless alright and not too limited.\n* **Concepts identification**: one potential limitation of the approach proposed is the fact that the concepts needed to be pre-defined rather than emergent through the model, which may conversely reduce the applicability or potential benefit of the approach in all cases where such info doesn’t exist. \n", "summary_of_the_paper": "The paper proposes a new version of transformer called concept transformer with the aim of improving the interpretability of its attention by computing cross attention between the inputs features and a set of concepts. This should make the model both more explainable, in the sense that it is easier for the human to interpret the attention weights of meaningful concepts, and also more faithful, in the sense that the attention scores given to particular concepts directly impact the final prediction of the model.\n\nUpdate: Following the author response I'd like to keep my positive score reflecting my view of the paper.", "summary_of_the_review": "Overall, I think the paper proposed a nice interesting model on an important problem, motivates the goal of the model in a compelling manner, presents the model with good detail and clarity, provides a nice mix of experiments of different types and on different datasets, and can be of benefit to the research community, and therefore I recommend its acceptance and wish best of luck to the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nGLM_Qn6CR", "reviewer": "Reviewer_wR5k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality and clarity: \nThe quality of the paper is good and it has a good motivation. The targeted goal is interesting and meaningful and the authors achieved this by linearly enforcing domain knowledge to weight allocations of attention mechanism. The proposed methodology is well-described over the paper and introduced experimental settings are properly designed with three public datasets. The written\nresults seem to be reasonable.\n\nOriginality:\nAlthough interesting, It’s a bit hard to say the introduced technique is quite new, since a simple linear combination of attention weights and concepts patches (domain knowledge) in Transformer is the main contribution. Also, the experimental results are relatively weak when comparing the main model with its baselines in accuracy performance.\n\n\nQualitative and quantitative analysis of model explainability between the proposed model and its baselines can be a good validation to show the benefit of their explanation technique. The authors also need to describe how concept patches are generated and the error bound generated from the concept patches should be considered for clarity.", "summary_of_the_paper": "The authors propose a transformer-based model which can enhance explainability of a deep learning model by inducing domain knowledge into the model in a form of cross-attention mechanism. This paper interestingly focuses on addressing the relationship between post-hoc  explainability and inherent  interpretable model.  The authors address the limitation of interpretable model, which focuses on the controversy over how much a human should trust model’s explanations for its decisions.\n", "summary_of_the_review": "Although the paper is interesting, it's considered that the proposed technique is somewhat incremental. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZCshjQiHsLu", "reviewer": "Reviewer_St45", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work investigates the important field of interpretable deep models. With their technical contribution, the authors present an interesting approach to extend previous approaches, such as Bottleneck Concept Models (Koh et al.), by explicit local and global concepts. \nThe proposed approach seems to be very clearly described. Figure 1 and the illustrative example on MNIST describe the approach really well.\n\nFurthermore, distinguishing between local and global concepts by the design of the architectures is a very interesting approach. Unfortunately, the paper only rudimentary focuses on this contribution. In this regard, the findings of the present work don’t show novel insights e.g. compared to Concept Bottleneck Models by Koh et al (ICML 2020) or the neuro-symbolic, object-centric approach by Stammer et al (CVPR 2021). \n\nIn particular, the takeaway message of Table 1 is not clear to me. In the main text, the authors mention that resulting explanations from ProtoPNet \"cannot be always guaranteed to be faithful to the classifier’s decision\", whereas the authors claim that the ConceptTransformer guarantees this, which is not clear to me. Are you only referring to ProtoPNet or Concept Bottleneck models in general? Can you elaborate on this further? Applying the evaluation of Margeloiu et al. (ICLR Workshop 2021) would further strengthen the authors’ claims.\n\nThe intention of the final evaluation on aPY is also not clear to me. Especially since here, only global concepts are used. What is the benefit compared to a more „simple“ classifier (c.f Concept Bottleneck Models by Koh et al.)?\n\n\n\nReferences: \n\nKoh et al. Concept Bottleneck Models, ICML 2020 (https://arxiv.org/abs/2007.04612)\n\nStammer et al. Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations, CVPR 2021 (https://arxiv.org/abs/2011.12854)\n\nMargeloiu et al. Do Concept Bottleneck Models Learn as Intended? ICLR Workshop 2021 (https://arxiv.org/abs/2105.04289)", "summary_of_the_paper": "The paper introduces a transformer-based architecture for (deep) concept models. Similar to previous approaches in this area, the authors propose a replacement of the model’s deep classifier head. Here, the authors suggest using the attention mechanism. With the introduced ConceptTransformer module they are able to utilise global as well as local (spatial) concepts.\nAfter introducing the architecture and supervised concept learning of the model, the model’s performance is demonstrated on three benchmark datasets. Where only one makes use of local and global concepts.", "summary_of_the_review": "As it is now presented the paper seems only to be a minor technical extension of Concept Bottleneck Models and I would recommend that the authors focus more on presenting as well as evaluating the benefit of their ConceptTransformer. \nThe authors state that the explanations by their introduced approach are plausible and faithful. However, no quantitive evaluation in this regard is performed (cf. Margeloiu et al. Do Concept Bottleneck Models Learn as Intended?). The introduced ConceptTransformers seems to be a technical extension to „simple“ linear classification heads of Concept Bottleneck Models. The advantage is not clearly described and evaluated. \nTherefore, the paper seems to me to be only a minor technical contribution. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a generalization of the standard Transformer attention mechanism in which keys and queries represent abstract concepts (which must be specified a priori). This in turn yields "concept embeddings" (and logits) as intermediate network outputs, providing a sort of interpretability. Reviewers agreed that this is a simple (in a good way) and interesting approach, and may lead to follow-up work that builds on this architecture. 

Some concerns regarding the relation of this method to prior work—in particular the "Concept Bottleneck" model—were raised and addressed in discussion; the authors might incorporate additional discussion in future drafts of the work.
Decision: Accept (Poster)