A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
Authors: DIPANJYOTI PAUL
Arpita Chowdhury
Xinqi Xiong
Feng-Ju Chang
David Edward Carlyn
Samuel Stevens
Kaiya L Provost
Anuj Karpatne
Bryan Carstens
Daniel Rubenstein
Charles Stewart
Tanya Berger-Wolf
Yu Su
Wei-Lun Chao
Venue: ICLR 2024 poster (2024)
Keywords: Explainability
Interpretability
Transformer
Fine-grained recognition
Attribute discovery
Primary area: visualization or interpretation of learned representations
TL;DR: Transformer based Interpretable Image recognition where each query in the decoder will learn class specific features.

Abstract: We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incorporate class information to make predictions, we investigate a proactive approach, asking each class to search for itself in an image. We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR). We learn ''class-specific'' queries (one for each class) as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. We name our approach INterpretable TRansformer (INTR), which is fairly easy to implement and exhibits several compelling properties. We show that INTR intrinsically encourages each class to attend distinctively; the cross-attention weights thus provide a faithful interpretation of the prediction. Interestingly, via ''multi-head'' cross-attention, INTR could identify different ''attributes'' of a class, making it particularly suitable for fine-grained classification and analysis, which we demonstrate on eight datasets. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site: https://github.com/Imageomics/INTR.

Published as a conference paper at ICLR 2024
A SIMPLE INTERPRETABLE TRANSFORMER FOR FINE- GRAINED IMAGE CLASSIFICATION AND ANALYSIS
Dipanjyoti Paul'
Arpita Chowdhury' Xinqi Xiong'
Feng-Ju Chang
David Carlyn'
Samuel Stevens'
Kaiya L. Provost' Anuj Karpatne"
Bryan Carstens'
Daniel Rubenstein"
Charles Stewart
Tanya Berger-Wool Wolf'
Yu Su
Wei-Lan Chao
The Ohio State University
2 =Amazon Alexa
"Virginia Tech
Princeton University
*Rensselaer Polytechnic Institute
Painted Bunting!! Do you see yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Answer
Yes
Yes
Yes
No
Figure 1: Illustration of INTR. We show four images (row-wise) of the same bird species Painted Bunting and the eight-head cross-altention maps columm--ise triggered by the query of ground-truth class. Each head learned to attend to a different (across columns) but consistent (across rows) semantic cue in the image that useful to recognize this bird species (c.g., attributes). The exception is the last row, which shows inconsistent attention. Indeod, this is misclassified case, showcasing how INTR interprets (wrong) predictions.
ABSTRACT
We present novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incor- porate class information to make predictions, asking each class to search for itself an image. We realize this idea via Trans- former encoder-decoder inspired by DEtection TRansformer (DETR). We learn "class-specific" queries (one for each class) as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. We name our approach INteppretable TRansforner (INTR), which is fairly easy to implement and exhibits several compelling properties. We show that INTR intrinsically encourages each class to attend distinctively; the cross-attention weights thus provide a faithful interpretation of the prediction. Interestingly, via "multi-head" cross-attention, INTR could identify different "attributes" of a class, making it particularly suitable for fine-gra mne-grained classification and analysis, which we demonstrate on eight datasets. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site: /ttps: com/ Imageomi cs cs/INTR
investigate proactive approach,
INTRODUCTION
Mainstream neural networks for image classification (He et al. [0166] SSmonyyn & Zisserman 2015
Krizhevsky et al., 2017) Huang et al.) 2019 (Szegedy et al. 2015)] Dosovitskiy et al. 2021 Liu et al. 2021) typically allocate most of their model capacity extract "class-agnostic" feature vectors from

---

Published as a conference paper at ICLR 2024
images, followed by fully connected layer that compares image feature vectors with "class-specific" vectors to make predictions. While these models have achieved groundbreaking aceuracy, their model design cannot directly explain where model looks for predicting particular class.
In this paper, we investigate a proactive approach to classification, asking each class to look for itself in an image. We hypothesize that this "class-specific" search process would reveal where the model looks, offering a built-in interpretation of the prediction.
At first glance, implementing this idea may need significant model architecture design and complex training process. However, we show that a novel usage of the Transformer encoder-decoder (Vaswani al., 2017) inspired by DEtection TRansformer (DETR) (Carion et al. 2020) can essentially realize this idea, making our model fairly easy to reproduce and extend.
Concretely, the DETR encoder extracts patch-wise features from the image, and the decoder attends to them based on learnable queries. We propose to learn "class-specific" queries (one for each class) as imput to the decoder, enabling the model to obtain celass-ppecffic" image features via self-attention and cross- -ooss-attetionn determining the patterns necessary to distinguish between classes; cross-attention then allows each class to look for the distinctive patterns in the image. The resulting "class-specific" image feature vectors (one for each class) are then compared with a shared "class-agnostic" vector to predict the label of the image. We name our model INterpetablle TRansformer (INTR). Figure 2 illustrates the model architecture. In the training phase, we learn INTR by minimizing the cross-entropy loss. In the inference phase, INTR allows to visualize the attenttntnnnonn maps triggered by different "class-specific" queries to understand why the model predicts or does not predict a particular class.
seff-attention encodes the contextual information among candidate classes,
On the surface, INTR may fall into the debate of whether attention is interpretable (Jain & Wallace 2019| Wiegreffe & Pinter 2019; Bibal et al. 2022). However, we mathematically show that INTR offers faithful attention distinguish between classes. In short, INTR computes logits by performing inner products between class-specific feature vectors and the shared class-agnostic vector. To classify an image correctly, the ground -oound-truth class must obtain distinctive class-specific image features to claim the highest logit against other classes, which is possible only through distinct gross-attention weights. Minimizing the training loss thus encourages each class-specific query to produce distinct cross- auss-atttntion weights. Manipulating the gross-attentin weights in inference, as done in adversarial attacks to attention- based interpretation (Serrano & Smith, 2019). would alter the prediction notably.
We extensively analyze INTR, especially in cross-attention. We find that the "multiple heads" in cross-attention could learn to identify different "attributes" of a class and consistently localize them in images, making INTR particularly well-suited for fine-grained classification. We validate this on multiple datasets, including CUB-200-2011 (Wah et al. (21T)), Birds-525 (Piosenkal 2023). Oxford Pet (Parkhi et al. (012)). Stanford Dogs (Khosla et al. 2011). Stanford Cars (Krause al., 2013),
FGVC-Aireraft (Maji et al.] (2013). iNaturalist- 2021 (Van Horn et al., [2021), and Cambridge butterfly
Moottjo-KKovaeeich Kovacevich et al. [2220). Interestingly, by concentrating the decoder's 's input visually
similar classes (e.g.. the mimicry in butterflies), INTR could attend to the nuances of patterns, even
matching those found by biologists, suggesting its potential benefits to scientific discovery.
It is worth reiterating that INTR is built upon a widely-used Transformer encoder- architecture and can be easily trained end-to-end. What makes it interpretable is the novel usage incorporating class-spectts information at the decoder's 's input rather than output. We view these as key strengths and contributions. They make INTR easily applicable, reproducible, and extendable.
2
BACKGROUND AND RELATED WORK
2.1 WHAT KIND OF INTERPRETATIONS ARE WE LOOKING FOR?
As surveyed in (Zhang & Zhu 2018 Burkart & Huber 2021 Carvalho et al. 2019 Das & Rad 2020 202
Buhrmester et 202 Linardatos et al. 2020). various ways exist to explain or interpret a model's prediction (see) Appendix A for more details), Among them, the most popular is localizing where the model looks for predicting a particular class. We follow this notion and focus on -raigra classification (e.g., bird and butterfly species). That is, not only do want to localize the coarse- grained objects (c.g., birds and butterflies), but we also want to identify the "attributes" (c.g.. wing patterns) that useful to distinguish between fine-grained classes. We note that attribute can
2

---

Published as a conference paper at ICLR 2024
be decomposed into "object part" (e.g., head, tail, wing, etc.) and "property" (e.g.. patterns the wings), in which the former is commonly shared across all classes (Wah et al. 2011). We thus expect that our approach could identify the differences within part between classes, not just localize parts.
2.2 BACKGROUND AND NOTATION
We denote an image and its ground-truti label by I and y, respectively. To perform classification over classes, mainstream neural networks learn feature extractor fe obtain a feature map X = (T( E Dx Here, O denotes the parameters; 1) denotes the number of channels: H and W denote the number of grids in the height and width dimensions. For instance, ResNet (He) ct al. 2016) realizes fe by convolutional neural network (ConvNet) with residual links; Vision Transformer (ViT) (Dosovitskiy et al. 2021) realizes it by Transformer encoder. Normally, this feature map is reshaped and/or pooled into a feature vector denoted by æ = Vect which then undergoes inner products with C' class-specific vectors {we The class with the largest inner product is outputted as the predicted label,
T
arg ca W
2.3 RELATED WORK ON POST-HOC EXPLANATION AND DELF-INTERPRETABL METHODS
Since this classification process does not explicitly localize where the model looks to make predictions, the model is often considered a black box. To explain the prediction, post-hoe mechanism needed (Ribeiro et al. 2016) Koh & Liang] 2017] Yuan et al.] [ et al.. 2022] Zhou et al. 2015). For instance, CAM (Zhou et al. 2016) and Grad-CAM (Selvaraju et al. (011] obtain class activation maps (CAM) by paokpproggg c class-specific gradients to the feature map. RISE (Petsiuk et al. 2018)
iteratively masks out image contents to identify essential regions for classification. These methods have been widely used. However, they are often low-resolution (e.g., blurred or indistinguishable across classes), computation-heavy. and not necessarily aligned with how models make predictions.
To address these drawbacks, another branch of work designs models with interpretable prediction processes, incorporating explicit mechanisms that allow for a direct understanding of the predictions (Wang al. 2021 Donnelly al. 2022 Rigotti et al. 2021 Kim et al. 2022: Bau 2017 Zhou et al. 2018). For example, ProtoPNet (Chen el al. 2019) compares the feature map X to "learnable prototypes" of each class, resulting in Teature vector æ whose elements are semantically meaningful: the d-th dimension corresponds to prototypical part of a certain class and [d] indicates its activation in the image. By reading æ and visualizing the activated prototypes, one could better understand the model's 's decision. Inspired by ProtoPNet, ProtoTree (Nauta et al. 202T) arranges the comparison to prototypes in a tree structure to mimic human reasoning; ProtoPFormer (Xue et al., 2022) presents a Transformer- basedd realization of ProtoPNet, which was originally based on ConvNets. Along with these interpretable decision processes, however, come specifically tailored architecture designs and increased complexity of the training process, often making them hard to reproduce, adapt, extend. For instance, ProtoPNet requires a multi-stage training strategy, each stage taking care of a portion of the learnable parameters including the prototypes.
INTERPRETABLE TRANSFORMER (INTR)
3.1 MOTIVATION AND BIG PICTURE
Taking into account the pros and cons of the above two paradigms, we ask, Can we obtain inter- pretability via standard neural network architectures and standard learning algorithms?
To respond to "interpretabilit"" we investigate proactive approach to classification, asking each class search for its presence and distinctive patterns in an image. Denote by S the set of candidate classes; we propose new classification rule,
(2) where 96 (f( c, S) represents the image feature vector extracted specifically for class in the context of S, and w denotes a binary classifier determining whether class c is present in the image I. Compared to Eqqation 1. the new classification rule in EEqutton 2 incorporates class-specific
= arg max.elc]
w T (f)
3

---

Published as a conference paper at ICLR 2024
information in the feature extraction stage, not in the final fully connected layer. As will be shown in subsection 3.4 this design is the key to generating faithful attention for interpretation.
To respond to "standard neural network architectures",
find that the Transformer encoder-decoder (Vaswani et al., 2017). which is widely used in object detection (Carion et al. 2020 Zhu ct al. 202T)
and natural language processing (Wolf et al., (020), could essentially realize Equation 2 Specifically,
the encoder extracts the image feature map X (T) For the decoder, we propose to learn C J==1
class-specific queries (=(]) input, enabling it to extract the feature vector gø (J)((T), =m (c) 8) for class e via cross-attention.
To ease the description, let us first focus cross-attention, the key building block in Transformer decoders in inlsussection 3.2 We then introduce our full model in Subsection 3.3
3.2 INTERPRETABLE CLASSIFICATION VIA CROSS-ATTENTION
Cross-attention. Cross-attention can be seen as a (soft) retrieval process. Given an input query vector Zm E R. it finds similar vectors from vector pool and combines them via weighted average. In our application, this pool corresponds to the feature map X. Without loss generality, let us reshape the feature map X D to X ,1,-.. NN BDXN That is, X contains N = H x W feature vectors representing each spatial grid in an image; each vector E, is D-dimensional.
With ii and X, cross vross-attention performs the following sequence of operations. First, it projects Zin and X to a common embedding space such that they can be compared, and separately projects X to another space to emphasize the information to be combined,
q W Zin E RR,, K = W E R PXN, V= W.X E DxN
Then, it performs inner product between q and K, followed by Softmax, compute the similarities between žin and vectors in X, and uses the similarities as weights to combine vectors in V lincarly,
Zout
x Softmax
KT q VD
JER
where vD is a scaling factor based on the dimensionality of features. In other words, the output of cross-attention is a vector Zon that aggregates information in X according to the input query Zin-
Class-specific queries. Inspired by the inner workings of cross-attention, we propose to learn C (c)
"class-specific" query vectors Zin = (1) ..... = E DDx one for each class, We expect each of these queries to look for the "class-specific" distinctive patterns in X. The output vectors (1) (C) E RDC thus should encode whether each class finds itself in the image,
Zom oul
Puul
KTS Q
Zout x Softmax(
E PxC.
where Q = Wg E R DxC
(5)
VD
We note that the Softmax is taken over elements of each column; i.e., in Equation 5) each column in Zm attends to X independently. We use superscript/subseript index columns Z/X.
Classification rule. We compare each vector in Zou to learnable "presence" vector w E RR" to determine whether each class is found the image. The predicted class is thus
9 = arg /
w
Z
(6)
Training. As each class obtains a logit
(c) out we employ the cross-entropy loss,
((I = - log exp(w
Z
(7)
Σn exp (w
coupled with stochastic gradient descent (SGD) to optimize the learnable parameters, including Zin. w, and the projection matrices WG. Wt. and W, in Equation 3/ This design responds to the final
piece of question in insubeection 3.1 "standard learning algorithms".
Inference and interpretation. We follow Equation 6 to make predictions. Meanwhile, each column of the cross-attentin weights Softmax( K Q in Equation 5 reveals where each class looks to find VD
4

---

Published as a conference paper at ICLR 2024
itself, enabling to understand why the model predicts does predict class. We note that this built-in interpretation does not incur additional computation costs like post-hoc explanation.
Multi-head attention. It is worth noting that standard cross-attention block has multiple heads. learns multiple sets of matrices (Wa.ri Wt. W in Equation 3) E {1, ... R), to look for different patterns in X, resulting in multiple Softmax( K, Q.) and Zou, in Equation 5 This enables vTi
the model to identify different "attributes" of a class and allows us to visualize them.
In training and inference, 1Z ) are concatenated row-wise, followed by another learnable matrix W. to obtain a single Zon as in Equution 5)
Zout
W. WolZout. 1.
Zout. R E R..
As such, Equation and Equation are still applicable to optimize the model and make predictions.
3.3 OVERALL MODEL ARCHITECTURE (SEE FIGURE 2 FOR AN ILLUSTRATION)
We implement our full INterpetablle TRansformer (INTR) model (cf. Equation 2) using Transformer decoder (Vaswani et al. 2017) on top of feature extractor fe that produces a feature map X. Without loss of generality, we use the DEtection TRansformer (DETR) (Carion et al.) (020) as the backbone. DETR uses a Transformer decoder of multiple layers; each contains a cross- -atos-tttnttion block. The output vectors of one layer become the input vectors of the next layer. In DETR, the input to the decoder (at its first layer) is set of object proposal queries, and we replace it with our learnable
"class-specific"" query vectors Zi (x ..... = ( E PxC. The Transformer decoder then
outputs the "class-specific" feature vectors Zou that will be fed into Equation 6
Using Transformer decoder rather than single cross-attention block has several advantages. First, with multiple decoder lay- ers, the learned queries Zi can improve over layers by grounding themselves on the image. Second, the self-attention block in each de- coder layer allows class-specific queries to exchange information to encode the context. (See Ap- pendix C for details.) As shown in Figure 15] the cross-atie auts--tttetiinn blocks later layers can attend to more distinctive patterns.
Decoder
Mult-head Attantion
Encodor
Selti Attention
Logits
Input Image
Class Specilic Quenes
Figure 2: Model architecture of INTR. See subsectin 3.2 for details.
Training. INTR has three sets of learnable parameters: a) the parameters in the DETR backbone, including fo: b) the class-specific input queries Zi E IR DxC to the decoder; and c) the class- vector w. We train all these parameters end-to-end via SGD, using the loss in Equation
3.4 How DOES INTR LEARN TO PRODUCE INTERPRETABLE CROSS-ATTENTION WEIGHTS?
We analyze how INTR offers interpretability. For brevity, we focus on the model in subsection 3.2
Attention vs. interpretation. There has been an ongoing debate on whether attention offers faithful interpretation (Wiegreffe & Pinter 2019 Jain & Wallace, 2019 Serrano & Smith 2019 Bibal et al.) 2022). Specifically, [Serrano & Smith 2019) showed that sigmificantly manipulating the attention weights inference time does not necessarily change the model's prediction. Here, we provide mathematical explanation for why INTR may not suffer from the same problem. The key is in our
classification rule. In Equation 6 we obtain the logit for class by w T z If e is the ground -truh label, it must obtain a logit larger than other classes c' * to make a correct prediction. This implies
(e') Zoul aull which is possible only the cross-attention weights triggered by [(e) are different from
those triggered by other class- spassfs queries 2/ (cf. Equation 4 and Equation for how (c) is constructed). Minimizing the training loss in [Equation 7 thus would Torce each learnable query vector
(c) Ve [C], be distinctive and able attend class-specific patterns in the input image.
5

---

Published as a conference paper at ICLR 2024
Unveiling the inner workings. We dig deeper to understand what INTR learns. For class obtain a high logit in Equation 6. 2 must have a large inner product with the class-agnostic un. We note (c) that Hout is a linear combination of V = [01.... , uNll E IR DxN (ef. Equation 4), and V is obtained
by applying a projection matrix W, the feature map X [-],-... FNl E I tx [qquation 3).
Let (c) Wa (c) and let ( Softmax( K" K"g(0) (0))
R. the logit w = can be rewritten as
VD
exp K (c) = w T Vala (r) x(w T 01,-.. w eNj w
(m))
x|w T W, .. w WotN]eeyy exp (Wix)) g(0)
x(81 sN exp((((() (c) WLF1, o W Σ
x a [m].
where Sn = = W, and (000m) x exp(g(e) (c) Wkrn). We note that does not depend on the class-specific query =
It only depends on the input image I, or more specifically, the feature map X and how it aligns with the vector w. In other words, we can view Sn as an "image-specific" salient
score for patch n. In contrast, (<x) [n]] depends on the class-specific query zin its value will be high if class e finds the distinctive patterns in patch n..
Building on this insight and Equation 9 if class is the ground-truth class, what its query Z needs to do is putting its attention weights (c) on those high-score patches. Namely, class e must find its distinctive patterns in the salient image regions. Putting things together, we can view the roles of W, and W2 as "disentanglement"" They disentangle the information in æn into "image-specific" and "classification-seecii
components - the former highlights "whether patch should be looked at"; the latter highlights "what distinctive patterns it contains". When multi-head cross-attention is used, each pair of (W., Wi) can learn to highlight an object "part" and the distinctive "property" in it. These offer the opportunity to localize the "attributes" of a class. See Appendix for more details.
3.5 COMPARISON TO CLOSELY RELATED WORK
ProtoPNet (Chen et al. 2019) and Concept Transformers (CT) (Rigotti et al., 2021). INTR is fundamentally different in two aspects. First, both methods aim to represent image patches by a set of learnable vectors (c.g.. prototypes in ProtoPNet; concepts in CT The resulting features for image patches are then pooled into a vector x and undergo a fully connected layer for classification. In other words, their classification rules still follow [Equation 1 In contrast, INTR extracts class- specific features from the image (one per class) and uses a new classification rule to make predictions (cf. Equation 6). Second, both methods require specifically designed training strategies or signals. For example, CT needs human annotations to learn the concepts. In contrast, INTR is based on a standard model architecture and training algorithm and requires no additional human supervision.
DINO-y1 (Caron et al. 2021). DINO-v1 shows that the "[CLS]" token of a pre-trained ViT (Doso- vitskly et al. (02T) can attend to different "parts" of objects via multi-head attention. While this shares some similarities with our findings in INTR, what INTR attends to are "attributes" that can be used to distinguish between fine-grained classes, not just "parts" that are shared among classes.
4 EXPERIMENTS
Table 1: Dataset statistics. (# images are rounded.)
Bird CUB BF Fish
Dog
Pet Car Craft
Dataset. We consider eight fine-gras -raiirai datasets from various domains, including Birds-525 (Bird) (Piosenka| 2023). CUB- 200-2011 1 Birds (CUB) (Wah et al. 2011).
# Train Img.
85K
6K
5K
45K
12K
4K
8K
3K
# Test Img.
3K
6K
1K
2K
9K
1K
8K
3K
# Classes
525
200
65
183
120
37
196
100
Cambridge butterfly (BF) (Montejo- Kovaijooi-oaevvchhhhe et al. 22200). iNaturalist- Satpellitttt2221--- 2021-Fishh (Fish) (Van Homn
ct al. 2021), Stanford Dogs (Dog) (Khosla et | 2011). Stanford Cars (Car) (Krause et 2013)). Oxford Pet (Pet) (Parkhi et al. 2012). and FGVC Aircraft (Craft) (Maji et al. 2013). We create the
BF dataset by considering the species-level labels from (Montejo- Kovacevich et al. 2020). For the Fish dataset, we extract species from the taxonomical Class named Animalia Chordata Actinopterygii
in iNaturalist-2021. Table provides the dataset statistics. See Appendix D for additional details.
'Eve though CT applies cross-attention, it uses image patches as queries to attend to the concept embeddings; the outputs of cross-attention thus fcatures for image patches.

---

Published as a conference paper at ICLR 2024
Model. We implement INTR on top of the DETR backbone (Carion al. 2020). DETR stacks Transformer encoder on top of a ResNet as the feature extractor. We use its DETR Re--ee version.
in which the ResNet-50 (He et al. 2016) was pre-trained on ImageNet-IK, IK (Russakvssyy et al. 2015
Deng et al. 2009) and the whole model including the Transformer encoder-decoder (Vaswani et al. 2017) was further trained on MSCOCO (Lin et al. 2014 We remove its prediction heads located on top of the decoder and add our class-agnostic vector w we remove its object proposal queries and add our C' learnable class-specific queries (e.g., for CUB, C' 200). See Figure 2 for an illustration and (sussection 3.3 for more details. We further remove the positional encoding that was injected into the cross-attention keys in the DETR decoder: we find this information adversely restricts our queries to look at particular grid locations and leads to artifacts. We note that DETR sets its feature map size D x H x W (at the encoder output) as 256 x Ho x Wa, where Ho and Wo are the height and width resolutions of the input image. For example, a typical CUB image is of a resolution roughly 800 x 1200; thus, the resolution of the feature map and cross- rross-attention map is roughly 25 x 38. We investigate other encoders and the number of attention heads and decoder layers in Appenddx F]
32 32
Visualization. We visualize the last (i.e., sixth) decoder layer, whose cross rross-attention block has eight heads. We superimpose the cross-attention weight (maps) on the input images.
Training detail. The hyper-parameter details such as epochs, learning rate, and batch size for training INTR are reported in Appendix F] We use the Adam optimizer (Kingma & Ba (014) with its default hyper-parameters. We train INTR using the StepLR scheduler with learning rate drop at 80 epochs. The rest of the haper-parameters follow DETR.
Baseline. We consider two sets of baseline methods. First, we use ResNet-50 (He et al. 2016) pre-trained on ImageNet- 1 and fine-tune it on each dataset. We then use Grad-CAM (Selvaraju et al. (017)) and RISE Petsiuk et al. (2018) to construct post-hoe saliency maps: the results are kept in Appendix F. Second, we compare to models designed for interpretability, such as ProtoPNet (Chen) al.] 2019), Proto (Nauta et al. (02T), , and ProtoPFormer (Xue al. 2022). We understand that these are by no means a comprehensive set of existing works. Our purpose in including them is to treat them references for what kind of interpretability INTR can offer with its simple design.
Evaluation. We reiterate that achieving a high classification accuracy is not the goal of this paper. The goal is demonstrate the interpretability. We thus focus our evaluation on qualitative results.
4.1 EXPERIMENTAL. RESULTS
Table 2: Accuracy(%) comparison.
Model
Bird
CUB
BF
Fish
Dog
Pct
Car
Craft
Accuracy comparison. It is crucial to em- phasize that the primary objective of INTR is to promote interpretability, not to claim high accuracy. Nevertheless, we report in Table 2 the classification accuracy of INTR and ResNet-50 on all eight datasets. INTR obtains comparable accuracy on most of the datasets except for CUB (12% worse) and Fish (9.2% better). We note that both CUB and Bird datasets focus on fine-grained bird species. The main difference is that the Bird dataset offers higher-quality images (e.g.. cropped to focus on objects). INTR's 's accuracy drop on CUB thus more likely results from its inability to handle images with complex backgrounds or small objects, not its inability to recognize bird species.
ResNet INTR
98.5
83.8
95.6
71.9
77.1
89.5
80.9 86.8
89.3
97.4
71.8
95.0
81.1
72.5
90.4
76.1
Comparison to interpretable models. We compare INTR to PNot Proto'Tree, and ProtoPFormer (Figure 3). For the compared methods, we show the responses of the top three prototypes (sorted by their activations in the image) of the ground-truth class. For INTR, we show the top three cross-attention maps (sorted by the peak un-normalized attention weight in the map) triggered by the ground-ur -true class. INTR can identify distinctive attributes similarly to the other methods. In particular, INTR capable of localizing tiny attributes (like patterns of beaks and eyes): unlike the other methods, INTR does not need to pre-define the patch size of prototype or attribute.
Tert
BUTR
ProtsPble
ProisPfomerr PrsteTww
Pank
Figure 3: Comparison to interpretable models. We show the responses of the top three cross- attention heads or prototypes (row-wise) of each method (column-wise) in Painted Bunting image.
Pleasee see subsection 4.2 for discussion on concerns about data leakage and unfair comparison.
7

---

Published as a conference paper at ICLR 2024
Dansor KChoe
kope Image
Vixel Demonetatioo
Datanel (Tlsn)
Inser Image
Vousa Demonstation
CLB Fietirore
(
Bid Blu Doons]
Alcur
Car
malioerlouus
12emmnn Shorthained
Ca
Rah
(Demsefthh/ Mecsenccttty
Zovalck)
Figure 4: INTR on all eight datasets. We show the top four cross cross-attention maps per test example triggered by the ground-truth classes (based on the peak un-normalized attention weights in the maps). As the indices of the top maps may not be the same across test examples, the attributes may not be the same in each column.
Red-winged Blackbärd!! you see yourself?
Reference
Head-1
Head-2
Head-3
Head-4
Head-5
Head-8
Head-7
Head-8
Answar
vinged
Flvddhii
No Crow)
American Orew
you yourself?
Reforonca
Head-1
Hoad-3
Head-4
Head-5
Head-6
Head-7
Head-8
No (Bottinore Onole)
O-chad 090e
Yes
Figure 5: INTR can identify tiny image manipulations that distinguish between classes. On the top. we remove the red spots of the Red-winged Blackbird. After that, INTR cannot correctly classify the image the parentheses in the Answer column highlight the predicted classes. On the bottom, we change the color of the bird's belly (Baltimore Oriole) to make look like Orchard Oriole, After that, INTR would misclassify as Orchard Oriole. Both results demonstrate INTR's 's sensitivity to visual attributes.
4.2 FURTHER ANALYSIS AND DISCUSSION ABOUT INTR
INTR can consistently identify attributes. We first analyze whether different cross-attention heads identify different attributes of class and if those attributes are consistent across images of the same class. Figure shsows a result (please see the caption for details). Different columns correspond to different heads, and see that each captures distinct attribute that consistent across images. Some of them are very fine-gra grained, such as Head-4 (tail pattern) and Head-5 (breast color). The reader may notice the less concentrated attention in the last row. Indeed, is misclassified case: query of the ground-truth class (i.e., Painted Bunting) cannot find itself in the image. This showcases how INTR interprets incorrect predictions. We show more results in Appendix G
INTR is applicable to variety of domains. Figure 4 shows the cross-attention results all eight datasets. (See the caption for details.) INTR can identify the attributes well in all of them, demonstrating its remarkable generalizability and applicability.
INTR offers meaningful interpretation about attribute manipulation. We investigate INTR's response image manipulation by deleting (the first block of Figure 5) and adding (the second block of Figure 5) important attributes. We obtain human-identiffe attributes of Red-winged Blackbird (the first block) and Orchard Oriole (the second block) from (Cor) and manipulate them accordingly.
8

---

Published as a conference paper at ICLR 2024
Heliconius melpomene, compared with all query
Heliconius melpomene, only compared with Heliconius elevatus
Figure 6: INTR can identify fine-grained attributes to differentiate visually similar classes. The test image (first column) is Heliconius melpomene. We show the cross-attention maps triggered by the ground-truth class when compared to all other classes (first row) and the visually similar Heliconius elevatus (second row). As shown in the sccond row, limiting the input queries visually similar classes enables INTR to identify the nuances of patterns, even matching those found by biologists. Specifically, at the bottom, we show the image of Hcliconius melpomene (blue box) and Heliconius clevatus (green box) and where biologists localize the attributes (purple arrows). The bottom images are taken from (Hel).
As shown in Figure 5| INTR is sensitive to the attribute changes; the cross attention maps change drastically at the manipulated parts. These results suggest that INTR's 's inner working is heavily dependent on attributes make correct classifications.
INTR can attend differently based on the context. As mentioned in (ubsection 3.3] the self- attention block in INTR's decoder could encode the context of candidate classes determine the patterns necessary to distinguish between them. When all the class-specific queries (e.g.. 65 classes in the BF dataset) are inputted the decoder, INTR needs identify sufficient patterns (e.g.. both coarse-graine goarse-grainddd and fine- (ne-grained) to distinguish between all of them. Here, we investigate whether limiting the input queries to visually similar ones would encourage the model to attend finer-grained attributes. We focus on the BF dataset and compare two species, Heliconius melpomene (blue box in Fngiurre 6) and Heliconius elevatus (green box in Figure 6), whose visual difference is very subtle. We limit the input queries by setting other queries zero vectors. As shown in Figure 6 this modification does allow INTR to localize nuances of patterns between the two classes.
Concerns regarding an MSCOCO-pre- tranndd backbone. We understand this may cause concern about data leakage and unfair comparison. We note that MSCOCO only offers bounding boxes for objects, not for parts, and it does not contain fine- grained labels. Regarding fair comparisons, our work is not to claim higher accuracy but to offer a new perspective. We use DETR to demonstrate that our idea be easily compatible with pre-trained encoder-decoder (foundation) models.
Limitations. INTR learns C' class-specific queries that must be inputted to the Transformer decoder jointly. This could increase the training and inference time if C is huge, e.g., larger than the number of grids N in the feature map. Fortunately, fine-grained classification (e.g.. for species in the same family or order) usually focuses on a small set of visually similar categories; C' is usually not large.
CONCLUSION
We present Interpretable Transformer (INTR), simple yet effective interpretable classifier building upon standard Transformer encoder-decoder architectures. INTR makes merely two changes: learn- ing class-specific queries (one for each class) as input to the decoder and learning class-agnostic vector on top of the decoder output to determine whether a class is present in the image. As such, INTR can be easily trained end-to-end. During inference, the cross-attention weights triggered by the winning class-specific query indicate where the model looks to make the prediction. We conduct extensive experiments and analyses to demonstrate the effectiveness of INTR in interpretation. Specifically, we show that INTR can localize not only object parts like bird heads but also attributes (like patterns around eyes) that distinguish one bird species from others. In addition, we present a mathematical explanation of why INTR can learn to produce interpretable cross-attention for each class without ad-hoc model design, complex training strategies, and auxiliary supervision. We hope that our study can offer new way of thinking about interpretable machine learning.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This research is supported in part by grants from the National Science Foundation (IIS-2107077 and OAC-2118240), We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center. We thank Lisa Wu (OSU) for fruitful discussion on datasets.
REFERENCES
Birds of the world: httpp://bbeeottheeooll...//////p
The cornell lab of httpl/ooyy.ttgee///....aaa..ooo.
La variété des http://www.....///.....o.o////aa
html
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretahiliity of deep visual representations. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 6541-6549, 2017.
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang. Thomas François, and Patrick Watrin. Is attention explanation? introduction to the debate. In Proceedings of the 60th annual meeting of the association for computational linguistics, pp. 3889-3900, 2022.
Vanessa Buhrmester, David Münch, and Michael Arens. Analysis of explainers black hox deep neural networks for computer vision: A survey, Machine Learning and Knowledge Extraction, 3(4):966 989, 2021.
Nadia Burkart and Marco F Huber. A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245-317, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Diogo V Carvalho, Fduardo M Percira, and Jaime S Cardoso. Machine learning interpretahility: A survey on methods and metrics. Electronics, 888):332, 2019.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. Advances neural information processing systems, 32, 2019.
Arun Das and Paul Rad. Opportunities and challenges in explainable artificial intelligence (xai): A survey, arXiv preprint ar0iv:2006. / 1371, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, 2009.
Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 00265-10275, 2022.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In International conference on learning representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Geoff Plciss, Laurens Van Der Maaten, and Kilian Q Weinberger. Convolutional networks with dense connectivity. IEEE transactions on pattern analysis and machine intelligence, 44(12): 8704-8716, 2019.
Sarthak Jain and Byron C. Wallace. Attention is not explanation. In Proceedings of the 2019 annual conference of the north american chapter of the association for computational linguisties, 2019.
10

---

Published as a conference paper at ICLR 2024
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- fine-grained image categorization: Stanford dogs. In Proceedings CVPR workshop on line-grained visual categorization (FGVC), 2011.
Sangwon Kim, Jaeyeal Nam, and Byoung Chul Ko. Vit-net: Interpretable vision transformers with neural tree decoder. International conference on machine learning, pp. 11162-11172, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69980 2014.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning. pp.
2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained catego- rization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Ilya Sutskever. and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision. pp. 1500-55,000 2014.
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1):18, 2020.
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022 2021.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschka, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5133 2013.
Gabriela Montejo-Kove Kovacevich, Eva van der Heijden, Nicola Nadeau, and Chris Jiggins. Cambridge butterfly wing collection batch 10, 2020. http::tts////ttt org/10 5281/ /tt////cdcc.o//ooo/oo//oooooocc/ 4289223
Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- fine-grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 14933-149453 2021.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 0508-305,5000 2012.
Roger Tory Peterson. A field guide to the birds: eastern and central North America. Houghton Mifflin Harcourt, 1999.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421/ 2018.
Gerald Piosenka. Birds 525 species- image classification. 05 2023. http:httpt:/wwtt www.wwwwwwl...om com/ Satasetss/picceenkee
bi birdd species
Yao Qiang, Deng Pan, Chengyin Li, Xin Li, Rhongho Jang, and Dongxiao 7hu. Attcat:
Explaining transformers via attentive class activation tokens. In Advances in neural information processing systems, 2022.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 144, 2016.
Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention-based interpretability with concept transformers. In International conference learning representations, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Adilya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journaa of computer vision, 115:211-252 2015.
Ramprasaath R Selvaraju. Michael Cogswell. Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEFE international conference computer vision, pp. 618-626, 2017.
11

---

Published as a conference paper at ICLR 2024
Solia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the annual meeting of the association for computational linguistics, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International conference on learning representations. 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357, 2021.
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmark ing representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12884-1222 12933, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Catherine Wah. Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200- 2011 dataset. 2011.
Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF international conference computer vision, pp. 895-904, 2021.
Sarah Wiegreffe and Yuval Pinter. Attention is not explanation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Menggi Xue, Qihan Huang, Haofei Zhang, Lechao Cheng, Jie Song, Minghui Wu, and Mingli Song. Protop- former: Concentrating prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208. 10431, 2022.
Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, and Dejing Dou. Explaining information flow inside vision transformers using markov chain. In eXplainable Al approaches for debugging and diagnosis., 2021.
Quan-shi Zhang and Bong-Chum Zhu. Visual interpretability for deep learning: survey. Frontiers of Information Technology Electronic Engineering, 19(1):7739, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralha. Object detectors emerge in deep scene cnns. In International conference on learning representations, 2015.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEFE transactions on pattern analysis and machine intelligence, 4109):111-2145,, 2018.
Xizhou Zhu, Weijie Su, Lewei La, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. International conference on learning representations, 2021.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
We provide details omitted in the main paper.
Appendix A
related work (cf. (sussection 2.1 of the main paper).
Appendix B
additional details of inner workings and visualization (ef. (subsection 3.2 and of the main paper).
subsection
Appendix
additional details of model architectures (cf. subsection 3.3 of the main paper).
Appendix D
details of dataset (ef. |scciinn 4 of the main paper).
Appendix E
details of experimental setup (ef. |section 4 of the main paper).
Appendix Fr
additional experimental results (subsection 4.1 of the main paper).
Appendix G
additional qualitative results and analysis (ef. (uusecction 4.1 of the main
paper).
Appendix H
additional discussion (cf. Section 5f the main paper).
A RELATED WORK
In recent years, there has been significant increase in the size and complexity of models, prompting surge in research and development efforts focused on enhancing model interpretability. The need for interpretability arises not only from the goal of instilling trust in model's predictions but also from the desire to comprehend the reasoning behind a model's predictions, gain insight into its internal mechanisms, and identify the specific input features it relies to make accurate predictions. Numerous research directions have emerged to facilitate model interpretation for human understanding. One notable research direction involves extracting and visualizing the salient regions in an input image that contribute to the model's prediction. By identifying these regions, researchers aim to provide meaningful explanations that highlight the relevant aspects of the input that influenced the model's ' decision. Existing efforts in this domain can be broadly categorized into post hoc methods and self-interpretable models.
Post hoc methods involve applying interpretation techniques after a model has been trained. These methods focus on analyzing the model's behavior without modifying its architecture or training process. Most CNN-based classification processes lack explicit information on where the model focuses its attention during prediction. Post hoc methods address this limitation by providing interpretability and explanations for pre-trained black box models without modifying the model itself. For instance, CAM (Zhou et al., (016)) computes a weighted sum of feature maps from the last convolutional layer based on learned fully connected layer weights, generating a single heat map highlighting relevant regions for the predicted class. GRAD-CAM (Selvaraju et al. 2017) employs gradient information flowing into the last convolutional layer to produce heatmap, with the gradients serving as importance weights for feature maps, emphasizing regions with the greatest impact on the prediction. Koh & Liang 2017) introduce influence functions, which analyze gradients of the model's loss function with respect to training data points, providing a measure of their influence on predictions. Another approach post hoc methods involves perturbing or sampling the input image. For example, LIME (Ribeiro et al. 2016) utilizes superpixels to generate perturbations of the input image and explain predictions of black box model. RISE (Petsiuk et al.) (018)) iteratively blocks out parts of the input image, classifies the perturbed image using pre- trained model, and reveals the blocked regions that lead to misclassification. However, post hoc methods for model interpretation can be computationally expensive, making them less scalable for real-world applications. Moreover, these methods may not provide precise explanations or comprehensive understanding of how the model makes decisions, affecting the reliability and robustness of the interpretation results obtained.
Self-interpretable models are designed with interpretability as a core principle. These models incorporate explicit mechanisms or structures that allow for direct understanding of their decision- making process. One direction is prototype-bart based models. Prototypes are visual representations of concepts that can be used to explain how model works. The first work of using prototypes to describe the DNN model's prediction is ProtoPNet (Chen et al. (019), which learns predetermined number of prototypical parts (prototypes) per class. To classify an image, the model calculates the
13

---

Published as a conference paper at ICLR 2024
similarity between a prototype and patch in the image. This similarity is measured by the distance between the two patches in latent space. Inspired by ProtoPNet, ProtoTree (Nauta et al.) (202T) is hierarchical neural network architecture that learns class-agnostic prototypes approximated by
a decision tree. This significantly decreases the required number of prototypes for interpreting a prediction than ProtoPNet.
ProtoPNet and its variants were originally designed to work with CNN-based backbones. However, they can also be used with ViTs (Vision Transformer) by removing the class token. This approach, however, has several limitations. First, prototypes are more likely to activate in the background than in the foreground. When activated in the foreground, their activation often scattered and fragmented. Second, prototype-based methods are computationally heavy and require domain knowledge to fix the parameters. With the widespread use of transformers in computer vision, many approaches have been proposed to interpret their classification predictions. These methods often rely on attention weights to visualize the important regions the image that contribute to the prediction. ProtoPFormer addresses this problem by applying the prototype-basee method to ViTs. However, these prototype- based methods are computationally expensive and require domain knowledge to set the parameters. ProtoPFormer (Xue et al. 2022) works solving the problem by applying the prototype- based method with ViTs. However, these prototype-base based works are computationally heavy and require domain knowledge fix the parameters. ViT-Net (Kim al. 2022) integrates ViTs and trainable neural trees based on ProtoTree, which only uses ViTs as feature extractors without fully exploiting their architectural characteristics. Another recent work, Concept Transformer (Rigotti et al., 2021). utilizes patch embeddings of an image as queries and attributes from the dataset as keys and values within a transformer. This approach allows the model obtain multi-head attention weights, which are then used to interpret the model's predictions. However, a drawback of this method is that it relies on -umann-eefied attribute annotations for the dataset, which can be prone to errors and is costly as necessitates domain expert involvement.
B ADDITIONAL DETAILS OF INNER WORKINGS AND VISUALIZATION
Interpretability vs. model capacity. We investigate whether the conventional classification rule in Equation 1 induces the same property discussed in subsection 3.4 We replace w =(u]) with
w. R Yout : ie.. we learn for each class a class-specific w.. This can thought of as increasing the model capacity by introducing additional learnable parameters. The resulting classification rule is
(c) W Fout
9 = arg max,g((]
(10)
Here. even if [(c) =(d') class can still claim the highest logit as long =(4) has a larger inner product with w. than other wT= Namely, even if the cross-attention weights triggered by different class-specific queries are identical as long as the extracted features in X are correlated strongly enough with class c, the model can still predict correctly. Thus, the learnable queries (ee)) Vc E [C], need not necessarily learn to produce distinct and meaningful cross-attention weights. Zm
Indeed, as shown in Figure 7 we implement variant of our approach INTR FC with its classification rule replaced by Equation 10 INTR produces more distinctive (column- wise)) and consistent (row-
wise) attention.
Visualization. In subsection 3.4 of the main paper, we show how the logit of class c can be decomposed into
w
(c) Zout
Σ sn x a (m).
where
8n = W, Wn
(11)
a n]] exp(g Wi..) = exp(= (c) W, Wi..).
The index n corresponds to grid location (or column) in the feature map X PXN
In the extreme case, one may consider the weights be uniform, i.c., A. at all spatial grids for all classes.
14

---

Published as a conference paper at ICLR 2024
Figure 7: INTR (left columns) INTR-FC (right columns). INTR produces better interpretations than INTR-FC. The bird species is Green Violetear.
to predict an input image as class the cross-attention map (c triggered by the class-specific query 2m should align with the image-specific scores [s1,-.. sx]. words, for an image that is predicted as class c, the cross-attention map ( very much implies which grids in image have higher scores. Hence, in the qualitative visualizations, only show the cross-attention map (e rather than the image-specific scores.
Based on Equation 11
other
We note that throughout the whole paper, INTR learns to identify attributes that are useful to distinguish classes without relying on the knowledge of human experts.
C ADDITIONAL DETAILS OF MODEL ARCHITECTURES
Our idea in subsection 3.2 can be realized by the standard Transformer decoder (Vaswani et al. 2017)
top of any "class-agnostic" feature extractors that produce a feature map X (e.g.. ResNet (He jet al. (016) or ViT (Dosovitskiy et al.) (021)). A Transformer decoder often stacks M layers of 1M the same decoder architecture denoted by { Each layer Lim takes
of C vector tokens as input and produces another set of C vector tokens as output, which can then be used as the input the subsequent layer Lm+1- In our application, the learnable "class-specific" query vectors
(C) E DxC are the input tokens to L1.
Zin in
Zin
Within each decoder layer is sequence of building blocks. Without loss of generality, let omit the layer normalization, residual connection, and Multi-Layer Perceptron (MLP) operating on each token independently, but focus on the Self-Attentin (SA) and the subsequent Cross-Attention (CA) blocks.
An SA block is very similar to the CA block introduced in (uusecctionn 3.1, The only difference is the pool of vectors be retrieved while a CA block attends the feature map extracted from the image, the SA block attends to its input tokens. That is, in an SA block, the X E DxN matrix in Equation 3 is replaced by the input matrix Zin RDx This allows each query token 2 to combine information from other query tokens, resulting in a new set of C' query tokens. This new set of query tokens is then fed into CA block that attends the image features in X to generate the "class-specific" feature tokens.
(c) RD
As Transformer decoder stacks multiple layers, the input tokens to the second layers and beyond possess not only the "learnable" class-specific information in Zn but also the class-specffic feature information from X. We note that an SA block can aggregate information not only from similar tokens ft but also from dissimilar tokens. For example, when W, is an identity matrix and W, - W a pair of similar tokens Zin will receive smaller weights than pair of dissimilar tokens. This allows similar query tokens to be differentiated if their relationships other tokens different, enabling the model to distinguish between semantically or visually similar fine-grained classes
'In this paragraph, this refers the similarity in the inner product space.
15

---

Published as a conference paper at ICLR 2024
Table 3: Statistics of the datasets from different domains.
Bird
CUB
BF
Fish
Dog
Pet
Car
Craft
Training
84635
5994
4697
45162
12000
3680
8144
3334
Testing
2625
5794
275
1830
8580
3669
8041
3333
Classes
525
200
65
183
120
37
196
100
D DETAILS OF DATASETS
We present the detailed dataset statistics in Table We download the butterfly (BF) dataset from the Helicomiine Butterfly Collection Records at the University Cambridge. These downloaded datasets exhibit class imbalances. To address this, performed selection process the downloaded data as follows: First, we consider classes with a minimum of B images, where B is set to 20. Subsequently, for each class, we retained at least K images for testing, with K set 3. Throughout this process, we also ensured that we had no more than M training images, where M is defined as 5 times the quantity (B K). The resulting dataset statistics are presented in Table 3
E DETAILS OF EXPERIMENTAL SETUP
During experiment, for all datasets, except for Bird, set the learning rate to 1 4, while for Bird, we use a learning rate of 5 x e Additionally, we utilize a batch size of 16 for Bird, Dog, and Fish datasets, and batch size of 12 for the other datasets. Furthermore, the number of epochs required for training is 100 for BF and Pet datasets, 170 for Dog, and 140 for the remaining datasets.
F ADDITIONAL EXPERIMENTAL RESULTS
Performance with different encoder backhones. In Inssubsection 4.1 we demon- Table 4: Performance of INTR using different encoders.
strate that INTR yields consistent results when compared to ResNet50. We further delve into an analysis of INTR's perfor- mance using various architectural config- urations, specifically by employing differ- encoder backbones. Our objective is to ascertain whether INTR can potentially achieve superior performance with an alter- native encoder backbone. To investigate this, we employ DeiT (Tourron et al. 2021) and ViT (Doso vitskiy al. 202T) models pre-trained on ImageNet- 1 and ImageNet-2IK datasets, respectively. Specifically, ly, we utilize Deil-Small (INTR-DeiT--S-11 1K) and ViT-Huge (INTR --T---2----) The results of our investigation are presented in Table 4 we see that using different encoders can impact the accuracy. Specifically, on CUB where we see huge drop of INTR in Table 2) using ViT-Huge backbone can achieve a 11.4% improvement.
Encoder Backbone
CUB
BF
Car
INTR (DETR-R50)
71.8
95.0
86.8
INTR (DeiT-S-
78.2
94.2
81.8
INTR (ViT-H-2IK)
83.2
95.7
88.4
Table 5: INTR with different numbers of decoder layers and attention heads, using the CUB dataset.
Algorithm
Number heads
Number of decoders
8
16
4
5
6
7
INTR
69.48
71.75
70.17
68.39
69.21
71.75
69.07
We further perform ablation studies different numbers of attention heads and decoder layers. The results are reported in Table 5) We find that the setup by DETR (i.e., 8 heads and 6 decoder layers) performs the best.
hhtpss//eennooomm//cc////11111
16

---

Published as a conference paper at ICLR 2024
Comparisons to post-hoc explanation methods. We use Grad-CAM (Selvaraju et al.] (017)) and RISE (Petsiuk et al., 2018) to construct post-hoc saliency maps on the ResNet-50 (He et al. 2016) classifiers. We also report the insertion and deletion metric scores (Petsiuk et al. 2018) to quantify the results. It is worth mentioning that insertion and deletion metrics were designed to quantify post-hoc explanation methods. However, here we show a com- parison between INTR, RISE, and Grad-CAM. We examine the CUB dataset images that are accurately classified by both ResNet50 and INTR, resulting in a reduced count of 3, 582 validation images. We generate saliency maps using Grad-CAM and INTR and then rank the patches assess the insertion and deletion metrics. For fair comparison, employ ResNet-50 as a shared classifier for evaluation. The results are reported in Table 6
Table Quantitative comparisons of interpre- tations utilizing ResNet common back- bone on CUB dataset. For insertion, the higher, the better. For deletion, the lower, the better.
Algorithm
Insertion
Deletion
GradCam
0.61
0.31
RISE INTR
0.56
0.30
0.786
0.01
We further compare the attention map of INTR (averaged over eight heads) to the saliency map of Grad-CAM and RISE (using ResNet-50). All methods can visualize the model's 's responses to different classes. For INTR, this is through the cross-attention triggered by different queries. In Fiiuure show a correctly (left) and a wrongly (right) classified example. The two test images on the top are from the same class (i.c., Rufous Hummingbird), and we visualize the model's 's responses to four candidate classes (row-wise; exemplar images are provided for reference); the first row is the ground- truth class. Resolution-wise, both INTR and Grad-CAM show sharper saliency. Discrimination-wiss INTR clearly identifies where each class sees itself each attention map highlights where the candidate class and the test image look alike. Such message not clear from Grad-CAM and RISE. Interestingly. in the case where both INTR and ResNet-50 predict wrongly (into the fourth class: Ruby-Throod
Hummingbird), INTR is able to interpret the decision: the similarity between the test image and the fourth class seems more pronounced compared to the true class. Indeed, the notable attribute the true class (i.c., the bright orange head) is not clearly shown in the test image in Figure 8 (b).
How do you interpret correct prediction this Ruffous Hummingbird image?
How do you interpret incorrect prediction of this Rufoous Hummingbird Image?
Cansskree
Cardilma Class
Name
Nario
Inage
Gofour Hasrongbed
Arna Aaserigleed
Ácsa
Chippio
Spernw
Sparcer
Buyy Vecuret
Ruby Hhrveted
(a) Test image with correct prediction.
(b) Test image with incorrect prediction.
Figure 8: Comparison to Grad-CAM to RISE. The test image at top. Each row is candidate class. The columns of INTR, Grad-CAM, and RISE show the model's 's' response to each candidate class in the test image.
G ADDITIONAL QUALITATIVE RESULTS AND ANALYSIS
Figure offers additional results of Figure 3 In Figure 3 of the main paper, we visualize the top /
three cross cross-attention heads or prototypes. Figure further shows all the prototypes or attention heads for the same test image featuring the Painted Bunting species.
To gain further insights into the detected attributes, we compare INTR with ProtoPFormer, a prominent method in our previous evaluations. We randomly picked five images from each of the four species sampled uniformly from the CUB dataset. Figure 10 shows the attention heads detected by these methods for four images, each from a different species. We validate the attributes detected by these methods through a human study. We provide detected attention heads and image-level attribute information (available in the CUB metadata) to seven individuals, who unfamiliar with the work. We instruct them to list all attributes they believe are captured by the attention heads. An attribute is deemed detected if more than half of the individuals identify it from the attention
17

---

Published as a conference paper at ICLR 2024
Teat Image
INTR
ProtoFNet
Proto
ProtofFomer
Rank
3
4
Figure 9: Extended Comparison with ProtoPNet, ProtoPFormer, and Proto
Image
Attention Heatmaps
Methods
INTR
ProtoPFormer
INTR
ProtoPFormer
INTR
ProtoPFormer
INTR
ProtoPFormer
Figure 10: Extended Comparison between INTR, and ProtoPFormer. We show attention heads comparison between INTR, and ProtoPFormer for four images, each from a different species.
heads. Our quantitative analysis reveals that INTR outperforms ProtoPFormer, achieving an attribute identification accuracy of 74.77 7% compared to ProtoPFormer's 42.2%.
Summer Tanager!! Do yourself?
Image
Head-1
Head-3
Answer
Tarager
[Scariet Tanager)
Summer Tanager
Yes
Figure 11: INTR identify tiny image manipulations that distinguish between classes. We change the color the Scarlet Tanager's wings and tail to make look like Summer Tanager. After that, INTR would misclassify it as Summer Tanager. The result demonstrates INTR's sensitivity to visual attributes.
18

---

Published as a conference paper at ICLR 2024
Heermann Gull
Ring Billed Guil
Slaty Backed Gull
Northern Fulmar
Laysan Albatross
Test Image Heermann Gull
Herring Gull
Western Gull
Caspian Tern
California Gull
Least
Figure 12: We present the top ten classes predicted by INTR for the test image in the Heermann Gull (CUB dataset), showcasing similarities appearances and genera among them.
Heermann Gull!! Do you see yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Answer
Test Class Image
Candidate
No
Class Image
Ring Billed Gull
Figure 13: INTR's Class-specific query able to discriminate similar species. The test image (first row) is Heermann Gull and the most similar candidate class (second row) Ring Billed Gull. We show the cross-attention maps of the ground-truth class image (first row) and candidate class image (second row) triggered by the test class ground-truth query. The query searches for class-specific attributes in both species. For instance, in Head-1 to Head-4 (purple box). both rows detect the common back, breast, tail, and belly pattern respectively. Head-8 (brown box) detects the red black-tipped bill from the test class but not the yellow ring bill from the candidate class.
In the main paper, we demonstrate the capability of INTR in detecting tiny image manipulations. focusing on the species Red-winged Blackbird and Orchard Oriole detailed in |Figure 5] We further extend our analysis to another species, Scarlet Tanager, in Figure 11 Specifically, we modified the Scarlet Tanager by altering its wing and tail colors to red, resembling the Summer Tanager. These alterations were conducted following the attribute guidelines (Cor). To quantitatively measure the effects, we randomly selected ten images from each species for manipulation. Our observations revealed that twenty-nine out of thirty cases resulted in a change in classification post- manipulation, indicating a success rate of 96.7%. This underscores INTR's 's ability to discern tiny image modifications that differentiate between distinct classes.
How does INTR differentiate similar classes? We explore the predictive capabilities and investigate INTR's ability recognize classes that share similar attributes. In Figure 12 show the top predicted classes by INTR for the test image Heermann Gull. The top ten predictions indeed exhibit similar appearances and genera, indicating the meaningfulness of these predictions.
We further investigate attributes detected by INTR that are responsible for distinguishing similar species. The attributes that INTR captures are local patterns (specific shape, color, texture) useful to characterize a species or differentiate between species. These attributes can be shared across species the species are visually similar. These can be seen in/Figure 13 a3) 14] In Figure 13] we applied the query of Heermann Gull to the image of Heermann Gull (first row) and Ring Billed Gull (second row). Since these two species are visually similar, several attention heads identify similar attributes from both images. However, at Head-8, the attention maps clearly identify the unique attribute of Heermann Gull that is not shown in Ring Billed Gull. Please see the caption for details. In Figure 14] present the cross-attention map activated by the ground-truth queries for two closely related species, Baltimore Oriole and Orchard Oriole. Additionally, we manually document some of the attributes by checking whether the attention maps align with those human- annotttdd attributes in the CUB dataset. This reveals that INTR can identify shared and discriminative attributes in similar classes.
19

---

Published as conference paper at ICLR 2024
Baltimore Oriole!! Do you see yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-1
Head-5
Head-8
Head-7
Head-8
Anewer
Yes
Artributes
orange wing. Yellow under tail
Orange
belly
Black nape
Orchard Oriole!! Do you see yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-1
Head-5
Head-6
Head-7
Head-88
Yes
Altributes
orange
Black throat
Orange and black belly
All-ourpose bill shape. Black
Mutt-colored wing
forehead
Black crown
Black eye
Figure 14: INTR identify shared and discriminative attributes in similar classes. We present the cross-attention map activated by query corresponding to ground-truth class two closely related species, the Baltimore Oriole and the Orchard Oriolc. Additionally, we document the attributes identified by saessspe queries, evaluated by humans. It is worth noting that certain attributes detected, such as black throat, breast color, etc., are sharod, given the similarity of these two species.
Class-specific queries are improved over decoder layers. As mentioned in |scciinn 4 and Ap pendix B our implementation of INTR has six decoder layers; each contains one cross-attention block. In qualitative results, only show the cross-attention maps from the sixth layer, which produces the class-specific features that will be compared with the class-agnostic vector for prediction (cf. [Equation 6). For the cross-attention blocks in other decoder layers, their output feature tokens become the input (query) tokens to the subsequent decoder layers. That is, the class-specific queries will change (and perhaps, improve) layers.
To illustrate this, we visualize the cross-aftention maps produced by each decoder layer. The results in Figure 15 The attention maps improve layers in terms of the attributes they identify to differentiate different classes.
Image
Head-1
Head
Head-3
Head-4
Head-5
Headi-8
Head-7
Heed-8
Layer
layer-1
layer-2
layer-3
layer-4
layer-5
layer-6
Figure 15: Attention maps from different INTR Decoder layers, different cross-altention heads the same image using the true query.
20

---

Published as a conference paper at ICLR 2024
Figure 16, Figure 17, Figure 18 Figure 19 Figure 20, Figure 21, and Figure 22 showcase the cross- attention maps associated with the dataset Bird, BF, Dog, Pet. Fish, Craft, and Car respectively, following the same format of Figure I Remarkably, different heads of INTR for each dataset successfully identify different attributes of classes, and these attributes remain consistent across images of the class. INTR provides the interpretation of its prediction different datasets across domains.
Starling!! you see yourself? you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Answer
Yes
Yes
Yes
Yes
Figure 16: Illustration INTR. We show four images (row-wise) of the same bird species Bali Starling and the eight-head cross-allention maps (column-wise triggered by the query of ground-truth class. Each head learned to attend to a different (across columns) but consistent (across rows) semantic cue in the image that is useful to recognize this bird species.
Paititia Neglecta!! Do you yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Answer
Yes
Yes
Yes
Figure 17:
Illustration of INTR. We show three images (row-wise) of the same butterfly species Paititia Neglecta and the eight-head cross-altention maps (column-wise triggered by the query of the ground-truth class. Each head learned attend a different (across columns) consistent (across rows) semantic cue in the image that is useful to recognize this butterfly species.
21

---

Published as a conference paper at ICLR 2024
Toy Terrier!!
Do see yourself? How do interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Answer
Yes
Yes
Yes
No
Figure 18: Illustration of INTR. We show four images (row-wise) of the same dog breed Toy Terrier and the eight-head cross-attention maps (column-wise) triggered by the query of the ground-truth class. Each head is learned to attend to different (across columns) but consistent (across rows) semantic in the image that is useful recognize this dog breed. The exception the last row, which shows inconsistent attention. Indeed, this is a misclassified casc, showcasing how INTR interprets (wrong) predictions.
Chihuahual! Do you see yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Answer
Yes
Yes
Yes
No
Figure 19: Illustration of INTR. We show four images (row-wise) of the same pet dog breed Chihuahua (from the Pet dataset) and the eight-head cross-attention maps (column-wist triggered by the query of the ground-truth class. Each head is learned to attend different (across columns) but consistent (across rows) semantic the image that useful recognize this pet breed. The exception is last row, which shows inconsistent attention. Indeed, this is a misclassified case, showcasing how INTR interprets (wrong) predictions.
Dicotylichthys punctulatus!! Do you see yourself? How you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-6
Head-7
Head-8
Anawer
Yes
Yes
Yes
Yes
Figure 20: Illustration of INTR. We show four images (row-wise) of the same fish species Dicotylichthys punctulatus and the eight-head cross-attention maps (column-wise) triggered by the query of the ground-truth class. Each head is learned to attend to different (across columns) but consistent (across rows) semantic in the image that useful to recognize this fish species.
22

---

Published as a conference paper at ICLR 2024
727-2001: Do see yourself? How do interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-6
Head-6
Head-7
Head-8
Answer
Yes
Yes
Yes
Illustration of INTR. We show four images (row-wise) of the same craft variant A319 and the eight-head cross-attention maps (column-wise) triggered by the query of the ground-truth class. Each head is learned to attend to different (across columns) but consistent (across rows) semantic in the image that is useful to recognize this craft variant. The exception is the last row, which shows inconsistent attention. Indeed, this is misclassified case, showcasing how INTR interprets (wrong) predictions.
Figure 21:
Acura ZDX Hatchback!! Do you see yourself? How do you interpret your decision?
Image
Head-1
Head-2
Head-3
Head-4
Head-5
Head-8
Head-7
Head-8
Anewer
Yes
Yes
Yes
No
Figure 22:
Illustration of INTR. We show four images now-wise) of the same car model Acura ZDX Hatchhack and the eight-head cross-attention maps (column-wise) triggered by the query of the ground-truth class. Each head is learned to attend to a different (across columns) but consistent (across rows) semantic cue in the image that is useful to recognize this car model. The exception is the last row, which shows inconsistent attention. Indeed, this misclassified case, showcasing how INTR interprets (wrong) predictions.
H ADDITIONAL DISCUSSIONS
In this section, we discuss how biologists recognize organisms, how ornithologists (or birders) recognize bird species, and how our algorithm INTR approaches fine-graud fine-grained classification that could benefit the community.
Biologists use traits - the characteristics of an organism describing its physiology, morphology. health, life history, demographic status, and behavior - as the basic units for understanding ecology and evolution. Traits are determined by genes, the environment, and the interactions among them. Peterson (1999) o created the modern bird field guide where he identified key markings to help the birder identify the traits that are distinctive to species and separate it from other closely related or aligned species. These traits are grouped into four categories: 1) habitat and context, 2) size and morphology. 3) color and pattern, and 4) behavior. |Figure I] shows that INTR extract the first three categories from the static images, while behavior typically requires videos.
'This field guide presents the idea of highlighting key features from images to identify species and distinguish them from closely related species. In the field guide, Peterson used expert knowledge to draw a synthetic representation of the species and pointed arrows to the key features that would focus birder's attention when in the field to few defining traits that would help the observer correctly identify species.
23

---

Published as a conference paper at ICLR 2024
butrosa
Paay Finsh
Humningbed
Albutroa
Flsy Firch
Hammingaed
We show the attention maps from cross-attention heads: Head-1 (left) and Hcad-4 (right). Row-wise: test images from different classes; column-wise: different class-specific queries. To make correct. predictions (diagonal), INTR must find and localize the property of an attribute (e.g., the dotted pattern of the tail from Head-1; the color pattern of the hird's head from Hcad-4) at the correct part (e.g., on the tail head).
Figure 23:
More specifically, bird field marks that are used by ornithologists often center around two main bird parts: the head and the wing Patterns of contrasting colors in these regions are often key to distinguishing closely related and therefore similar-looking species. Painted bunting males are nearly impossible to confuse with almost any other species of bunting the stunning patches of color are highlighted in the results presented in Figure 1] For more dissimilar species, aspects of well habitat and behavior.
the shape and overall color pattern are generally more important The characteristic traits of the three species in Figure 23 correspond broadly to these features the long pointed wings of the albatross; the plump body of the finch; the diminutive feet of the hummingbird. Overall, INTR can capture features at both of these scales, mimicking the process used by humans guided by experts. This is of great potential impact because the analysis of traits is critical for biologists to understand the significance of patterns in the evolutionary history of life. Specifically for fine-grained species, our approach can aid biologists rapid identification and differentiation between species, refining and updating taxonomic classifications, and gaining insights into relationships between species.
Please he referred
Cor and Bin
24


Reviews:
[{"review_id": "N2sNNkpCzy", "reviewer": "Reviewer_4rTr", "summary": "The paper introduces Interpretable Transformer (INTR), a classifier that builds on the standard Transformer architecture. It leverages the cross-attention and a set of learnable class-specific queries to introduce by-design visual explanations for each class. The interpretability of INTR comes from the ability to track the cross-attention weights during inference, showing which parts of an image are being considered when making a prediction. The model proves to be effective in not only identifying parts of objects like bird heads but also in distinguishing between species by recognizing subtle attributes. The decrease in classification accuracy compared to ResNet is not a big deal but the evaluation that is mostly qualitative makes me question about its utility. Found details below.", "questions": "N/A\n\nPost-rebuttal reviews:\n\nI genuinely appreciate your great efforts put into this rebuttal!\n# Real utility of the explanations.\nAuthors insisted that:\n\n> In our current paper, the targeted real utility is indeed automatic attribute/trait identification and discovery for organisms.\n\nand in Sect. G, they provided more context:\n\n> These traits are grouped into four categories: 1) habitat and context, 2) size and\nmorphology, 3) color and pattern, and 4) behavior. Figure 1 shows that INTR can extract the first\nthree categories from the static images, while behavior typically requires videos.\n\nHowever, looking at Fig.1, it is unclear how INTR extracts size and morphology information. For added context and background, simply highlighting background pixels provides only loose support for INTR's ability to extract habitat-related traits. There is a need for a systematic evaluation of INTR's efficiency in trait discovery. Currently, most of the claims about its utility are subjective and unconvincing..\n\n# Evaluate the explanations using proxy metrics in XAI.\n\nAs authors also stated that: \n> For self-interpretable methods like ProtoPNet, ProtoPFormer, ProtoTree, and our INTR, which build specific classifiers, XAI metrics are seldom reported as they may not be fair across different methods.\n\nThe current evaluation using deletion and insertion as proxy metrics appears to be of limited significance. A more meaningful benchmark would involve comparing the explanations with other concept-based, self-interpretable classifiers, rather than relying on post-hoc explainers. \n\n# Performance of INTR.\n\nDoes increasing M to 2 equate to doubling the number of heads, in terms of the concepts to be learned? In Fig.20's bottom row, while increasing M can rectify misclassifications, the resultant explanations tend to be noisier. I hope authors will revise this for the next version.\n\n# Overall\n\nI again thank the authors for the incredible efforts in the rebuttal. Given that the authors partially addressed my concerns, I will increase the score from 5->6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The presentation is clear and structure is easy-to-follow.\n- I like the idea of manipulating distinctive features in Figure 5 and believe the authors could relates to Causal inference.\n- I like the honesty in Table. 2 where the authors show the bad performance of INTR over different datasets.", "weaknesses": "- As this paper’s evaluation contains mostly qualitative results (e.g. showing the visualizations and explanations of INTR), it would be great if they can show the real utility of the explanations for a human-related task [a]. In this case, the decreases in accuracy shown in Table. 2 can be completely negligible. Otherwise, I still see ResNet clearly better than INTR (although this is not a major point).\n- The authors may also want to evaluate the explanations using proxy metrics in XAI.\n- It has been unclear to me whether the bad performance of INTR stems from the capability of query vectors or not? It would be great if the authors have ablation studies for the query vectors (e.g. for classes with high inter-class variations, a simple vector could not be representative for the whole class). Also, how do they affect the explanations.\n- The number of heads is equivalent to the number of concepts in ProtoPNet and its variants. For CUB, I believe the number of distinctive features is less than 8 (to my experience). Then I believe this number of heads should be adjusted according to the dataset.\n\n[a] visual correspondence-based explanations improve ai robustness and human-ai team accuracy", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vezh5UJKbv", "reviewer": "Reviewer_dzqa", "summary": "This paper introduces a method called feature accentuation. It is a new explainability method that it indicates which pixels in the image are relevant for the final decision of the model. As well as which kind of features activate relevant neurons.", "questions": "Please refer to weakness points.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A strength of this method is not needed auxiliary generative models and being seeded to images. \nAlso, the release as an open-source library as part of Lucent will make it accessible to those who would like to use it for their applications or built on top of it.\n\nTo overcome some gaps from previous related research, this methods incorporates several techniques  to avoid that the modified image that accentuates some feature changes how it activates the neuron’s with respect to the original seed image as a regularisation term in the loss. There is an analysis on which layers play an important role to avoid undesired distortions, and it is reported that enforcing it in earlier layers yields better visualizations in early layers. An analysis in the impact of regularisation, parametrisation and augmentation techniques from the literature applied in this method is conducted, highlighting the right combination of those factors. Additionally, to improve relevance of the feature representations, a global normalisation is proposed\n\nThe experiments reported use circuit coherence assessment from another paper, which is a reasonable measure.", "weaknesses": "The other applications showcased are also of high importance, but the results become more difficult to assessed. How confirmation biased is overcomed with this method? It is still based on visualisations which need human assessment. The what is based on visual information that is difficult to parse for a human. How useful then it really is still an open question. This is already mentioned in limitations, but it is a strong self-critic that should be given more thought on how to overcome those. How to in corporate over tools for the interpretation is also not clear.\n\n\n\nMINOR:\nThere is a reference missing with a question mark. There are a couple of blank space missing to segment words.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O9VJgvgrLG", "reviewer": "Reviewer_Z5oh", "summary": "The authors present a method that uses Cross-Attention in a Transformer Encoder-Decoder setup to provide interpretable insights how classification results are obtained. By learning class-specific queries as input to the decoder paired with a shared class-agnostic weight vector, the method checks in an intuitive way whether each class can be located in an image, and which attributes within the image lead the ‘classifier’ to believe a certain class is present/dominant – providing a faithful way of interpreting how the decision was made (via visualization of the respective attention patterns).", "questions": "**Main Questions connected to previously mentioned 'weaknesses'**: \n\n- Related to **Influence of Encoder**: As previously detailed, the CA shows to which locations of the feature map the classifier is paying attention to – which does not necessarily have to directly correspond to the input image, especially for architectures with global receptive fields. Do the authors agree with this? What are the (potential) implications of this towards the interpretability of the presented approach?\n\n- Related to **Efficiency**: Do the authors see any possibility to improve efficiency of the method in terms of preventing quadratic growth w.r.t. the number of classes? (and thus increasing the ease of applicability towards datasets with more classes)\n\n- Related to **Pre-training influence**: Did the authors experiment with training from scratch, e.g. on ImageNet, as well? (One alternative could also be using a (frozen) pre-trained backbone and train the 'remainder' of the architecture (Transformer decoder) on only ImageNet); Do the authors have any insight as to how much training on a detection task might help with focusing on detail, and whether this makes any difference to 'simple' classification pre-training? \n\n\n---\nSome additional questions (mainly out of curiosity):\n\n- I’d like to get some more insight regarding the importance of the shared ‘w’ vector to perform the classification; While I do understand the intuition, would the result of learning distinct representations differ if the representation itself was just averaged, i.e. ‘w’ would simply be 1/len(w)? \n\n- Simply out of interest: Did the authors encounter images with multiple objects during their experiments? If so, was their method able to actually locate these individually within the image if queries for the (multiple) ‘correct’ classes existed? If not, how would the authors expect the method to behave – ‘correctly’ finding e.g. 2 objects in an image, or do the authors expect that such use cases will cause issues? (Such a case could potentially be constructed by simply concatenating 2 partial images)\n\n---\nComment:   \n- It might be worth changing the way how HxW is described: The authors consistently describe ‘H’ and ‘W’ as the ‘number of grids’ – however, I’d strongly suggest to instead refer to the elements as “grid cells”, since at least I would see the entire image/matrix HxW as the grid, with H and W indexing the ‘grid cells/elements of the grid’\n\n\n-----\n**Post rebuttal update:**  \nI'd like to genuinely thank the authors for their detailed answers and the effort invested into this rebuttal!  \nMy main concerns regarding the encoder's influence (receptive field) and pre-training have been partially addressed;  \nI do however see some remaining challenges in the presented insights regarding reduced interpretability when using a different backbone: While I agree with the authors' statement that reuse of pretrained backbones is legitimate (and should be encouraged), the results show that using another (non-DETR) pretrained backbone (e.g. the presented ViTs in Fig.22) significantly hinders the efficacy of the presented INTR; This might be due to misalignment, but note that if the method should be useful, it must be applicable to a variety of backbones of interest, not just to one specific combination that has been optimized for DETR; Whether the method transfers well onto other architectures of choice still remains somewhat open/not entirely satisfyingly resolved.  \n$\\rightarrow$ That said, I do think the paper presents a novel and insightful take on interpretability. After reading the other reviews, I decided to stick with my initial rating.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Quality:  \n- Method is described in sufficient detail, including the intuitions underlying each component; Authors additionally do a good job in providing enough mathematical background to clearly follow the inner workings in a concise way;\n- Conducted experiments clearly outline and support the intended contributions, mostly via illustrations of ‘qualitative’ results\n- The level of detail provided in Section 3.4 is very helpful to further understand underlying motivations and inner workings of the method; \n### Clarity:  \n- Very easy to read and follow, very well-structured manuscript; The author’s do an excellent job in posing questions that are then answered to clearly articulate their goals; Good use of illustrations to create compelling story line\n### Originality & Significance: \n'Originality' is (as often) slightly tricky here: The underlying idea of using cross-attention between learnt queries that try to locate itself in the image is pretty much directly adopted from DETR – as the authors themselves correctly state.   \n&#8594; The originality of this work hence mainly lies in the application to use this technique for better interpretability, which is paired with/supported by their way of performing classification to support faithfulness of the involved attention patterns;   \nConsidering these points, I do support the author’s claim of originality in this sense, especially thanks to the explicit consideration of the faithfulness aspect.", "weaknesses": "**TLDR:** While one could potentially criticize the novelty in terms of methods, I do think this work clearly outlines a different and interesting perspective how CA can be used to increase interpretability in a (more) faithful way. Paired with the well-structured manner the manuscript is prepared, I lean towards recommending acceptance of this work — and am happy to further increase my score if the authors can address my questions & concern (mainly 1st weakness).\n\n---\n*Missing discussion of the encoder’s potential influence*:   \nThe (potential) influence of the encoder is not really discussed in this work; Standard DETR already employs a quite large CNN to ‘encode’ the information into embeddings, and one could argue that certain ‘decisions’ as to what is important are already made there. Note that the receptive field of each element of the feature map emitted by the encoder is substantial (potentially even global, esp. when ViT is used) and therefore could incorporate information of various locations throughout the actual input image.     \n&#8594; Please correct me if I’m wrong, but the CA essentially shows us to which locations of the feature map the classifier is paying attention to – which does not necessarily have to directly correspond to the input image? I’d like to see some discussion/clarification of this point.\n\n---\n*Known and stated weaknesses/limitations, but still worth pointing out*: \n-  Self-attention in the Transformer Decoder quickly becomes prohibitive – as the authors correctly state. It however still worth noting that the complexity in memory and time grows quadratically with the number of classes due to the self-attention operation performed across the class-specific query tokens.  \n  \n    &#8594; I’d be curious to know whether the authors might have some suggestion whether all classes are required to be included as decoder input ‘all the time’, or whether some pre-selection could be performed to reduce the computational complexity, at least at inference time.\n\n\n- Influence of pre-training: The authors state that they employ the DETR model pretrained on ImageNet and MSCOCO – which is a lot of data that importantly provides various different aspects of images (classification, as well as (multi-object) detection data);   \nNote: The authors acknowledge this fact in the manuscript, but it is still worth pointing out since it is a potential point of concern that’s introducing some uncertainty as to how backbones that are trained on less data or ‘only ImageNet’ might perform when paired with INTR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bb5YGOKwKH", "reviewer": "Reviewer_8cK8", "summary": "This paper presents an interpretable learning framework using Detection Transformer (DETR) for fine-grained classification, namely INTR.\n\nSpecifically, learning \"class-specific\" queries, performing the cross-attention between the queries and the feature maps to obtain \"class-specific\" features, then a \"class-agnostic\" \"presence\" vector to determine where this query (class) is found in the image or not.\n\nThis paper also presents a mathematical explanation to unveil how the proposed framework learns.\n\nThis paper also presents a few ways to analyze the interpretable classification results, such as image manipulation to monitor changes in cross-attention & the classification score, and fine-grained attributes visualization analysis.\n\nThe experiments are mainly performed with fine-grained recognition datasets.", "questions": "1. What is the \"accuracy\" in detecting the attributes (e.g., Sec 4.2 of [1])? What are the captured attributes? Are these attributes shared or different among the different classes (e.g., it can detect the stripe of a bird, are they the same stripes or different in colors/patterns)? Why is that?\n\n2. Are other models not able to identify attributes like INTR does? How do you quantitatively measure it?\n\n3. How do you systematically perform the image manipulation experiments? You should also measure the result quantitatively.\n\n4. I understand that the attention map is not regularized by any loss functions so the attention maps may look weird. But how do you explain the attention focus on the background, but sometimes focus on the \"attributes\" (e.g. Fig 13)? How can we know what attributes will fire up which heads?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "originality: While not novel (directly incorporating DETR), this paper rethinks the purpose of DETR and applies it to interpretable learning, also providing a valid mathematical explanation to show why and how the queries learn.\n\nquality: This paper has presented experiments to show the benefit of the proposed interpretable framework. the method is straightforward and is easy to implement.\n\nclarity: The paper is well written. The results are quite clear. But some presentations can be improved, for instance, in fig 5, the author can also show the changes in the attention map (like the one in fig 6). For Fig 4, I am not sure what to look at.\n\nsignificance: Interpretability is a well-established problem but we still don't have a completely interpretable framework. I think this paper is somehow significant to affect the community. The idea of this paper is simple, this may inspire future works to improve upon this work.", "weaknesses": "- Lack of interpretability analysis. As mentioned by the author this paper is not about accuracy, hence I shall expect more quantitative/qualitative experiments to fully understand inner works. Since the author says INTR can detect the \"attributes\", the author can measure how precise the model can detect. What is the \"accuracy\" in detecting the attributes (e.g., Sec 4.2 of [1])? What are the captured attributes? Are these attributes shared or different among the different classes (e.g., it can detect the stripe of a bird, are they the same stripes or different in colors/patterns)? Why is that? I think all these results can be shown in the main paper.\n\n- Comparison with other interpretable models. The only comparison I noticed is Fig 3. Are other models not able to identify attributes like INTR does? How do you quantitatively measure it? From what I see in this figure, ProtoPFormer is also able to locate local attributes, but there are not enough samples here.\n\n- Image manipulation experiments seem not systematic enough, how do you measure the changes? E.g., after manipulating the \"part\", how many images changed the classification results? Are they still the correct or the wrong results? \n\n- The attention map now looks a bit not \"clean\" enough, when some attributes cannot be detected from the image, I suppose the attention should be empty instead of looking at the background, if it is looking at the background, does it mean it is a feature? or it is overfitting? The author did not further analyze it.\n\n- the sentence \"Fortunately, fine-grained classification usually focuses on a small set of visually similar classes; C is usually not large.\" may be not true, iNaturalist [2] has 10000 classes, and even the \"bird\" classes alone have 1478 classes.\n\n[1] Zixuan Huang and Yin Li. Interpretable and Accurate Fine-grained Recognition via Region Grouping. CVPR 2020.\n[2] https://github.com/visipedia/inat_comp/tree/master/2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)