Learnable Visual Words for Interpreting Image Recognition Models
Authors: Wenxiao Xiao
Zhengming Ding
Hongfu Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: To interpret deep models' predictions, attention-based visual cues are widely used in addressing *why* deep models make such predictions. Beyond that, the current research community becomes more interested in reasoning *how* deep models make predictions, where some prototype-based methods employ interpretable representations with their corresponding visual cues to reveal the black-box mechanism of deep model behaviors. However, these pioneering attempts only either learn the category-specific prototypes and deteriorate with their generalization ability deterioration or demonstrate several illustrative examples without a quantitative evaluation of visual-based interpretability narrowing their practical usages. In this paper, we revisit the concept of visual words and propose the Learnable Visual Words (LVW) to interpret the model prediction behaviors with two novel modules: semantic visual words learning and dual fidelity preservation. The semantic visual words learning relaxes the category-specific constraint, enabling the generic visual words shared across multiple categories. Beyond employing the visual words for prediction to align visual words with the base model, our dual fidelity preservation also includes the attention-guided semantic alignment that encourages the learned visual words to focus on the same conceptual regions for prediction. Experiments on six visual benchmarks demonstrate the superior effectiveness of our proposed LVW in both accuracy and interpretation fidelity over the state-of-the-art methods. Moreover, we elaborate on various in-depth analyses to further explore the learned visual words and the generalizability of our method for unseen categories.

Under review as a conference paper at ICLR 2023
LEARNABLE VISUAL WORDS FOR INTERPRETING IMAGE RECOGNITION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
To interpret deep models' predictions, attention- based visual cues are widely used in addressing why deep models make such predictions. Beyond that, the current research community becomes more interested in reasoning how deep models make predictions, where some prototype-based methods employ interpretable represen- tations with their corresponding visual cues to reveal the black-box mechanism of deep model behaviors. However, these pioneering attempts only either learn the category-specins prototypes with their generalization ability deterioration or demonstrate several illustrative examples without a quantitative evaluation of visual- based interpretability narrowing their practical usage. In this paper, we revisit the concept of visual words and propose the Learnable Visual Words (LVW) to in- terpret the model prediction behaviors with two novel modules: semantic visual words learning and dual fidelity preservation. The semantic visual words learning relaxes the seoegorr--peccffii constraint, enabling the generic visual words shared across multiple categories. Beyond employing the visual words for prediction to align visual words with the base model, our dual fidelity preservation also includes the attention-guider semantic alignment that encourages the learned visual words focus on the same conceptual regions for prediction. Experiments on six visual benchmarks demonstrate the superior effectiveness of our proposed LVW in both accuracy and interpretation fidelity over the state-of-th of-te-of-theaarr methods. Moreover, we elaborate on various in-depth analyses to further explore the learned visual words and the generalizability of our method for unseen categories
INTRODUCTION
Model interpretation aims to explain the black-box base model in semantically understandable way and preserve high fidelity with model outputs (Li et al. [02T Molnar et al. 2020: /Zhan et 2021b.. Although interpretative models are not designed to pursue higher performance than the base model, they are of great importance, especially in the deep learning era. With human- understandable explanations, model designers can diagnose errors and potential biases embedded in deep models; model users are confident and relaxed to rely on model predictions. This trust is built on the assumption that the interpretation model is faithful to the original deep networks. For tabular data, the raw features with physical meanings are used to interpret the model prediction. LIME (Ribeiro) et al. 2016) and SHAP (Lundberg & Lee 2017) are two representative explainable models, which learn linear models to locally fit the base model's outputs with sample perturbations. For visual data, attention-based visual cues are widely adopted in interpreting deep models' predictions (Correia & Colombini 202T). Grad-Cam (Selvaraju et al. 2017 2020) is popular technique for visualizing the region where a convolutional neural network model looks at. uses the saregory-speciffi gradient information flowing into the final convolutional layer of CNN to produce a coarse localization attention map of the important regions/pixels the image. Following this direction, several studies further employ weakly supervised information directly on attention maps during the training stage to
reduce the attention bias (Li et al. 2018 Srinivas & Fleuret 2019 Chaudhari et al. 2021).
Beyond the above studies addressing why deep models make their predictions at the generally coarse level, some pioncering attempts have been taken to reason how deep models make predictions at the fine-grained level. ProtoPNet (Chen al. 2019) learns a predetermined number of prototypes per
Our code is available at http:///sshia.ooo...mmmmaeeeeeee Visual- Visual-Words

---

Under review as a conference paper at ICLR 2023
category and proposes prototypical part network with a hidden layer of prototypes representing the activated patterns. By learning these prototype-basee based interpretable representations, ProtoPNet explains the model reasoning process by dissecting the query image into several prototypical parts and interpreting these prototypes with training images in the same category. Later, ProtoTree (Nauta et al. 202T) and ProtoPShare (Rymarczyk al. (022b) extend ProtoPNet by decision trees and prototype pruning to achieve global interpretation and reduce the model complexity, respectively. Unfortunately, these studies either learn the calegory-specific prototypes that deteriorate their generalizing capacities or only demonstrate several illustrative examples without quantitative evaluation of visual- -ased interpretability (Figure 2 demoostratee their discovered meaningless or irrelevant prototypes despite their high recognition accuracy). Moreover, they only evaluate on two object-cropped visual datasets and leave their performance on other general visual datasets and samples from unseen categories on the shelf. It is crucial to note that an interpretation model should be evaluated by whether the interpretation model is loyal to the base model in all aspects of the outputs, rather than solely focusing on if it achieves higher accuracy or delivers human rognition-consisten explanations.
Contributions.
The process of ProtoPNet recalls us of the bags-of-visual words (Csurka et al. 2004 Sivic & Zisserman 2003), a popular image representation technique before the deep learning It treats an image as a document and visual words are defined by the perpoonn/ddescrrptoos/ptttt histogram over the
that are used to construct vocabularies. Then the image can be represented occurrences of these visual words. It is worth noting that these visual words with semantic meanings across different images can also be used for model interpretation, which is similar to the prototypes in ProtoPNet. The major difference lies in that conventional visual words are usually hand-crafted, pre-defined, and independent of the downstream learning tasks. In light of this, we propose Learnable Visual Words (LVW) to overcome the aforementioned drawbacks of the prototype-based interpretative methods. Technically, our model consists of two modules, semantic visual words learning, and dual fidelity preservation. The semantic visual words learning relaxes the category-specine constraint, enabling the generie visual words to be shared across different categories, while the dual fidelity preservation encourages the learned visual words to behave similarly to the base model in both prediction and model attention. This novel attention fidelity ensures that the learned visual words attend to the same areas of sample image when the base deep network makes its predictions. Our major contributions are summarized as follows:
In the semantic visual words learning, we relax its calegory-specific constraint and further simplify ProtoPNet to achieve cross-category visual words and increase the generalization of model interpretation, rather than adding any new terms.
In the dual fidelity preservation, we encourage the learned visual words to preserve high fidelity with the base model in terms of both prediction and model attention. This dual fidelity helps our model identify an interpretation that loyally presents the base network. Additionally, we further design measurement to quantitatively evaluate the visual-based interpretation.
We demonstrate the superior effectiveness of our model on six visual benchmarks over the state-of-the-ae prototype-based methods in both accuracy and visual-based interpretation and explore the generalization of our model by interpreting unseen categories.
2 RELATED WORK
Visual Image Understanding, Research efforts in interpretable explanations of a Convolutional Neural Network (CNN) can be generally divided into posthoc and self-imterpretable genres. Posthoc methods attempt to build an extra explainer for the pre-trained black-box model to interpret its prediction. Approaches including saliency visualization based on backpropagation (Springenberg
et 2014: Zhou et al. [016)] Zhang et al., 2018, Bach et al. 2015] Sundararajan et al. 2017
Shrikumar et al., 2017]] Smilkov et 2017: Fong & Vedaldi 2017 Rebufh et al. 2000) and activation maximization caused by perturbation (Simonyan et al.
Zeiler & Fergus 2014
Petsiuk et al. 2018 Fong et al., 2019 Kapishnikov et al. [211) Dabkowski & Gal 2017 Ancona et al. 2017) to identify the most influential parts for the black-box model's 's prediction. However, visualizing the salient areas does not explain how the black box makes such decisions. Other posthoc methods (Ghorbani et al. 2019 Zhang et al.) 2021a Olah et al.) 2018 Akula et al. 2020 Yeh jet al.] |020] Koh et al. 2020; Kim et al. 2018 Chen et al. 2020) obtain interpretable concept activation vectors from pre-segmented feature maps and interpret the CNN model with these concepts.
2

---

Under review as a conference paper at ICLR 2023
Alternatively, self- self-interpretable methods aim to directly learn explanatory representations during training, instead of disentangling the pre-trained black box. ProtoPNet (Chen et al., 2019) introduces case-based study that interprets the reasoning process of the black-box model with category-specti prototypes associated with image patches from training data, which answers how deep models make predictions by linking the test image with interpretable prototypes. Recent studies (Rymarczyk et al.)
2021a Donnelly et al. 2021) extend ProtoPNet in various directions. For instance, ProtoTree (Nauta) al. 2021) combines prototype learning with decision trees, resulting in an interpretable decision path consisting of prototypes. Similarly, HPnet (Hase et al. 2019) hierarchically organizes prototypes to classify objects every level in predefined class taxonomy. ProtoPShare(Rymarcy)
al., 2021bb further groups similar prototypes discovered by pre-trained ProtoPNet model with a data- dependent merge-pruning strategy. Meanwhile, TesNet (Wang et al. 2021) extracts prototypes from class-specific embedding subspaces defined on the Grassmann Manifold. Lately, Deformable ProtoPNet(Dai et al., 2017) explicitly captures pose variations and context by integrating deformable
convolutional networks in their model.
Interpretation Model. A great number of researchers are dedicated to the attempts of opening the black-box deep networks. Early works model interpretation try to learn set of rules decision tree globally as the interpretation for a pre-trained deep model. Some methods (Quinlan 2014] Odajima al. 2008: Nayak] 2009 Krishnan et al. 1999 Pedapati et al.! 2020) extract global set of rules or a decision tree from the training samples generated by the base network. Later works seek to find local logic rules for one specific sample or a small group of samples. Dhurandhar et al.) (2018) construct such rules by identifying the import features, which is represented by sparse permutation of input features that is sufficient produce the same prediction as the base model, while Wang et al. (2018) propose to look for critical data routing paths of the network for each input by assigning sparse non- non-negative weights to each channel on each layer. On the other hand, attribution-bass explanation methods interpret deep networks by assigning important scores to the input features. Most of these attribution methods can be separated into two groups: gradient- based attribution and model agnostic attribution. [Bachrens et al. (2010) explain the classification decision of k-NN and SVM with gradient-basse attribution, while other methods (Springenberg et al. 2014 Bach et al. 2015 Sundararajan al.. 2017: (Srinivas & Fleuret, 2019) calculate gradient-related saliency maps as attributions. Alternatively, model agnostic methods like LIME (Ribeiro et al., 2016) and MAPLE (Plumb et al. 2018) interpret individual model predictions based on locally approximating the model around a given prediction with a local linear explanation model. Other methods (Lundberg
Lee 2017 Ancona et al. 2019) adopt Shapley values for locally measuring the attribution of input features. Lately, SpRAy (Guo et al., 2019) and MAME (Hatt et al.) (219)) combine the individual attributions obtained from local methods to generate a global ittribution-basee explanation.
3 METHOD
3.1 FRAMEWORK OVERVIEW
We introduce the framework of our Learnable Visual Words (LVW) model to extract meaningful semantic information for interpreting the base CNN models in Figure which consists of visual words learning and dual fidelity preservation. The visual words learning module aims to extract semantics shared across all categories, while the dual fidelity preservation module guides these learned visual words to focus on the regions of the image that base CNN models attend to when making predictions and also preserves the similar predictive ability of the base model. In order to learn cross-class visual words, our semantic learning module simplifies ProtoPNet (Chen et al. 2019) by removing its separation loss, which is used for finding category-specific prototypes for each category. We keep ProtoPNet's clustering loss so that the learned visual words are closely related to the training images, and at the same time, contain various shared semantic information from all classes. The dual fidelity preservation module assures the learned visual words preserve high fidelity with the base model in terms of both prediction and attention in similarity-based approach. For each image sample, the similarity scores between one visual word and all patches of the sample image's 's backbone convolution output are calculated, resulting in a similarity heatmap that represents how strong this visual word matches different parts of the image, the same as ProtoPNet. The maximum value of the similarity heatmap for each visual word is then used for making the final prediction by the fully connected layer with cross-entropy loss. To achieve interpretability for the prediction made by the CNN backbone, the attention-guided guided alignment module combines the similarity heatmaps of
3

---

Under review as a conference paper at ICLR 2023
Input Image
Visual Words L earning
Dual Fidelity Preservation
Figure 1: Illustration of our pro- posed Learnable Visual Words (LVW) model for image recog- nition interpretation. Visual Words Learning module simpli- fies ProtoPNet and learns cross- category semantic visual words to describe the samples; the Dual Fidelity Preservation mod- ule enhances the learned visual words to preserve high fidelity with the base model in terms of both prediction and attention.
Simplified ProtoPNet
Class Prediction
Moose
Attention Guided
Aligmment
Allention
Model
Cluster
Separation
Semartis Intigralie alloo
Base Backbone
the sample image's top i visual words with max nax-pooling and guides the pooled heatmap with the base CNN model's class-level attention. With attention alignment, the learned visual words focus on the same areas used by the base model for making predictions, preserving the base model's attention.
3.2 LEARNABLE VISUAL WORDS FOR INTERPRETABLE MODEL
Since our model simplifies ProtoPNet in visual words learning, we first introduce the base backbone model and ProtoPNet to provide enough preliminaries and then elaborate on our innovations.
Preliminaries. Let the base backbone model (e.g.. ResNet (He et al. 2016). VGG (Simonyan Zisserman 2014)) be trained in a supervised setting with N labeled training samples X [z,,,, 1siSN associated with C' different categories. ProtoPNet (Chen al. 2019), , prototype- based explanation method for a CNN model, consists a convolutional layer f. a prototype layer 9 and prediction layer h. It first extracts the feature map Z; =xxx,) with dimension of H W x D from the last convolutional layer's output of the base model. Then ProtoPNet learns total and W'<W. Their prototype layer computes an activation map consisting of the inverted L2-distance between one learned visual word fn and all HI, x W1 x D-shape patches of Z. Then the activation map is resized back to the original image size, resulting in an activation heatmap A to represent how strong Uy matches each part of the input image z₁. ProtoPNet takes the maximum value in A². i.c., max (A.) as similarity score between the visual word 2 and the input image. Later, a fully- colleyeed layer makes the prediction based on similarity scores g o fxz)
M prototypes visual words V {uz 1 with the shape of x'>>"'>>>, H'
mmxx(A)) .... max (AA)) associated with all M learned visual words. Note that the base model's 's attention map of A, for sample image It can be calculated with existing attention method like Grad-Cant (Selvaraju al. 2017 2020) or GAIN (Li et al. 2018). By aligning the base model's attention A, and the attention maps of the k top visual words that are most semantically closest to Zj, our proposed model aims to learn visual words that keep high fidelity with the base model in terms of both prediction and attention, providing a more reasonable explanation for understanding its reasoning of making predictions.
Visual Words Learning. Our visual words learning module is simplified version of ProtoPNet where we keep its clustering loss but discard the separation loss. The clustering loss encourages all training images to be close some learned visual words; the other hand, removing the separation loss relaxes the eategory-specifee constraint and allows the learned visual words to be shared across categories. Mathematically, the cluster loss is defined as:
C.
min mim minmin Vy =EZ,
11112
where z a single patch of shape H' XX' X Z..
Attention Guided Semantic Alignment. Beyond employing the visual words for prediction, our dual fidelity preservation also includes the attention- -uided semantic alignment, which requests the learned visual words to focus the same region the base CNN model. To achieve this goal of attention fidelity, the alignment module combines the activation heatmaps of the top visual words with the highest similarity after max-pooling, resulting in single activation heatmap A that
4

---

Under review as a conference paper at ICLR 2023
represents how strong these k visual words are present in the input image The attention alignment abides by the loss term as follows:
Lg N
11A 144-A111 -
where A, is the input class-level attention the base model on input image Is, and A is the concatenated activation heatmap from the top l visual words that are semantically closest to 24.
3.3 TRAINING PROTOCOL AND OBJECTIVE FUNCTIONS
We adopt the same end-to-end training procedure as ProtoPNet[Chen et al. 2019) with three stages: (1) optimization before the last layer; (2) projection of visual words; (3) optimization of the last layer.
Optimization Before the Last Layer. In this stage, we fix the fully connected layer h and train the convolutional layers f and visual words layer g. For each category. the connections between M/c visual words are set to and the rest connections are set to -0.5, encouraging the model to learn some visual words that are closely related to this category. Different from ProtoPNet, we
remove the separation loss so that the learning module does not force the learned visual words to be eeegory-speciffie The overall objective function for this stage
min f:g
E Cotsh f(z))) + aLe + BLa.
N
where f((x4) is vector containing similarity scores between M visual words to Tgs a and / are two trade-off parameters.
Projection of Visual Words. During this stage, we project each visual word uy onto the nearest patch across all training images to further ensure that the learned visual words contain various semantic information from the training data. The project can be described as follows:
Uj
min =E2
1yl1?
(4)
where the collection of all latent patches of shape H' XD contained X.
Optimization of the Last Layer. In this stage, we only optimize the fully connected layer h with fixed convolutional layers and visual words layer g. This training stage uncovers the semantic associations of each learned visual word across all categories. Following ProtoPNet, we also add L regulaarizzaton during this training stage and the objective function is:
1 N
min
Σ Carlhogof(zi) +Y
Σ |w],
(5)
WEW,
where W,, is the collection of all weights in the fully connected layer and is the trade-off parameter.
EXPERIMENTS
We demonstrate the performance of our learnable visual words in two aspects: classification ability and attention-based model interpretability. We first introduce the experimental setup, then report the algorithmic performance with extended examples, and finally provide various in-depth analyses.
4.1 EXPERIMENTAL SETUP
Datasets. We include six datasets for performance evaluation including STLIO (Coates et al. 2011 Oxford 102 Flower (Nilsback & Zisserman, 2008), Oxford-IIIT Pets (Parkhi et al. (012). Stanford Dogs (Khosla et al. 2011), Food-101 dataset (Bossard et al. 2014), and AwA2 (Xian et al. 2019). Their detailed descriptions can be found in Appendix|A.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Performance of different interpretative models by accuracy and IoU coverage on six datasets
Method
STLJO
Orfond Flower
Oxfond-H11
Pets
Stanford Dogs
Food-10!
AWA2
Acc
IoU
Acc
loll
Ace
IaU
Ace
loUl
Ace
IoUl
Ace
IoU
RosNet-44 He al.
2016
57.40
1.00
98.17
1,00
92.28
1.00
79.03
1,00
81.60
1,00
93.66
1.00
ProtoPNet Chen
2019
0.2199
0.2190
0.2294
ProtoTice Noirl o1 di M
89.58
0.3228
30,44
0.3418
62.41
0.5113
41.00
0.4267
36.37
04173
86.54
0.197
PhotoPStare
(2 Revmnrg 2V cral
b02
$7.01
0.2722
87.04
0.2958
74.70
0.3467
68.70
0.4585
12.37
0.1826
81.10
0.2571
TonNet [Wang al.
S7.66
0,5444
91.69
0,6708
85.53
0,7058
75,47
0,6387
75.94
0.5817
90.82
0.6665
Ours
Align.
0.2309
0.3722
0.2583
70.72
0.2748
15.03
91.16
0.2536
Ours
89.75
0.6465
91.81
0.7967
86.20
0.8086
73.81
0.7933
76.33
0.6969
90.85
0.7781
Baseline Models. We compare our model to four baseline methods including ProtoPNet (Chen et al. 2019), ProtoTree (Nauta et al. 202T] ProtoPShare (Rymarezyk et al. 2022bb and TesNet (Wang et al. 2016) as the base CNN backbone for all models including our model. For ProtoPNet, ProtoPShare, and TesNet, 10 prototypes are selected per category. For ProtoTree, we train a tree of depth 10 so that each sample image will be associated with 10 prototypes in the test stage. For our method, we choose M=5xC visual words, where C' is the class number.
202T. We choose ResNet-34 34 (He et al.
Evaluation. Recognition accuracy is used to evaluate the predictive ability of each model. An interpretation model should not only provide human-understanddbbl explanations of the base model but also keep loyal to the base model, so that the model understanding can be used for downstream tasks like model diagnosing and debugging. Thus we also propose an Intersection over Union (IoU) coverage metric as an objective measurement for the model's interpretability as follows:
1oUg.m MM,(A) n (A**
(6)
M,(
where A is the class-level attention from the base model, and A is the combined attention acquired from the m most related visual words. Here, M,, means the binary masking function that masks out all the values that smaller than the g-th quantile of the attention map as 0 and keeps the rest values as Here we set k 10 and q = 50 by default. Essentially, the designed IoU metric represents the amount of overlap between each sample's 's base model attention and the top k visual words' attention. Thus, we use the average loU across all test samples to measure the model's ability in explaining the decision made by the base CNN model. As obtaining the class-level attention maps from the base network only requires class labels, our model does not require any extra information for training. Also, our model does not have access to any attention maps of the testing samples. Thus, we believe evaluating our model against the baselines with this proposed IOU metric is a fair comparison.
Visualization Protocol. We adopt a similar protocol of visualizing learned visual words as Pro- toPNet (Chen et al. 2019). For visual word fyy globally, we identify the training image zj whose convolutional output Z, has the highest activation of u.. The global visualization of vj is generated by overlaying the up-sampled activation heatmap A, on St. For any test image x' locally, the activation heatmap A₁ of visual word ti on x' represents the presence of U on x'.
4.2 ALGORITTMIC PERFORMANCE
Here we comprehensively evaluate our learnable visual words model and three prototype-based interpretable recognition models with classification accuracy as well as the proposed IoU coverage metric. We also visually demonstrate a case study on Food- 10/ dataset.
Quantitative Evaluation. Table I shows the recognition accuracy and the proposed IoU coverage of all models on six datasets, in which we highlight the best results in bold. In addition to our complete model, we also report the performance of our model without ireettion-guided alignment in order to check the efficacy of this module. In general, our proposed model achieves comparable recognition accuracy on these six datasets with the base ResNet-34 model. Our model sacrifices at most 6.36% accuracy loss Oxford Flower (Nilsback & Zisserman 2008), while, to our surprise, improving ResNet-34 by 2.35% % on STLIO (Coates et 201 dataset. Compared with the three baseline models, our full model attains the best recognition accuracy in 5 out of 6 datasets, only second to model without attention alignment on AWA2 (Xian et al. 2019), which boosts prediction accuracy over
We notice new variants of ProtoPNet, Deformable ProtoPNet (Doneelly et al. 2021). We do not include this method for comparison, since this method uses Deformable Convolutional Networks( Dai et al. 2017). which employ different architecture from the hase CNN models used in our experiments.

---

Under review as a conference paper at ICLR 2023
(a)
Test Image
Delected Visual Words
Visual Words on Training Samples
Greek Salad
Capreso Salad
Madame
Proto
522 Caprese Salad
Greck Salad
Grock Salad
PooooTae
Apple Fic
Chicken Curry
Gredi Sabd
PhotaPStare
TeN
Greek Salad
Salad
Greek Salad
ReSNel
Ours Toll: 0.7537
ProluPNee IoU: 0.5438
ProloPShare
TeaNel IoU: 0.3937
Allention
IoU: 0.1908
Figure 2: Visual demonstration of our model and other prototype-based based methods for the same base model (ResNet-34) interpretation Food-101. (a) shows the detected visual words prototypes the test image and corresponding training samples, where the bounding boxes are generated according the activated regions of visual words prototypes. We the same technique in ProtoPNet Chen et al. (2019) for bounding box generation. (b) shows the ResNet attention map and the integrated attention maps of visual words or prototypes with their IoU values.
ProtoPNet (Chen et al. 2019) only by removing the class-specific constraint. We assert that our model preserves, and possibly improves, the predictive ability of baseline ResNet-34. However, accuracy alone cannot properly estimate the more evidence in the next Visual Demonstration paragraph). Therefore, we also evaluate all methods with our proposed loU coverage metric to check if the explanation keeps the original meaning of the base network's reasoning process. Our model exceeds all baseline models by relatively large margin in terms of IoU coverage. At minimum, the best- pessfrrrfoning baseline model in terms IOU, by at least 0. on all datasets, while our model significantly outperforms the rest of the bascline models, especially and STLIO (Coates et al., 2)0I). where our model's 's IOU can be twice as the IoU of the aacond-best baseline method. These results suggest that our learned visual words preserve the base model's
interpretability of different models (We will provide
model outperforms TesNet(Wang
al. 2021), which is
datasets like Oxford Flowers
7

---

Under review as a conference paper at ICLR 2023
0.8
75
0.8
0.78
74
Accuraggtt)
38 0.76
PretePiat
PrstaPiit
Shate
Phato
0.2
PreteTrne
0.2
Prsudless
TeNee
1oU
Avcuracy
Ours
15
5d
%
25
60
75
90
%
25
50
75
90
(a) IoU Acc on Stanford-Der -Dgg
(b) STLIO
(c) Oxford Flower
0.5
0.8
0.8
0.6
0.6
0.4
PrstuPNet
PrstePNet
Prenitharm
PretefSlam
PritaShaa
0.2
0.8
PretoTins
0.2
Prstafinse
ToaNet
ToNee
Oure
Oure
9
25
60
75
90
%
26
60
75
90
%
25
50
75
90
(d) Oxford- LIlT Pets
(c) Food-101
(f) AWA2
Figure 3: (a) Performance of method with different numbers of top visual words k. (b-f) IoU coverage for ours and baseline models with different quantile thresholds on STLIO, Oxford Flower, Oxford-IIfT Pets, Food-10) and AWA2.
attention and provide interpretation that is loyal to the reasoning of the base model. Notice that our model without attention alignment outperforms ProtoPNet on all six datasets in terms of accuracy, which verifies our assumption that different categories might share common visual semanties. On the other hand, the IoU coverage of the model without attention alignment is worse than ProtoPNet on datasets like Stanford-Dogs (Khosla et al.. 2011) and Food- (Bosard al.. 2014) despite the improved accuracy, indicating that only relaxing the -ptegor--specific constraint does not enhance its interpretability, where the cross-category assumption might not be suitable for fine finegrriied classes. Also notice that the comparison between ProtoPNet and our model without attention alignment, which is essentially ProtoPNet after removing the separation loss, illustrates that the separation loss does not affect the accuracy or IOU in any deterministic way.
Visual Demonstration. We compare our model with the four prototype- based methods visually with case-study example in Figure 2 We choose one test image from Food-101 [Bosardd et al. (2014) datasets and select the three most related visual words or prototypes extracted by each method. We visualize them in the test images as well as their corresponding projected training images with the same protocol as ProtoPNet et al. 2019) in Figure 2 a). Our method detects the visual words semantically corresponding to "cheese crumbs," "tomatoes," and "green vegetables" across three training categories. On the other hand, although the baseline models can also uncover prototypical patches from the sample images, the detected prototypes are more likely to focus on meaningless or irrelevant semantics which provides little explanation for the prediction. For example, all four baselines identify prototypes related the plate located on the edge of the test image. Those prototypes might help classify this test image as "greek salad," but they are unable to interpret this decision. We conjure that without the guidance of base model attention, these prototype-basee methods rely on those prototypes of the base model's attention area to make predictions, especially on the uncropped image data. In Figure (b)) both the loU coverage values and the combined attention maps illustrate that the ResNet attention is well covered by visual words, while the baseline methods focus more on the edge area of the test image. From this visual demonstration, we can observe that our learnable visual words provide more reasonable interpretation of the reasoning process of the base model than competitive methods.
different datasets. One key hyperparameter in our model is the number of top visual words i used in attention- guided semantic alignment. We choose different values of k from I to 50 and conduct experiments
Hyperparameter Analysis. We explore model's sensitivity to hyperparameters
8

---

Under review as a conference paper at ICLR 2023
Test Image
Detected Visual Words
Visual W Words on Training Samples
Unseen Class Liger
Seen Lion Flerce Hlair
Sten Lion Face
Seen Classs Tiger Strip
Uassen Class
Seen Class
Soen C
Class
Rhinocaras
Sheep Body
Elephant
Figure 4: Illustrative examples of our learnable visual words trained on AWA2 for interpreting image samples of Liger and Tapir. Their categories are unobserved in the training set.
on Stanford-Dog dataset. We train our model for each choice of k. and randomly select 3 different subsets original test set, each containing 50% of total test images, to check our model robustness against different test data. As shown in Figure 3]. prediction accuracy does not fluctuate for various choices of The IoU drops for smaller like or 3 since we only encourage very few visual words to focus on the class-level attention. For each k, the error between three random test sets is insignificant, considering our large loU improvement. Moreover, the choice of quantile threshold in designed measurement could also significantly affect the final results of IoU during the testing stage. To evaluate the robustness and generalizing ability, we plot the IoU coverage for different quantile thresholds on the rest of the five datasets. Our model consistently outperforms the four baseline models on all benchmarks as shown in Figure (esure]b-f) b(f)). For 90, the attention of baseline models barely overlaps with the base model attention except for TesNet( et 202I). while our method still maintains at least 0.2 IoU coverage, which evinces our proposed method uncovers the most attended semanties in the base model.
Interpreting Samples From Unseen Categories. In light of the visual words across different categories, we extend our method to those categories that potentially share semantics with existing training categories, but are unseen by the base model. In Figure 4 we choose "Liger" and "Tapir" as the unseen categories and test them on our model trained on AWA2 dataset. Ligers are zoo-bred hybrid offsprings of male lion and female tiger, which possess features of both parents. Tapirs are rare creatures that are often confused with pigs or anteaters, but their closest living relatives actually rhinoceros and horses. As expected, our model successfully uncovers several shared semantics between the unseen classes with classes that exist the training data. The face, fierce hair, and strip pattern the body of this liger tracked back to the learned visual words from the lion and tiger categories in the first row of Figure 4 Although our model does not predict this unseen image liger, does help us better understand this image: an animal with the face and fierce hair of lion, as well as tiger's stripe on its body. Similarly, our model can interpret this tapir as creature, which has the ear of a rhinoceros, the body of sheep, and the face of elephant.
CONCLUSION
To sum up, we presented Learnable Visual Words (LVW), novel approach that aims to cast a light on how deep models make predictions by extracting cross-category visual words and aligning learned visual words with the base deep models' visual attention. LVW streamlines ProtoPNet (Chen et al. 2019) by relaxing its class-specifi constraint. Its attention-guided semantic alignment preserves high fidelity with the base deep model on the level of predictive ability and model attention. This fidelity ensures our model loyally interprets the reasoning of the networks, allowing these learned visual words to guide the downstream tasks like model diagnosis. We conducted extensive experiments on six benchmarks to quantitatively coverage metric against three state-of-an art prototype-base methods. The experimental results evinced the efficacy and generalization of our L.W method over several benchmarks.
model with recognition accuracy and our proposed IoU

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arjun Akula, Shuai Wang, and Song-Chun Zhu. Cocox: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-basee based attribution methods for deep neural networks. arXiv preprint urXiv:1711.06100 2017.
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In International Conference Machine Learning. PMLR, 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 2015.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 2010.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. An attentive survey of attention models. ACM Transactions on Intelligent Systems and Technology, 2021.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2020.
Adam Coates. Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011.
Alana de Santana Correia and Esther Luna Colombini. Attention, please! survey of neural attention models in deep learning. arXiv preprint arXiv:2103.16777 2021.
Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. In Workshop on Statistical Learning in Computer Vision, ECCV, 2004.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in Neural Information Processing Systems, 2017.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEFE international conference on computer vision, pp. 764-773, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai and Li Fei-Fei, Imagenet: A large-scale hier- archical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2009.
Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Advances in neural information processing systems, 2018.
Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. arXiv preprint a5Xiv:2/11./5000, 2021.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. IEEE International Conference on Computer Vision, 2019.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In IEEE International Conference on Computer Vision, 2017.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 2019.
Zhe Guo, Xiang Li, Heng Huang, Ning Guo, and Quanzheng Li. Deep learning-based image segmentation multimodal medical imaging. IEEE Transactions Radiation and Plasma Medical Sciences, 2019.
Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. Interpretable image recognition with hierarchical prototypes. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2019.
Mathieu Hatt, Chintan Parmar, Jinyi Qi, and Issam EI Naqa. Machine (deep) learning methods for image processing and radiomies. IEEE Transactions on Radiation and Plasma Medical Sciences, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pallern Recognition, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Vi egag and Michael Terry. Xrai: better attributions through regions. IEEE International Conference on Computer Vision, 2019.
Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Li Fei-Fei. Novel dataset for Fine-Grained Visual Categorization,
fine-graines image categorization. In First Workshop IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2011.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Muchine Learning, 2020.
Krishnan, Sivakumar, and Bhattacharya. Extracting decision trees from trained neural networks. Pattern recognition, 1999.
Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Raymond Fu. Tell me where to look: Guided attention inference network. JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang. Ji Liu, Jiang Bian, and Dejing Dou. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond. arXiv preprint arXiv:2103. 1000......,.....1111 2021.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 2017.
Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. Interpretable machine learning brief history, sftt--of-theane art and challenges. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2020.

---

Under review as a conference paper at ICLR 2023
Meike Nauta, Ron van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Richi Nayak. Generating rules with predicates, terms and variables from the pruned neural networks. Neural Networks, 2009,
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
Koichi Odajima, Yoichi Hayashi, Gong Tianxia, and Rudy Setiono. Greedy rule generation from discrete data and its use in neural network rule extraction. Neural Networks, 2008.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 2019.
Tejaswini Pedapati, Avinash Balakrishnan, Karthikeyan Shanmugam, and Amit Dhurandhar. Learning global transparent models consistent with local contrastive explanations. Advances in neural information processing systems, 2020.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.0742)) 2018.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. Model agnostic supervised local explana- tions. Advances neural information processing systems, 2018.
J Ross Quinlan. C4. 5: progrums for machine learning. Elsevier, 2014.
Sylvestre-Alvise Rebuffi, Ruth Fong, Xu Ji, and Andrea Vedaldi. There and back again: Revisiting backpropagation saliency methods. IEEE/CV Conference on Computer Vision and Puttern Recognition, 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016
Dawid Rymarczyk, Lukasz Struski, Michal Górszczak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zielinski. Interpretable image classification with differentiable prototypes assignment. arXiv preprint arXiv:21 2021a. URL httptpppp//.xi....oaa/ wwwpeiiiioorrrgaaoo.o abs http:://aaaa...oo///b/////......
Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2021b.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In JEEE International Conference on Computer Vision, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. International Journal of Computer Vision, 2020.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning. 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
12

---

Under review as a conference paper at ICLR 2023
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In IEEE International Conference on Computer Vision, 2003.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:/706.0388 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox. and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. Advances in neural information processing systems, 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017.
Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In IEEE International Conference on Computer Vision, 2021.
Yulong Wang, Hang Su, Bo Zhang, and Xiaolin Hu. Interpret neural networks by identifying critical data routing paths. In proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Yongqin Xian, Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, 2014.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 2018.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin P. Rubinstein. Invertible concept-based explanations for CNN models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021a.
Yu Zhang, Peter Tiño, Aleš Leonardis, and Ke Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics Computational Intelligence, 2021b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2016.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASET
We include six datasets for performance evaluation. (1) STLIO (Coates et al. 2011) is a popular image classification benchmark containing 13,000 labeled images from 10 object classes 5,000 images used for training while the remaining images are kept for testing. (2) Oxford 102 Flower (Nilsback & Zisserman 2008) a fine-grained image classification dataset including 102 flower categories. Each class consists of between 40 and 258 images. Based on its original split, we use its training set for training and its validation set for testing- (3) Oxford- IIl'' Pets (Parkhi et al. 2012) has 37 categories including various breeds of dogs and cats, with roughly 200 images for each class and it is often used for fine- mie-grained image classification tasks. (4) Stanford Dogs (Khosla et al. 2011 contains 20,580 images with fine-grained annotation for 120 different dog breeds. The sample images are divided into 12,000 images for training and 8,580 images for testing. (5) Food- 101 dataset (Bossard et al.) 2014)
is a large-scale fine-gre aie-ggrined classification benchmark, which contains 101 food categories with 750
training and 250 test images per category. (6) AwA2 (Xian et al. 2019) contains 37,322 images in 50 animal categories. We choose this dataset specifically for checking if our method generalized well for non- non-fine-gen fine-grained grained datasets. Notice that all sample images are not cropped, even for those datasets that come with Region of Interest annotation.
A.2 IMPLEMENTATION DETAILS
We implement our model in PyTorch (Pazkee et al. 2019) using one NVIDIA Titan GPU. For all datasets, we fine-tune the ResNet-34 base model pre- trainea on ImageNet (Deng et al. 2009) for each dataset, and then Grad-Cam (Selvaraju et al. 2017: 2020) attention map is extracted with
the fine-tuned ResNet. All the images are resize to 224x224x3 except for STLIO. We keep the original image size for STLIO as all its images have the same size. Random horizontal flip is applied during training as the only data augmentation procedure. The shape of convolutional output f(x) is == with D=128 channels. The shape of each visual word is set to ='= The learning rate of the backbone feature extractor is set as 0.0001, while the learning rate of the other parts is set as 0.003. a, /3 and 7 are set to 0.8, 10 and 0.0001, respectively. The number of visual words & is set to 10 empirically for attention-gun guided alignment training. All datasets are trained for 200 epochs, and we project visual words after every 10 epochs of training. We use ReLU as the activation function except for the last fully connected layer, in which we use the sigmoid function.
A.3 ALGORITHMIC PERFORMANCE ON VGG BACKBONE
Our method can also be used for other types of convolutional backbones. such as VGG (Simonyan & Zisserman 2014), DenseNet (Huang et al., 2017), and potentially ViT (Dosovitskiy et al. (020). We replace the ResNet-34 backbone with VGG-19. We also trained a base VGG-19 model on the Oxford-IIIT Pets (Nilsback & Zisserman 2008 and obtain the Grad-CAM as the input attention for method. Here we report the performance of diffferent interpretation methods with the VGG backbone on Oxford-IIIT Pets in Table 2 Our method still outperformances all baseline methods in terms of accuracy and IOU coverage.
Table Performance of different interpretative models with VGG-19 backbone onOxford-III IIIT Pets
Oxford-IIIT Pets
Method
Acc
loU
VGG-19 (Simonyan & Zisserman, 2014
89.06
1.00
ProtoPNet (Chen et al. 2019
86.24
0.3696
ProtoTTre (Nauta et al. 2021
50.14
0.3305
ProtoPShare (Rymarczyk et al. 2021b
87.04
0.2722
TesNet (Wang al. 2021
87.53
0.5688
Ours w/o Atte. Align.
87.57
0.2321
Ours
87.93
0.6426
14

---

Under review as a conference paper at ICLR 2023
Test Image
Base A Mention
Our Visual Words
Visual Words on Training Samples
Ground Truth Squirrel
Base Prediction Walrus
Walrus
Rhinoceros
Seal Skin
Valrus
Skin
Head
Ground Truth
Base Prediction Leopard
Tiger Strip
Leopold
Lepardd
Spot
Figure 5: Examples where base model makes wrong predictions based on incorrect attention. The first column the test imgge and the second column is the base model's attention map and prediction. The third column contains the detected visual words the sample and the prediction of our model.
Test Image
Base Attention
Detected Visual Words
Visual Words on Training Samples
Ground Truth Bat
Prediction Bat
Siamese Cat Fur
Rabbit Black Spot
Stamese
Background
Ground Truth Grizzly Bear
Base Prediction G Grizzly Bear
Racoon
Racoon Face
Giant Panda Background
Raceon
Fur
Figure 6: Failure cases where our model makes wrong interpretations based on the correct attention of the base model.
A.4 FAILURE CASE ANALYSIS
In this section, we share some failure cases for both the base model and our interpretative model to identify the potential limiting factors of our method.
Failure cases of the base model. Firstly show two examples where the base model makes wrong predictions because incorrect attention in Figure In the upper example, a "squirrel" image is predicted "walrus" by the base model. The attention map shows that the base model focuses the foreground soda can and tree stump instead of the squirrel. Guided by this attention, our model finds that the visual word representing the walrus skin is similar the texture of the tree, and the visual words representing the rhinoceros head and seal skin are similar to the texture/color of the soda can. The lower example is labeled as "mouse" even though the salient object Unsurprisingly, the base model predicts this image as a "leopard" based on the foreground object. In this case, our model related the strips and spots of the foreground animal with the tiger's strips and the leopard's 's black spots. In both failure examples, our visual words provide a reasonable explanation of how the base model makes wrong prediction, which validates the effectiveness of our visual words in model diagnosis. Notice that the last visual word representing the black fur of the otter finds the mouse despite it being located outside the base attention. conjure that the high l similarity between these two patches causes this visual word to deviate from the base model's 's attention.
cat-like animal.
Failure cases of our interpretative model. Next, we will show two failure examples of our model when interpreting the correct predictions of the base model in Figure 6 In both cases, the base model makes the correct predictions and the base model provides reasonable attention maps for the predicted categories. However, the main object only occupies very small area in each example, which might
15

---

Under review as a conference paper at ICLR 2023
Bird Airplane
6.0
Ship Truck
Class Average Act vation
Average Activation
5.0
4.0
3.0
2.0
1.0
0.5
1.0
0.0
10
20
40
50
0
20
30
40
50
(a) Visual Words Activation of Similar Categories
(b) Visual Words Activation
of Dissimilar Categories
Figure 7: Visual words exploration. (a) demonstrates the visual word distributions on two similar categories and (b) demonstrates the distributions of two dissimilar categories, where the x-axis denotes the 50 learned visual words.
only contain limited semantic information. small number of visual words will be enough to extract such information, while other visual words might align with the background noise. For example, in the upper "bat" example, the first visual word finds the similarity between the color a "siamese cat" and the color of the "bat", while the other two visual words seem only to focus the background. Similarly, the visual words in the second example fail to connect the small attention area in the test image the "grizzly bear" category, resulting in incorrect prediction and explanation.
Based on these examples, seems that learning the prototypes solely on the 22 similarity guided by the attention map may impair the performance of our method, especially for extreme Figure Finding a better similarity measurement for aligning semantics or refining the attention map with other techniques e.g., posture detection, would be interesting extensions for our future work.
shown in
A.5 VISUAL WORDS EXPLORATION
We explore the learnable visual words by inspecting the interrelation between different classes. As we discussed in Section 3.2, the output of visual words learning module g fx;; can be viewed as an activation vector representing how strong each visual word is associated with training image x;. By averaging the vectors of all training samples in the category c, we acquire new vector representation of the category defined by its similarity with the M learned visual words. We choose pairs of classes, one similar and one dissimilar, and plot each pair's visual words based activation vectors As demonstrated, the similar pair of "Bird" and "Airplane" has relatively high activations on at least ten visual words, which suggests these shared visual words' presence in both categories, while those unshared visual words differentiate one class from another. On the other hand, the dissimilar pair "Truck" and "Ship" barely agree on any visual words.
together in Figure7(h)&&((
16


Reviews:
[{"review_id": "zbek_7diLb", "reviewer": "Reviewer_zev4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a learnable visual word method to interpret the model prediction behaviors. The authors proposed 2 new modules: semantic visual word learning and dual fidelity preservation. The semantic visual word removes the separation loss from the protopnet, and the dual fidelity preservation adds attention alignment with the grad-cam outputs. Experiments show the proposed method achieves good performance on the metric. ", "summary_of_the_review": "This paper provides two improvements over the prior ProtoPNet's architecture: 1: remove the separation loss; 2: add attention-guided alignments. The proposed IoU metric is a little bit concerning since the model directly optimizes for this target compared to other methods. ", "strength_and_weaknesses": "[Strength]\n\n- The paper is well-written and clear. \n\n- Experiments results show the proposed method is better than previous methods. \n\n[Weakness]\n\n- My major concern with the proposed method is its novelty. The proposed LVW basically follows ProtoPNet's objective and removes the separation loss. The ProtoPNet's objective is also learnable. \n\n- The metric of IoU is also problematic, it's unfair to compare to other methods using this metric since the proposed method trained with l2 normalization loss. \n\n- What is the performance with separation loss? It seems there is no such number in the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oi-sBBKZiV", "reviewer": "Reviewer_HEcT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to learn visual words to explain deep neural networks specifically for the task of image recognition. Certain visual words are learned based on the target category and these words are encouraged to be shared between different categories. Moreover, they also include the attention guided semantic alignment that encourages the leaned visual words to focus on the same conceptual regions for prediction. They show results on several benchmark datasets on both accuracy and interpretation fidelity. They also show that their method is generalizable to ground visual words for unseen categories,.", "summary_of_the_review": "I have some concerns regarding the contributions and the training process adopted to encourage interpretability. It is not very clear if the scores are high because of correct attention map guidance or something else. An in-depth evaluation of learned visual words, failure cases and how the attention maps differ from baseline when these are wrong would be interesting to see. ", "strength_and_weaknesses": "Strengths - The paper tackles the problem of understanding and interpreting  deep neural networks by learning visual words and grounding the words aligned with the attention of the base model. They perform experiments on several benchmarks and present  several analysis. \n\nWeaknesses - \n\n1. The idea about aligning the maps of visual words to the base model attention map will prevent stronger deviations from learning wrong attentions but at the same time also will prevent the model to learn correct disambiguations. How do you ensure that the models learn part speciifc attentions for the learned visual words ? \n\n2. Also what would happen when the base model's attention is completely wrong ? \n\n3. The contributions are very thin compared to previous literature  and the improvements in the IoU metric is obvious since the model's objective is to improve the overlap between attentions. But does this metric correctly state that the model is more interpretable ? \n\n4. It would be worthwhile to share some failure cases of the model to get a more in depth idea of what are the limiting factors.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "IxGh-tQm7H", "reviewer": "Reviewer_Tm8e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a learnable visual word model that consists of two modules to predict model prediction behaviors. Concretely, a semantic visual words learning model is proposed to relax the category-specific constraint and thus enable the generic visual words shared across multiple categories. A dual fidelity preservation module is proposed to encourage the learned visual words to focus on the same conceptual regions for prediction.\n", "summary_of_the_review": "I think the originality is limited, and the experiments seem not sufficient.", "strength_and_weaknesses": "Strength \n1. I think model interpretability is quite an interesting task. \n2. The paper is well-written and easy to understand.\n\nWeakness\n1. It seems the novelty is limited as it is merely a simplification of the current work ProtoPNet. It simply removes the separation loss. However, the motivation for removing this loss is not well clarified. Besides, as it is a key contribution of this work, but its effectiveness is also not verified in the experiments.\n2. It seems that semantic alignment is also used in many model interpretability works. Its originality should also be better clarified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject