Self-Interpretable Model with Transformation Equivariant Interpretation
Authors: Yipei Wang
Xiaoqian Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: interpretable machine learning
transformation equivariance
computer vision

Abstract: With the proliferation of machine learning applications in the real world, the demand for explaining machine learning predictions continues to grow especially in high-stakes fields. Recent studies have found that interpretation methods can be sensitive and unreliable, where the interpretations can be disturbed by perturbations or transformations of input data. To address this issue, we propose to learn robust interpretation through transformation equivariant regularization in a self-interpretable model. The resulting model is capable of capturing valid interpretation that is equivariant to geometric transformations. Moreover, since our model is self-interpretable, it enables faithful interpretations that reflect the true predictive mechanism. Unlike existing self-interpretable models, which usually sacrifice expressive power for the sake of interpretation quality, our model preserves the high expressive capability comparable to the state-of-the-art deep learning models in complex tasks, while providing visualizable and faithful high-quality interpretation. We compare with various related methods and validate the interpretation quality and consistency of our model.

Self-Interpretable Model with Transformation Equivariant Interpretation
Yipei Wang, Xiaoqian Wang"
Elmore School of Electrical and Computer Engineering Purdue University
West Lafayette, IN 47907
wang4865@purdue.edt edu, joymangSpurdue.em edu
Abstract
With the proliferation of machine learning applications in the real world, the de- mand for explaining machine learning predictions continues to grow especially in high-stakes fields. Recent studies have found that interpretation methods be sensitive and unreliable, where the interpretations can be disturbed by per- turbations transformations of input data. To address this issue, propose to learn robust interpretations through transformation equivariant regularization self- interpretteeee model. The resulting model is capable of capturing valid interpretations that are equivariant to geometric transformations. Moreover, since our model self-interpretable. it enables faithful interpretations that reflect the true predictive mechanism. Unlike existing self-interpretale models, which usu- ally sacrifice expressive power for the sake of interpretation quality, our model preserves the high expressive capability comparable to the sfate-of-the-ar deep learning models in complex tasks, while providing visualizable and faithful high- quality interpretation. We compare with various related methods and validate the interpretation quality and consistency our model.
Introduction
Deep learning (DL) models have been a great success various domains of applications, including object detection, image classification, etc. However, many applications suffer from the overfitting problem, which is usually due to the lack of various training data. For scenarios with limited data access, data augmentation is usually applied to alleviate the overfitting problem. As one of the most simple but effective data augmentation methods, geometric transformation plays an important role in exploring the intrinsic visual structures of image data 44
34]. Transformation equivariance refers to the property that data representations learned from the model capture the intrinsic coordinates of the entities [15], i.e., transformations on the data will result in the same transformations to the model representations. Building raasformation-equivaal
DL models desired in many kinds of applications, such medical image analysis [1]] reinforcement learning [27]. etc.
Although DL models can exert excellent performance in various tasks, DL models are usually expressed as black boxes. Therefore, DL models can have great performance in complex tasks but lack an explanation of the results [1]. In low-risk tasks such as adaptive email filtering, the direct deployment of black-box models without reasoning might be acceptable. However, for high-risk peeision-mmkknn tasks such disease diagnosis and autonomous vehicles [18), the applied model needs to be more convincing than black box. On the one hand, by faithfully explaining the model behavior, can ensure the end user intuitively understands and trusts the DI. model. On the other hand, the explanation of black-box models can provide insights into the relationship between input
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and output, thereby improving model design. However, with the rapid growth in computational power, DL models are designed to be more and more complex to meet the performance [35], and the most advanced DI, models can have billions of trainable parameters [36]. High complexity leads to the complete black box for human beings, which results lack of trust in the model. The demand for building more reliable and easy-to-understand DL. models is growing rapidly.
Depending on the stages where predictions and interpretations are conducted, the methods can be divided into two opposing categories: self-interpretable models and post-hoc models [30] Unlike post- hoc models, which generate interpretations to pre- trained black-box models, self-interpretable models aim to build models that are intrinsically interpretable themselves. The main difference between these two categories is that for post-hoe models, the interpretation and prediction are obtained in two different stages. The interpretation is separately obtuined after the black- box models are trained. Therefore, interpretations obtained from post-hoc models are considered to be more fragile, sensitive, and less faithful to the predictive mechanism [1 19 50] In contrast, self-interpretable models make interpretations at the same time as predictions, thus revealing the intrinsic mechanism of the models, and are thereby preferred by users in high-stakes tasks 40) Besides, considering how powerful and common transformation can be in data augmentations, it is reasonable to take the robustness of interpretation to transformations into consideration when designing and evaluating the interpretations. Robust interpretation towards transformation implies two requirements: 1) the predictive mechanism indicated by the interpretation should remain the same after transformation (e.g., the highlighted region should remain the same despite the transformation); 2) the location of interpretation should change according to the transformation. These two requirements naturally lead to transformation equivariance on interpretation. The twtasffrrmationeeuivarraane property will enhance robust and faithful interpretation, where the interpretation is aware of the transformations and preserves the predictive mechanism. This correspondence between transformation equivariance and faithfulness suggests that eeff-interpretable models may perform better than post-hoc models in transformation awareness given their higher faithfulness. And the experiments also demonstrate this.
Although self-interpretable models surpass post-hoc models in faithfulness and stability, there are non-negligible challenges in building self-interpretable models. First, the interpretations may need additional regularization to be in forms that rational humans. This process usually involves prior domain knowledge provided by human experts [2] 39.. Besides, since the interpretability is intrinsic, specific constraints are required in the models to ensure the interpretabilltyy. The prediction power of such models will be damaged since it is essentially adding constraints to optimization problems. It is acknowledged that the increase of the interpretation quality is likely decrease the performance of prediction results 13 31. As consequence, self-interpretable models are usually less expressive compared with black-box models, which can be interpreted by post-hoc models.
Our Model: In this paper, we develop a transorrnation-cquiaaaaaa self geff-interpretable model for classification tasks. As reff-interpretable model, our method makes predictions and generates interpretations of the predictions at the same stage. In other words, the interpretations are directly involved in the name our method as SITE SSeff-Interpretahi model with Transformation Equivariant Interpretation). In SITE, we generate data-dependent prototypes for each class and formulate the prediction the inner product between each prototype and the extracted features. The interpretations can be easily visualized by upsampling from the prototype space to the input data space.
prediction process, and are therefore faithful to the final results. We
Besides, we introduce transformation regularization and reconstruction regularization to the pro- totypes. The reconstruction regularizer regularizes the interpretations to be meaningful and com- prehensible for humans, while the transformation regularizer constrains the interpretations to be transformation equivariant. We validate that SITE presents understandable and faithful interpretations without requiring additional domain knowledge, and preserves high expressive power in prediction.
We summarize the main contributions through this work as:
To our best knowledge, we are the first to learn transformation equivariant interpretations.
We build a self-interpretable model SITE with high-quality faithful and robust interpretation.
SITE preserves the high expressive power with comparable or better accuracy than related black-box models.
We propose velf-consistency score, new quantitative metric for interpretation methods. It quantifies the robustness of interpretation by measuring the consistency of interpretations to geometric transformations.
2

---

2 Related Work
Machine learning interpretation can have different goals, such as attribution 24 interpretable
clustering (28), interpretable reinforcement learning [29]. disentanglement [43] etc. Our method lies
in the attribution category, thus we mainly review the related interpretation methods for attribution.
Attribution methods target at identifying the contribution of different elements in the prediction. Based on if the prediction and interpretation obtained in the same stage, the methods can be divided into post-hoe interpretation and ielfrintterrettable methods.
For post-hoc interpretation, the prediction results are obtained by black-box model while the interpretation is obtained separately to explain the predictive mechanism of the black box. Among the different post-hoc interpretation techniques, backpropagation methods [60] 52 45 37
trace from the output back to the imput to determine how the different elements in the input contribute to the prediction result. Class Activation Mapping (CAM) [60] visualizes the feature importance in convolutional neural networks by mapping the weights in the last fully connected layer to the input layer via upsampling. Score- CAM learns weighting scores for the activation maps by integrating the increase in confidence for an improved CAM visualization [52]. While for approximation methods [3], the interpretation is obtained by fitting an interpretable model to the black-bor box prediction around the target sample. Deconvolution methods (58) interpret a convolutional neural network via image deconvolution. For perturbation-base
14], the methods interpret the feature importance by imposing perturbation to certain feature and checking the changes in the output. Moreover, Shapley values [24] have been used to calculate the feature importance due to the nice properties preserved by Shapley values. For post-hoc interpretation, since the prediction and interpretation are separated, the prediction can be obtained by highly expressive black-box model to handle complex tasks. However, the post-hoc interpretation may not capture the true predictive mechanism of the black box and is less reliable 19
interpretation 22
Different from post-hoc interpretation, interprrrtee models target at building white boxes that intrinsically interpretable. which are able to conduct prediction and interpretation at the same time. A self-interpretable model preserves faithful interpretation since the model itself is a white hox. However, the self-interpretation constraints can limit the expressive power, thus sacrificing prediction performance. For example, in order to build an interpretable decision set (22), there is a constraint on the number of rules for the sake of interpretation, which restricts its application to complex tasks. Recent models propose to build self-interpretable models with neural network (2 3
20 [33 [54]
and kernel methods 77). FRESH 17 focuses on the interpretability for natural language processing tasks. SENN [3) Concept Bottleneck Models (20) generate interpretations in high-level spaces instead of the raw pixel space. ProtoPNet 6 provides interpretations in the pixel space, but focuses more on the local patches corresponding to the local areas of the image instead of the global interpretation. NAM 2 provides the same kind of interpretations
SITE. combines neural networks with additive models to facilitate the seff-interpretation via component function. But it decouples all pixels, which results in low expressiveness. Moreover, attention models have been widely used to build interpretable predictions [26). However, recent works find that the interpretation via attention weights can fail to identify the important representations
Different from the related works, our goal is to build a self-interpretable model that learns faithful interpretation and has high expressive power. Previously the transformation equivariance property has been studied in prediction via deep neural networks. Many recent studies integrate the transformation cquivariance in object detection, with the goal of building convolutional neural networks that are equivariant to image translations. The models learn features equivariant to translation and rota- tion [56 55 10. 3D symmetries (5T). and build sets with symmetric elements for general equivariant tasks 225 Despite transformation equivariance in the prediction. these methods may not guarantee the transformation equivariance in the interpretation, i.e., the prediction mechanisms of transformed and untransformed inputs may be inconsistent. We thus introduce the interpretation equivariance to complement the prediction equivariance, To the best of our knowledge, we are the first to learn transformation-cquiaaiin interpretations to ensure faithful and robust interpretation.
3 Building Transformation Equivariant Interpretation in SITE
In this section, we introduce the structure of SITE. For notations, all normal lowercase letters stand for numbers; all bold lowercase letters stand for tensors; all normal uppercase letters stand
3

---

SITE
regularization
Fi(T(())
upsampling
TOO
Figure 1: An illustration of our SITE model. SITE can take both original image and transformed image T(x) as input. The input first fed the feature extractor F1, then SITE generates c prototypes Wi,..-, We through generator G. Finally, both the prediction and interpretation come from the Hadamard product between the latent representation Fi (T)) and each prototype. The interpretation is obtained by upsampling the Hadamard product, and the prediction is obtained by the element- wise summation of SITE ensures transformation equivariant interpretation by constraining the interpretations before and after transformation.
for operations (including functions, networks, etc); and all curly uppercase letters denote sets and families. Additionally, all Greek letters will be explained when they are introduced in context.
3.1 Formulation
For image classification tasks, suppose that x E R' denotes the input image, one-hot vector y e {0, _ denotes the label, and ý [0, 1] denotes the predicted class probabilities. We clarify that p is the product of the number of channels, the width, and the height of the image x, while denotes the number of classes. Generally, traditional classifier F with a feature extractor Fi and a simple classifier F2, where F1 x + z and F2 z - y. Here E IRd denotes the extracted latent representations of x, and usually has lower dimension (d < p). The extractor F1 usually consists of convolutional neural networks or ResNet structures, and F/ consists of fully connected layers. The goal is minimize the classification loss min yxyx (F (F(x).
be decomposed into F = F2 Fi
F=FooP;
where X, are the input data set and the target set, and Lee denotes the cross-entropy loss function.
Traditional methods in (I) is not intrinsically interpretable w.r.t. the contribution of features in x to the prediction y. order to address this, in SITE we build generative model = [G G..] that maps the latent representation prototype corresponds to specific class. We formulate the final prediction as the inner product of the latent representation z and each prototype [G,(z)} That is,
prototypes {w.)-= C Rd where w, Gz) Each
y = (G((z) z) = a ([G)(2) z, G2(z) 2, 2...., G.......z)T) =2]")
(2)
where a is softmax activation. The prediction ý is the similarity between the latent representation and the generated prototype (z). Thus we have the modified classification loss
Lats Exex.yey (G (x)) Fi)))
(3)
Note that we formulate the prediction result for class i as ú. (z) to our formulation of y in (2), we can explicitly capture the contribution of elements in z to the final prediction by the Hadamard product between w; and z as w; the interprelation of the i-th prediction result, such that the contribution of different elements to each prediction result is clear from the interpretation. For instance, w/ = 1,... c.j = 1.... ,, denotes the contribution of the j-th element of the i-th class. Based our formulation of w. the interpretation from our model preserves the completeness property. That is, the summation of the importance scores of all features equals the prediction result. This is introduced [48), and also known as the local accuracy in (24] This property assures that the interpretation is related the corresponding prediction in the numerical sense.
aw, z). According
@ Naturally we take w; as
Proposition. 1
The interpretation obtained from optimizing Tuts ensures the faithfulness (i.e., which shows the true predictive mechanism of the model), but may not ensure that the interpretation is human-

---

understandable. In order to build high-quality interpretation, we propose to regularize the prototypes G 1. c with the following. For an imput image x, we enforce each generated prototype G (x)) to be similar to its corresponding class's latent representation (x;):
L 1=22 ExEXX Luce (x)(F, (i(x)))
(4)
(=1
where Lace denotes the binary cross-entropy loss, and 2r C X denotes the set of input data that belongs the i-th class.
In addition, we propose to regularize on the transformation equivariance property of interpretation from our SITE model. Let T3 denote pre-defined parametric transformations as described in [16]. We want SITE to learn interpretations that are equivariant to the transformations. Here 3 denotes the randomly sampled parameters from pre-defined parameter distribution B. This is because affine transformation operator T can be parameterized by an x 3 matrix J.. During the training process, suppose that the random transformation Tp is known and first transform the input image by randomly
have access to its inverse
Ty' In the feed-forward process of training, sampled transformations Te (x). B, then feed it the model G F1. So the prediction result on the transformed image is ((F((I(())) Fi (T)( The prototypes of the transformed input image ((F)(T((x)) can be transformed back by the inverse transformation Tz We build the
(G (F)()(((())))) 1, 1. and
reconstruction loss between the transformed prototypes Tp" the latent representations of x; E i -,.... :c, respectively:
L2 ExEx Luce (T (G((F((T(())))))G(((()))
(5)
By integrating the equivariance property (5) with transformation Tg, e B in the interpretation regularization in (4). we propose the transformation loss as:
Tarans ExExxxx Lace ( (TT) FI(x:)
Hence, propose the objective of SITE with classification loss and transformation loss follows:
min
EB
E ExEX Trae (T (TT(((((T)))) (((
G.F.
A Σ (T
where À a hyper-parameter that balances the training paces between the classification loss and the transformation loss. The first term in objective (7) ensures transformation-aware classifier, while the second term ensures cransormationneqqivarrr x; X2 and the expectation over B can properly approximated by Monte Carlo sampling.
interpretations. In practice, the expectation over
3.2 Visualization Methods
In the previous subsection, we obtain the self- self-interpretable model G o Fi, and the corresponding interpretation w; for input x. However, since the interpretations w; E IRd are not in the original image space, the direct visualization of w; will be less meaningful.
Notice that the interpretations w; are approximations of the output space of the feature extractor F₁. it is natural to visualize it by visualizing H (W:). where H Rd - IPP is an approximated inverse of Fr. And since F1 based convolutional neural networks, simple but judicious choice for H would be the bilinear upsampling function. On the one hand, the output space of F, will preserve the relative relationship between features. And on the other hand, the Lipschitz continuity of H can preserve all the intrinsic properties in w, Finally the interpretation w, is visualized in the original space of the input images by overlaying on the input x as heatmap.
3.3 Transformation Self-Consistency Scores
In order to measure the transformation equivariance of an interpretation method properly, we propose a numerical metric, namely the self-consistency score. measures the self-consistency [49] of an attribution interpretation method. For given input data x and a parameterized transformation
5

---

Figure 2: Interpretations of SITE on MNIST dataset. For each digit, the first column shows the randomly transformed images. The following c = 10 columns are the prototypes (wi)i=i (the first and third rows), and the interpretation heatmaps (w; O x (the second and fourth rows).
T3, let [(x denote the interpretation of x, then the self-consistency score (I( is defined as the cosine similarity between the transformation of the interpretation the transformed images Tx(x) as UI( = 2x=_ (Txx)() where S(. the cosine similarity. The expectation
and the interpretation to
the transformation family TB is approximated by the Monte Carlo sampling method. However, note that in practice T3 (xx)) transforms the interpretation directly and will introduce zero padding in the corner of interpretation heatmaps. (xTx(x)) transforms the input data before the prediction so that the interpretation is not padded. To eliminate the influence from the padded area, we introduce a transformation mask my {0, 1) where m' = 0 for the padding area of Ta (x), and m = otherwise. Thus the conssccoyyyyyncy score is calculated by (I) Exx (mm C TT([(x)))
m2 O (xTx(x))).
(8)
4 Experiments
In this section, we conduct experiments on image classification tasks with and without transformations. The experiment results demonstrate the high-quality interpretations and the validity of SITE. Please refer to the Appendix for more details about the experimental setup.
4.1 Experiments on MNIST
First, we implement SITE on MNIST dataset. Since SITE Go Fi shares the same backbone structure F1 with the traditional classifier F F2 o F,, we clarify that SITE does not sacrifice prediction power for interpretability, Please refer to Sec. 4.3 for more details.
The interpretations of SITE on MNIST are shown in Fig. The Hadamard product decides that the interpretations are essentially the pixel-wise similarities between the input digit x and prototype w; The interpretation to each prototype can be treated how and where do the prototype w, and look similar. Therefore, x will be classified to the class where the prototype is the most similar to the input data. Besides, we can observe that the interpretations of SITE preserve good transformation equivariance property thanks to the transformation regularization. The interpretations are transformed automatically with the input data while preserving the shape of the highlighted region.
4.2 Experiments on CIFAR-10
Interpretations of SITE
For CIFAR-I0, input fed the feature extractor Fi. Then the generator G takes the latent representation z as input and generates the = 10 data-dependent prototypes [wr)e Then the visualizable interpretation of the input is defined // ([w,. where is the bilinear upsampling,
and i' arg sise w. z is the predicted class for input
The interpretation results are of SITE on CIFAR-10 are shown Fig. 3 Here we sample three images of plane, dog, and car for demonstrations. Each image is transformed by a constrained

---

011
0008
0006
-004
-002
0.00
Figure 3: Interpretations of SITE on CIFAR-10 dataset. The first row shows the original images (odd columns) and their random affine transformed version (even columns). The second row shows the interpretation heatmaps overlaid on corresponding images.
affine transformation T3 To that sampled independently. And in the bottom row, we overlay the interpretations (w,, z) corresponding images. It can be found clearly that SITE successfully highlights the main parts of objects on sampled images comprehensible way to humans. For instance, in the dog image, SITE highlights the silhouette of the dog in both transformed and untransformed images. And comparing the odd columns and the even columns, it's clear that the interpretability of SITE preserves great self- self-consistency during transformations [9 This can be treated as the robustness to transformations. Please refer to the Appendix for more examples. Besides, would like to clarify that SITE does not sacrifice expressive power for interpretations. We take the ResNet-18 18 classifier as the benchmark since SITE takes the same structure as the feature extractor. Given the same transformation family TB. SITE and ResNet-18 backbone model achieve comparable validation accuracy of 89% on randomly transformed images. And on untransformed
images, SITE even demonstrate higher expressiveness, Please refer Sec. 4.3 for more details.
Comparison with Post-Hoc Methods
We carry out comparison experiments with various attribution methods that interpret feature contribu- tions to the prediction results. The comparing methods include: back- sockppropagation methods such as Brad-CAM [1,, excitation back baok-propag [59], guided back- propagation [47), gradient [46], De- ConvNet [58], and linear approximation. And also there perturbation methods such randomized input sampling (RISE) (32 and extremal perturbation (EP) [1]. To illustrate the comparison results heatmaps of the same settings visualize the interpretations of all methods. Since the interpretations of different models are obtained in very different ways, the visualizations of them are performed separately. Hence the heatmaps only demonstrate the relative importance of pixels within each interpretation itself. The comparison of the interpretation results is shown in Fig. 4 where we present the interpretations the predictions of the given image, respectively. First, we clarify that for the sake of consistency, all post-hoc interpretations shown in Fig. 4 are obtained by applying the post-hoe interpretation models mentioned above to SITE. Since SITE is trained on the transformed dataset, all post-hoc interpretations are reasonable to the transformed image. However, their interpretations of the untransformed image are affected by the transformation. SITE does the best in capturing the main body of the ship in the untransformed image. It also preserves the best self-consistency. fact, according to the self-consistency scores (x over the whole validation set of CIFAR-10, 10, SITE outperforms all post-hoc methods, and is thereby more robust to transformations.
consistently,
The comparison of self self-consistency scores is shown in Table I Due to the inefficient computation of perturbation methods, here omit the calculations of RISE method and extremal perturbation method. For completeness, We also include the self-consistency scores the post-hoc methods on the backbone model (ResNet-18). It is also trained on the transformed training set.
Finally, we carry out the mask- k dixiless experiments to demonstrate the equivariance SITE as a self interrrrreee model. This experiment is implemented by masking k pixels of the input data based on the interpretations provided. For each interpretation model, we obtain a series of masked subset of the original dataset based on the interpretations. Here sample the first 1000 images from the validation set of CIFAR-10, and perform the mask-k-pixels experiments on all interpretation methods mentioned above except for the two perturbation methods. Besides, also add the case
7

---

(a) Orig. Image
(b) SITE
(c) Grad-CAM
(d) Guided BP
(c) Excitation BP
(f) Trans. Image
(g) SITE
(h) Grad-CAM
(i Guided BP
(i) Excitation BP
(k) DeConvNet
(l) Lnr. Approx.
(m) Gradient
(n) RISE
(o) EP
(p) DeConvNet (q) Lnr. Approx.
(r) Gradient
(s) RISE
(Ú EP
Figure 4: Interpretation comparison CIFAR-10 10 dataset. The odd rows show the interpretations on the original image. while the even rows show the interpretations on the transformed image. Note that the model is trained on transformed images, thus all interpretations on the transformed images are relatively reasonable. However, most interpretations are highly disturbed on untransformed image, while SITE preserves the most transformation equivariant interpretations.
Table Self-consistency scores x (7) E [-1,1] of interpretation methods. A higher score indicates better self-consistency. The first is the scores for SITE, and the second row the scores for the backbone model (ResNet-18) 18). Both models are trained on transformed data. Perturbation methods RISE and EP are omitted in this experiment due the inefficient computation.
SITE
Grad-CAM
Guided BP
Excitation BP Gradient
Linear Approx.
DeConvNet
ex (SITE)
0.8860
0.8817
0.7830 0.8168
0.1159 0.2460
0.7174 0.6926
0.8485 0.4183
0.8591
ox (backbone)
0.8416
0.7721
where pixels are randomly masked. In order demonstrate the transformation equivariance, here we mask the least & important pixels according to the interpretations to untransformed images, and feed the random transformations of the masked images the classifier. We present the trend of the prediction accuracy and the log-odds ratio (LOR) of the predicted logits to the true classes Fig. 5(a). It is expected that the more slowly curve drops, the better equivariance the interpretation possesses. Hence, can be found that SITE outperforms all other post-hoc interpretation methods. Besides, observe that when very few pixels (< 20%) masked, the decrease of SITE is almost negligibly faster than some post-hoc methods. We explain this phenomenon by presenting a typical example in Fig. b). Generally, the least important pixels are located at the corners, therefore. those masks are eliminated when the corners are hidden after the transformation, as shown in the top two images Fig. ()b). This results in almost mask at the beginning. As the proportion of masked pixels increase, this phenomenon is gradually alleviated, as shown in the other four images in Fig. (]))). Furthermore, we validate the faithfulness of SITE compared with post-hoc methods on Benchmarking Attribution Methods (BAM) dataset [57] Please refer to the Appendix for details.

---

STF CredCAM
0.00
Sisted HP
38 prode
-022
.9.4 Gred an!
Linier Adprox DeComviatt
Predier Fon accuracy
Random
grodes maskked
-0.75
-1.25
138 Sodde min Int
-1.50
0.3
0.2
-2.00
c.4 Propotion of pixcis masked
Propotion of pixats masked
Transfo med
(a) Mask-k Experiments
(b) Sampled Masked Images
Figure 5: (a) The trend of accuracy (left) and log-odds ratio (right) with various proportions of pixels masked; (b) Images where Ik (top), 3k (middle) and 1 Ik (bottom) pixels (out of 16.4k pixels) are masked. The left and right columns are for the original and the transformed images, respectively.
Table 2: Accuracy comparison among self-interpretable models. We implement SITE on different backbone models and show the performance of the black-box backbone in parenthesis.
Decision Tree
Random Forest
Logistic Regression
SITE-CNN (Blackbox CNN)
SITE-ResNet (Blackbox ResNet)
XGBoost
NAM
MNIST
0.886 0,229
0.970
0.929
0.975
0.935
0.988 (0.981)
CIFAR-10
0.396
0.357
0.450
0.370
0,840 (0.828)
0.892 (0.862)
4.3 Expressiveness
Since there is inevitable trade-off between expressiveness and interpretability. most existing self-interpretable models have relatively low accuracy on image datasets such as MNIST and CIFAR. Here we compare the expressiveness of SITE and existing self-interpretable models including simple models (trained using sklearn) like Decision Tree, Random Forest, Logistic Regression, and complex models like XGBoost [9] (trained using xgboost), Neural Additive Model (NAM) (2] The results of XGBoost are reported in [33. We include SITE with backbones of different levels of complexity to demonstrate the scalability of SITE. The CNN backbone contains 235k parameters for MNIST and 1 1.2m parameters for CIFAR-10, while the ResNet backbone is the ResNet- used in all previous experiments. The backbone models share the same structures and the same (transformed) training data as the corresponding SITE feature extraction. The test is performed on the untransformed validation set. As shown in Table SITE outperforms all other self-interpretable models by a large margin. It has even higher expressiveness than the backbone model. We give this credit to the regularization the self-interpretation 5
Conclusions
In this paper, we propose a self-interpretable model SITE with transfornation-cquivariaaa inter- pretations. We focus the robustness and self-consistency of the interpretations of geometric transformations. Apart from the transformation equivariance, as a self-interpretable model, SITE has comparable expressive power the benchmark black-box classifiers, while being able to present faithful and robust interpretations with high quality. It is worth noticing that although applied in most of the CNN visualization methods, the bilinear upsampling approximation is rough approximation. which can only provide interpretations in the form of heatmaps (instead of pixel-wise). It remains an open question whether such interpretations can be direct to the input space (as shown in the MNIST experiments). Besides, we consider the translation and rotation transformations in our model. In future work, we will explore the robust interpretations under more complex transformations such as scaling and distortion. Moreover, we clarify that SITE is not limited to geometric transformant mransformation (that we used in the computer vision domain), and will explore SITE other domains future work.

---

Acknowledgement
This work was partially supported by NSF IIS #1955890, Purdue's 's Elmore ECE Emerging Frontiers Center. The applications of various post-hoc methods are implemented through the TorchRay toolkit. We are grateful to the anonymous NeurIPS reviewers for the insightful comments.
References
[1] Adebayo, J. Gilmer, M. Muelly. I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. arXiv preprint arXiv: 11.800003992 2018.
[2] R. Agarwal, N. Frosst, X. Zhang. R. Caruana, and G. E. Hinton. Neural additive models: Interpretable machine learning with neural nets. arXiv preprint arXiv:2004.13912, 2020.
[3] D. Mevase and T. S. Jaakkola. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7786-7795, 2018.
[4] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7):e0130140, 2015.
[5] A. Boopathy, S. Liu, G. Zhang, C. Liu, P.-Y. Chen, S. Chang, and I.. Daniel. Proper network interpretability helps adversarial robustness in classification. In International Conference on Machine Learning, pages 1014-1023, PMLR, 2020.
[6] C. Chen, O. Li, C. Tao, A. J. Barnett, J. Su, and C. Rudin. This looks like that: Deep learning for interpretable image recognition. arXiv preprint arXiv: 1806. 10574, 2018
[7] H. Chen, X. Wang, C. Deng, and H. Huang. Group sparse additive machine. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 197-207, 2017.
[8] Chen, L. Song. M. Wainwright, and M. I. Jordan. L-shapley and e-shapley: Efficient model interpretation for structured data. arXiv preprint 180iv:1808.000 026600, 2018.
[9] т. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[10] x. Cheng, Q. Qiu, R. Calderbank, and G. Sapiro. Rotdef: Decomposition of convolutional filters for ooation-equivarial deep networks. In International Conference on Learning Repre- sentations, 2018
[11] B. Chidester, T. Zhou, M. N. Do, and J. Ma. Rotation equivariant and invariant neural networks for microscopy image analysis. Bioinformatics, 35(14) 35(14)i530-i537, 2019.
[12] F. Bossi--Vler and B. Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: / 100.008608 2017.
[13] M. Du, N. Liu, and X. Hu. Techniques for interpretable machine learning. Communications of the ACM, 63(1) (3(1):68 (:80):6877-8 2019.
[14] R. Fong, M. Patrick, and A. Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2950-2958. 2019.
[15] G. E. Hinton, A. Krizhevsky, and S. D. Wang. Transforming auto-encoders. In International Conference Artificial Neural Networks, pages 44-51. Springer, 2011.
[16] M. Jaderberg, K. Simonyan, Zisserman, and K. Kavukcuoglu. Spatial transformer networks. arXiv preprint arXiv:1506.022 02025, 2015.
10

---

[17] S. Jain, S. Wiegreffe, Y. Pinter, and B. C. Wallace. Learning to faithfully rationalize by construction. arXiv preprint arXiv:2005.00115 2020.
[18] J. Kim and J. Canny. Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE International Conference on Computer Vision, pages 2942-2950, 2017.
[19] P.-. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 267-280. Springer, 2019.
[20] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
[21] I. Lage, A. S. Ross, B. Kim, S. J. Gershman, and F. Doshi-Veless Human-in-the inter- pretability prior. arXiv preprint arXiv:1805.115777 2018.
[22] H. Lakkaraju, S. H. Bach, and J. Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1675-1684, 2016.
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
[24] S. M. Lundberg and S.-l. Lee. A unified approach to interpreting model predictions. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 17768-777, 2017.
[25] H. Maron, O. Litany, G. Chechik, and E. Fetaya. On learning sets of symmetric elements. In International Conference on Machine Learning, pages 6734-6744 PMLR, 2020.
[26] A. K. Mohankumar, P. Nema, S. Narasimhan, M. M. Khapra, B. V. Srinivasan, and B. Ravindran. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4206-4216, 2020.
[27] A. K. Mondal, P. Nair, and K. Siddiqi. Group equivariant deep reinforcement learning, arXiv preprint arXiv:2007.00 03437, 2020.
[28] T. Monnier, T. Groucix, and M. Aubry. Deep cecafformation-ivvaaiin clustering. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 7945- 7955, 2020.
[29] A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, and D. J. Rezende. Towards interpretable reinforcement learning using attention augmented agents, arXiv preprint 19Xiv:1906.025000 2019.
[30] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications interpretable machine learning. Proceedings the National Academy of Sciences, 1 116(44):22011
2019.
[31] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Interpretable machine learning: Definitions, methods, and applications. arXiv preprint arXiv:1901.04597 2019.
[32] V. Petsiuk, A. Das, and K. Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint /0X14 2018.
[33] N. Ponomareva, T. Colthurst, G. Hendry, S. Haykal, and S. Radpour. Compact multi-class boosted trees. In 2017 IEEE International Conference on Big Data, pages 47-56. IEEE, 2017.
[34] G.-J. Qi, L. Zhang. F. Lin, and X. Wang. Learning generalized transformation equivariant representations via autoencoding transformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2020.

---

[35] I. Rahwan, M. Cebrian, N. Obradovich, J. Bongard, J.-F. Bonnefon, C. Breazeal, J. W. Crandall, N. A. Christakis, I. D. Couzin, M. O. Jackson, et al. Machine behaviour, Nature, 568(7753):477- 486, 2019.
[36] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv: 2021.
[37] S.-A. Rebuffi, R. Fong, X. Ji, and A. Vedaldi. There and back again: Revisiting backpropagation saliency methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8839-8848, 2020.
[38] M. T. Ribeiro, S. Singh, and C. Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144, 2016.
[39] L. Rieger, C. Singh, W. Murdoch, and B. Yu. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In International Conference on Machine Learning, pages 8116-8126. PMLR, 2020.
[40] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, ((5):206-215, 2019.
[41] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626 2017.
[42] S. Serrano and N. A. Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, 2019.
[43] Y. Shen, C. Yang, X. Tang, and B. Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2020.
[44] C. Shorten and T. M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
[45] A. Shrikumar, Greenside, and A. Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145-3153. PMLR, 2017.
[46] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.608: 2013.
[47] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 14/2,680, 2014.
[48] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.
[49] K. S. Tai, P. Bailis, and G. Valiant. Equivariant transformer networks. In International Conference Machine Learning, pages 6086-6095. PMLR, 2019.
[50] S. Teso. Toward faithful explanatory active learning with self-explainable neural nets. In Proceedings of the Workshop on Interactive Adaptive Learning, pages 4-16 CEUR Workshop Proceedings, 2019.
[51] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor field networks: Rotation-ana and -equaaiiiiniuuviiaintt neural networks for 3d point clouds. arXiv preprint arXiv: 0e..18002881111 2018
[52] H. Wang, Z. Wang, M. Du F. Yang, Z. Zhang, S. Ding, Mardziel, and X. Hu. Score-cam: Score- weighted visual explanations for convolutional neural networks. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition Workshops, pages 24-25, 2020.
12

---

[53] R. Wang. X. Wang, and D. I. Inouye. Shapley explanation networks. In International Conference on Learning Representations, 2020.
[54] X. Wang, H. Chen, Yan, K. Nho, S. Risacher, A. J. Saykin, L. Shen, H. Huang, and ADNI. Quantitative trait loci identification for brain endophenotypes via new additive model with random networks. Bioinformatics, 34(17xi866-i874, 2018.
[55] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10402-10413, 2018.
[56] D. E. Worrall, S. J. Garbin, D. Turmukhaabetov,, and G. J. Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[57] M. Yang and B. Kim. Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv: / /900 1907009707, 2019.
[58] M. D. Zeiler and Fergus. Visualizing and understanding convolutional networks. In European Conference Computer Vision, pages 818-833. Springer, 2014.
[59] J. Zhang, S. A. Bargal, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084-11 2018.
[60] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
[61] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-144 14644, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] We will provide the code and instructions at request. The data used in the experiments are public available. See the Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the beginning of Section 4
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
13

---

(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "i2XzzcYc36T", "reviewer": "Reviewer_mRVR", "summary": "This paper proposes a self-interpretable model for images, which can produce interpretations robust to geometric transformations by learning it with transformation equivalent regularization.\nThe contributions of the paper are to develop the aforementioned model with high-quality faithful and robust interpretation, while preserving high expressive power, and to propose a new quantitative metric for interpretation method, called self-consistency score.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe main originality of the paper is to propose to learn a self-interpretable model to be robust to geometric transformation of images.\nThe essence of the proposed model is in Eq. (7), and the proposed model simply uses the geometric transformation introduced in [16] in the equation, \nThe formulation of the proposed model may be new, but it seems that the formulation is a straightforward combination of the existing techniques.\n\n### Quality\n\nI think the proposed model is technically sound, except for some points stated in Clarity section.\nAlso, the experimental result shows that the proposed model can produce appropriate interpretations for geometrically transformed images as expected.\n\nHowever, I'm concerning about the superiority of the proposed model compared to the post-hoc interpretation methods.\nI think that if the backbone network was trained on data with data augmentation including geometric transformation, the post-hoc interpretation methods can produce interpretations as well as the proposed model.\nHowever, Figure 4 shows that some of the comparing methods are sensitive to the geometric transformation.\nIs the backbone network used in the comparing post-hoc interpretation methods the same as that of the proposed model? \nIf this is true, was the backbone network trained on data with data augmentation including geometric transformation?\n\nIn addition, I have another concern about the computational efficiency of the proposed model.\nIn Eq (7), it seems that the computational cost of the second term is very high.\nThe authors say \"In practice, the expectation over xi \\in Xi and the expectation over B can be properly approximated by Mont Carlo sampling.\", but I could not find the explanation of how many samples need to approximate the expectation.\nIn the experiment, how many samples are used in the computation of the expectation? Also, did you investigate the tradeoff between the number of the samples and performances such as self-consistency score and accuracy?\n\n### Clarity\n\nThe paper is well-organized and easy to follow.\nHowever, I think the following points are unclear.\n\nIn the second term of Eq. (7), why is binary cross entropy used for measuring discrepancy of two continuous vectors in the feature space? \nIntuitively, I think that Euclid distance is the first choice for measuring the discrepancy.\n\nFor transformation $T_\\beta$, $T_\\beta^{-1}$ is not exactly inverse of $T_\\beta$, because $T_\\beta$ transforms a point to another point in the original input space, while $T_\\beta^{-1}$ transforms a point in $\\mathbb{R}^d$ to a point in the space of $\\mathbf{z}$.\nThe authors should state the difference between $T_\\beta$ and $T_\\beta^{-1}$.\n\nIn Section 3.2, the authors state that the interpretations can be visualized by bilinear upsampling.\nThis seems that $F_1$ is assumed to be the function that return the output of the convolution layer in CNN.\nWhen extracting features from images using CNN, one often uses the output of the fully-connected layer in the CNN. \nTo avoid misreading, I think the authors should clearly state which layers in the CNN can be used for $F_1$.\n\n### Significance\n\nIn the experiment, it seems that the proposed model achieves better accuracy and interpretability.\nHowever, I think that the comparison with the existing models in the following perspectives is not enough.\n\nIs the backbone network used for the comparing post-hoc interpretation methods trained with enough data augmentation including geometric transformation?\n\nWhy was not the proposed model compared with attention-based methods, such as class activation map (CAM)? Since both the proposed model and the attention-based methods do prediction and interpretation in a single model, I think that the authors should compare the proposed model with at least one attention-based method in the experiment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned the limitations of the study.\nThere is no mention about the potential negative societal impact, but I think that's OK.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8h"}, "extra_text": {}}, {"review_id": "MQWT4MZBno3", "reviewer": "Reviewer_FoQp", "summary": "This paper presents an approach that learns a 'self-interpretable' model, which produces explanations that are invariant to certain user defined transformations. The attribution maps of traditionally trained models change when certain transformations (like rotation etc) are applied to the input, which could be seen as a disadvantage of these methods. Here the paper counteracts this by learning models that are self-interpreting (via higher level prototypes) and whose attribution maps are invariant to user-defined attributions.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThis paper tackles an original problem within the model/dnn interpretability literature. Self-explaining models have been proposed in the past, however, none of that previous work has emphasized equivariance, which is the primary focus of this work. The paper also proposes a self-consistency score for explanations. As far as I am aware, this score is novel and could be used in future work.\n\n### Quality\nThe paper tackles an important problem. However, I think the paper conflates two problems at once. First, the need for explanations to be robust to transformations is important; however, this only matters *if* the model in question is robust to such transformations. If a model is not robust to a transformation, then we shouldn't expect explanations from that model to also be robust to transformations well. The formulation proposed should also be compared to self-interpretable variants instead of just post hoc explanations alone.\n\n### Clarity\nThe paper is relatively easy to follow and free of major typos, but certain references could have been made more clear in different places. I\"ll go into detail on this at the end of this review.\n\n### Significance\nThe insight that attributions are not transformation invariant is significant and one that I believe future work will address as well. Equivariant self-interpretable networks is also an important and interesting direction as well. As it stands these two ideas are somewhat muddled in this work, but it could be that with better presentation the message of this work would be quite significant. Figure 3 and 4 are quite nice illustrations of the benefits of the proposed approach in practice.\n\n### Additional Comments/Questions\n1) The self interpretable claim. This paper claims that the prototypes that are output by the presented network are self-interpretable, but it does not test or justify this claim. While other papers might have done this as well, it is not clear that if the prototypes derived in this work were presented to an end-user they would find it interpretable. \n\n2) Why should we expect transformation invariant attribution from models that are not themselves transformation invariant? We now know that if one rotates an image and such image is presented to a network that has not learned such rotation transformation, then the prediction of the model can change as well. For such models one would expect the attributions to also not be rotation invariant. In the caption of figure 4. it says that the model is trained on transformed images ; however, does the prediction of the model stay the same even when these transformations are applied to the test image(s)?\n\n3) Comparison to attribution methods. While I understand that the attribution methods that the authors compare to allow them to demonstrate the effectiveness of their method, I think self-interpretable methods are the better comparisons here. See the work: \"This looks like that\" by Chen et. al. and Concept Bottleneck Models by Koh et. al. These approaches should form better comparisons for this work in addition to the post hoc methods.\n\n4) Interpreting the self-consistency score. The idea of this score is interesting; however, based on table 1 it is a bit difficult to understand or contextualize what these scores should mean. Excitation BP seems quite low, but the rest are above 0.5...does that mean that these methods are reliable? Figure(s) 3 and 4 say otherwise, so could it be that the score doesn't not really help distinguish reliable from unreliable methods?\n\n5) Section 3.1 stops without going into detail about how the model is actually trained. This information is important even if it is with sgd type methods with standard autodiff. This would help with future replication.\n\n#### Minor typos & Writing Feedback\nLine 193: Mont Carlo -> Monte Carlo\nLine 177: instead of referencing [16], it would've been better to discuss or mention the specific transformations you are referring to here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should include a more explicit limitations section in their paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "JkfY_76E_Xz", "reviewer": "Reviewer_zd9s", "summary": "In applications, the authors show that the model does not sacrifice its predictive accuracy compared to less interpretable models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors introduce an interesting idea for making a self-interpretable model that can achieve reasonably good predictive performance.\n\nThe authors do a good job motivating why self-interpretable models are useful, i.e. that \"self-interpretable models make interpretations at the same time of predictions, thus reveal the intrinsic mechanism of the models, and are thereby preferred by users in high-stakes tasks.\"\n\nThe authors then propose to follow *interpretation transformation equivariance*. This often seems quite reasonable, but the authors should note when this is a desirable property. For example, if the introduced transformations introduce artifacts into the image (e.g. large swatches of black background in the cifar images), it may no longer be reasonable to assume that the prediction mechanism should not rely on these regions.\n\nMy main criticism of the is work is that the authors should also reference (and maybe compare to) this recent paper on [self-interpretable neural networks](https://arxiv.org/abs/1806.10574) (chen et al. 2018) which use prototypes at a granular level. This papers also helps to provide localization which can provide transformation equivariant interpretation.\n\nThe author's might also want to discuss in more detail their choice to use bilinear upsampling to visualize $\\hat w$ - is this the best choice? Did the authors try any other visualizations, e.g. learning an inverse function directly?\n\nMinor points:\n\n- there are minor typos and grammatical errors throughout the text that could be improved.\n- Fig 5a xlabel has typo", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not seem to *explicitly* address limitations and societal impact, although both come across as reasonable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "E56RgGnXA2", "reviewer": "Reviewer_XcMQ", "summary": "The paper proposes SITE Self-Interpretable model with Transformation Equivariant Interpretation.\n\nFeatures are first extracted by feature extraction network such as CNN/Resnet F(x) such that F(x)=z SITE uses generative model G that maps z to c prototypes each corresponds to a specific class. y_hat = softmax(G(z)^T x) where y_hat_i = softmax(w_i^Tz) so  w_i is the interpretation of the i-th prediction. The paper then proposes to to regularize the prototypes for an input image x, they enforce each generated prototype to be similar to its corresponding class’s latent representation. They also propose to regularize on the transformation equivariance property of interpretation produced by the model.\n\nThe paper also proposed self-consistency score to compare different interpretation methods. For an input x and transformation T and interpretation I(x) the self-consistency score is defined as the cosine similarity between the transformation of the interpretation to x and the interpretation of the transformed images cosine similarity(T(I(x)),I(T(x)) . Higher indicates a more robust interpretation.\n\nThe paper compared SITE interpretation with post hoc interpretability methods and the accuracy maintained by SITE with that of inherently interpretable architecture that exhibit a decrease in accuracy due to their interpretable nature. Comparison was done on MNIST and CIFAR10.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The use of transformation for interpretation is original and novel.\n- There is a clear advance in terms of interpretation robustness when comparing SITE to post hoc methods.\n- In comparison to inherently interpretable architectures SITE was able to maintain model accuracy while providing reasonable interpretation.\n- The paper is well written and clear.\n- The paper is technically sound. \n- The proposed Self-consistency score can be used to compare different methods in the future.\n\nWeakness:\n- SITE was not compared with other self explaining networks such as SENN [1], INVASE [2], FRESH [3].\n- Experiments were done on pretty basic datasets MNIST, CIFAR; will SITE be able to provide good explanations while maintaining accuracy for on more realistic datasets like imagenet, food101, and bird260.\n- SITE does not provide feature-based explanations which are often desired, this makes applying SITE on other domains such as language and time-series not straightforward.\n- Computational cost of SITE compared to training a blackbox Resnet was not discussed.\n- Masking and measuring accuracy drop can result in false results since the accuracy drop might be the result of the masking not the removing of informative/non-informative features[4].\n- The relationship to previous work in Self-Interpretable Models is not clearly stated.\n\n[1] David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 7775–7784, 2018\n\n[2] Yoon, Jinsung, James Jordon, and Mihaela van der Schaar. \"INVASE: Instance-wise variable selection using neural networks.\" International Conference on Learning Representations. 2018.\n\n[3] Jain, Sarthak, et al. \"Learning to faithfully rationalize by construction.\" arXiv preprint arXiv:2005.00115 (2020).\n\n[4] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, 2019.\n\nThe main reason behind my score is the limited empirical results only very simple datasets were used and other self explanatory networks were not compared. \n\n\n\n\n\n\n################# Post rebuttal\nThank you for addressing my concerns. Given the author response I am happy to increase my score 5->6", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of training self-interpretable models. It first shows that existing approaches change the interpretation with input transformations even though the model is invariant to them. It then proposes a new method that fixes this problem. While the reviewers initially had concerns about the empirical evaluation, the rebuttal clarified most of the issues and all of them unanimously recommend acceptance. So, I am recommending the paper for acceptance. At the same time, there are also some concerns such as comparison with Chen et al. work that the rebuttal did not address properly. I would ask the authors to fix this and other issues raised in the reviews below in the final version.
Decision: Accept (Poster)