Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes SITE Self-Interpretable model with Transformation Equivariant Interpretation.

Features are first extracted by feature extraction network such as CNN/Resnet F(x) such that F(x)=z SITE uses generative model G that maps z to c prototypes each corresponds to a specific class. y_hat = softmax(G(z)^T x) where y_hat_i = softmax(w_i^Tz) so  w_i is the interpretation of the i-th prediction. The paper then proposes to to regularize the prototypes for an input image x, they enforce each generated prototype to be similar to its corresponding class’s latent representation. They also propose to regularize on the transformation equivariance property of interpretation produced by the model.

The paper also proposed self-consistency score to compare different interpretation methods. For an input x and transformation T and interpretation I(x) the self-consistency score is defined as the cosine similarity between the transformation of the interpretation to x and the interpretation of the transformed images cosine similarity(T(I(x)),I(T(x)) . Higher indicates a more robust interpretation.

The paper compared SITE interpretation with post hoc interpretability methods and the accuracy maintained by SITE with that of inherently interpretable architecture that exhibit a decrease in accuracy due to their interpretable nature. Comparison was done on MNIST and CIFAR10.
main_review: Strength:
- The use of transformation for interpretation is original and novel.
- There is a clear advance in terms of interpretation robustness when comparing SITE to post hoc methods.
- In comparison to inherently interpretable architectures SITE was able to maintain model accuracy while providing reasonable interpretation.
- The paper is well written and clear.
- The paper is technically sound. 
- The proposed Self-consistency score can be used to compare different methods in the future.

Weakness:
- SITE was not compared with other self explaining networks such as SENN [1], INVASE [2], FRESH [3].
- Experiments were done on pretty basic datasets MNIST, CIFAR; will SITE be able to provide good explanations while maintaining accuracy for on more realistic datasets like imagenet, food101, and bird260.
- SITE does not provide feature-based explanations which are often desired, this makes applying SITE on other domains such as language and time-series not straightforward.
- Computational cost of SITE compared to training a blackbox Resnet was not discussed.
- Masking and measuring accuracy drop can result in false results since the accuracy drop might be the result of the masking not the removing of informative/non-informative features[4].
- The relationship to previous work in Self-Interpretable Models is not clearly stated.

[1] David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 7775–7784, 2018

[2] Yoon, Jinsung, James Jordon, and Mihaela van der Schaar. "INVASE: Instance-wise variable selection using neural networks." International Conference on Learning Representations. 2018.

[3] Jain, Sarthak, et al. "Learning to faithfully rationalize by construction." arXiv preprint arXiv:2005.00115 (2020).

[4] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, 2019.

The main reason behind my score is the limited empirical results only very simple datasets were used and other self explanatory networks were not compared. 






################# Post rebuttal
Thank you for addressing my concerns. Given the author response I am happy to increase my score 5->6
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents an approach that learns a 'self-interpretable' model, which produces explanations that are invariant to certain user defined transformations. The attribution maps of traditionally trained models change when certain transformations (like rotation etc) are applied to the input, which could be seen as a disadvantage of these methods. Here the paper counteracts this by learning models that are self-interpreting (via higher level prototypes) and whose attribution maps are invariant to user-defined attributions.
main_review: ### Originality
This paper tackles an original problem within the model/dnn interpretability literature. Self-explaining models have been proposed in the past, however, none of that previous work has emphasized equivariance, which is the primary focus of this work. The paper also proposes a self-consistency score for explanations. As far as I am aware, this score is novel and could be used in future work.

### Quality
The paper tackles an important problem. However, I think the paper conflates two problems at once. First, the need for explanations to be robust to transformations is important; however, this only matters *if* the model in question is robust to such transformations. If a model is not robust to a transformation, then we shouldn't expect explanations from that model to also be robust to transformations well. The formulation proposed should also be compared to self-interpretable variants instead of just post hoc explanations alone.

### Clarity
The paper is relatively easy to follow and free of major typos, but certain references could have been made more clear in different places. I"ll go into detail on this at the end of this review.

### Significance
The insight that attributions are not transformation invariant is significant and one that I believe future work will address as well. Equivariant self-interpretable networks is also an important and interesting direction as well. As it stands these two ideas are somewhat muddled in this work, but it could be that with better presentation the message of this work would be quite significant. Figure 3 and 4 are quite nice illustrations of the benefits of the proposed approach in practice.

### Additional Comments/Questions
1) The self interpretable claim. This paper claims that the prototypes that are output by the presented network are self-interpretable, but it does not test or justify this claim. While other papers might have done this as well, it is not clear that if the prototypes derived in this work were presented to an end-user they would find it interpretable. 

2) Why should we expect transformation invariant attribution from models that are not themselves transformation invariant? We now know that if one rotates an image and such image is presented to a network that has not learned such rotation transformation, then the prediction of the model can change as well. For such models one would expect the attributions to also not be rotation invariant. In the caption of figure 4. it says that the model is trained on transformed images ; however, does the prediction of the model stay the same even when these transformations are applied to the test image(s)?

3) Comparison to attribution methods. While I understand that the attribution methods that the authors compare to allow them to demonstrate the effectiveness of their method, I think self-interpretable methods are the better comparisons here. See the work: "This looks like that" by Chen et. al. and Concept Bottleneck Models by Koh et. al. These approaches should form better comparisons for this work in addition to the post hoc methods.

4) Interpreting the self-consistency score. The idea of this score is interesting; however, based on table 1 it is a bit difficult to understand or contextualize what these scores should mean. Excitation BP seems quite low, but the rest are above 0.5...does that mean that these methods are reliable? Figure(s) 3 and 4 say otherwise, so could it be that the score doesn't not really help distinguish reliable from unreliable methods?

5) Section 3.1 stops without going into detail about how the model is actually trained. This information is important even if it is with sgd type methods with standard autodiff. This would help with future replication.

#### Minor typos & Writing Feedback
Line 193: Mont Carlo -> Monte Carlo
Line 177: instead of referencing [16], it would've been better to discuss or mention the specific transformations you are referring to here.
limitations_and_societal_impact: The authors should include a more explicit limitations section in their paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a self-interpretable model for images, which can produce interpretations robust to geometric transformations by learning it with transformation equivalent regularization.
The contributions of the paper are to develop the aforementioned model with high-quality faithful and robust interpretation, while preserving high expressive power, and to propose a new quantitative metric for interpretation method, called self-consistency score.
main_review: ### Originality

The main originality of the paper is to propose to learn a self-interpretable model to be robust to geometric transformation of images.
The essence of the proposed model is in Eq. (7), and the proposed model simply uses the geometric transformation introduced in [16] in the equation, 
The formulation of the proposed model may be new, but it seems that the formulation is a straightforward combination of the existing techniques.

### Quality

I think the proposed model is technically sound, except for some points stated in Clarity section.
Also, the experimental result shows that the proposed model can produce appropriate interpretations for geometrically transformed images as expected.

However, I'm concerning about the superiority of the proposed model compared to the post-hoc interpretation methods.
I think that if the backbone network was trained on data with data augmentation including geometric transformation, the post-hoc interpretation methods can produce interpretations as well as the proposed model.
However, Figure 4 shows that some of the comparing methods are sensitive to the geometric transformation.
Is the backbone network used in the comparing post-hoc interpretation methods the same as that of the proposed model? 
If this is true, was the backbone network trained on data with data augmentation including geometric transformation?

In addition, I have another concern about the computational efficiency of the proposed model.
In Eq (7), it seems that the computational cost of the second term is very high.
The authors say "In practice, the expectation over xi \in Xi and the expectation over B can be properly approximated by Mont Carlo sampling.", but I could not find the explanation of how many samples need to approximate the expectation.
In the experiment, how many samples are used in the computation of the expectation? Also, did you investigate the tradeoff between the number of the samples and performances such as self-consistency score and accuracy?

### Clarity

The paper is well-organized and easy to follow.
However, I think the following points are unclear.

In the second term of Eq. (7), why is binary cross entropy used for measuring discrepancy of two continuous vectors in the feature space? 
Intuitively, I think that Euclid distance is the first choice for measuring the discrepancy.

For transformation $T_\beta$, $T_\beta^{-1}$ is not exactly inverse of $T_\beta$, because $T_\beta$ transforms a point to another point in the original input space, while $T_\beta^{-1}$ transforms a point in $\mathbb{R}^d$ to a point in the space of $\mathbf{z}$.
The authors should state the difference between $T_\beta$ and $T_\beta^{-1}$.

In Section 3.2, the authors state that the interpretations can be visualized by bilinear upsampling.
This seems that $F_1$ is assumed to be the function that return the output of the convolution layer in CNN.
When extracting features from images using CNN, one often uses the output of the fully-connected layer in the CNN. 
To avoid misreading, I think the authors should clearly state which layers in the CNN can be used for $F_1$.

### Significance

In the experiment, it seems that the proposed model achieves better accuracy and interpretability.
However, I think that the comparison with the existing models in the following perspectives is not enough.

Is the backbone network used for the comparing post-hoc interpretation methods trained with enough data augmentation including geometric transformation?

Why was not the proposed model compared with attention-based methods, such as class activation map (CAM)? Since both the proposed model and the attention-based methods do prediction and interpretation in a single model, I think that the authors should compare the proposed model with at least one attention-based method in the experiment.
limitations_and_societal_impact: The authors mentioned the limitations of the study.
There is no mention about the potential negative societal impact, but I think that's OK.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In applications, the authors show that the model does not sacrifice its predictive accuracy compared to less interpretable models.
main_review: The authors introduce an interesting idea for making a self-interpretable model that can achieve reasonably good predictive performance.

The authors do a good job motivating why self-interpretable models are useful, i.e. that "self-interpretable models make interpretations at the same time of predictions, thus reveal the intrinsic mechanism of the models, and are thereby preferred by users in high-stakes tasks."

The authors then propose to follow *interpretation transformation equivariance*. This often seems quite reasonable, but the authors should note when this is a desirable property. For example, if the introduced transformations introduce artifacts into the image (e.g. large swatches of black background in the cifar images), it may no longer be reasonable to assume that the prediction mechanism should not rely on these regions.

My main criticism of the is work is that the authors should also reference (and maybe compare to) this recent paper on [self-interpretable neural networks](https://arxiv.org/abs/1806.10574) (chen et al. 2018) which use prototypes at a granular level. This papers also helps to provide localization which can provide transformation equivariant interpretation.

The author's might also want to discuss in more detail their choice to use bilinear upsampling to visualize $\hat w$ - is this the best choice? Did the authors try any other visualizations, e.g. learning an inverse function directly?

Minor points:

- there are minor typos and grammatical errors throughout the text that could be improved.
- Fig 5a xlabel has typo
limitations_and_societal_impact: The authors do not seem to *explicitly* address limitations and societal impact, although both come across as reasonable.
review_title: null

Meta-review: This paper considers the problem of training self-interpretable models. It first shows that existing approaches change the interpretation with input transformations even though the model is invariant to them. It then proposes a new method that fixes this problem. While the reviewers initially had concerns about the empirical evaluation, the rebuttal clarified most of the issues and all of them unanimously recommend acceptance. So, I am recommending the paper for acceptance. At the same time, there are also some concerns such as comparison with Chen et al. work that the rebuttal did not address properly. I would ask the authors to fix this and other issues raised in the reviews below in the final version.
Decision: Accept (Poster)