Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The authors proposed a self-explainable deep net architecture that could be used for text categorization. The main idea is to force the network to extract "excerpts", from the input text, each corresponds to a concept, which are also learned for interpretation. The classification is finally made based off of the learned concept, which is a binary vector. All three steps are learned in an end-to-end manner. The learning of concepts is regularized to make sure the concepts are consistent and non-overlapping. The idea sounds interesting and the experimental results support the usefulness of the proposed method on a variety of datasets. My sole concern is about the sensitivity analysis of the explanation, i.e. how robust is the explanation with respect to the perturbations that do not change the classifier prediction. It has been discussed in the literature that many explanation methods suffer from this sensitivity issue. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a classification model focused on interpretability. The model, Explaining model Decision through Unsupervised Concepts Extraction (EDUCE), is applied to a text classification task, while the authors argue in the appendix that this is also applicable to a wider problem, such as image classification. 

The model is composed of three parts: the first part is detecting salient spans of text relevant to the text classification problem, the second part assigning each salient span a concept label, and the third part which does the classification task based on the binary concept feature label. The models’ loss is composed of two parts: (A) minimizing the cross-entropy of text classification loss and (B) minimizing the cross-entropy of concept classification loss. For (A), as the first and second part of the model introduce discrete choices, they use a RL with Monte-Carlo approximation of gradients. 

The system is evaluated under two measure: 1) classification accuracy and 2) concept accuracy. They define the concept accuracy as follows: after training, they train a classifier that takes output (in the form of <salient span, their concept label>) of the model from the test portion of the data. They split this output into train and test, and report the test accuracy. This aims to show how consistent is the labeling of the salient spans for different methods: if the concept label set correctly merged together semantically similar spans, this “concept accuracy” would be higher. This is a new metric they are proposing. While it is interesting, I would like to see *some* studies on how this correlates with human’s judgements on how interpretable the model is. The paper is introducing a new measure *and* new model, and it’s hard to be persuaded the model is doing well based on this new measure, when there is little ground to know what this measure really measures. 

Overall, I’m not impressed with the models’ performances. The aspect rationale annotated beer sentiment dataset, presented by Lei et al (2016), has provided one of few opportunities to evaluate interpretability / rationale model quantitatively. The paper evaluates on this measure, which is included in the appendix, and the results are pretty disappointing compared to the existing models such as Lei et al’s initial baseline or Bastings et al. While the paper argues this method isn’t necessarily designed for this task unlike the other methods, I’m not sure this is necessarily the case. Bastings et al could be applied to other tasks that model is evaluated on, such as DBPedia and AGNews classification. The difference comes on how easy it is to interpret the methods, as these other rationale-based text processing methods would make use of captured words, while EDUCE would make use of detected “concept” clusters. Currently, the only real baselines are the ablations of its own model. 

Table 3 is quite interesting, different “concepts” capture different aspects fairly well. 

Not having a concept loss actually helps the classification accuracy. Would the concepts learned without concept loss qualitatively very different? This goes back to my original point that their new measure of "concept accuracy" is vague. 

Other comments and Q: 
- Figure (3), the visualization is a bit confusing cause it is unclear whether it is each span is a set of spans or a single span. Also, I would recommend making figures colorblind friendly, if possible. 
Q: what kind of classifier was used for the evaluation metric “concept accuracy” classifier? I don’t think it’s mentioned. 
Q: why are you sampling a test set for DBPedia experiments? Is it for efficiency reason?
Q: how sensitive is model’s performance to the hyper parameters, especially the number of concepts?
Q: the current baseline classifier is a simple BiLSTM one, which definitely perform a lot worse than recent pre-trained LMs such as BERT. Would it be easy to use this method on top of richer representation such as pertained LM outputs?
Q: how would this connects to saliency map literature in computer vision? I guess these would be mostly “a posteriori” explanations? Discussion would be helpful. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper introduces a new concept-based interpretability method that lies in the family of self-interpretable models (i.e. it's not a post-hoc method). Self-interpretability is achieved by a two-stage model: First, a concept-extractor finds the related pieces of consecutive words (excerpts) in a given text that are related to a concept among a set of given concepts (if any), then the model makes its predictions solely based on the presence or absence of concepts (binary).  The most useful part of the algorithm is that there is no need for concept annotations. The work then experimentally shows that their method, although does not outperform a non-self-interpretabile baseline but has better performance and interpretation compared to rival methods.
The paper is quite well-written. The introduced method is well-justified and the use of concept-based self-interpretable models is very useful to the field. It is also interesting to see that the performance is comparable to non-self-nterpretable baselines which would make a case for the use of self-interpretable models. I have two main concerns with this work. First, there is the novelty concern. The idea of unsupervised extraction of concepts for interpretability was introduced before (https://arxiv.org/pdf/1902.03129.pdf) and is not discussed by the authors (although the utilized terminology is very similar).  Authors should make a much more comprehensive discussion of what already exists in the concept-based interpretability literature and make the contribution of this work more clear (unsupervised concept extraction for a self-interpretable model instead of post-hoc interpretations). Secondly, although some of the objective experimental results (BeerAdvocate results, to some degree) suggest that the method is indeed capable of discovering meaningful, consistent, and separable concepts on its own, there is not enough discussion of why it actually should be the case. It's very easy to assume the introduced training procedure to extract separable but meaningless concepts(i.e. the excerpts of a concept are separable from that of other concepts and are consistent with each other in the eyes of the network while they are not consistent with a concept in the eye of human rationale; both loss terms will be minimized but there is no human-interpretability. One big issue with the experiments section is the A Posteriori Concept Acc metric as it only measures the separability and consistency of discovered concepts; first of all, it can be high while the extracted concepts are meaningless to humans, secondly, using it as a measure of comparison to rival methods is not quite fair as the introduced method is directly optimizing for this metric. The subjective results are interesting but not convincing. Adding a section for human subject experiments (following the previous work in concept-based interpretability) would make the results section much crisper. It also seems unlikely that all of the discovered concepts follow the desiderata and the fact that this is not mentioned makes it even more difficult to assess the amount of cherry-picking in the mentioned subjective results; there should be a discussion of cases of failure and why it happens. My score will change accordingly as the authors address the raised issues.

A few questions and suggestions:

* It would be more interesting to show subjective results of the model for cases of mistake; does the interpretability make sense when the model's prediction is wrong?

* It seems necessary to discuss how much the results are sensitive to selecting C? If the performance is robust, it would be interesting to see what happens for a large C? Does it discover more fine-grained concepts or most of the additional concepts would be null?

* For the model architecture, why did the authors choose to use the straight-through estimator and not other methods (e.g. concrete layers, ...)

* Some of the figures should be larger (much larger)

Thanks again for a well-organized and easy-to-follow paper.
review_title: Official Blind Review #3

Meta-review: This paper introduces a method for building interpretable classifiers, along with a measure of "concept accuracy" to evaluate interpretability, and primarily applies this method to text models, but includes a proof of concept on images in the appendix.

The main contributions are sensible enough, but the main problems the reviewers had were:
A) The performance of the proposed method
B) The lack of human evaluation of interpretability, and 
C) Lack of background and connections to other work.

The authors improved the paper considerably during the rebuttal period, and might have addressed point C) satisfactorily, but only after several back and forths, and at this point it's too late to re-evaluate the paper.  I expect that a more polished version of this paper would be acceptable in a future conference.

I mostly ignored R1's review as they didn't seem to put much thought into their review and didn't respond to requests for clarifications.
Decision: Reject