Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose ProtoryNet,  a prototype-based model for paragraph classification that associates each sentence in the paragraph with a relevant prototypical sentence from the training data. The idea is interesting and the ability to decompose sentiment scores over each sentence + find prototypes for each helps to build user understanding of the model prediction. Thank you to the authors for the submission.

However, I have two concerns---baseline comparisons and model details---that prevent me from assigning a higher rating.

**Baseline comparisons**

1. Across 5 sentiment classification datasets, the authors find that ProtoryNet substantially underperforms a standard BERT model, in some cases obtaining more than double the error. This seems like a substantial price to pay the ability to associate each sentence with a prototype. The authors write that "Note, however, that DistilBERT was pre-trained on a massive corpus of text data... Hence, [it] should only be used for [a] sanity check". However, ProtoryNet also seems to build on top of standard pre-trained BERT embeddings, which have also been derived from a massive corpus of text data (and actually, the comparison should favor ProtoryNet, since DistilBERT is a smaller model than the standard BERT model). Could the authors elaborate on why they believe that this comparison is unfair? If it is unfair, the authors should perform a comparison that is as similar as possible to ProtoryNet but without the prototype parts, i.e., train a model on top of the same BERT embeddings and see how that performs.

2. Related to the above question, it is common to fine-tune BERT models on the dataset of interest. Was this done here for DistilBERT? What about for ProtoryNet? And if not, why not?

**Model details**

3. It was difficult to follow all of the model details; perhaps consider reorganizing and clarifying the writing. For example, it was unclear how the prototypes are actually chosen until late in the paper, whereas it should have been explained in S3.1. The notation in S3.1 has a few minor errors. For example, if the entire sentence is encoded as $\mathbb{R}^V$ then it seems like $V$ is not just the size of the vocabulary, but the size of the vocabulary to the power of the length of the sentence? Also, how were the hyperparameter values and coefficient values chosen? Is prototype projection also done at the end of training?

4. There are many modeling decisions that seem somewhat ad-hoc or non-standard. It seems like it might be possible to simplify the model significantly, or if not, it would be nice for the effects of these decisions to be better studied. For example: (a) mean-squared error is used even on binary classification problems; (b) the loss function is complicated by diversity and prototypicality terms, but the sensitivity analysis reveals that the accuracies are basically indistinguishable even when we completely remove those terms; (c) the sparsity transformation was approximated by a softmax that seems basically indistinguishable from a step function since $\gamma \geq 10^6$, so does it actually matter? (Note that ReLUs are also not differentiable.) (d) How useful is the LSTM at the end, if it generally goes over only ~4 sentences? 

**Update**

Thank you to the authors for the revisions, and great to know that the experimental results have improved significantly. In the absence of an updated manuscript, it is difficult to update my score appropriately, so I will leave it as it currently is. However, I think the work is promising and that an updated manuscript that incorporates the new experimental results and more carefully teases apart the contributions of the different components (as the authors have started to do in this rebuttal period) would be impactful. Thank you to the authors again for all of their hard work.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: this paper presents an RNN sequence classifying model that generates a prototype for each sentence in a paragraph. The generated prototypes help explain the model's prediction. The method embeds each sentence, matches to prototypes, then runs through an LSTM before making a prediction. Experiments found improved accuracy compared to a previous model that generates only one prototype for a paragraph. A user evaluation also found improvement in interpretability.

Strengths:
-  The paper is well written and easy to understand.
- Generating prototypes is a promising direction for improving the interpretability of RNNs and other neural nets models. The idea of generating a prototype trajectory for sentences in a paragraph is interesting and novel to my knowledge.
 - The architecture and training methods are technically sound.
- The experiments show positive improvement in prediction accuracy.

Concern on user evaluation:
- There is some improvement but the error bars are large, it is not clear if the differences are statistically significant.
- With a prototype for each sentence, users will need to read more. So a more fine-grained explanation will increase cognitive load. User evaluation can potentially investigate this.
review_title: Good paper, minor concern

final_score: rating: 7: Good paper, accept
review_content: This paper presents ProtoryNet, a framework for text data that classifies and explains the prototypes' results.   The key concept, that is the novelty of the work, is that this framework is based on sentence prototypes, called prototype trajectory in the paper. In particular, instead of working at the entity of the text level, the text is split into sentences and each sentence is analyzed by itself.  The structure of the framework is composed of a layer that encodes a text's sequences, followed by a prototype layer in which is computed the similarity among each sentence and the prototype trajectories. At this point, the sentences are represented in one-hot encoding: for each sentence, there is a bunch of zero and then a one for the most similar sentence prototype. This representation is used for the classification of the sentence, done using an LSTM structure. In this setting, the interpretation is given by exploiting the prototypes matched for the text under analysis. 

The idea presented in the paper is really interesting: it allows for an interpretation based on prototypes that is simple and understandable while achieving acceptable prediction performance.

Pros
It is well written and easy to read. There is only a repetition of a “the” at the end of page 7.
The idea of the trajectory prototypes is quite interesting, especially because it can be employed for other kinds of sequence data.
The explanations obtained are compelling: the results obtained from the human evaluation showed excellent results, especially in the context of local explanations for non-experts.
 I appreciated the presentation of the framework using a picture due to the complicated structure.

Cons
A few words more about the diversity and the prototypically would have been useful in the objective functions. The reader is able to get a general idea, but maybe in the appendix, there could be something more to understand the claims fully.
The prediction performance of the model is fine, but not excellent. In my opinion, some experiments more about the prototype initialization would have been helpful in understanding if it can be a possible source of errors. 
The evaluation of the explanations is only w.r.t. ProSeNet. However, there are other methods to compare with, such as LIME and SHAP, to name agnostic methods, but also attention-based explanations, such as LRP (Layerwise-Relevance-Propagation), NeuroX and Integrated Gradients (that are not cited even in the related work section).
What about the robustness of the explanations? Similar sentences are going to be represented by the same prototype or by prototypes that are similar? An analysis of the robustness of the explanations would be useful (fidelity, hit, and similar metrics could be used, but also methods such as ROAR-RemOve And Retrain or KAR-Keep and Retrain)
What about other classificators? The last part of the framework uses a LSTM to predict the sentiment of the sentence. However, due to the structure based on similarity, other classificators such as logistic regression or decision tree may be tested in the same fashion of shapelet-based classifiers. The explanation may also benefit from the use of such methods due to the additional interpretable information they provide.

[LIME] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).
[SHAP] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. In Advances in neural information processing systems (pp. 4765-4774).
[LRP] Patil, A., Wadekar, A., Gupta, T., Vijan, R., & Kazi, F. (2019, July). Explainable LSTM Model for Anomaly Detection in HDFS Log File using Layerwise Relevance Propagation. In 2019 IEEE Bombay Section Signature Conference (IBSSC) (pp. 1-6). IEEE.
[LRP] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.
[INTGRAD] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365.
[SHAPELET] Lines, J., Davis, L. M., Hills, J., & Bagnall, A. (2012, August). A shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 289-297).
review_title: A framework for text data that classifies that explains through prototypes' results

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses the problem of explainable AI by trying to build models which are inherently interpretable (as opposed to post-hoc methods that interpret black-box models after being trained).

In particular, they focus on a recent paper on prototype-based networks, and a model called prosenet. They introduce an extension, protorynet, which fares better on longer documents by applying prototypes to each sentence (rather than the whole document), leading to a "prototype trajectory", showing how the prediction evolves over the course of the sentence. 

They show improvements in prediction accuracy across 5 datasets, analyse the hyperparameters and conduct a human study.

Strengths:
I really like the concept of prototype networks, as it is simple and I could see it being accessible to non-technical users. The authors are to commended for undertaking a proper evaluation, including human experiments, which are often painful to conduct but quite helpful.

The proposed extension is straightforward and easy to understand. As someone unfamiliar with the original paper from Ming et al, I found it easy to get up to speed. The authors are also good at plainly describing the details of their approach, and running the requisite ablation studies (e.g. between sigmoid/exponential). 

Weaknesses:
1. The paper's prior work section is incomplete, and makes factually incorrect statements about the state of the field. In particular, the authors focus on attention-based techniques, ignoring the considerable amount of work on other approaches, and incorrectly saying that other approaches "often turn out to be gibberish". Attention is a particularly strange subset of interpretations to focus on, given that the community has recently started to argue about whether attention is an explanation at all [7]

In general, I would suggest the authors look at papers from the “Interpretability and Analysis of Models for NLP” track at ACL and the blackboxNLP workshop at ACL. For concrete starting points, I'll restrict myself to ICLR papers, dating back to 2017 [2-5]. General attribution methods, such as integrated gradients [6] are also pertinent.

I don't think the paper can be published without a reasonable related work section, hence the "clear reject" score. If this were fixed, I'd upgrade to a "weak reject", i.e. 5.

2. I am concerned that this model may be outperformed by simple, bag of words approaches. For IMDB, the original paper [1] has a variety of results hitting 88% accuracy, while the results reported for protorynet are 85%. If a bag of words model outperforms protorynet, that makes it a less desirable model. It is possible that preprocessing differences account for this, though.

Can the authors give some clarity on this, for IMDB as well as the other datasets? Ideally, there would be an additional column in the results table providing results for a simple, non-neural, baseline.

3. I worry that a lot of the added accuracy does not help the model's accuracy. In particular, the ablation charts in Figures 5 and 8 show that when alpha=beta=0, the model's accuracy is within ~.2% of the best accuracy, so do we need those additional loss terms? I also suspect that averaging the outputs, rather than feeding them through an RNN would be comparably accurate, and simpler. 

4. While I appreciate the human studies, the confidence bars are quite wide, so that most of the findings are not statistically significant.

5. For the model diagnosis human experiment, the users were only shown three examples. How were these examples chosen? That is not very many, so I worry that either those examples were chosen to be ones where protorynet was better. Ideally they would be chosen randomly, subject to some reasonable criteria.

Nitpicks:
- Distillbert is an odd SOTA to choose, as it is designed to have fewer parameters. Something like RoBERTA would likely have stronger results, and be more representative of SOTA.

[1] https://ai.stanford.edu/~amaas/papers/wvSent_acl2011.pdf
[2] https://arxiv.org/abs/1911.06194
[3] https://arxiv.org/abs/1812.04801
[4] https://arxiv.org/abs/1801.05453
[5] https://arxiv.org/abs/1801.05453
[6] https://arxiv.org/abs/1703.01365
[7] https://arxiv.org/abs/1902.10186
review_title: Incremental improvement on interesting concept with a thorough eval

Meta-review: The authors introduce an RNN model, ProtoryNet, which uses trajectories of sentence protoypes to illuminate the semantics of text data.

Good points were brought up and addressed in discussion, which have improved the paper - including a helpful suggestion from Rev 3 to fine-tune BERT sentence embeddings in ProtoryNet, which led to significant performance gains.

Unfortunately the tone of discussion with one reviewer slipped below the respectful standards to which we aspire, but rest assured that only substantive points on the paper were considered.

Reviewers were split but in discussion converged to leaning against acceptance, allowing the authors to reflect on, and incorporate new results carefully in an updated manuscript.
Decision: Reject