This Reads Like That: Deep Learning for Interpretable Natural Language Processing
Authors: Claudio Fanconi
Moritz Vandenhirtz
Severin Husmann
Julia E Vogt
Venue: EMNLP 2023 Main (2023)
Keywords: interpretability
natural language processing
deep learning

Abstract: Prototype learning, a popular machine learning method designed for inherently interpretable decisions, leverages similarities to learned prototypes for classifying new data. While it is mainly applied in computer vision, in this work, we build upon prior research and further explore the extension of prototypical networks to natural language processing. We introduce a learned weighted similarity measure that enhances the similarity computation by focusing on informative dimensions of pre-trained sentence embeddings.  
Additionally, we propose a post-hoc explainability mechanism that extracts prediction-relevant words from both the prototype and input sentences. Finally, we empirically demonstrate that our proposed method not only improves predictive performance on the AG News and RT Polarity datasets over a previous prototype-based approach, but also improves the faithfulness of explanations compared to rationale-based recurrent convolutions.

This Reads Like That: Deep Learning for Interpretable Natural Language
Processing
Claudio Fanconi"
Moritz Vandenhirtz'
Severin Husmann ETH Zürich shusmann @ethz.ch
Julia E. Vogt ETH Zürich
ETH Zürich fanconic@ ethz.ch
ETH Zürich
moritz.vandenhiite @ inf enf.ethz.ch
Julia. @ enf.cth..ch
Abstract
maintaining high predictive performance (Chen et al., 2019).
Prototype learning, popular machine learning method designed for inherently interpretable decisions, leverages similarities to learned pro- totypes for classifying new data. While it is mainly applied in computer vision, in this work, we build upon prior research and further ex- plore the extension of prototypical networks to natural language processing. We introduce a learned weighted similarity measure that en- hances the similarity computation by focusing on informative dimensions of pre-trained sen- lence embeddings. Additionally, we propose a post-hoc explainability mechanism that ex- tracts prediction-reles relevant words from both the prototype and input sentences. Finally, pirically demonstrate that our proposed method not only improves predictive performance on the AG News and RT Polarity datasets a previous prototype-based approach, but also improves the faithfulness of explanations com- pared to rationale- based recurrent convolutions.
In this work, we focus on transforming and en- hancing the prototype-basee model introduced by Chen et al. (2019) from the computer vision do- main to the natural language processing (NLP) do- main. Prototype-based methods for NLP, ex- plored in previous works (Friedrich et al., 2022: Ming et al., 2019; Huang et al., 2022; Wang et al., 2022), involve classifying test samples based on their similarity to prototypical training samples. To improve upon these prior adaptations, we propose two key enhancements. Firstly, we introduce a learned weighted similarity measure that enables the model to attend to the most informative as- pects of the pre-trained sentence embedding. This improvement ensures that the method benefit from pre-trained language models (Reimers and Gurevych, 2019), which have shown superb empir- ical results, and extract task-relevant information, thereby enhancing its performance. Secondly, we enhance the interpretability mechanism by conduct- ing post-hoc analyses to determine the importance of individual words in both test samples and sen- tence prototypes. Furthermore, we investigate the faithfulness of these explanations, examining their ability to faithfully capture and represent the under- lying model's 's decision-makee process.
1 Introduction
Due to the increasing utilization of artificial neu- ral networks in critical domains such as medicine and autonomous driving, the need for interpretable and explainable decisions made by these models is extremely important (Samek and Müller, 2019; Rudin, 2019). While extensive rescarch in this field has focused on post-hoc explanations for black-box machine learning models (Bach et al., 2015; Kim al.. 2018; Bau al., 2017: Lapuschkin al., 2019), a recent line of investigation has challenged this paradigm by proposing network architectures that inherently possess interpretability (Girdhar and Ramanan, 2017; Chen et al., 2019; Li et al., 2018; Brendel and Bethge, 2019). These interpretable models aim to base their decisions on a limited set of human- comprehend the decision-making process while
In summary. our contributions in this paper in- volve domain-specific adaptations of inherently interpretable prototype-based models from com- puter vision to the NLP domain, enhancing their performance through a learned weighted similar- ity measure, improving explainability via post-hoc analyses of word importance, and investigating the faithfulness thereof. These advancements aim further the understanding and trustworthiness of prototype- methods in NLP.
human-understame understandable features, enabling users
'The code is publicly available at fanconic/thin hetammicthhissssssssseee relir_eadd__eeeeee like_that. that.
com/
"Equal contribution

---

2 Related Work
To provide a meaningful interpretation in the in- put space, we periodically project the prototypes their nearest training sample of the same class. For the last three epochs, the prototypes are projected and fixed, thus enabling visualization of the proto- types contributing to an input's prediction at test time. To ensure the useful spreading of prototypes. we employ the loss terms proposed by Friedrich et al. (2022). In Appendix A.4, conduct abla- tion study assess the effect of the different terms, More information about the experimental setup and implementation can be found in Appendix A.2.
The three main interpretability mechanisms for deep neural networks discussed in this work are post-hoe explainabilityy rationalc-based models, and prototype-based models.
Post-hoc explanation methods correlate inputs and outputs explain model's behavior using gradient visualization (Sundararajan al., 2017; Smilkov et al., 2017) and attention mechanisms (Bahdanau et al., 2013; Li et al., 2017: Husmann et al., 2022; Abnar and Zuidema, 2020). However, the faithfulness of these methods is uncertain, as learned attention weights can lack correlation with gradient-based measures of feature importance, and different attention distributions can yield equivalent predictions (Jain and Wallace, 2019). basedd models (Lei et al., 2016; Jain et al., 2020; Bastings et al., 2019) improve explana- tion faithfulness by selecting rationales that serve as explanations for predictions. While rationales indicate contributing input features, they do not re- veal how these features are combined or processed within the neural network.
3.1 Learned Weighted Similarity Measure
Wanting to leverage the strong empirical perfor- mance of pre-trained language models, the issue arises that their sentence embeddings contain in- formation relevant to multiple tasks (Reimers and Gurevych, 2019), whereas are only interested in one (e.g. sentiment classification). Thus, the sim- ilarity to prototypes, that is computed directly on these pre-trained embeddings, should only depend on class-relevany information. To achieve this, we propose a new similarity measure that assigns indi- vidual weights to the dimensions of the embedding, inspired by the work of Vargas and Cotterell (2020) on gender bias as a linear subspace in embeddings. This allows the network to focus dimensions it deems more important for our specific task, rather than treating all dimensions equally. In this mea- sure, we calculate the similarity (i.e. &g distance or cosine similarity) between the embeddings and pro- totypes, considering the weights assigned to each dimension. For cosine similarity, we multiply each dimension of the embeddings by the corresponding learned weight, as shown in Equation 1:
Prototype-based networks (Chen et al., 2019) emulate human-likk reasoning by finding similari- ties between input parts and prototypes to make pre- dictions. However, existing NLP-based prototype methods (Friedrich et al., 2022; Ming et al., 2019; Huang et al., 2022; Wang et al., 2022) require full model control and lack the ability to visualize im- portant words in sentences, failing the criterion of sparse explanations (Rudin, 2019). Our proposed method addresses these limitations. Further details on related work can be found in Appendix A.1.
3 Methods
In our approach, we first compute the sentence em- bedding using pretrained language models BERT (Reimers and Gurevych, 2019), GPT2 (Radford al., 2019; Muennighoff, 2022), MPNet (Song et al., 2020), and RoBERTa (Sanh et al., 2019) to capture the semantic representation of the input sen- tence. Then, we utilize a prototypical layer to mea- sure the similarity between the sentence embedding and learned prototypes. These prototypes act as learned prototypical points in the embedding space, representing the different classes, to which they have been assigned equally. The similarity scores are multiplied by the learned prototype weights to arrive at the final, inherently interpretable predic- tion.
sim(w, u, v) = Σ, Wuuuu
En win? v Ee wayu
where w represents the weight vector, and and v are the embeddings whose similarity is being cal- culated. During training, we enforce non-negative reasoning by clamping the weights of O. This ensures that negative reasoning is disal- lowed. Similarly, for the f2 similarity, we weigh the embedding dimensions according to the weight vector:
minimum
sim(w w, u, v)
Σ (winyu))
(2)

---

3.2 Visualizing Important Words
Dataset
Backbone BERT
Similarity
RT reviews
AG News
Instead of the sentence-level approach proposed by Friedrich et al. (2022), which visualizes the whole prototypes as explanation, we introduce word-level visualization for improved interpretabil- ity through sparse explanations (Rudin, 2019). On a high level, we visualize the words of sample that were most important for the similarity to the set of prototypes, which had the biggest effect on the pre- diction. This is achieved by computing the discrete derivative of the similarity to these protolypes with respect to each input token. We iteratively remove the token from the input that decreases the similar- ity the most, as upon reversion, this token is most important for the similarity. Likewise, for visual- izing important words of prototypes, we consider the similarity to the input sample. By iteratively selecting and removing the tokens, we visualize the most important words for a given prototype and test sample. The former can be interpreted as a form of abstractive rationales (novel words are generated to support the prediction), while the latter are extrac- tive rationales (important words from the input data are extracted to support the prediction) (Gurrapu et al., 2023).
cosine
80.46 0.43
78.67 +: 0,21 86.13 1 0.12
w. cosine
82.14 1 0.20
& distance
79.20 0.36
73.36
0.89
w. & distance
80.02 0.44
73.11 +: 0.82
GPT2
cosine
73.41 + 0.41
85.77 + 0.20
w. cosine
75.44 + 0.05
87.25 + 0.13
& distance
69.39 1 1.78
78.36 1: 1.46
w. E distance
69.88 1: 0.46
79.36 1: 1.36
MPNet
cosine
$1.09
0.49
86.49
0.32
w. cosine
79.55
0.82
88.09 0.28
& distance
75.87 + 4.21
79.84 + 1.50
w. Eg distance
77.69
+ 2.26
79.73 + 1.38
RoBERTa
cosine
75.58 1: 2.37
86.24 + 0.06
w. cosine
77.51
+ 0.29
87.45 + 0.07
Es distance
72.57
1.79
80.48 0,90
w. la distance
73.34
3.55
81.58 0.85
Table I: Test accuracy (%) when using weighted ("w.") similarity scores, against using normal similarity scores with different sentence transformers. The experiments are averaged over five runs and the standard eror is re- ported, The best performing method within weighted and non weighted similarity measure is marked in bold.
with different random seeds, and the mean and stan- dard deviation thereof are reported. In the second experiment, calculated the test set confidence intervals (CIs) through l'000-fold bootstrapping proxy, due to the excessive computational cost.
Unlike previous approaches (Friedrich et al., 2022), we do not impose restrictions on the relative position of the words or fix the number of visual- ized words in advance. Instead, we dynamically determine the number of words to be visualized based on the explained percentage of the full dis- tance. We define full distance as the change in similarity after having removed the top n tokens (where n = 10), hereby implicitly assuming that the similarity between any two sentences can be erased by removing tokens. Then, we visualize the smallest subset of removed tokens that together account for least q [o, 1] (in case q 0.75) of the full distance. This approach provides more flexibility and adaptability in visualizing the impor- tant words of a sentence and its prototypes without requiring predefined specifications of their struc- ture. It offers a more nuanced understanding of the reasons behind the similarity between test sample and the prototypes and thus of the decision-making process for predicting sample's 's label.
4.1 Learned Weighted Similarity Measure
To explore the impact of the weighted embedding dimensions in the similarity measure, we carry out an experiment in which we compare the per- formance of the weighted with the unweighted model on the Rotten Tomato (RT) movic review dataset (Pang et al., 2002), as well as the AG news dataset (Zhang et al., 2016). The test accuracy results are displayed in Table 1. The results demon- strate that in 13 out of the 16 experiments, the model profits in performance from focusing on the task-relevant sentence embedding dimensions. The largest change can be observed when using stan- dard cosine similarity on the AG news dataset with an accuracy of 78.86% (+0.21 0.21 %p) increasing by almost 8%-poisss to 86. 18% 1.2 %p) when us- ing learned embedding dimensions. We observe, that for both datasets, the unweighted as well as the learned weighted cosine similarity performed bet- ter than &g or learned weighted b similarity. Thus, for the remaining experiments, we will focus on the model using weighted cosine similarity.
4 Results
In this section, we present experiments demonstrat- ing the importance of our proposed method. In the first experiment, the models were run five times

---

Test sample and its keywords
Prototype and most important words
This insufferable movie meant make you think about
Plodding, poorly written, murky and weakly acted. the pic- ture feels as everyone making it lost their movie mojo.
existential suffering. Instead, it'll only put you to sleep.
Because of an unnecessary and clumsy last scene,' swimfan'
Plodding. poorly written, murky and weakly acted. the pic-
left me with very bad feeling.
ture feels as if everyone making it lost their movie mojo.
Campanella's is competent direction and his excellent cast over-
.riggtt,, intelligent, and humanly funny film.
come the obstacles a predictable outcome and screenplay that glosses over rafacl's evolution.
Between them, de niro and murphy make showtime the most
bright, intelligent, and humanly funny film
savory and hilarious guilty pleasure of many recent movie season.
Table 2: Exemplary test samples from the movic review dataset by Pang et al. (2002) with the corresponding most important prototype from training. The most important words as determined by model visualized in color. The highlighted words within the test sentence are extractive rationales, while the ones within the prototype can be thought of as abstractive rationales.
4.2 Visualizing Important Words
for the model's behavior. Sufficiency is quanti- fied as the difference between the probability of the predicted class for the original input and the probability of the same class when making predic- tions solely based on the rationales. measures the extent to which the extracted rationales are suffi- cient for a model to make its prediction (DeYoung et al., 2020). Comprehensiveness is similarly com- puted, but instead of considering the probability obtained with only the rationales, it subtracts the predicted probability of the input without any ra- tionales. Comprehensiveness captures the degree to which the extracted rationales capture all tokens the model uses for prediction.
To demonstrate the effectiveness of our visualiza- tion approach, we first show qualitative samples from the test set of the movie review dataset (Pang et al., 2002). Table 2 presents test set sentences with prototypes and the corresponding most im- portant words from the sentence itself (extractive rationales), but also from the prototype (abstractive rationales). Thanks to our innovation, we go be- yond simply visualizing the prototypical sentence and show the most important words from the proto- type as well as from the sample we are comparing it to. We enhance the interpretability mechanism, show to which words from the prototype the test sample most similar to and which words from contributed most the similarity. Hence, the most significant words from the same prototype can vary depending on the test samples, since the rationale for their similarity may differ.
The previous work of Friedrich et al. (2022) lacks the extraction of specific rationales for test sentences. Thus, we employ the recurrent convolu- tional neural network (RCNN) of Lei et al. (2016) as a baseline that extracts rationales at the word- level from the test samples. Similarly, for our ap- proach, we extract the important words by focusing on the similarity the three prototypes that con- tribute most to the prediction from the test samples. The results of the evaluation on the Rotten Tomato movic review dataset (Pang et al., 2002) are pre- sented in Table 3. Therefore, in this experiment we focus solely on extractive rationales to compute the comprehensiveness and sufficiency.
4.3 Faithfulness
To evaluate the faithfulness of our model's visual- ization mechanism, we adopt the metrics of com- prehensiveness and sufficiency, following the ap- proaches of Jacovi and Goldberg (2021) and DeY- oung et al. (2020). These metrics serve as prox- ies for assessing whether the visualized important words, referred to as rationales, faithfully account
Model
Comp. (%p)
Suff. (%p).
Accuracy (%)
Rationales RCNN (I.i al., 2016)
19.12
5.81
73.84
95%-CI
(18.20, 20.01)
(5.07, 6.58)
(71.97, 75.75)
Ours (BERT)
52.90
2.63
82.65
95%-CC
(51.79, 54.12)
(1.63, 3,54)
(80.98, 84.33)
Table 3: Comprehensiveness (Comp.), Sufficiency (Suff.), and Accuracy results on the movie review dataset, including bootstrapped 95% confidence intervals of the test set in gray.

---

Both models exhibit notably low sufficiency, with our method achieving a value of 2.63% points (95%-CI: 1.63%p. 3.54%p) and the baseline model achieving 5.81% % points (95%-CI: -I: 5.07%p. 6.58%p). These findings suggest that our proposed network extracts rationales that are utilized for pre- diction more sufficiently than the baseline. Addi- tionally, our model demonstrates comprehensive- ness score that is more than 33% points higher than the baseline model. This observation suggests that while both methods extract relevant rationales, only our method visualizes the full set of rationales on which the model bases its prediction.
our primary objective in this work is not empha- size predictive performance compared to larger and more complex models, but rather to explore the in- terpretability mechanism of prototypical networks in the context of NLP.
Secondly, although prototypical networks ex- hibit an inherently interpretable network architec- ture, they still rely a black box encoder. As demonstrated by Hoffmann al. (2021), manip- ulating the model inputs can lead to interpreta- tions that may not align with human understand- ing. Nonetheless, our research represents a signifi- cant step towards achieving explainable and inter- pretable Al.
5
Conclusion
Finally, the proposed post-hoe method for quan- tifying word-level importance is computationally expensive. It involves iteratively removing individ- ual words from an embedding, which necessitates multiple forward passes through the network.
Our work presents several significant contributions. Firstly, we advance the rescarch on extending pro- totypical networks to natural language processing by introducing a learned weighted similarity mea- sure that enables our model to focus on the crucial dimensions within sentence embeddings while dis- regarding less informative context. This allows us to reap the benefits of pre-trained language models while retaining an inherently interpretable model. This enhancement leads accuracy improvements over the previous work by Friedrich et al. (2022) on both the RT movie review dataset and the AG news dataset. Additionally, we propose a post- hoc visualization method for identifying important words in prototypical sentences and in test sen- tences. Our results demonstrate improved faithful- ness in capturing the essential factors driving the model's 's predictions compared Lei et al. (2016). Collectively, our findings highlight the efficacy of learned weighted similarity measure, the value of our post-hoe visualization technique, both leading to the enhanced faithfulness of the proposed model. These advancements contribute to a deeper under- standing of the interpretability of neural networks in natural language processing tasks.
References
Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Annual Meeting of the Association for Computational Linguistics.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wo- jeiech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise rele- vance propagation. PLOS ONE, 10(7)=0013 30140.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409,044
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Con- ference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2977.. Association for Computational Linguisties.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quanti- fying interpretability In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549.
deep visual representations.
A Aeknowledgements
MV is supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MB22.00047.
Wieland Brendel and Matthias Bethge. 2019. Approx- imating CNNs with bag-of- -fcal-ferrsss models works surprisingly well on imagenet. In Interna- tional Conference on Learning Representations.
Limitations
Chaofan Chen, Oscar Li. Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. 2019. This looks like that: Deep learning for interpretable image recog- nition. In Advances in Neural Information Process- ing Systems, volume 32, pages 8930-8994 Curran Associates, Inc.
This study has certain limitations. Firstly, our fo- cus is on two well-balanced classification datasets, and our performance is not competitive with non- interpretable of classifiers. However,

---

Jay De Young, Sarthak Jain, Nazneen Fatema Rajani. Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. FRASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458, Online. Association for Computational I Linguistics.
testing with concept activation vectors (teav). In In- ternational Conference on Machine Learning, pages 2668-2677. PMLR.
Sebastian Lapuschkin. Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2019. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10(1):1-8.
Felix Friedrich, Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. 2022. Interac- tively providing explanations for transformer lan- guage models. In Proceedings of the 1st Conference of Hybrid Human Artificial Intelligence (HHAT) and in Frontiers in Artificial Intelligence and Applicu- tions.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 107-117, Austin, Texas, Association for Computational Linguisties.
Rohit Girdhar and Deva Ramanan. 2017. Attentional pooling for action recognition. In Advances Neural Information Processing Systems, volume 30.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Un- derstanding neural networks through representation erasure.
Sai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, and Feras A. Batarseh. 2023. Ratio- nalization for explainable nlp: a survey. Frontiers in Artificial Intelligence, 6.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. 2018. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32.
Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. 2021. This looks like that... does it? shortcomings of latent space prototype interpretabil- ity in deep networks. In ICML 2021 Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable Al.
Zachary C. Lipton. 2018. The mythos of model in- terpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57.
Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. 2019. Interpretable and steerable sequence learning via pro- totypes. Proceedings of the 25th ACM SIGKDD In- ternational Conference on Knowledge Discovery & Data Mining.
Youcheng Huang, Wenqiang Lei, Jie Fu, and Jiancheng Lv. 2022. Reconciliation of pre-trained models and prototypical neural networks in few-shot named en- tity recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1793-1807, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint w:Xiv:2202.08904
Severin Husmann, Hugo Yèche, Gunnar Ratsch, and Rita Kuznetsova, 2022. On the importance of clini- cal notes in multi-modal learning for EHR data. In NeurlPS 2022 Workshop on Learning from Time Se- ries for Health.
Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the ACL-02 Conference on Empirical Methods Natu- ral Language Processing Volume 10, EMNLP '02, page 19-86. USA. Association for Computational Linguistics.
Alon Jacovi and Yoav Goldberg. 2021. Aligning
Faithful Interpretations with their Social Attribution. Transactions of the Association for Computational Linguistics, 9:294-310 310.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. CoRR, 11t.19922.10888
Nils Reimers and Tryna Gurevych. 2019, Sentence-bert. Sentence embeddings using siamese bert-networks.
Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and By- C. Wallace. 2020. Learning to faithfully ra- tionalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4459 4473. Association for Computa- tional Linguistics.
Cynthia Rudin. 2019. Stop explaining black box ma- chine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215. 215
Wojeiech Samek and Klaus- Robert Müller. 2019. To- wards explainable artificial intelligence. In Explain- able AI: interpreting, explaining and visualizing deep learning, pages 5-22. Springer.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018, In- terpretability beyond feature attribution: Quantitative

---

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
A
Appendix
A.1 Supplemental Related Work
Various methods have been proposed to address the explainability and interpretability of deep neu- ral networks. In this section we are discussing the existing work, focusing especially on post-hoc explainability, artional-based interpretability, and prototype-based interpretabilliyy
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. 2017. Smoothgrad: removing noise by adding noise.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. In Proceedings of the 34th International Conference Neural In- formation Processing Systems, NIPS'20, Red Hook,
A.1.1 Post-hoc Explainability
NY, USA. Curran Associates Inc.
Post-hoc explanation methods are focused find- ing correlations based on the inputs and outputs. Thus, explanations are commonly only created ter forward passes of black-box models. One such method is e.g. to visualize the gradients of neu- ral networks over the inputs with heat maps (Sun- dararajan et al., 2017; Smilkov et al., 2017). Alter- natively, what has recently become popular with the rise of transformer architectures in NLP, are post-hoc explanations based on attention mecha- nisms (Bahdanau et al., 2015). For example Li al. (2017) use the attention layers to detect important words in sentiment analysis tasks. However, Jain and Wallace (2019) point out some shortcomings in the explainability of attention heads and thus need to be handled with caution.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan, 2017. Axiomatic attribution for deep networks. In Proceed- ings of the 34th International Conference on Machine Learning Volume 70, ICML'11 17. page 3319-3328. JMLR.org.
Francisco Vargas and Ryan Cotterell. 2020. Exploring the linear subspace hypothesis gender bias miti- gation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2902-2913, Online. Association for Computational Linguistics.
Jianing Wang, Chengyu Wang, Chuanqi Tan, Minghui Qiu, Songfang Huang, Jun Huang, and Ming Gao. 2022. SpanProto: A two-stage span-based prototyp- ical network for few-shot named entity recognition. In Proceedings the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 3466-3476, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
A.1.2 Rational-based Models
Lei et al. (2016) introduced so-called rationals. These pieces of the input text serving as justi- fications for the predictions. They used modular approach in which a selector first extracts parts of a sentence followed by the predictor module which subsequently makes classifications based on the ex- tracted rationals. Both modules are trained jointly in an end-to-end manner. They argue that using the selector, this fraction of the original input should be sufficient to represent the content. Nonethe- less, this selector modular approach has several drawbacks (Jacovi and Goldberg, 2021). Ja- covi and Goldberg (2021) argue that by training this end-to-end fashion the selector may not encode faithful justifications, but rather "Trojan explana- tions". These may be problems like the rational directly encoding the label. Another illustrated fail- case is that the selector module makes im- plicit decision manipulate the predictor towards decision, meaning that the selector dictates the decision detached from its actual purpose to select important information for the predictor by extract- ing a rationale that has a strong class indication
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016. Character-level convolutional networks for text clas- sification.

---

it deems to be correct. For these reasons, Jacovi and Goldberg (2021) suggest a predict--elect-verify model in which the model classifies based on the full input text but the selected rationale should in itself be sufficient to get to the same prediction, building up the extracted rational until this is the case.
Tranking Image
Prosagee
Prekengse ls d4s1
Activatin Map
A.1.3 Prototypical Models
Prototype-basek networks have recently been in- troduced for computer vision tasks to improve in- terpretability. In the paper "This Looks Like That: Deep Learning for Interpretable Image Recogni- tion" (Chen et al., 2019), the authors argue that their ProtoPNet has a transparent, human-like reasoning process. Specifically. ProtoPNet tries to find sim- ilarities between parts of an input image and parts of training images, called prototypes (hence the title of their paper), and classifies the input image based on this. Figure 1 shows an example for the classification of birds.
Figure An example depicting the reasoning produced by ProtoPNet for computer vision.
A.2 Experimental Setup
In the first experiment, where we examine the learned weighted similarity measure, we em- ployed the berr-largenlll-mean-toeee variant of Centence-BERT, following the code provided by Friedrich et al. (2022). For the MPNET, we uti- lized the 'all-mpnet-base-22 sentence transformer; for RoBERTa, we employed the "all-distilroberta- v1' variant; and for GPT2, we utilized the pre- trained transformer model "Muennighoff/GGPT L.MM-weigheedmean--l--bttf (all of which are available in the Hugging Face library). Across all training runs, we adopted the ADAM optimizer with learning rate of 0.005 and weight decay of 0.0005. Our models underwent 100 epochs of training, with batch size of 128. We implemented a learning rate scheduler that reduced the learning rate by a factor 0.5 every 30 epuchs if there was no improvement in the validation loss. Similar to Friedrich et al. (2022), our model lacks an early stopping mechanism, as the projections only com- mence after 50% of the epochs have passed. A projection step, involving the projection of pro- totype back to a training sample, was conducted every 5th epoch. In the final three epochs, only the fully connected layer was trained using posi- tive weights to prevent negative reasoning, aligning with Friedrich et al. (2022)'s code implementation. Recognizing the impact of vector magnitudes on l distance compared to cosine similarity, we ad- justed the dimension weighting vector by applying sigmoid function and then multiplying it by two, constraining the values to the range of (0, 2). For each class, we defined 10 prototypes, leading to a total of 20 prototypes for the movie classification and 40 for the AG News dataset. Furthermore, the
ProtoPNets have already been discussed in the setting of NLP by Friedrich et al. (2022) and Ming et al. (2019). Friedrich et al. (2022) change the encoder of the network from convolutional neural network to pretrained attention-based language model. Furthermore, they enhance the model with additional losses in the prototype layer receive better prototypes on a word and sentence-level pre- diction. Ultimately, their so-called i/Proto-The -Trex is applied to classification and explanatory interac- tive learning (XIL). Ming et al. (2019) apply the concept of prototypes to sequential data and focus on different variants of RNNs. They identify that for sequential data it is often the case that multiple prototypes similar. A solution they introduce an additional diversity loss that penalizes prototypes that are close to each other. However, RNNs have the problem that it is difficult to project the proto- types only to the relevant parts of a sequence. To overcome this limitation they use CNNs that can easily capture subsequences of an input. Further de- tails will be explained in Section 3. Like with post- hoe and nutional-basce methods, prototypes have their shortcomings in their interpretability mecha- nism (Hoffmaan et al., 2021). If there are subtle differences in the way the test and training data acquired, it can lead to misguiding interpretations (such as JPEG compression for images).

---

encoder layers are frozen during training, therefore the gradient only flows through the fully connected layer and the prototypical layer.
where 4 is the prediction, yi is the true label, E Id are the latent variables of the sentence- embedding, p E R are the prototypes, and w E RCxd are the weights of the last layer. Furthermore, for the loss to be complete, following must uphold: A > 0, E { 5}. (Please note, that by the time of writing, the formulas of the losses in the paper of Friedrich et al. (2022), as well as some in- terpretations included some mistakes. However, in their code they are implemented correctly.) To eval- uate whether each of the loss terms is necessary we carry out an ablation study. In Table 5, the results of our loss ablation study are presented. We show the accuracy of our best-performing model on the movie review dataset with the different loss terms individually removed (A; = 0, * i) from the full loss term well the top k (0....,3) pro- totypes removed. This allows inspecting whether each loss term had its desired effect.
A.3 Interpretability vs. Performance
As there is often a trade-off between the perfor- mance and the interpretability of deep neural net- works, we analysed the predictive performance of standard BERT on the movics and AG News dataset. Instead of appending prototype layer after the transformer backbone, we directly added a fully connected (FC) layer to the classification output. We trained it with the cross entropy loss, and kept the whole experimental setup the same as in the previous experiments. The results are show in Table 4. We see that while the non-interpretable baseline performs better as expected, as it does not have underlying architectural constraints, the differ- ence is not as prominent. Additionally, one could trade-off interpretability and accuracy by allowing for more prototypes, which would increase accu- racy but make the method less interpretable as the simutability decreases (Lipton, 2018).
In general, we observe that for the baseline model with all loss terms included, the model largely relics on the top 2 prototypes, as without them accuracy drops to 27.89%.
The clustering loss, Clst(z,, p) Rd Rdd - R. should ensure that each training sample is close to at least one prototype whereas the separation loss, Sep(z, 2, p) Rd x Ed - I, should encourage training samples stay away from prototypes not of their own class (Li et al., 2018). In the results, we can see that seemingly the full loss (A: * O, í E (1....,5) 5}) relies more on the top 2 prototypes, as accuracy is much lower for the model with the clustering term removed than for the model with the full loss term. This suggests that without this term the model may rely on many prototypes to get to estimate instead of only on the most similar prototype it should encourage.
Dataset
Backbone
Final Layer
RT reviews
AG News
BERT
FC
85.12 1 0.25
90.29
0.15
Prototype
82.14 0.20
86.38
0.09
GPT2
FC
77.87 + 0.22
89.47
0.14
Prototype
75.44 +: 0.05
87.25
0.13
MPNet
FC
85.12 + 0.25
89.35
0.05
Prototype
79.55 0.82
88.09 = 0.28
RoBERTa
FC Prototype
84.37
0.08
89.14
0.05
77.51 0.29
87.45
0.07
Table 4: Test set accuracy (%) comparison of the per- formance vs. interpretability trade-off, hy removing the prototypical layer and connecting the embedding layer directly to the classification output with a fully connected (FC) layer. The results are averaged over five runs, and the standard error is reported in gray. The best performing variant of a model is denoted in bolt.
Results after removing the separation loss term show a slightly higher accuracy of 83.37% and a stronger drop-off when removing the prototypes.
The steeper drop-off may imply that it has the de- sired effect of separating prototypes of different classes appropriately.
A.4 Loss Ablation Study
Here, we analyze the importance of the losses that we and Friedrich et al. (2022) are using:
The goal of the distribution loss, Dist(z, p) Rd x Rd - R, is to ensure that each prototype is close to sample in the training set. The removal of this loss does not seem to hurt the performance. Hence, we believe is not necessary and could be (3) removed. An explanation of this behavior might be that while thcoretically the maximum is specified over the whole training set, in the code of Friedrich
n L CE(te, 9:) + Clst(z, p)
+ +A2Sep(z, p) + AaDist(z, p) +iversss(pp) As//w/l

---

Top prototypes removed
Removed loss term
k
0
k = 1
k
2
k
3
Baseline (full loss)
82.46
80.16
62.15 15
27.89
Clustering loss (A1 0)
82.56
58.03
38.91
29.61
Separation loss (A2 0)
83.37
66.94
40.11
23.10
Distribution loss (Ag 0)
82.99
82.70
78.77
57.21
Diversity loss (A4 0)
82.41
80.21
56.49
25.11
Weights' & loss (As 0)
82.51
79.59
55.20
25.16
Table 5: Loss ablation study showing accuracy when removing different number of top prototypes and using our best model with weighted cosine similarity on the RT polarity dataset.
et al. (2022) that adopted, the maximum is only taken over batch. Requiring each prototype to be close training sample from each batch might be too much to ask for. Further research might be going into a relaxation this loss.
The idea of the diversity loss, Divers(p)
Rdd -> R, is to push prototypes from the same class apart to learn different facets of the same class (Friedrich et al. 2022). Compared to the baseline, the results show that indeed accuracy is higher when removing the top k prototypes and the diversity loss term seems to have the desired effect prototypes learning different facets of the same class.
Finally, dropping the weights' l eeeegulaariation term does seem to slightly worsen the performance when dropping to prototypes.


Reviews:
[{"review_id": "ezZjATk9Q6", "reviewer": "Reviewer_AbxD", "summary": "", "questions": "[A] How exactly does your experimental setup look like? What SBERT model did you use (there are lots of different models available for SBERT) as this affects your weight matrix? Did you perform any hyper-parameter tuning? What are your other training parameters (e.g., number of epochs, batch-size etc.)?\n\n[B] Are your results averaged across several runs with different seeds? \n\n[C] \"Lack of rationales for test sentences\" (line 279): but Friedrich et al (2022) do also evaluate word-level models instead don't they? Given that GPT-2 outperforms SBERT, what would then the advantage of your approach be?\n\n[D] What prototypes would be generated by the approach proposed by Friedrich et al (2022)? \n\n[E] Is there really no better baseline that has been published since Lei et al (2016)? What about the work from Jacovi and Goldberg (2021) that you mention?\n\n[F] Will the code be published?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful questions and critical comments. We address the questions separately in **bold** and our response is denoted in normal font.\n\n**Much of the experimental setup is unclear. For instance, the used model is not reported, there is no reporting on the used hyper-parameters and the baseline seems rather weak. The submitted code indicates the authors used some bert (1024-dim) and roberta (768-dim) models but it is unclear which ones exactly (cf. the list of available models in [1]). One would have to guess the parameters from the .yaml files and hope for the same results.**\n\nWe apologize for any confusion that may have arisen due to the lack of clarity in our initial manuscript version. Please refer to the supplementary setup provided below, which will also be included in the manuscript's appendix:\n\nIn the first experiment, where we examine the learned weighted similarity measure, we employed the `bert-large-nli-mean-tokens` variant of Sentence-BERT, following the code provided by Friedrich et al. (2022). For the MPNET, we utilized the `all-mpnet-base-v2` sentence transformer; for RoBERTa, we employed the `all-distilroberta-v1` variant; and for GPT2, we utilized the pretrained transformer model `Muennighoff/SGPT-125M-weightedmean-nli-bitfit` (all of which are available in the Hugging Face library). Across all training runs, we adopted the ADAM optimizer with a learning rate of 0.005 and a weight decay of 0.0005. Our models underwent 100 epochs of training, with a batch size of 128. We implemented a learning rate scheduler that reduced the learning rate by a factor of 0.5 every 30 epochs if there was no improvement in the validation loss. Similar to Friedrich et al. (2022), our model lacks an early stopping mechanism, as the projections only commence after 50% of the epochs have passed. A projection step, involving the projection of a prototype back to a training sample, was conducted every 5th epoch. In the final three epochs, the prototypes were projected and fixed, and only the fully connected layer was trained. We only allow for positive weights in the fully connected layer as well as the dimension weighting in order to prevent negative reasoning, aligning with Friedrich et al. (2022). Recognizing the impact of vector magnitudes on L2 distance compared to cosine similarity, we adjusted the dimension weighting vector by applying a sigmoid function and then multiplying it by two, constraining the values to the range of (0, 2). For each class, we incorporated 10 prototypes, leading to a total of 20 prototypes for the movie classification and 40 for the AG News dataset.\n\n**[A] How exactly does your experimental setup look like? What SBERT model did you use (there are lots of different models available for SBERT) as this affects your weight matrix? Did you perform any hyper-parameter tuning? What are your other training parameters (e.g., number of epochs, batch-size etc.)?**\n\nThank you for pointing this out. I hope with the aforementioned response we could clarify our experimental setup. We shall add the text to our manuscript’s appendix\n\n**[B] Are your results averaged across several runs with different seeds?**\nNo, within the initial manuscript, we did not compute averages across distinct runs or various seeds. Instead, we conducted test set bootstrapping to incorporate confidence intervals, effectively capturing the inherent aleatoric uncertainty within the data. However, in response to your feedback, we have subsequently re-executed our experiments on five separate occasions using diverse seeds, except for the extractive sufficiency and comprehensiveness, as it is very computationally intensive, thus we kept the bootstrapped test interval. Here you can see the results for the first experiment:\n\n|                 |                      | Dataset                   ||\n|-----------------|----------------------|---------------------------|--------------------------------|\n| **Backbone** | **Similarity**  | **RT reviews**       | **AG News**               |\n| S-BERT          | Cosine               | 80.46 $\\pm$ 0.43          | 78.67 $\\pm$ 0.21               |\n|                 | Weighted cosine            | **82.14** $\\pm$ 0.20 | **86.13** $\\pm$ 0.12      |\n|                 | $\\ell_2$ distance    | 79.20 $\\pm$ 0.36          | **73.36** $\\pm$ 0.89      |\n|                 | Weighted $\\ell_2$ distance | **80.02** $\\pm$ 0.44 | 73.11 $\\pm$ 0.82               |\n| S-GPT2          | Cosine               | 73.41 $\\pm$ 0.41          | 85.77 $\\pm$ 0.20               |\n|                 | Weighted cosine            | **75.44** $\\pm$ 0.05 | **87.25** $\\pm$ 0.13      |\n|                 | $\\ell_2$ distance    | 69.39 $\\pm$ 1.78          | 78.36 $\\pm$ 1.46               |\n|                 | Weighted $\\ell_2$ distance | **69.88** $\\pm$ 0.46 | **79.36** $\\pm$ 1.36      |\n| MPNet           | Cosine               | **81.09** $\\pm$ 0.49 | 86.49 $\\pm$ 0.32               |\n|                 | Weighted cosine            | 79.55 $\\pm$ 0.82          | **88.09** $\\pm$ 0.28      |\n|                 | $\\ell_2$ distance    | 75.87 $\\pm$ 4.21          | **79.84** $\\pm$ 1.50      |\n|                 | Weighted $\\ell_2$ distance | **77.69** $\\pm$ 2.26 | 79.73 $\\pm$ 1.38               |\n| RoBERTa         | Cosine               | 75.58 $\\pm$ 2.37          | 86.24 $\\pm$ 0.06               |\n|                 | Weighted cosine            | **77.51** $\\pm$ 0.29 | **87.45** $\\pm$ 0.07      |\n|                 | $\\ell_2$ distance    | 72.57 $\\pm$ 1.79          | 80.48 $\\pm$ 0.90               |\n|                 | Weighted $\\ell_2$ distance | **73.34** $\\pm$ 3.55 | **81.58** $\\pm$ 0.85      |\n\n**Table 1**: Performance comparison of using weighted similarity scores, against using normal similarity scores over different sentence transformers. The experiments are averaged over five runs and the standard error is reported. The best performing method within a weighted and non weighted similarity measure is marked in bold. Here, we added three additional backbones as an ablation study, to demonstrate our results also with different sentence transformers, requested by reviewer L24G. From the results in the table, we can see that weighting the dimensions in the similarity measure helps improving the performance in almost all cases.\n\n\n**[C] \"Lack of rationales for test sentences\" (line 279): but Friedrich et al. (2022) do also evaluate word-level models instead don't they? Given that GPT-2 outperforms SBERT, what would then the advantage of your approach be?**\n\nThank you for pointing this out. Yes, they do indeed also evaluate word-level models. Nevertheless, the reason why we do not compare our model on sufficiency and comprehensiveness with Friedrich et al. (2022) is because of a slight difference of interpretation of “rationales” and how they calculated their metrics. More precisely, in our calculation we are using **extractive** rationales, whereas in Friedrich et al. (2022) are using a form of **abstractive** rationales (see Gurrapu et al, 2023, https://arxiv.org/pdf/2301.08912.pdf, Fig 2.)\nFriedrich et al. (2022) mention in their paper that “Since our prototypical explanations are already short sequences, removing the entire prototype is similar to removing the rationale in the explanation.” (5. Faithful Prototypical Explanations). How we understand this, is that to calculate sufficiency and comprehensiveness, they do not remove any parts from a test sentence, but rather remove a prototype that is originating from the training set. Hence, during a forward pass the network would still see the whole test sample, which is a form of abstractive rationales.\nIn our proposed method, we can have both, abstractive and extractive rationales. Similar to Friedrich et al. (2022) we can just remove a prototype to calculate the sufficiency and comprehensiveness. However, with our proposed method, we can also have extractive rationales, which we focus on in the results. In order to have a comparable baseline, we followed De Young et al. (2020) baseline, when they proposed the ERASER baseline and the sufficiency and comprehensiveness metric.\n\nAdditionally, note that Friedrich et al. (2022) require a priori assumptions on their word-level prototypes. That is, they have to fix the number of words per prototype, as well as their relative positioning (eg adjacent words / every second word). Our method on the other hand does not require such restrictive assumptions and instead learns the number of words as well as their relative positive by itself, thus, allowing for much more flexibel prototypes. We believe this is the main advantage of the proposed method over Friedrich et al. (2022).\n\n\n**[D] What prototypes would be generated by the approach proposed by Friedrich et al (2022)?**\n\nThis is an interesting question, but maybe a bit difficult to display. We present the reviewer here below the prototypes of the weighted cosine similarity BERT model (ours) vs the prototypes of the cosine similarity BERT model (Friedrich et al. (2022)) on the movie review dataset. Please note, that here we are not highlighting the important words, as we have no reference test sample to compare the prototypes to. Nevertheless, compared to Friedrich et al. (2022), depending on the test sample we could highlight the most important words in the prototype, which the previous method was not able to.\n\nPrototypes Weighted Cosine Similarity (ours):\n\n*positive prototypes*:\n- \"the real star of this movie is the score, as in the songs translate well to film, and it's really well directed.\"\n- \"a distinguished and thoughtful film, marked by acute writing and a host of splendid performances.\"\n- \"an invaluable historical document thanks to the filmmaker's extraordinary access to massoud, whose charm, cultivation and devotion to his people are readily apparent.\"\n- “a very charming and funny movie.”\n- “very well - written and very well - acted.”\n- \". . . bright, intelligent, and humanly funny film.\"\n- \"a very well - made, funny and entertaining picture.\"\n- “certainly the performances are worthwhile.”\n- \"it's a wise and powerful tale of race and culture forcefully told, with superb performances throughout.\"\n- “jones has delivered a solidly entertaining and moving family drama.”\n\n*negative prototypes*:\n- \"the documentary is much too conventional--lots of boring talking heads, etc .--to do the subject matter justice.\"\n- “the leads we are given here are simply too bland to be interesting.”\n- \"unfortunately, the picture failed to capture me . i found it slow, drab, and bordering on melodramatic.\"\n- “i was perplexed to watch it unfold with an astonishing lack of passion or uniqueness.”\n- \"even as lame horror flicks go, this is lame.\"\n- “there is no pleasure in watching a child suffer . just embarrassment and a vague sense of shame.”\n- “a bland animated sequel that hardly seems worth the effort.”\n- \"this is a poster movie, a mediocre tribute to films like them!\"\n- “too clunky and too busy ribbing itself to be truly entertaining.”\n- \". . . a bland, pretentious mess.\"\n\n\n**Table 2**: Prototypes on the movie classification dataset, when using a weighted cosine similarity measure. \n\n\nPrototypes Cosine Similarity (Friedrich et al.):\n\n*positive prototypes*:\n- \"the performances are uniformly good.\"\n- \"the result is something quite fresh and delightful.\"\n- \"brings an irresistible blend of warmth and humor and a consistent embracing humanity in the face of life's harshness.\"\n- \". . . very funny, very enjoyable . . .\"\n- \"it's a masterpiece.\"\n- \"the film brilliantly shines on all the characters, as the direction is intelligently accomplished.\"\n- \"a riveting documentary.\"\n- \"everywhere the camera looks there is something worth seeing.\"\n- \"a thoughtful, moving piece that faces difficult issues with honesty and beauty.\"\n- \"moving and vibrant.\"\n\n*negative prototypes*:\n- \"plodding, poorly written, murky and weakly acted, the picture feels as if everyone making it lost their movie mojo.\"\n- \"the character of zigzag is not sufficiently developed to support a film constructed around him.\"\n- \"doesn't add up to much.\"\n- \"this extremely unfunny film clocks in at 80 minutes, but feels twice as long.\"\n- \"anyone not into high - tech splatterfests is advised to take the warning literally, and log on to something more user - friendly.\"\n- \"thoroughly awful.\"\n- \"simplistic, silly and tedious.\"\n- \"the comedy is nonexistent.\"\n- \"horrible\"\n- \"banal and predictable.\"\n\n\n**Table 3**: Prototypes on the movie classification dataset, when using a cosine similarity measure (non-weighted). \n\n\n**[E] Is there really no better baseline that has been published since Lei et al (2016)? What about the work from Jacovi and Goldberg (2021) that you mention?**\n\nWe opted to utilize the work of Lei et al. (2016) due to its specific focus on extractive rationales. As it has been proposed by De Young et al. (2020) as a reference for evaluating comprehensiveness and sufficiency on the ERASER benchmark, we deem it a reasonable baseline. Please note that we believe it would be unfair to compare to transformer architectures that only offer interpretability post-hoc, as our model architecture is inherently interpretable [1].\n\n[1] Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1, 206–215 (2019). https://doi.org/10.1038/s42256-019-0048-x\n\n**[F] Will the code be published?** \n\nCertainly! The code package, along with the data we employed, was already made accessible to the reviewers. Our code had previously been publicly available, but we opted to set the GitHub repository to private in adherence to the privacy period stipulated by EMNLP. Rest assured, we have intentions of reverting the repository to public status as soon as we are permitted to do so.\n\nAgain, we thank the reviewer for the insights and comments, and hope that we could satisfy some of these.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work investigates an interesting topic and propose a useful extension that inherently induces interpretability on a token-level.\n\nI assume the code will be published given that it is included in the supplementary material.", "reasons_to_reject": "Much of the experimental setup is unclear. For instance, the used model is not reported, there is no reporting on the used hyper-parameters and the baseline seems rather weak. The submitted code indicates the authors used some bert (1024-dim) and roberta (768-dim) models but it is unclear which ones exactly (cf. the list of available models in [1]). One would have to guess the parameters from the .yaml files and hope for the same results.\n\n\n\n[1] https://sbert.net/docs/pretrained_models.html", "questions_to_authors": "[A] How exactly does your experimental setup look like? What SBERT model did you use (there are lots of different models available for SBERT) as this affects your weight matrix? Did you perform any hyper-parameter tuning? What are your other training parameters (e.g., number of epochs, batch-size etc.)?\n\n[B] Are your results averaged across several runs with different seeds? \n\n[C] \"Lack of rationales for test sentences\" (line 279): but Friedrich et al (2022) do also evaluate word-level models instead don't they? Given that GPT-2 outperforms SBERT, what would then the advantage of your approach be?\n\n[D] What prototypes would be generated by the approach proposed by Friedrich et al (2022)? \n\n[E] Is there really no better baseline that has been published since Lei et al (2016)? What about the work from Jacovi and Goldberg (2021) that you mention?\n\n[F] Will the code be published?", "paper_topic_and_main_contributions": "This work proposes an extension for prototypical networks by introducing a weight matrix (that is learned during training) to capture similarities between the prototype and original embedding. The authors investigate the extension of the proposed weight matrix for two similarity metrics, namely, L2 distance and cosine similarity. In their experiments, they show that the additional weighting increases the performance on two datasets and finally, showcase the usefulness of their approach for visualizing important prototypes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Style: I'd suggest to do some hedging throughout the paper. For instance, the statement 'extend prototypical networks to [NLP]' in the abstract in the context with mentioning computer vision gives an impression that you are the first to do so. \n\nline 580: the -> that?"}}, {"review_id": "kwI9YdkM4b", "reviewer": "Reviewer_L24G", "summary": "", "questions": "A) It would be very useful to see the full set of prototypes and rationales that the method generated. The supplemental materials appear to contain this information in a difficult-to-use format.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the interesting points and comments. Our response is denoted in normal font, whereas the questions/comments are in **bold** font.\n\n**A) It would be very useful to see the full set of prototypes and rationales that the method generated. The supplemental materials appear to contain this information in a difficult-to-use format.** \n\nWe apologize for the inconvenience caused by the usability issues with the visualization of the prototypes and their corresponding key words. We have made the necessary improvements in the code, resulting in the production of a clearer CSV file. Additionally, as the reviewer expressed interest in the prototypes themselves, we have implemented code to include an extra CSV file containing these prototypes. For your convenience, we present below the set of 20 prototypes derived from the weighted cosine model applied to the movie dataset. Please note, that here we are not highlighting the important words, as we have no reference test sample to compare the prototypes to.\n\nPrototypes Weighted Cosine on Movie Dataset:\n\n*positive prototypes* \n- \"the real star of this movie is the score, as in the songs translate well to film, and it's really well directed.\"\n- \"a distinguished and thoughtful film, marked by acute writing and a host of splendid performances.\"\n- \"an invaluable historical document thanks to the filmmaker's extraordinary access to massoud, whose charm, cultivation and devotion to his people are readily apparent.\"\n- \"very well - written and very well - acted.\"\n- \"a very charming and funny movie.\"\n- \". . . bright, intelligent, and humanly funny film.\"\n- \"a very well - made, funny and entertaining picture.\"\n- \"certainly the performances are worthwhile.\"\n- \"it's a wise and powerful tale of race and culture forcefully told, with superb performances throughout.\"\n- \"jones has delivered a solidly entertaining and moving family drama.\"\n\n*negative prototypes*\n- \"the documentary is much too conventional--lots of boring talking heads, etc .--to do the subject matter justice.\"\n- \"the leads we are given here are simply too bland to be interesting.\"\n- \"unfortunately, the picture failed to capture me . i found it slow, drab, and bordering on melodramatic.\"\n- \"i was perplexed to watch it unfold with an astonishing lack of passion or uniqueness.\"\n- \"even as lame horror flicks go, this is lame.\"\n- \"there is no pleasure in watching a child suffer . just embarrassment and a vague sense of shame.\"\n- \"a bland animated sequel that hardly seems worth the effort.\"\n- \"this is a poster movie, a mediocre tribute to films like them!\"\n- \"too clunky and too busy ribbing itself to be truly entertaining.\"\n- \". . . a bland, pretentious mess.\"\n\n\n**Table 1**: Prototypes on the movie classification dataset, when using a weighted similarity measure. Please note, that here we are not highlighting the important words, as we have no reference test sample to compare the prototypes to.\n\n**The method is only evaluated on a single encoder model, and on two classification datasets. Experiments on a larger range of models/datasets would be necessary for the claims to be fully convincing.**\n\nCertainly, we agree with the reviewer's observation. In response, we have introduced three additional backbones to showcase our findings (namely RoBERTa, GPT2, and MPNET). Presented below are the results obtained using five distinct random seeds, along with the corresponding mean and standard error values. The experimental setup were kept the same throughout the experiments and will be included in the manuscript.\n\n|                 |                      | Dataset                   ||\n|-----------------|----------------------|---------------------------|--------------------------------|\n| **Backbone** | **Similarity**  | **RT reviews**       | **AG News**               |\n| S-BERT          | Cosine               | 80.46 $\\pm$ 0.43          | 78.67 $\\pm$ 0.21               |\n|                 | Weighted cosine            | **82.14** $\\pm$ 0.20 | **86.13** $\\pm$ 0.12      |\n|                 | $\\ell_2$ distance    | 79.20 $\\pm$ 0.36          | **73.36 $\\pm$ 0.89      |\n|                 | Weighted $\\ell_2$ distance | **80.02** $\\pm$ 0.44 | 73.11 $\\pm$ 0.82               |\n| S-GPT2          | Cosine               | 73.41 $\\pm$ 0.41          | 85.77 $\\pm$ 0.20               |\n|                 | Weighted cosine            | **75.44** $\\pm$ 0.05 | **87.25** $\\pm$ 0.13      |\n|                 | $\\ell_2$ distance    | 69.39 $\\pm$ 1.78          | 78.36 $\\pm$ 1.46               |\n|                 | Weighted $\\ell_2$ distance | **69.88** $\\pm$ 0.46 | **79.36** $\\pm$ 1.36      |\n| MPNet           | Cosine               | **81.09** $\\pm$ 0.49 | 86.49 $\\pm$ 0.32               |\n|                 | Weighted cosine            | 79.55 $\\pm$ 0.82          | **88.09** $\\pm$ 0.28      |\n|                 | $\\ell_2$ distance    | 75.87 $\\pm$ 4.21          | **79.84** $\\pm$ 1.50      |\n|                 | Weighted $\\ell_2$ distance | **77.69** $\\pm$ 2.26 | 79.73 $\\pm$ 1.38               |\n| RoBERTa         | Cosine               | 75.58 $\\pm$ 2.37          | 86.24 $\\pm$ 0.06               |\n|                 | Weighted cosine            | **77.51** $\\pm$ 0.29 | **87.45** $\\pm$ 0.07      |\n|                 | $\\ell_2$ distance    | 72.57 $\\pm$ 1.79          | 80.48 $\\pm$ 0.90               |\n|                 | Weighted $\\ell_2$ distance | **73.34** $\\pm$ 3.55 | **81.58** $\\pm$ 0.85      |\n\n**Table 2**: Performance comparison of using weighted similarity scores, against using normal similarity scores over different sentence transformers. The experiments are averaged over five runs and the standard error is reported. The best performing method within a weighted and non weighted similarity measure is marked in bold.\n\n\n**The paper does not evaluate the magnitude of interpretability tax associated with the method.**\n\nGiven that this concern was raised by another reviewer as well, we have taken the initiative to incorporate an extra experiment in the appendix. In this experiment, we contrast the prototypical network structure with direct classification subsequent to the encoder (connecting a fully connected layer from the sentence embedding dimension to the classification outputs), thereby bypassing the interpretability mechanism. The results of this comparison can be found below for your reference:\n\n|                 |                      | Dataset                        |                                |\n|-----------------|----------------------|--------------------------------|--------------------------------|\n| **Backbone** | **Final Layer** | **RT reviews**            | **AG News**        |\n| S-BERT          | FC                   | **85.12** $\\pm$ 0.25        | **90.29** $\\pm$ 0.15 |\n|                 | Prototype            | 82.14 $\\pm$ 0.20                 | 86.38 $\\pm$ 0.09          |\n| S-GPT2          | FC                   | **77.87** $\\pm$ 0.22        | **89.47** $\\pm$ 0.14 |\n|                 | Prototype            | 75.44 $\\pm$ 0.05                 | 87.25 $\\pm$ 0.13          |\n| MPNet           | FC                   | **85.12** $\\pm$ 0.25        | **89.35** $\\pm$ 0.05 |\n|                 | Prototype            | 79.55 $\\pm$ 0.82                 | 88.09 $\\pm$ 0.28          |\n| RoBERTa         | FC                   | **84.37** $\\pm$ 0.08        | **89.14** $\\pm$ 0.05 |\n|                 | Prototype            | 77.51 $\\pm$ 0.29                 | 87.45 $\\pm$ 0.07          |\n\n**Table 3**: Comparison of the performance vs interpretability trade-off, by removing the prototypical layer and connecting the embedding layer directly to the classification output with a fully connected layer.\n\nWe see that while the non-interpretable baseline expectedly performes better, as it does not have underlying architectural constraints, the difference is not as prominent. Additionally, one could tradeoff interpretability and accuracy by allowing for more prototypes, which would increase accuracy but make the method less interpretable as the simutability [2] decreases.\n\n[2] Zachary C. Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (May-June 2018), 31–57. https://doi.org/10.1145/3236386.3241340\n\n\n**The baseline model which is chosen for comparisons is quite old, predating BERT by several years. It is not clear how to interpret the current method's improvement over this baseline.**\n\nUnfortunately, due to the constraints of the rebuttal period, we could not implement more recent extractive rationale baselines, which we leave for future work. We would like to add that our work focusses on an inherently interpretable method, which is a different direction than post-hoc explainable transformer methods, thus, we do not compare to such works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is simple, reasonable, and (to the best of my knowledge) novel.\n\n- The method is general and can be applied straightforwardly to other models and classification tasks.\n\n- The experiments are well designed, and the results are convincing (for the datasets that the method is tested on).\n\n- The paper is very well written.", "reasons_to_reject": "I like the paper overall, and I think the contribution is probably sufficient for a short paper. The concerns below could be addressed in follow-up work.\n\n- The method is only evaluated on a single encoder model, and on two classification datasets. Experiments on a larger range of models/datasets would be necessary for the claims to be fully convincing.\n\n- The paper does not evaluate the magnitude of interpretability tax associated with the method.\n\n- The baseline model which is chosen for comparisons is quite old, predating BERT by several years. It is not clear how to interpret the current method's improvement over this baseline.", "questions_to_authors": "A) It would be very useful to see the full set of prototypes and rationales that the method generated. The supplemental materials appear to contain this information in a difficult-to-use format.", "paper_topic_and_main_contributions": "The paper investigates a method for interpretable text classification. The method uses prototypes for classification. The prototypes themselves are constructed from training samples, giving them a straightforward text interpretation. The method can identify the most important words from the prototypes, further improving interpretability. The paper performs evaluations on two classification datasets, showing that the model can extract interpretable word sequences for the model's classification decisions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zTgUR8gytr", "reviewer": "Reviewer_pffd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the interesting points  and comments. Our response is denoted in normal font, whereas the questions/comments are in **bold** font.\n\n**While it is clear and rightly pointed by the authors that this work is not competitive with non-interpretable state-of-the-art classifiers, its would be important to have at least one such SOTA baseline for a better understanding of the price being paid for the interpretability achievements and how that could be mitigated. This of course is being said on the principle of making the paper argument complete, as one could obtain such an answer by examination of relevant literature.**\n\nThis is an interesting point, and frequently, there exists a trade-off between the performance and interpretability of deep learning models. The primary focus of this paper is to enhance interpretability. We acknowledge that our predictive performance might not be directly comparable to that of the current SOTA models, which lack these explainable mechanisms. Nonetheless, upon evaluating the predictive performance, it becomes evident that the predictions remain competitive and non-random.\n\nTo address these concerns, we have included an additional experiment. In this experiment, we aim to illustrate the interpretability cost incurred by incorporating an interpretable mechanism, such as the prototype layers. We replaced the prototypical layer with a fully connected layer connected to the classification output. The results are presented below. As we observe, despite not achieving SOTA status, the introduction of this supplementary interpretability component results in a decrease of 3-6%p in accuracy, which is a reasonable price to be paid for increased interpretability. Note that one can adjust this tradeoff by using more prototypes which reduces model interpretability but increases accuracy.\n\n|                 |                      | Dataset                        |                                |\n|-----------------|----------------------|--------------------------------|--------------------------------|\n| **Backbone** | **Final Layer** | **RT reviews**            | **AG News**        |\n| S-BERT          | FC                   | **85.12** $\\pm$ 0.25        | **90.29** $\\pm$ 0.15 |\n|                 | Prototype            | 82.14 $\\pm$ 0.20                 | 86.38 $\\pm$ 0.09          |\n| S-GPT2          | FC                   | **77.87** $\\pm$ 0.22        | **89.47** $\\pm$ 0.14 |\n|                 | Prototype            | 75.44 $\\pm$ 0.05                 | 87.25 $\\pm$ 0.13          |\n| MPNet           | FC                   | **85.12** $\\pm$ 0.25        | **89.35** $\\pm$ 0.05 |\n|                 | Prototype            | 79.55 $\\pm$ 0.82                 | 88.09 $\\pm$ 0.28          |\n| RoBERTa         | FC                   | **84.37** $\\pm$ 0.08        | **89.14** $\\pm$ 0.05 |\n|                 | Prototype            | 77.51 $\\pm$ 0.29                 | 87.45 $\\pm$ 0.07          |\n\n**Table 1**: Comparison of the performance vs interpretability trade-off, by removing the prototypical layer and connecting the embedding layer directly to the classification output with a fully connected layer.\n\n\nAgain, we thank the reviewer for the comments, and hope that we could satisfy some of these.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The interpretability matter is highly relevant to the NLP community. The paper is a straightforward read and easy to follow. The results point to qualitatively high interpretability, but the price to pay for that is not quite clear (see reasons to reject).", "reasons_to_reject": "While it is clear and rightly pointed by the authors that this work is not competitive with non-interpretable state-of-the-art classifiers, its would be important to have at least one such SOTA baseline for a better understanding of the price being paid for the interpretability achievements and how that could be mitigated. This of course is being said on the principle of making the paper argument complete, as one could obtain such answer by examination of relevant literature.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an adaptation of prototype-based learning from the computer vision domain to NLP, introduces a learned weighted similarity measure, and proposes a post-hoc explainability method that allows highlighting the most important words in an inference as well as the corresponding words in the matching prototype. The similarity measure model performs a type of disentanglement, by separating and aligning embedding factors relevant to each prototype. The important word visualisation approach, despite being coarse and computationally expensive, can also be a valuable source of information from the base models. The results indicate that the proposed approach can successfully extract the information claimed in an interpretable manner.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L.188: \"similarity the input\" -> \"similarity *to* the input\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors extend the prototypical network approach to NLP to obtain an inherently interpretable model and propose a learnable weighting scheme for computing semantic similarity. They evaluate their approach on two text classification datasets (RT, AGNEWS) atop one encoder model (SBERT) and show that their proposed weighting scheme improves upon baseline similarity metrics. They further show that prototypical networks improve upon faithfulness — as estimated through comprehensiveness and sufficiency compared to a baseline.

The reviewers agree that the paper is well written and easy to follow (pffd, L24G). The reviewers further commend the simplicity, usefulness and relevance of the proposed method (AbxD, L24G, pffd).

The main concerns raised by the reviewers were (1) the lack of an analysis of the “interpretability tax” (L24G, pffd), (2) an unclear experimental setup (AbxD) and (3) a narrow experimental evaluation (L24G).
In the discussion period, the authors have addressed issues (1, 2, 3) with detailed comments and additional experiments, which they have committed to add to their paper — the author response has thus alleviated these concerns of the reviewers.

Another concern raised by the reviewers (L24G, AbxD) was the lack of relatively novel baselines, as the work of Lei et al is relatively outdated — as more novel baselines would help contextualize the contributions of the authors’ work. Some more recent and very relevant inherently interpretable methods which were not cited by the authors and can also be used as baselines are [1], [2]. While due to the speed of the field it is possible to miss relevant work, these papers are relatively well known in the field of explainable NLP and their omission is somewhat of a concern.

Furthermore, reviewer AbxD accurately notes that the paper might benefit from some hedging with respect to the strength and generality of claims made. Apart from reviewer AbxD’s comment, in the abstract (and in the conclusion) the authors claim that “… our proposed method not only improves predictive performance…” — however this improved predictive performance holds with respect to the baselines without the learned weighting scheme, and not in general. The experimental scope should be clearly delineated in scope of the paper.

The conclusion also does not mention the extension of prototypical networks to NLP, which should be mentioned as a contribution. 

[1] Jain, Sarthak, et al. "Learning to faithfully rationalize by construction.” (2020)

[2] Bastings, Jasmijn, Wilker Aziz, and Ivan Titov. "Interpretable neural predictions with differentiable binary variables." (2019).
Decision: Accept-Main