Interactively Generating Explanations for Transformer Language Models
Authors: Patrick Schramowski
Felix Friedrich
Christopher Tauchmann
Kristian Kersting
Venue: ICLR 2022 Submitted (2022)

Abstract: Transformer language models are state-of-the-art in a multitude of NLP tasks. Despite these successes, their opaqueness remains problematic. Recent methods aiming to provide interpretability and explainability to black-box models primarily focus on post-hoc explanations of (sometimes spurious) input-output correlations. Instead, we emphasize using prototype networks directly incorporated into the model architecture and hence explain the reasoning process behind the network's decisions. Moreover, while our architecture performs on par with several language models, it enables one to learn from user interactions. This not only offers a better understanding of language models but uses human capabilities to incorporate knowledge outside of the rigid range of purely data-driven approaches.

Under review as a conference paper at ICLR 2022
INTERACTIVELY PROVIDING EXPLANATIONS FOR TRANSFORMER LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer language models are state-of-the-art in a multitude of NLP tasks. Despite these successes, their opaqueness remains problematic. Recent methods aiming to provide interpretability and explainability to black-box models primar- ily focus on post hoc explanations of (sometimes spurious) input-output correla- tions. Instead, we emphasize using prototype networks directly incorporated into the model architecture and hence explain the reasoning process behind the net- work's 's decisions. Moreover, while our architecture performs on par with several language models, it enables one to learn from user interactions. This not only offers a better understanding of language models but uses human capabilities incorporate knowledge outside of the rigid range of purely data-driven approaches.
INTRODUCTION
Transformer language models (LMs) are ubiquitous in NLP today but also notoriously opaque. Therefore, it is not surprising that a growing body of work aims to interpret them: Recent evalu- ations of approaches of saliency methods (Ding & Koehn 2021) and instance attribution methods Pezeshkpour et al. (2021) find that, while intriguing, for the same outputs, different methods assign importance to different inputs. Furthermore, they are usually employed post hoc, thus possibly en- couraging reporting bias (Gordon & Van Durme 2013). The black-box character of LMs becomes especially problematic, as the data to train them might be unfiltered and contain (human) bias. As result, ethical concerns about these models arise, which can have a substantial negative impact on society as they get increasingly integrated into our lives (Bender et al., 2021..
Here, focus providing case-based reasoning explanations during the inference process, di- rectly outputting the LM's predictions. In doing so, we avoid the problems mentioned above of post hoc explanations and help to reduce the issue of (human) bias. To increase the interpretability of the model, we enhance the transformer architecture with a prototype layer and propose Prototypical- Transformer Explanation (Proto-Trex) Networks. Proto-Trex networks provide an explanation prototypical example for specific model prediction, which is similar to (training-)sampler with the corresponding label.
Our experimental results demonstrate that Proto-Trex networks perform on par with non- interpretable baselines, e.g. BERT (Devlin et al., 2019) and GPT (Radford et 2019). terms of explanations, we show promising results with learned prototypes providing helpful explanations for the user to understand better the LMs decision-making which, in turn, increases trust and reliabil- ity. To further enhance the prototypical network with human supervision, we propose an interactive learning setting (iProto-Trex) that allows users of any knowledge to give feedback and improve the model (Ribeiro et al., 2016), moving beyond purely data-driven approach.
To summarize, our contributions are as follows: We (i) introduce prototype networks for transformer LMs that provide explanations and (ii) show that they on par with non non-interpretable baselines on classification tasks on different architectures. Furthermore, to improve prototype networks, we (iii) provide novel interactive prototype learning setup accounting for user feedback certainty.
We proceed as follows. We start by briefly reviewing related work of interpretability in NLP. Then novel interactive learning setup combining
introduce Proto-Trex networks, including planatory interactive learning with prototype networks. Before concluding, we discuss faithful ex- planations and touch upon the results of our experimental evaluation.

---

Under review as a conference paper at ICLR 2022
Class
Q+ LM
Class
Q+ LM
Expl.
i/Proto-Tree Trex
post-hoc Expl.
Explainer
Q: Staff is available to assist, but limited to the knowledge and understanding the individual.
Post hoc
Staff is available assist, but limited to the knowledge and understanding of the individual.
Proto- Trex
They literally treat you like you are bothering them. No customer service skills.
iProto Trex
They offer bad service.
Figure I: i/Proto- Trex compared to a post hoc explanation on sentiment classification. The input query (top row) is classified (as negative), and three explanations are provided. The first (post hoc) explanation is provided by LIT (Tenney et al.) (020)) with LIME (Ribeiro et al. (0166). The intensity of the color denotes the influence of certain word: Blue (red) color indicates positive (negative) sentiment. Middle and bottom row explanations are provided by (interactive) Proto- Trexo-Trex networks.
2 TOWARDS THE EXPLAINABILITY OF TRANSFORMER LANGUAGE MODELS
To open the black-box of transformers, we use (PProtoTree networks built upon post hoc interpre- tation methods, case-based reasoning approaches, and explanatory interactive learning.
Post hoc interpretability. Various (post hoc) interpretability methods focus on different parts of the transformer architecture. |Atanasova et al. (2020) and [eelinkvv & Glass) (2019) provide overviews of this fast developing field. Generally, there are methods that analyze word representations (Voita
al. 2019a), the attention distribution throughout the model (Jain & Wallace 2019; Wiegreffe & Pinter 2019), and the (attention and classification) model heads (Voita et al. 2019b Geva et al., 2021). Other approaches such as Geva et al. (2020) focus on the forward layers. Gradient-
based approaches, such as asSuudaraaraaan et al. (2017) and Smilkov et al. (2077) can generally be used to trace gradients, while influence functions (Koh & Liang 2017) trace model parameter changes throughout LM (Han al.] 2020). While backtracking all the model weights might be possible, such explanations can only tell us which part of the input they are looking at (Chen et al./ 2019) and humans are ill-equipped interpret them (Stammer al. 202I). So instead, we aim for more intuitive and sparse explanations: dell-descriptive but short sequences as prototypes. These are not only able reveal the parts of the input is looking at but also show prototypical cases similar to those parts.
Case-based Reasoning in Deep Neural Networks. Our work relates most closely to previous work demonstrating the benefits of prototype networks in Computer Vision (Li et al.
2018
Chen et al.
2019) as well as for sequential data by combining them with RNNs (Hase et al. 2019 Ming al, 2019). More precisely, in contrast to post hoc interpretation methods, prototype networks employ explanations in the, egg. classification process, i.e. classify sample by comparing several parts which could be words of a sentence- to (leared) prototypical parts from other samples for given class. If things look similar, they are classified similarly. Humans behave very similarly, which is called case-based reasoning (Clifton & Frohnsdorff] 200T). The prototypical parts provided by the network help the user to understand the classification. Tab. [T] illustrates the benefit of proto- typical explanations. One can observe that the post hoc explanation can leave the user clueless, while the prototypical explanations can help better understand the network's 's decision. Accordingly, prototypes are an additional method in the interpretability toolbox, extending current post hoc meth- ods. As described, the reasoning process of prototypical networks is qualitatively similar to that of humans (Clifton Frohnsdorff] 2001 Chen al., 2019). previous approaches, however, the training of such networks was entirely data-driven. Therefore, like Chen et al.) (2019), in our work, we not focus on quantifying unit interpretability of prototypical networks, but extend learning the reasoning process of our network by Explanatory Interactive Learning (Schramowski et al., 2020).
Explanatory Interactive Learning (XIL) incorporates explanatory supervision in the learning pro- cess by involving the human user. More precisely, human users can ask the model to explain pre- diction. They can then respond by correcting the model if necessary, providing a slightly improved bbut not necessarily optimal- feedback the explanations (Stammer et al. 2021). Interactive learn- ing and, in particular, explanatory interactive learning showed not only to increase the explanation quality of deep black-box models but also the trust of the user (Ross al. 2017 Selvaraju al.,
2

---

Under review as a conference paper at ICLR 2022
Transfic Transform former
Transfun Transforme former
Softmax
Softmax
slnke..p
Jincar
Jinear
simnte.
MintP.R
Input
f
Prototype Layer P
R
Ouiput
Input
Prototype Layet
x
Ourput
(a) Sentence-Level
(b) Word-Level
Figure Architecture of the Prototypical- Protoypical-Transfoomm Transformer Explanation Network (Proto-Trent (a) Sentence-level case uses transformer to compute sentence-level embedding explanations, while (b) word-level case provides multiple word-level explanations and therefore requires an additional word-sclection layer.
2019 Teso & Kersting, 2019 methods, we avoid tracing gradients. That makes our approach particularly efficient, as we only change prototypes (cf. 4).
Schramowski al. 2020). In contrast previous (post hoc) XII.
PROTO-TREX: PROTOTYPE LEARNING FOR TRANSFORMER LMS
The prototype network (Proto-Tree Trex) architecture builds upon large-scale transformer LMs, summa- rized in Fig. 2 In the following, we describe the general idea of prototypical networks and our Hansformer-basee prototype architecture and its modules in more detail.
3.1 PROTO-TREX NETWORKS
General Idea. Proto-Trex is a prototype network, where prototypes are representative observations in the training The classification is based on the neighboring prototypes, i.e. input query is classified as positive when the network thinks it looks similar to a certain (near) positive proto- type within the training set. In more detail, the reasoning and classification processes use the same module, i.e. they use the similarity between query and prototypes. As both reasoning and classifi- cation use the same module (in parallel), we consequently have an interpretable model. In particular, (interpretable) prototypical explanations help verify Right for the Right Reasons (Ross et al. 2017).
Architecture. Specifically, the model, in our case the pre-trained transformer f. creates context embedding from the input sequence This embedding contains useful features for the predic- tion and is then passed on to the prototype layer. This layer computes the similarity between the embedding and each of the trainable prototypes Pj from the set of prototypes P. The prototypes are learned during the training process and represent prototypical (sub-) Sub-)sequeecees in the training data. We use transformer models both the sentence-level (a) and the word-level (b), resulting in a single input representation for the sentence-level transformer and for word-level. To compare the word-level embeddings e; with the prototypes, subsequences according to the desired prototype length k. The resulting similarity values are passed to the final linear layer g with weight matrix w, and no bias, mapping from (the number of prototypes) to the number of classes. Finally, the output values are passed on to a softmax producing class probabilities and predictions t.
have patch them (2 (patch(e:)) into
3.2 PROTO-TREX LOSS
Optimization of prototype networks aims at maximizing both performance and interpretability. To this end, our
loss combines performance and interpretability with prior knowledge of the prototype network's structure. Let us illustrate this for the word-level case (the sentence-level case results from e e-z). L:=
min 1 Σ CE(t, vi) Ci p) AeSep(z, p) + >gDistt(2, p) AaDivers(p) As [gll P,w, n
3

---

Under review as a conference paper at ICLR 2022
where X weights the influence of the different terms and n is the number of training examples. The first term is the cross-entropy (CE) loss optimizing the predictive power of the network. The second term (CIst) clusters the prototypes w.r.t the training examples of the same class, maximizing simi- larity to them (we rewrite maximization as minimization terms), and the separation loss minimizes the similarity other-class instances:
Clst(z, p) n JPP ZEZ
1 min min sim(z.P,): Sep(z. p) = 1 n JP; ZE2
Σ min min sim Pf) (2)
n
Together, Clst and Sep push each prototype to focus more training examples from the same class and less on training examples from other classes. Both are motivated by ProtoPNet (Chen et alal., 2019). To get prototypes that are distributed well in the embedding space, we introduce two additional losses, a distribution loss (Li et al. 2018), assuring that a prototype is nearby each training example, and a diversity loss (Ming al. 2019)
m
1 m
Distr(z, p)
m7 Σ
min min sim p,); Divers(p) = m Σ
min sim P,) -PEEP
(3)
sEiL,n ZEZ
In contrast to the other terms, the diversity loss does not compute similarities between embeddings and prototypes but between prototypes themselves. It is another way of distributing prototypes in the embedding space as it maximizes the distance between prototypes, preventing them from staying at the same -not necessarily optimal- location. This is especially helpful in the case of multiple prototypes for a single class, encouraging them to represent different facets of the class. If they otherwise too similar, information is gained and resulting in redundant prototypes together with the class-specific loss encouraged by the cluster loss (Clst) and the separation loss (Sep). this helps compute prototypes that focus solely on their class. Otherwise, prototypes leading to negative reasoning. Also, we clamp the weights of the classification layer with min (Wy- to avoid negative reasoning (Chen et al., 2019). Finally, the last term of the Proto- Trex loss (Eq. [ is an Ll-regularization term of the last layer (g), which prevents the network from overfitting or relying too much on single prototype.
can get ambiguous
3.3 SIMILARITY COMPUTATION
Computing similarities is essential aspect of Proto-Trex. For prototypes to represent certain aspects or features of the input distribution in the embedding space, we compute the similarity sim(e, p) between embedded training example and prototype. A distance minimization can replace each similarity maximization max ai p) = min dist(c. (. p). We here follow common practice and explore both the L2- or the cosine similarity:
P,llz
L2-norm
sim(e, p.))
e'p
cosine similarity
|
where the index denotes specific prototype. For each training example, there is an embedding which is compared to all m prototypes p. i.e. we get m similarity values for each embedding.
The 1.2-norm assumes Gaussian prior, which be wrong assumption, which is why also investigate cosine similarity. While 1.2-norm computes the distance between two vectors, cosine similarity measures the angle between them. Both, but especially cosine similarity, choices for NLP tasks (Manning et al. (2008). However, cosine similarity is more robust here than L2-norm as the distance's 's magnitude between vectors has no influence due to normalization.
natural
3.4 SELECTION FOR WORD-LEVEL PROTOTYPES
Since learning prototypes for LMs pre-trained on word-level representations is more involved than for the sentence-level, let us focus on them here; the sentence-level case naturally follows from the discussion. Word-level prototypes are generally sensible as explanations should consist of sparse sequences, at best focusing only on subsequences of the input sentence. As sequences can also be ambiguous and contain little information, enforce the prototype to be similar to the relevant subsequences, containing the most information. But how do we select the most informative words?
combine different word embeddings (patch()) and
4

---

Under review as a conference paper at ICLR 2022
The food was delicious and the service great The food was delicious and the service great
self-
The food was delicious and service great
attention
food
delicious
service great
The food was delicious and the service great The food was delicious and the service great
patch
food delicious (food service (..))
(b) Word selection with self-attention. First, tention weights are calculated for each word, and most attended-to words are selected; then all patches w.r.t. prototype length are calculated.
(a) Sliding windows approach: Convolutions sliding over a sequence, (1) without and (2) with additional dilation in the convolutional window.
Figure 3: Word selection for word-leve Proto-Tres with (a) convolution and (b) attention.
naive approach would be to simply compute all possible patches (word combinations) of the input sequence and compare them to the prototypes to find the best patch for classification the most important subsequence should then be similar to a certain prototype. Unfortunately, for long input sequences with length / it becomes hard to compute all possible word patches of length k. in this case 2 = (t) To solve this problem without losing too many valuable patches, the following two approaches are sensible ideas.
(a) Sliding windows naturally reduce the number of patches. A sliding window is convolution that selects a certain part of the input according to the window size and then sliding to the next part. The main disadvantage of sliding windows is the relatively rigid structure of window. This is problem- atic in the context of NLP tasks that often have long-range dependencies. We introduce dilation to loosen this. Dilation facilitates looking direct word neighborhoods but also more distant ones to capture long-range dependencies. This "convolutional" approach is illustrated in Fig. 3Ja). We note that applied to the contextualized word embeddings of transformer LMs, the convolutional ap- proach without dilation should contain not only local information but also some global information. Unfortunately, this information is stored in the embeddings and cannot be visualized easily.
As an alternative, the (b) Self-Attention approach adds a self-attention layer after the transformer f and before the prototype layer p filter irrelevant words (with low attention-scores), cf Fig. ()). The purpose of this self-attention layer differs from the one in the transformer LM itself. The here used attention mechanism selects the most important words, and the embedding representation remains untouched. To still provide as much information and variety as possible, the number of selected words nh of the attention layer which are passed into the distance computation is hyper- parameter, and we chose it to be twice much as the length of single prototype however clamped by the threshold kim: nu = mén mar(kumm k). 2k). The threshold can be set w.r.t. computational efficiency. This means having a prototype of length l- = 4 and kim 10, the attention layer selects n 88 words yielding in this example Z (t) -0 patches for the distance computation.
3.5 DRCODING VIA NEAREST NEIGHBOR PROJECTION
Proto-Trex networks encode prototypes in the embedding space. Consequently, they cannot simply be decoded from the transformer embedding space, as this space and textual data are categorical and continuous. To overcome this, we assign, i.e. project, each prototype to its nearest neighbor the training data (where the textual representation is available) in an intermediate training step. Thereby, we ensure that the prototype really represents what it actually looks like, which also increases the interpretability of the Proto-Tres Trex network. The final training step (after the nearest neighbor projec- tion) then fine-tunes the classification head to adapt to the projected prototypes. Doing so has the advantage of being decoded precisely on the spot and providing more certainty for the explanatory power. The dinadvantage. however, is that a prototype may be located sub-optimally, which could lead to performance losses. We faced this issue especially for word-level prototypes and found the projection incligible for these highly contextualized word representations. Instead, for word-level, only use the nearest neighbor for approximating the prototype to provide the explanation.
4 INTERACTIVE PROTOTYPE LEARNING
Recent case- based reasoning approaches in Computer Vision and NLP usually assume a large vol- ume of data for training (Li et al., 2018 Chen al. 2019 Hase et al. 2019), with little no
5

---

Under review as a conference paper at ICLR 2022
iProto-Trex
interact
taste dessert was great.
The place was busy however the service was awesome!!
Prototypes
(positive)
awessmn food
anvesome food
Hellogud Food
pizza was super delicious and had a lot of onlons on top.
LM
awful location
ho one was taking care, am really disappointed with them.
nice atmosphere
rude service
explain
Figure 4: Interactive Prototype Learning: iProto-Trex classifies the input and gives the user an explanation based on prototype. The explanation is highlighted in bright blue. If the user is dissatisfied with the given explanation, they can replace it with a self-chosen sequence.
user feedback during the model building process. However, user knowledge and interaction, in par- ticular via explanations, can be valuable already in the model building and understanding phase, significantly reducing the amount of data required, avoiding Clever-Hans moments early and increasing the explanation quality of the model and, in turn, the trust of the user, see c.g. Teso & Kersting (2019); Schramowski et al. (2020).
Interactive Proto-Trex. To address this, we propose an explanatory interactive learning approach for Trex networks, called iProto-Tres
illustrated in Fig. Specifically, carried out during the training progress by freezing and evaluating the current state or post hoc training, the interaction takes the following form. At each step, the Proto-Trex networks provide prototypes explana- tions of a classification. The user responds by correcting the learner, if necessary, in our case, the prototypes. For this, iProto-Trex provides range of opportunities for interacting with prototypes.
Specifically, iProto-Trex distinguishes between weak-knowledge and grrong-knowledge interac- tions. In strong-knowledge interactions, users are certain about their feedback (which could require high-level expertise). In this case, IProto-Trex provides options to remove and add prototypes. Re- moving a prototype is also helpful if the network has learned redundant prototypical sequences or prototypes that cover all critical aspects of the task at hand. If users are content with all the present prototypical sequences, they can also add new prototypes (Replacing a prototype the same as removing plus adding prototype). After these interactions, the subsequent classification layer is retrained to integrate and balance the new prototype setup in its decision decision-making. making.
In weak-knowledge interactions, users only state content discontent with prototype based on their intuition. They do not need to know what a replacement should exactly look like, which is good trade-off between user knowledge and loss optimization of the network. To this end, PProto-Trex uses re-initialization and fine-tuning instead of providing an explicit replacement, users freeze prototypes they like while relearning those they are dissatisfied with. The difference between these approaches is that users can express how well the current prototype represents a particular task aspect. Another form of weak-knowledge interaction, which iProto-Trex provides only for sentence- level prototypes, is pruming the sequence length of prototypical explanations, i.e. compressing it the essentials. In order limit meaning changes of sequence, threshold is provided for the cosine similarity of the pruned and the original version. We initially set this threshold to 0.8, but users can set the threshold as they like, resulting strong- stoollekn interaction.
Soft User Feedback. iProto-Trex's 's interaction methods, especially the knowl-kn ones, require users to quite confident about their feedback -they have to be experts. To elevate this burden, we propose soft feedback mechanism, using a loss based on the prototype similarity as
Linteract >6 max (max(sim(pola, pn) Pau
with c E [0, 1]. Instead of directly replacing the selected prototype Pold. this soft interaction loss pushes the prototypes to be similar to the desired prototype Pnew as suggested by user. Moreover, users can control how strong iProto- Proto-Trex&toe should incorporate their interventions by setting a certainty value e between 0 (low certainty) and 1 (high certainty).
5 FAITHFULNESS OF PROTOTYPICAL EXPLANATIONS
A question that naturally arises when dealing with explainable AI is how to evaluate the quality of an explanation. One of the key metrics in NL.P and Computer Vision is the faithfulness of explanations

---

Under review as a conference paper at ICLR 2022
(Jacovi & Goldberg 2021 De Young et al. 2020), i.e. whether given explanation represents the true reasoning process of the black-box model. We apply and adjust this metric to text-baseds prototype networks. First, evaluate the faithfulness of Prex as whole Young al. 2020) by showing that the class probabilities change significantly if we perturb the input. We follow their approach of computing comprehensiveness and sufficiency. To calculate comprehensiveness, we remove the rationale in each input sequence and evaluate the changes in the class probabilities for the prediction. Second, we perturb the prototype layer by removing the top explanation for a given test sample and compute the loss in accuracy. This helps to identify to what extent the decision was actually based on the explanation given. Since our prototypical explanations are already short sequences, removing the rationale in the explanation is similar to removing the entire prototype.
6
EXPERIMENTAL EVALUATION
Our intention here is to investigate how good prototypes help to understand transformer L.s.. To this end, we evaluated P/PrrtooPPiiiie -rex explanations on three benchmark datasets: MovieReview (Pang et al. Open Yclp and Jigsaw Toxicity 2 We compared five pre-trained LMs (GPT-2 (Radford et al. BERT (Devlin et al. 2019). DistilBERT, (Sanh et al. 2020), SBERT (Reimers & Gurevych
2002
2019
2019
and the text-encoder of CLIP (Radford et al. 2021)) to investigate three questions:
(Q1) How much does adding prototype layer affect the performance of (non-interpretable) LMs, i.e. a classification head defined by two fully-comnected non- non-linear layers?
(Q2) How does the performance change after and during interaction with the model explanations? In particular, we investigated the different modules of Proto-Trex networks on sentence- and word-level and the interaction between users and the prototypical explanations.
(Q3) How faithful are the given explanations, i.e. how well does the given explanation represent the true reasoning of the black-box model?
We present qualitative and quantitative results and refer to the Appendix for additional details on the experiments and our implementation. as well as additional qualitative results. If not stated otherwise, the Proto-Trex architecture includes sentence- and word-level embeddings, the convolution module without dilation to select word-tokens, and cosine similarity compute the similarity between pro- totypical explanations and input query. We optimized the prototype and classification module with the proposed loss (Eq. D and initialize the prototypes randomly. In each experiment, the number of prototypes is m = 10, and the largest variant of the corresponding pre-trained LM is evaluated.
(Q1,2) Trade off accuracy and interpretability. Tab. summarizes the experimental result of Proto-Trex network based on different sentence- and level LMs. As one can see and expected from the literature, interpretability comes along with a trade-off in accuracy. The trade-off is gen- erally higher on sentence-level LMs, partially due to the nearest neighbor projection (cf. Appendix for direct comparison). However, the difference between traditional LMs and Proto-Trex LMs is often marginal and task-dependent (e.g. DistiBBRRT on Yelp and Movic). Surprisingly, in the case of BERT and GPT-2 (on Yelp and Toxicity), the Proto-Trex network is outperforming the baseline LMs. Most interestingly, one can see that the user may boost the performance of the corresponding Proto-Tres network interactively (iProto- -rex). Overall, i/Proto-Tres is competitive with state-of- the-art LMs while being much more transparent.
(Q1) Ablation study. Next, we investigate different Proto-Trex module choices for performance, cf. Tab. [D), and explanation provision impact. While input- and prototype similarity computation does not explicitly correlate with the prototypes learned -yet we assumed it to be the better choice-, the word-selection module also impacts the explanation outcome. Furthermore, cosine similarity is not only more accurate but also converges faster. In terms of accuracy, the convolutional word selection outperforms the attention module. More importantly, we also observe an advantage for the provided explanations; namely, the attention module tends to select punctuation and stop words. According to Ethayarajh (2019), this is because punctuation and stop-words are among the most context-specific word representations-thee are not polysemous themselves but can have an infinite number of contexts. contrast, the convolution module easier to interpret as words are coherent.
bbttss//www.yccp.co/dttaaa
7

---

Under review as a conference paper at ICLR 2022
b)
Word-Selection
Similarity
a)
Language Model
Yelp
Movie
Toxicity
Proto- Trex LM
Conv.
Attn.
Cos.
L2
inter. word- level sent. -ovo
SBERT
94.92
84.56
084.40
BERT
92.10
89.66
92.10
91.34
SBERT (Proto-Trem,
93.59
80.05
73.19
GPT-2
95.32
94.16
95.32
94.99
CLIP
93.78
75.49
80.82
DistilBERT
92.71
91.09
92.71
92.05
CLIP (Proto-Tree Trex)
87.16
63.52
67.75
SBERT
93.59
93.13
BERT
89.41
61.45
76.88
CLIP
87.
86.88
BERT (Pexto-Trex)
92.10
75.51
79.35
GPT-2
93.78
87.05
81.56
GPT-2 (Proto-Trex)
95.32
84.57
84.14
Faithfulness
Acc.
DistilBERT
92.91
79.62
81.57
Proto- Proto-Trex LM
Comp.
Suff.
before
after
DistilBERT (Proto-Trex.
92.71
78,64
83.39
SBERT
0.22
-0.08
80.05
69.66
SBERT (iProto-Trex)
93.81
80.24
73.36
iSBERT
0.21
-0.08
80.24
70.76
GPT-2 (iProto-Tree
e95.25
84.80
84..1
GPT-2
0.12
0.02
84.57
49.91
iGPT-2
0.13
0.02
84.80
55.74
Table 1: Quantitative Results. a) Average Accuracy of Proto-Trex with different LMs on different datasets. Best ("""") and runner-up (**"") are bold. b) Ablation study of Proto-Trex module choices on Yelp dataset. c) Evaluation of i/Prrtoo Trexo regarding faithfulness on MovieReview dataset. Mean values over 5 runs reported. The confidence intervals can be found in Appendix Tab. 5
Importance
Query: I really like the good food and kind service here.
0.52-8.07 4.20
P8: Oliver Rocks! Great hidden gem in the middle of Mandalay! Great friends great times
P4: Incredibly delicious!!! Service was great and food was awesome. Will definitely
sentence- level
0.84-3.01= 2.55
come back!
P6: found this place very inviting and welcoming. The food is great so as the servers.. love the food and the service is phenomenal!! Well done everyone..!
0.90-1.21= 1.10
P2: This place is really high quality and the service is amazing and awesome! Jayden was very helpful and prompt and attentive. Will come back the quality is good and service made it!
0.79-1.16= 0.92
P4: Incredibly delicious!!! Service was great and food was awesome. Will definitely
interaction
0.84-5.02 4.22
come back!
0.92-2.10 1.94
P6: found this place very inviting and welcoming. The food is great the servers.
0.76-2.13= 1.62
P2: This place is really high quality and the service is amazing and awesome!
0.73-0.71= 0.52
PIO: Fun and friendly atmosphere, fantastic selection. The sushi so fresh
Table 2: Provided Explanations by Proto-Trex networks. Top-four explanations for the query (top row) are provided for sentence-level networks. Colored boxes illustrate the advantage of pruning sentence explanations (interaction). Importance scores (left, bold) are calculated with cosine simi- larity and classification weight.
(Q1,2) Provided explanations. To analyze the provided explanations, we consider the SBERT based Proto-Trex networks, of. Tab. [ trained the Open Yelp dataset. Tab. [ shows how Proto- Trex provides users with explanations. These explanations are given as prototypical sequences that correspond to query. In addition, Proto-Trex provides corresponding importance scores. indicating the significance of an explanation for the classification. We show four sentence-level prototypes for the query that users can quickly extract the important aspects that help understand the classification. However, one observe that the sentence-level explanations are difficult to interpret about the query, which also demonstrates the demand for (further) interaction (cf. colored boxes with prototypical explanations P6 and P2 in Tab. 2 The pruned Proto-Trex provides less ambiguous and easier to interpret sequences.
sometimes lacking sparsity and
(Q2) Interactive prototype learning. In the previous experiment, pruning has already shown the benefits of adapting explanations to users' preferences. In order to further evaluate our interactive learning setup, incorporated certainty (c) and evaluated its effectiveness. Since sentence-level
8

---

Under review as a conference paper at ICLR 2022
Type of interaction
Acc.
Prototype
Horrible customer service and service does not care about safety features. That's all I'm going to say. Oh they also don't care about their customers Terrible delivery service. People are mean and don't I care about their cus- tomers service. I will not ever come back to this place. Also the food is small, very good and too expensive.
no interaction
93.64
re-initialize
93.80
Terrible delivery service. People are mean and don't care about their cus-
soft replace (0.5)
93.81
tomers service. I will not ever come back to this place. Also the food is small, very good and too expensive.
really don't it recommend this place. The food not good, service bad.
soft replace (0.9)
93.79
The entertainment is so cheesy. Not good
soft replace (1.0)
93.79
They offer had service.
Table 3: Interactive learning: Different user interaction methods with accuracy on Yelp reviews. In- teraction changes a redundant prototype into a user-selected one incorporating more user knowledge without significantly impacting the accuracy.
prototypes allow for better coherence, this is where we will focus the interactive learning. Tab. shows an influential, iee. high importance value, sentence-level explanation (yet no interaction) for negative restaurant reviews on the Open Yelp dataset. Assuming user is dissatisfied with an expla- nation yet uncertain about what good explanation would entail: Applying re-initialization (most uncertain interaction technique) confirms the user's intuition of "weak" component, as the slight increase in accuracy indicates. More importantly, we can already observe that users can influence the network's decision process based on their preferences without performance loss. However, the revised explanation is still not sufficiently interpretable. Therefore, we considered incorporating ex- plicit user feedback here. In this case, the phrase "They offer a bad service" served as soft replace- ment for the model's provided explanation, and the user applied it with different levels of certainty. First, a low certainty value (c 0.5) results in the same explanation as before. This is because the similarity of any two prototypes with clearly negative sentiment is higher than a certainty threshold of 0.5. As the user gets more certain, he gradually increases the threshold. Finally (c 1). the user simply replaced the prototype to obtain his desired solution without trade-off in accuracy. In summary, our results demonstrate that interactive learning is a solid method to counter the network's incapability in consistently providing interpretable prototypes along with high accuracy.
(Q3) Faithful prototypes. To analyze faithfulness, we first follow Young et al. (020) and use the MovieReview dataset as they provide human-annotated rationales for this dataset. We com- pare the classification probabilities of the samples (De Young et al., 2020) with the ones where the rationale is removed (comprehensiveness) and with the ones with only the rationale (sufficiency). Tab. [c) shows that i/Proto- Trex scores high for comprehensiveness. indicating that the network is focusing on the input rationale for the classification, while low sufficieney scores indicate that the surrounding context has little to even poor impact on the prediction. This shows that i/Proto-Tres are generally faithful classifiers. i.e. the aaman--nnotated rationales agree with the internal rationales in the model. Furthermore, we remove the top (prototype) explanation for the classification and exam- ine the change in accuracy after the removal to show that is the prototype that faithfully captures the rationale. Tab. [c) shows a drastic decrease in accuracy for both models after the explanation removal, confirming the faithful contribution of prototypical explanations to the classification.
7 CONCLUSION
Large-scale transformer LMs, like other black-box models, lack interpretability. We presented meth- ods (prototype networks) to incorporate case-based reasoning to explain the LM's decisions. Despite the explanatory power of prototypical explanations, challenges regarding the quality of their inter- pretability still exist. Previous applications lack human supervision, although case-based reasoning is a wahhhunninnsn inspiredd approach. Therefore, we propose an interactive prototype learning setup to over- come these challenges and improve the network's 's capabilities by incorporating human knowledge with the consideration of knowledge certainty. Future work covers improving methods for patching the input for word-level prototypes enable projection for higher interpretabilityy

---

Under review as a conference paper at ICLR 2022
8 ETHICS AND REPRODUCIBILITY STATEMENT
The article presented here approaches the explainability of transformer language models through case-based reasoning in the form prototypical explanations. The interpretable prototype module that we learned is built on pre-trained transformer language model. As these models are trained on data that not publicly available, it remains an open question whether any kind of bias is inherent to the model other potential misuse emerge as these models are trained with weak to no supervision.
whole (Bender et al. 2021). In addition, concerns about privacy violations and
The code to reproduce the results can be found in our publicly available repository. Furthermore, we provide the trained Proto-Trex models to obtain explanations.
REFERENCES
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9256-3274, November 2020.
Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics (TACL), 7:49-72, March 2019.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Madeleine Clare Elish, William Isaac, and Richard S. Zemel (eds.), Conference Fairness, Accountability, and Transparency (FAccT), pp. 610-623, 2021.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. In Proceedings of the 2019 Conference on Advances Neural Information Processing Systems (NeurlPS), pp. 121 2019.
James R. Clifton and Geoffrey Frohnsdorff. Applications of computers and information technology
18. In Handbook of Analytical Techniques Concrete Science and Technology, pp. 765-799. William Andrew Publishing, Norwich, NY, 2001.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies ((NAACL)), pp. 4171-4186, 2019.
Jay Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 4443-4455 Association for Computational Linguistics, July 2020.
Shuoyang Ding and Philipp Koehn. Evaluating saliency methods for neural language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (NAACL), pp. 5034-5052, June 2021.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the ge- ometry of bert, elmo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 55-65, 2019.
Mor Geva, Roei Schuster. Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint 201.:22012..4913,, 2020.
Mor Geva, Uri Katz, Aviv Ben-Arie, and Jonathan Berant. What's in your head? emergent behaviour in multi-task transformer models. arXiv preprint arXiv:2104.06129 2021.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Gordon and Benjamin Van Durme. Reporting bias and knowledge acquisition. In Pro- ceedings of the 2013 Workshop on Automated Knowledge Base Construction (AKBC), pp. 25-30, 2013. ISBN 9781450324113.
Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 5553-5563, July 2020.
Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. Interpretable image recognition with hi- erarchical prototypes. Proceedings of the AAAI Conference Human Computation and Crowd- sourcing, 7(1):32-40, Oct. 2019.
Alon Jacovi and Yoav Goldberg. Aligning Faithful Interpretations with their Social Attribution. Transactions of the Association for Computational Linguistics, 9:294-310, 2021.
Sarthak Jain and Byron Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pp. 3543-3556, June 2019.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1885-1894, 2017.
Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the 2018 Conference on Artificial Intelligence (AAAI), pp. 3530-3537, 2018.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008. ISBN 0521865719.
Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. Proceedings of the 25th International Conference on Knowledge Discovery and Data Mining (KDD), pp. 903-913, 2019.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 79-86, 2002.
Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, and Sameer Singh. An empirical comparison of instance attribution methods for NLP. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pp. 967-975, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. CoRR, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd International Conference on Knowledge Discovery and Data Mining (KDD), pp. 1144, 2016. ISBN 978 1700342222
11

---

Under review as a conference paper at ICLR 2022
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi- Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence (IJCAI), pp. 2662-2670, 2017.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1910011110001100000 2020.
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nat Mach Intell, pp. 476-486, 2020.
Ramprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry P. Heck, Dhruv Batra, and Devi Parikh. Taking HINT: leveraging explanations to make vision and language models more grounded. In 2019 International Conference on Computer Vision (ICCV), pp. 2591-2600, 2019.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:2012.09699 2017.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revis- ing neuro-symbolic concepts by interacting with their explanations. In Proceedings of the 2021 Conference on Computer Vision and Pattern Recognition (CVPR), 2021. to appear.
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 3319-3328, 06-11 Aug 2017.
Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. The language in- terpretability tool: Extensible, interactive visualizations and analysis for NLP models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pp. 107-118, October 2020.
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning- Proceedings of the 2019 Conference on AI, Ethics, and Society (A/ES), pp. 239-245, 2019.
Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the trans- former: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP)-JCNLLP), pp. 4406, November 2019a.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL.), pp. 5797-5808, July 2019b.
Sarah Wiegreffe and Yuval Pinter. Attention is not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-JJCNLP), pp. 11-20, Hong Kong, China, 2019. Association for Computational Linguistics.
A APPENDIX
A.1 DATASETS
The benchmark datasets mentioned above provide a fixed test set except the Yelp Open dataset. For the Yelp Open dataset we randomly select 200 000 training examples and split them into train (70%). validation (15%) and test (15%) set. We split the Jigsaw Toxicity train set into train (20%) and validation (80%) set. Before the split, we filter out long sequences (num tokens > 40) for each dataset. This is required because transformer-based LMs can only handle sequences of limited
12

---

Under review as a conference paper at ICLR 2022
length, especially CLIP, and long sequences also cause the other sequences of set to be padded to the same length. which, in turn, produces many padding tokens. For the tokenization we use the GPT2-tokenizef" We apply grid search for hyperparameters optimization. We report the cross- validated results. For the faithfulness computation we use the eraserbenchmark repository and dataf)
A.2 TRAINING
The Proto-Trex networks trained with and evaluated different datasets. We use PyToreh for the implementation. We optimize our model with Adam optimizer with hyperparameters 3 (0.9, 0.999) and 10-8 We use base learning rate of Iroase 0.001 and apply learning rate warm-up and scheduling that is here a linear decay. The learning rate is then given as
step;
c step:
Ir = Iraase min
Cwap
Ewup
where step; is the current step, e the total number of epochs and ewap the number of warm-up epochs. The warm-up takes place for wwup min (10, 2g) epochs. Warm-up reduces the dependency of early optimization steps that may cause difficulties in the longer run. Also, we weigh the cross-entropy loss with the class occurrences to re-balance the influence of an unbalanced dataset. The same is done in the accuracy computation where a balanced class accuracy is computed. For regularization, apply an Ll-regularization a network may tend to overfit. To counteract this, we evaluate the model every 10th epoch on the validation set and keep the model that yielded the best validation result. Furthermore, class mask to assign each class to the same number of prototypes. This is used and enforced by the class-specific losses (CIst and Sep). Changing the balance of the class assignment can sensible to correct for imbalance in the dataset or focus on a specific class its sentiment is more polysemous or generally more relevant. We show the hyperparameters used for our experiments, found by grid search. The found hyperparameters were often very similar across models and datasets as the parameters mainly aimed interpretability, not only performance. Note, not all hyperparameters were grid-searched simultaneously. We first searched for the best combination of À to À5 and searched for ÀS separately because the interaction loss depends on interaction and does not directly impact the model performance during training. The best performing value for λ was 0.5. For the attention-based based word selection we choose the number heads to be and k = 4; kism = 10.
the weights of the last linear layer. During the training process,
set up
A.3 SENTENCE- VS. WORD-LEVEL LOSS
For clarification, we present the loss terms for sentence-level (/Proto-Tre networks in more detail: L:=
min P.w. n
CE(t,, v) + Clst(e,, p) + AzSep(e, p) + AgDistr(e, p) DeDivers(p) + As [,wl
(5)
where
1
Clst(e, ==
min sim(e; Py)
(6)
p.p
Sep(e, ()) := min sim(e; Pj)
7
i=1 e
Distr(c, p) ==
min sim(c, p,) [E[,m]]
7
httass/www..rrserreenccnaat https://pytorch.orrg/
13

---

Under review as a conference paper at ICLR 2022
X2
A3
M
A
{0, 0.1, 0.2, 0.5}
(0,0.1,0.22
0.5}
10,0.1,00 0.2)
(0,0.1,
0.33
(0,0.001}
{0, 0.1, 0.5, 1.0]
2)
LM
A2
A
A
A5
SBERT
0.5
0.2
0.2
0.3
0.001
Movie
CLIP
0.5
0.2
0.2
0.3
0.001
BERT
0.2
0.2
0.2
0.3
0.001
GPT-2
0.2
0.2
0.1
0.3
0.001
DistiBBERT
0.2
0.2
0.1
0.3
0.001
3)
LM
X2
X
X4
SBERT
0.5
0.2
0.1
0.3
0.001
CLIP
0.5
0.2
0.2
0.3
0.001
Yelp
BERT
0.5
0.2
0.1
0.3
0.001
GPT-2
0.5
0.2
0.2
0.3
0.001
DistilBERT
0.5
0.2
0.2
0.3
0.001
4)
LM
A1
Xx
A
Às
SBERT
0.2
0.1
0.1
0.1
0.001
Toxieity
CLIP
0.2
0.1
0.2
0.1
0.001
BERT
0.2
0.2
0.2
0.1
0.001
GPT-2
0.2
0.2
0.1
0.1
0.001
DistiBBERT
0.2
0.2
0.1
0.1
0.001
Table 4: 1) Search space for hyperparameter grid search. 2-4) Hyperparameter setup for Proto-Tres models on the three different datasets.
and
Divers(p) 1 m
Σ
min sim(pj.P)
(9)
m ppppP
A.4 RESULTS WITH CONFIDENCE INTERVALS
Due to a lack of space we shifted the results with confidence intervals to the Appendix which you see in tab. 5
A.5 PROJECTION
We show the impact of projecting the prototypes onto their nearest neighbor in Tab. A.5 The projection evaluated for sentence-level Proto-Trex networks. One can see the trade-off between interpretability and accuracy introduced by projection.
A.6 PROVIDED EXPLANATIONS
Here we show the explanations that Proto-Tres provides. For each network, we use 10 prototypes, and for word-level networks, we use prototype length of 4 tokens. First we extend the results of Tab. 2 for the word-level case, shown in Tab. 7 Then we show prototype lists for the models on different datasets. In Tab. g we show prototypes for Proto-Trex based on the GPT-2 transformer and in Tab. the prototypes based the SBERT transformer. Both tables show the prototypes for the Yelp Open and MovieReview dataset. The results for the Jigsaw Toxicity dataset can found in external document in the codebase. A CONTENT WARNING: the content in this document can be disturbing due to highly toxic texts! These results do not represent the authors' opinion and show prototypical explanations provided solely by Proto-Trex.
Tab. 10 shows the pruned prototypes of -Proto-Trex from Tab. I(... Pruning cuts off the words at the end of sequence that go beyond two sentences or 15 tokens total.
14

---

Under review as a conference paper at ICLR 2022
Language Model
Yelp
Movie
Toxicity
Sent sent.-leve -levele
SBERT
94.92.0.01
84.56
c84.40:0.00
SBERT (Proto-Trex)
10.19
80.05
13.19
CLIP
93.78+0.00
75.4
80.82
CL.IP (Proto-Trex)
1..1
63.5220.66
67.75+2.10
BERT
59.41+22.00
61.45-+1.16
76.88+1.33
inter. word-level
BERT Proto-Trex)
92.10+0.0s
75.51
79.35
GPT-2
93.78.0.41
+87.05:0.33
81.5620.58
GPT-2 (Proto-Trex)
00.32.0000
84.57:0.31
84.14410.88
DistilBERT
90.01
19.6220.15
$1.57=0.11
DistilBERT (Proto- -rex)
92.71+0.03
78. 64.444.11
83.39+0.47
SBERT (iProto-Trex)
93.81
80.24
73.3610:78
GPT-2 (iProto-Tres)
10.12
c84.80:0.17
84.51110.9
b)
Serd-Seeection
Similarity
Proto Trex LM
Conv.
Attn.
Cos.
BERT
92.10 92.1020.08
89.66=0.92
92.10=0.08
91.3422.225
GPT-2
95.32
94.1 16
95.32=0.m
94.9940.00
DistilBERT
92.71
21.09
92.71
92.05:10.39
SBERT
93.59=0..
20.59=0010
98. 13200.4
CLIP
87.16
86.88:11.45
Faithfulness
Acc.
Proto- Proxo-Trex LM
Comp.
Suff.
before
after
SBERT
0.22
0.08
80.05
69.66
iSBERT
0.21
-0.08
80.24
70.76
GPT-2
0.12
0.02
84.57
49.91
1GPT-2
0.13
0.02
84.80
55.74
Table 5: Extension of Tab.
1 with respective confidence intervals.
Language Model
Yelp
Movie
Toxicity
SBERT (Proto- Trex)
94.1322.15
83.23
83.1120.19
SBERT (Proto-Trex) with projection
93.5940.16
80.05=0.26
73.19=0.71 80.74=0.01 67.75
CLIP (Proto-Trex)
93.41
73.62=0.20
CLIP (Proto-Trex) with projection
87.16+1.58
63.52+0.c0
Table
6:
Impact
of
Projection on
Proto-Trex
networks.
We showcase all experiments the user interaction in Tab. which an extension of Tab. The extension includes (1) retraining the classification layer for the same number of epochs to provide a fairer comparison with the interaction methods and to exclude changes accuracy simply due to training for more epochs, (2) pruning the original, (3) removing the original, (4) adding new proto- type without changing the original, (5) replacing the original with the user-chosen alternative "They offer a bad service." without using the interaction loss and (6) fine-tuning the original prototype.
In Fig. 12 we shows the full explanations with importance scores for the exemplary query in the motivation (cf. Fig. 1). It highlights the advantage of using prototype networks and the benefit of interactive learning.
15

---

Under review as a conference paper at ICLR 2022
Importance
Query: really like the good food and kind service here.
0.43-17.13=7.31 P6: Excellent baby back ribs. Creamed corn was terrific too Came in the aftemoon
word-level
and was casy to get in and have a relaxing meal. Excellent service too!!
0.56-12.14 6.80 P4: Food was great...service was excellent! Will be eating there regularly.
0.39-12.89 5.03 PIO: great experience each time come in The employees are friendly and the food
is awesome.
Table 7: Provided Explanations by Proto-Trex networks. Highlighted words mark matching subse- quences between query (top-row) and corresponding top three most similar prototypes (word-level). Importance scores (left, bold) are calculated with cosine similarity and classification weight.
Pl This places not authentic and the pho is not that good. Very small uncomfortable dining area and the service was horrible.
P2
Great amhiance with menu that short and sweet
The food was delicious and loved ginger beer! Excellent service. Definitely recommend this place!
P3
Waited over hour for food. Only one person making sushi. Left without food, they still charged for the one beer we had. Bad server + no food horrible exp.
P4
Food was great...srrvice was excellent! he eating there regularly.
P5
Called it. This place didn't stand chance with terrible management, gross food and slow service, Too bad for the workers.
P6
Came in the afternoon and was easy Don't 't waste your money this restaurant. Took my daughter to celebrate her birthday. They didn't even sing Happy Birthday. The food was horrible.
Excellent baby back ribs. Creamed corn terrific too get and have a relaxing meal. Excellent service too!!
P7
P8
Great food, great service. Chicken tikka was awesome with fresh peppers. Iced tea had surprising and refreshingly different flavor.
P9
Worst customer service No even said hello after going in separate times. am avid spender have spent money there the past. Will not shop there again!
P10
A great experience each time come in The employees are friendly and the food awesome.
(a) Yelp
P1
There's too much falseness the sccond half, and what began as intriguing look youth fizzles into dull, ridiculous attempt at heart-tugging.
P2
Parts of the film feel bit too much like infomercial for ram dass's latest book aimed at the boomer demographic.
mostly it's work that, with humor, warmth, and intelligence, captures a interestingly lived.
P3
Godard's ode to tackling life's wonderment is rambling and incoherent manifesto about the vagueness of topical excess... In praise of love remains a ponderous and pretentious endeavor that's unfocused and tediously exasperating.
P4
It's a lovely film with lovely performances by buy and accorsi. throws quirky characters, odd situations, off-kilter dialogue at us. all if say, "Look this!
P5
This an interesting movie!" but the film itself ultimately quite unengaging
P6
Like Tarantino movie with heart, alias Betty is richly detailed, deftly executed and utterly absorb- ing.
P7
bland, obnoxious 88-minute infomercial universal studios and ancillary products.
P8
The film fearlessly gets under the skin of the people involved... this makes it not only a detailed historical document, but an engaging and moving portrait a subculture.
P9
A sad and rote exercise in milking played-out idca a straight guy has to dress up in drag that shockingly manages to be even worse than its title would imply.
P10
inventive, absorbing movie that's hard classify as it is hard to resist.
(b) Movic
Table 8: Proto-Trex GPT-2 Prototypes. The prototypes are received with GPT-2 for the Yelp Open (a) and Revie (b) dataset. The prototypical subsequences provided by GPT-2 are highlighted in color.
16

---

Under review as a conference paper at ICLR 2022
Pl
This place is horrible. The staff is rude and totally incompetent. Jose horrible and is poor excuse for customer representative.
P2
This place is really high quality and service is amazing and awesome! Jayden was very helpful and was prompt and attentive. Will come back as the quality is so good and the service made it! Terrible delivery service. People are mean and don't I care about their customers service. I will not back this place. Also the food is small, not very good and expensive.
P3
P4
Incredibly delicious!!! Service was great and food was awesome. Will definitely come back!
P5
They sat us 30 minutes late for our reservation and didn't get our entree for 1.5 hours after being seated. The service was terrible.
P6
found this place very inviting and welcoming. The food great the servers.. love the food and the service is phenomenal!! Well done everyone..!
P7
Horrible customer service. Not helpful at all and very rude. Very disappointed and will not go back to this location.
P8
Oliver Rocks! Great hidden gem in the middle of Mandalay Bay! Great friends great times
P9
They literally treat you like you are bothering them. No customer service skills. Substandard work. Fun and friendly atmosphere, fantastic selection. The sushi is so fresh and the flavors are WOW.
P10
(a) Yelp
P1
...in the pile of useless actioners from mtv schmucks who don't know how to tell a story for more than four minutes.
P2
the solid filmmaking and convincing characters makes this a high water mark for this genre.
P3
dull, lifeless, and amatcurishly assembled.
P4
it's wise and powerful tale of race and culture forcefully told, with superb performances throughout. stale, futile scenario.
PS
P6
real winner smart, funny, subtle, and resonant.
P7
plodding, poorly written, murky and weakly acted, the picture feels as if everyone making lost their movic mojo.
P8
enthralling, entertaining feature.
P9
fails in making this character understandable, in getting under her skin, in exploring motivation...
well before end, film grows as dull its characters, about whose fate hard care. this delicately observed story, deeply felt and masterfully stylized, is triumph for its maverick director.
P10
(b) Movie
(a) and MovieReview (b) dataset. Table Proto- Trex SBERT Prototypes. The prototypes are received with SBERT for the Yelp Open
Pl
This place is horrible. The staff is rude and totally incompetent. Jose horrible and is This place really high quality and the service is amazing and awesome! Jayden was It care about their customers service. will Incredibly delicious!!! Service was great and food was awesome. Will definitely come back!
P2
P3
Terrible delivery service. People are mean
P4
P5
They sat us 30 minutes late for our reservation and didn't 't get our entree for found this place very inviting and welcoming. The food is great the servers..
P6
P7
Horrible customer service. Not helpful all and very rude. Very disappointed and will not go
P8
Oliver Rocks! Great hidden gem the middle Mandalay Bay!
p9
They literally treat you like you are bothering them. No customer service skills. Substandard work.
P10
Fun and friendly atmosphere, fantastic selection. The sushi is so fresh and the flavors are
Table 10: Pruning Prototypes with iProto-Trex. It shows the prototypes from Tab, 9(a) after pruning has been applied. Pruning reduces the sequences in length while preserving the sentiment. The accuracy remains the same (cf. Tab. TT). Prototypes 4 and 9 are not pruned as this would alter the sentiment too much.
17

---

Under review as a conference paper at ICLR 2022
Method
Acc.
Prototype
interaction
93.64
Horrible customer service and service does not about safety features. That's all I'm going to say. Oh they also don't ' care about their customers
retrain
93.64
Horrible customer service and service does not care about safety features.
That's all I'm going to say. Oh they also don't care about their customers
prune
93.64
Horrible customer service and service does not care about safety features.
That's all 'm
remove 93.61
add
93.79
They offer bad service.
replace
93.78
They offer a bad service.
re-initialize
93.80
Terrible delivery serviee. People are mean and don't care about their cus- tomers service. will not ever come back this place. Also the food is small, not very good and too expensive.
fine-tune
93.81
Terrible delivery service. People are mean and don't care about their cus- tomers service. I will not ever come back to this place. Also the food is small, not very good and expensive.
soft replace (0.5)
93.81
Terrible delivery service. People are mean and don't care about their cus- tomers service. I will not ever come back to this place. Also the food is small, not very good and expensive.
soft replace (0.9)
93.79
really don't recommend this place. The food not good, service bad. The entertainment is so cheesy. Not good
soft replace (1.0)
93.79
They offer bad service.
Table 1I: User Interaction with iProto-Trex. Each row shows a different interaction method with the balanced accuracy the test set conducted Yelp Open dataset. The interaction methods are able to remove the unwanted prototype while incorporating more knowledge and hence more interpretability along with higher accuracy.
Importance
Query: Staff is available assist, but limited to the knowledge and understanding of the individual
0.58-6.95-22 4.01
They literally treat you like you are bothering them. No customer service skills. Substandard work.
Proto- Trex
0.38-2.61=1.00 1.000.61=1.00
They sat us 30 minutes late for our reservation and didn't 't get entree for 1.5 hours after heing seated. The service was terrible.
0.36-2.02 1.36-2.02=0.77 0.73
This place horrible. The staff rude totally incompetent. Jose was horrible
and is a poor excuse for a customer representative.
iProto- -Irex
0.50:3.25=1.66
They offer a bad service
1.58
They us 30 minutes late for reservation and didn't get entree for 1.5 hours after being seated. The service was terrible.
0.36-3.01 3.00
This place is horrible. The staff is rude and totally incompetent. Jose was horrible and is poor excuse for customer representative.
Table 12: Top-3 Explanations for Query in the Introduction (Fig. 1}. The first three rows depict explanations with sentence-level Proto-Trex with the SBERT transformer, while the last three show the top three explanations iProto-Tres On the left side, one can see the importance scores for the classification.
18


Reviews:
[{"review_id": "a2p_xWJ-XhX", "reviewer": "Reviewer_hWyb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The paper touches upon several interesting threads: interpretability, case based reasoning and using user interactions for improving models. It is well written, with a comprehensive set of experiments. \n\nWeaknesses:\n- Hyperparameters: My initial impression of the method is that it involves far too many hyperparameters (various $\\lambda$ for combining losses, choosing similarity metrics etc). How hard was tuning hyperparameters?\n- Results:  \n    - different models respond differently to the proposed approach. For instance, we see large performance gains on BERT but performance drops for SBERT. It would be good to know why this happens.\n    - User interaction results are not convincing. From Table-3, it seems like the interactions barely help, even though the prototype changes sometimes. Could the authors comment more about this?\n    - Can the authors provide with confidence intervals for all results so we can compare across different settings better?\nPrototype quality: prototypes do not look very convincing. I glanced at the prototypes in the Appendix and they seem to be quite redundant / of low diversity. For instance, from Table-6 a lot of the prototypes either model “bad service and bad food” or “good service and good food” (P2/P4/P6/P8/P10). However yelp also has examples with “bad service, good food”, “good food, bad service”, examples not related to restaurants at all but the prototypes do not uncover these. \nFaithfulness: How competitive are these numbers with post-hoc methods? I think for presentation purposes, it would be good to also compare this with SHAP/LIME/IG. \n\nOverall, I think this paper is currently borderline. However, I’m happy to increase my score if the weaknesses can be quickly addressed —- the main ones are providing evidence that the learnt prototypes can be diverse and cover a lot more of the “modes” in the training distribution, providing a more comprehensive comparison with posthoc interpretability approaches, and confidence intervals.  \n", "summary_of_the_paper": "This paper introduces a method for improving interpretability of black box transformer based text classifiers. The approach is based on “case based reasoning” where the network classifies an input by comparing it against a library of learnt prototypes (each with a corresponding label) and classifying the input based on a weighted similarity with the prototypes. Thus, in a sense it is more interpretable than an end-to-end classifier since a user can directly look at the similarity scores with the prototypes to understand the labeling decision. The approach also promises to be more “faithful” than post-hoc interpretability methods since the predictions are based directly on similarity scores. The paper also shows that end-users of the system can interact with it by either editing prototypes (if they are experts) or providing weak feedback about which prototypes are good, and this feedback can be integrated to re-learn better prototypes. From experiments we see that their approach is competitive with standard end-to-end finetuning while being more interpretable. Based on sufficiency and comprehensiveness scores, we see some evidence that the explanations from prototypes are faithful, though it is unclear if these scores are competitive.\n", "summary_of_the_review": "While i think this work combines many interesting threads in interpretability, using user feedback and case based reasoning, the paper in its current state is not ready. I’m happy to increase my score / recommendation if the weaknesses are addressed, however. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UyqPTuf7h0d", "reviewer": "Reviewer_cN7n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper seems motivated by a prior NeurIPS 19 work \"This Looks Like That...\" in the sense that the architecture and loss designs are sourced from there. The nice thing about this paper is that it focuses on NLP tasks, so the framework could potentially benefit the explanation community. The early part of this paper is very straightforward and intuitive. Related works have limited coverage. The explanation generation part is vague. The experiment section is weak. Analysis on generated explanation is also weak.\n\nThe architecture relies on using prototypes which are nicely discussed in this paper. The problem if improving interpretability without trading off downstream task F1 is interesting since the trade-off was common among prior works. However a couple confusing points still. \n\n1. There are many lines of explanation works, such as those use prompt engineering, information bottleneck, and purely generative approaches. This paper has limited coverage on these topics. Touching different approaches is important here since the way ProtoTrex handle explanation might not easily extend to all other cases.\n\n2. Even though this design was from the NeurIPS 19 paper, but in the task of NLP, how does the prototype embeddings compare against the label-wise weights in the final classification layer? This is to imagine that, without the use of the complicated loss in Eq 1, how does simply treating the label-wise embeddings as prototypes perform?\n\n3. The explanation generation (sec 3.5) needs elaboration. It seems this paper uses prototype embedding to find a training example as it nearest neighbor, and then use this data point as explanation to its prediction. This design has certain limitations: a) not context/example dependent; b) this is hardly generation, instead, it is more in line with salience-based explanation works. Tab2 indeed shows some examples with explanation that partially depends on the input example. But no idea how they were generated.\n\n4. Tab 1b is confusing. I don't understand what each number means. This paper very briefly went over some statements without getting into details.\n\n5. I don't see a solid explanation evaluation in this paper. Tab 1c shows rationale performances however these numbers are quite low compared with prior works (e.g. Paranjape's work at EMNLP 20). And not sure if rationale performances are based on token or sentence selection. Either way, this evaluation has nothing to do with generation. And when it comes to generative explanations, ideally, there should be some human-based evaluation over a subset of testing data. But no such thing in this paper.", "summary_of_the_paper": "This paper aims to model explanation and task prediction such that task performances are not (or less) traded off for interpretability. It proposes a novel framework for transformer models where classification and explanation generation are based on shared prototype embeddings which are learnt from training data by a combination of losses. The framework is also compatible with settings that requires human in the loop for extra supervision on prototype learning. Experiment results show that adding the proposed ProtoTrex benefit task performances on 3 sentiment classification tasks.", "summary_of_the_review": "I think the architecture has novelty when it comes to NLP tasks. And this work could benefit the explanation community. However, I found the experiment results are confusing. To the best degree, it offers marginal improvement over the best baselines in terms of task F1. When it comes performance of explanation, I only see confusing numbers, thus no conclusion can be made. Analysis on generated explanation is another weak point since it's absent.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LKT0dRimfeI", "reviewer": "Reviewer_yfzh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- S1 - The idea of choosing a similar example from the train data as an explanation is an interesting idea.\n\nWeaknesses and Questions:\n- W1 - I'm quite unsure about the set of prototypes. It seems not described in the main contents. I checked examples of prototypes in the appendix but the 1:1 label distribution (positive vs. negative) seems to me they are human-picked.\n- W1-Q1 - If the set of prototypes pre-defined, do we really need to train the prototype? what if we just compute sentence similarity between input and the prototype?\n- W2 - The main table is not clear, especially on faithfulness. What if we just consider the random prototype as a prototype? what is the faithfulness of that random prototype then?\n- Q2 - Just curious, in the introduction, why the post-hoc explanation is potentially reporting-bias? Isn't this method also potentially reporting-bias since it only uses samples from train data as prototypes?", "summary_of_the_paper": "The premise of the paper is that example train instances with the corresponding label can be an effective explanation of the model's prediction rather than post-hoc explanation which is not deterministic. The framework jointly trains label prediction and prototype clustering. Then, the framework computes the similarity between input and prototypes to retrieve the most similar prototype as an explanation while predicting the label as well. Moreover, the framework can reflect human-in-the-loop feedback on the prototype.\nThe primary result of the paper is that their framework can show the proper explanation by a similar example while preserving the performance.", "summary_of_the_review": "My overall disposition towards the paper is indifferent although the paper proposes an interesting idea. It is a bit difficult to follow the method due to a lack of information. More details are needed especially on experimental settings, and methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1S3j26IfTj2", "reviewer": "Reviewer_QkZJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The proposed Proto-Trex/iProto-Trex are technically sound.\n2. Through several example cases, the proposed model could generate reasonable explanations.\n\n\nWeaknesses:\n\n1. The training loss of the model seems to be too complicated. In Eq.1, there are 6 parts for the overall training loss, and each of them associates with a lambda term (I checked the Appendix, but failed to find how you set lambda terms). Each of them sounds reasonable, but unfortunately, there is no specific ablations on how these parts attribute to the final prediction/explanation. \n2. According to the illustrations at the end of Section 3.4, there are many hyper-parameters should be set (along with lambda terms in training loss). How to determine these hyperparameters? If we move to a new dataset, we might lost in tuning these hyperparams.\n3. As the model incorporates additional modules (compared to pure classification model), it is questionable how efficient is Proto-Trex (both for parameter size and inference time).\n\n\nComments:\n\n1. It is unclear where are the results from in Table 1b and 1c. Yelp? Movie? Toxicity?\n", "summary_of_the_paper": "This paper proposes Proto-Trex model to increase the interpretability of the text classification systems. The proposed model mainly adds a bunch of prototype layers to learn the similarity between the query and prototypes. They also propose an extension caleld iProto-Trex to interactively learn from users' feedbacks. Experimental results and example cases show that the Proto-Trex could give comparable classification accuracy compared to plain classification model (w/o explanation) and could provide reasonable explanations. ", "summary_of_the_review": "The paper addresses an important issue in explainable natural language processing models. The proposed model is reasonable, and the results seem to be OK. However, many issues are left unclear, such as its efficiency, hyper-parameter tuning, etc. I tend to lean towards weak rejection at the moment.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method—"Proto-Trex"—that incorporates prototype networks into text classification architectures to facilitate model explanations via presentation of similar training examples. There was agreement that the direction here is promising and the work contains some nice ideas and a good initial set of evaluations. However, the presentation can be improved substantially to better situate the contribution with respect to related work (and clarify the specific contributions on offer here), and to clarify the key technical details of the proposed approach.
Decision: Reject