Interpretable Sequence Classification Via Prototype Trajectory
Authors: Dat Hong
Stephen Baek
Tong Wang
Venue:  (2021)
Keywords: interpretable
RNN
prototypes

Abstract: We propose a novel interpretable recurrent neural network (RNN) model, called ProtoryNet, in which we introduce a new concept of prototype trajectories. Motivated by the prototype theory in modern linguistics, ProtoryNet makes a prediction by finding the most similar prototype for each sentence in a text sequence and feeding an RNN backbone with the proximity of each of the sentences to the prototypes. The RNN backbone then captures the temporal pattern of the prototypes, to which we refer as prototype trajectories. The prototype trajectories enable intuitive, fine-grained interpretation of how the model reached to the final prediction, resembling the process of how humans analyze paragraphs. Experiments conducted on multiple public data sets reveal that the proposed method not only is more interpretable but also is more accurate than the current state-of-the-art prototype-based method. Furthermore, we report a survey result indicating that human users find ProtoryNet more intuitive and easier to understand, compared to the other prototype-based methods.

INTERPRETABLE SEQUENCE CLASSIFICATION VIA PROTOTYPE TRAJECTORY
ABSTRACT
We propose a novel interpretable recurrent neural network (RNN) model, called ProtoryNet, in which we introduce new concept of prototype trajectories. Moti- vated by the prototype theory in modern linguistics, ProtoryNet makes prediction by finding the most similar prototype for each sentence in text sequence and feeding an RNN backbone with the proximity of each of the sentences to the proto- types. The RNN backbone then captures the temporal pattern of the prototypes, to which we refer as prototype trajectories. The prototype trajectories enable intuitive, fine-gratee lineigaiied interpretation of how the model reached to the final predic- tion, resembling the process of how humans analyze paragraphs. Experiments conducted on multiple public data sets reveal that the proposed method not only is more interpretable but also is more accurate than the current state-of-the-art prototype-based method. Furthermore, human users find ProtoryNet more intuitive and casier to understand, compared to the other prototype-basd methods.
report survey result indicating that
INTRODUCTION
Recurrent neural networks (RNN) have been widely adopted in natural language processing. RNNs achieve the state-of- the-art performance by utilizing the con- textual information in a "memory" mecha- nism modeled via hidden/cell states. Albeit the benefit, however, the memory mecha- nism obstructs the interpretation of model decisions: as hidden states are carried over time, various pieces of information get in- tertwined across time steps, making RNN models "black box" inherently.
Food delicious in this place. @Bu: the service is super low and we waited for more than half an hour for our dessert.
Sontimmnn Seore
0.9
When we asked waitress about it, she incredibly rude to d We will not back.
0.2
0.05
0.1
Pretotype for
"Great food
Protolypu for "There always long the food."
Prototype for "The service very bad."
Pretotype for O: "Overall,
worth it."
Figure 1: Prototype trajectory- based explanation.
The black box nature of RNNs has motivated a body research works aiming to achieve the interpretahiliity One approach to leverage certain architecture design in the DNNs such as ciention-hased methods. As will be discussed in Section 2 the attention- based approaches (Karpathy) al., 2015; Strobelt al., [2017] Choi et al.) 2016)] Guo et al. 2018) visualize the RNN using the attention mechanism, which weighs the importance of each hidden state element. However, while few of them could be quite illuminative, the attention weights are, fact, not always intelligible. Rather, they often turn out to be gibberish collection of numbers that does not possess much sensical interpretations. fact, recent research has been considering attention weights as not explanations (Jain & Wallace 2019). Furthermore, the analysis of attention weights requires a certain level of understanding how RNNs work in theory. Hence, novice user may find difficult to understand and, thus, the broader use in real-world applications might not be feasible.
The other is prade approaches (Ming et al., 2019), which use prototypes to explain the decision more intuitively. The process is analogous to how, for example, human doctors and judges make decisions on a new case by referring to similar previous cases: for given sequence, a prototype- based approach looks up few representative examples, or prototypes, from the data set and deduces a decision. From the interpretability standpoint, such prototypes then provide intuitive clues and evidences of how the model has reached conclusion in a form that even layperson understand.

---

However, the existing prototype-based methods find the prototypes the whole- paroge-p level, making it difficult to break down the analysis at the individual sentence level, e.g., the connections and flows of individual sentences constituting a paragraph. Moreover, there may not be suitable prototype when the length of a sequence is too large (e.g. a long paragraph), as longer sequences have greater degrees of freedom and, thus, harder to find matching prototype, evidenced later in Section
Here, we advocate the idea that the sentence level prototyping (as opposed to the paragraph level prototyping in the previous literature) produce more desirable outcomes, namely better interpretability and higher prediction accuracy. We propose a novel architecture, called ProtoryNet. in which we introduce a new concept of the prototype trajectory. Given one or more paragraphs, ProtoryNet looks up the nearest prototype for each sentence and computes the proximity. The prototype proximity values are then fed into an RNN backbone, which then captures the latent patterns across sentences by means of the trajectory of prototypes. The prototype trajectory, therefore, illuminates the semantic structure of text sequence and the logical flow therein, and, hence, provides highly intuitive, useful interpretation of how the model has predicted an output, as can be witnessed in Figure !
In fact, the prototype theory in modern linguistics supplies a strong justification for the proposed idea of prototype trajectory. In the prototype theory, linguists view "a sentence as the smallest linguistic unit that can be used to perform complete action," (Alston 1964) and analyze texts with individual sentences as building blocks. Linguists assume that the sentences of a category are distributed along continuum: one extreme of this continuum are sentences having maximal number of common properties; while on the other extreme are sentences that have only one or very few of such properties (Panther& & Köpcke| 2008). Here, the "ideal" sentence that possesses the maximally shared common properties can be considered as a prototypical sentence, or a prototype of the category. Thus, some sense, this paper takes a meaningful first step towards mathematically formalizing the prototype theory in modern linguistics and its analysis methods by incorporating the above view in a computational framework and emulating how linguists analyze a text.
As such, ProtoryNet permits explanation of the dynamics of the subsequences. In addition, since the technical details are hidden in the prototypes, a non-technical user can comprehend the interpretation. However, when necessary, technical users, i.e., the ones that are more knowledgeable about RNNs, can still look at the coefficients in RNN, similar to how the attention approaches visualize RNNs, as the proximity vectors feeding the RNN backbone are essentially one-hot encoded (i.e., zero everywhere except the k-th position for prototype k), making it convenient to trace how coefficients are related each prototype.
fine ine-grained understanding of sequence data alongside an intuitive
2 RELATED WORK
In addition to model- aggossic black-box explainers such as LIME (Ribeiro et al., 2016) and SHAP(Lundberg & Lee 2017). various post hoc explanation methods have been proposed for
DNN models, such as Integrated Gradients (Sundararajan et al. 2017). DeepLift (Shrikumar et al.
2017) and NeuroX (Dalvi et al. 2019. Specifically, to understand RNN models, T Tang et al. (2018) proposes a hicrarchical explanations for neural networks to capture interactions and JJi et al. 2019
adapts the idea to text classification quantify the importance of each word and phrase. For sentiment
analysis. MMrdoch et al.] (2018) proposes contextual decomposition method for analyzing individual predictions made by standard LSTMs, and the method is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTMs final prediction.
In addition to the external explanation methods, the prior efforts bring interpretability to RNNs can be categorized as allention- and prototype-based approaches. Bahdanau, Cho, and Bengio (Bahdanau et al. 2014) proposed an encoder-decoded type machine translation algorithm, in which they implemented an attention mechanism in the decoder. By the means of the alignment probabilities and the association energy, reflecting the importance of given word in predicting a translated word, they let the attention mechanism to weigh which part of the source sentence the model needs to pay attention to. This not only improved the performance of the model by relieving the burden of the encoder having to compress all the information about the source sentence into fixed-length vector, but also inherently visualized how the translation was conducted through the alignment matrix (e.g., Figure 3 of Bahdanau et al. (2014)). Similarly, Rocktäschel et al. (2015) analyzed word-to- word-to-word attention weights for achieving insights into how a long short term memory (LSTM) classifier reasons about entailment. /Zhang al. (2017) proposed language model read and explore discriminative
2

---

image feature descriptions from reports to learn a direct mapping between lexical components and image pixels via attention. Similar strategies can be found a number of other works (e.g. [smail et al.] ( Choi et al. (0166) that employ the idea of the attention mechanism. However, the attention-based approaches are mostly intended for expert users. Many non-technical users in the real world, who lack basic knowledge of how RNNs work (or even neural networks in general), may find them difficult to understand.
Instead, prototype-based approaches argue that the intuitiveness of interpretation can be significantly enhanced by visualizing the reasoning process in terms of prototypes. In fact, prototype-based reasoning has long history as fundamental interpretability mechanism in traditional models (Cupello & Mishelevich] [988]] Fikes & Kehler 1985] Kim et al.) 2014). More recently, the idea of prototype--aan
interpretation attracted several authors in neural network research. For instance, Chen et al. (2019) incorporated the concept of prototypes to convolutional neural networks. A prototype layer was added after convolutional layers to compare the convolution responses at different locations with prototypes. From this, users can understand, for example, a bird is classified as 'red-bellied woodpecker' because it has the typical prominent red tint at the belly and the top of its head as well the black and white stripes its wings.
More closely related to the present work. ProSeNet (Ming et al. 2019) added a prototype layer in RNN. ProSeNet computed the similarities between input sequence (usually short prose) and prototypes and produced the final prediction as a linear combination of the similarities. Despite the built-in interpretability of ProSeNet, however, an issue might arise when the sequence was too long. The original paper (Ming et al. 2019) validates ProSeNet only on paragraphs shorter than 25 words. However, it is easily fathomable that ProSeNet may fail to assimilate a long paragraph data, due to large degrees of freedom that complicate matching of a prototypical example. This may render some practical concerns. For instance, in sentiment classification, even if a paragraph is classified as "negative," it could consist of several twists of sentiments along sentences (e.g., sarcastic use of positive proses). With an increased length, such kinds of twists can get harder to be represented with a prototype, thus making the interpretation difficult and the explanation less credible. This claim is further supported by findings in modern linguistics, which suggests that sentences, instead of paragraphs, should be regarded as the basic elements for text analysis (Panther & Köpcke 2008).
Nonetheless, we share our view with ProSeNet that the prototype based method is worth paying attention to. The benefit of prototype-base reasoning resides in the fact that it hides technical details by encapsulating them with prototypical examples while being still tractable numerically when desired. Hence, novice users understand how the reasoning was achieved in RNNs long as they can comprehend the prototypes, lowering the barrier for those numerous non-technical users who may use RNN-hased applications in the real world. On the other hand, numerical weights assigned to prototypes alongside their association with the "nuts and bolts" of RNNs still allow experts to perform ie-depth analyses of how model has drawn prediction.
3 METHOD
3.1 THE PROTORYNET ARCHITECTURE
Suppose a data set DD = (X(()) ( N) of size N, comprised of text sequences (0) and the corresponding labels y(0) Here, note that the superscript (i) may be dropped for notational convenience hereinafter, unless necessary. Each instance X can be understood as a sequence of sentences E RVV at t-th position, yielding the representation X = (xx))_ where V is the size of vocabulary and T := |X| is the number of sentences in the sequence X. RRC is multi-hot encoded vector representing the class labels associated with the sequence X, iee, the e-th element ye of y equals if the label is associated with X 0 otherwise. C' is the total number of classes.
ProtoryNet interfaces with text data via a sentence encoder (Figure 2h) modeled as a mapping r: R R', where J is the dimension of sentence encoding specified by the user. That is, the encoder takes each sentence X, E X and produces a sentence embedding e; = r(x:). In this paper, the development of the encoder is beyond the scope of this paper and, hence, we employ preited BERT-based sentence encoder with mean-tokens pooling (Reimers & Gurevych 2019).
where J = 768 by default.
3

---

Food was delicious x, place.
Cii
Pii There always wait for the food
3 [[Sck]
S
But the service super low we wained for more X for our dessert.
[0.61, ...99.....0.11
[0...0. 0.92.....0
(aanyeaau,)) uunding
C2
Ps: Overall was not worth it :
[0.91.
0.34]
[0.91, O.... O.....0)
HC
When asked the waitress about it. she was incredibly rade us.
[0.21, 0.44. 0.89]
........... 0.89]
[0.45, 0.91, 18.............................. 0.38)
[0. 0.91.... O.....
Pae' The service very bad
We will back.
X4
a) Sequence encoder
b) Prototype layer
RNN layer d) Fully connected layer
Figure 2: The architecture of ProtoryNet.
The sentence embeddings e, are fed into the prototype layer (Figure 2p), in which set of trainable
prototypes = {Px R' = ......... K} are compared with e,, where := (P] is the number of prototypes specified by the user. Then, given a distance metric RR + R+ the proximity 8t,k the
sentence embedding e₁ to a given prototype P is measured as $t,k == s(e,, P&) = exp dier.ps)
1
where is user-specified constant. Between two popular choices for the distance metric namely the cosine distance and the Euclidean distance, we find that there is no significant difference between the two. Hence, we use the Euclidean distance in our experiments for convenience.
Note here that the intermediate throughput of the prototype layer is the similarity matrix S [s of the size T' x K, associating the t-th sentence with the k-th prototype. The rows of the similarity matrix then constitute the input to the LSTM backbone at time step t (Figure 2 which then finally produces an output prediction. Here, the sparsity transformation is performed on the similarity matrix before feeding it into the LSTM backbone by setting each row zero except for the position where 8t,k is the maximum. That is, each row of the transformed similarity matrix S would be of the same topology as the one-hot encoded vector, whose elements equal St, k. == arg max, & 8t,k and otherwise. The sparsity transformation of to enhances the interpretability of the architecture, by enforcing each sentence be matched with no more than single prototype and, thus, disentangling the information. This accomplished only at a small cost of accuracy, as observed from an ablation study Appendix.
Motivating Example The text data in Figure eexemplifies the use of ProtoryNet for sentiment analysis (text classification). In this example, the task is to predict whether the review a restaurant positive not. The input text data X is comprised of T' 4 sentences, in this particular case, and the label y is the binary sentiment label of the review, either "positive" (1, 0) or "negative" ([0, 1).. To this, ProtoryNet converts the text data into sentence embeddings, each of which then matched with the closest prototype. Observe, in the figure, that the prototypes that ProtoryNet produced are, indeed, morphosyntactically equivalent to the corresponding input sentences, well-exemplifying them semantically. The one-hot-like similarity vectors between the sentences and the prototypes are then fed into the LSTM backbone, which captures the patterns and trends in the trajectory of prototypes and, finally, predicts the final sentiment label, which, in this case, is "negative." 3.2 OBJECTIVE FUNCTIONS
The training objectives of ProtoryNet entail four different terms aiming to achieve both the prediction accuracy and the interpretability. Below are the details of their definitions.
Accuracy The accuracy loss is defined as the square loss between the predicted value and the ground truth label, promoting the model to make accurate predictions:
N
Luve (D) = N
Σ y0)
500
(1)
4

---

Diversity To ensure diverse and non-overlapping prototypes, define the diversity loss term added to enforce the minimum mutual distance among the prototypes:
dmin min d(pk, Pkz),
(2)
Laiv (D) == a (1(6 dmin))
(3)
where (-) is the sigmoid function and 1) is smoothing constant, which we 1) empirically. The constant 5 E R., is positive real number defined by the user, to enforce the minimum separation among prototypes. Hence, when the distances among the prototypes do not meet the minimum separation requirement i.e., dmin < 6. the 77(ō dmin) term will have some positive value, making the diversity loss term Laiv active; on the other hand, when the minimum separation requirement met and thus, Amin > 8, then the sigmoid function will pull the loss term to zero. Note that smaller will make such transition by the sigmoid function smoother.
Prototypicality The accuracy and the diversity terms alone, it is observable a prominent tendency of prototypes diverging away from the sentences during training. Such a behavior introduces overfitting, in which prototypes become less generalizable, as the prototypes lose their representativity of category. Hence, we introduce the prototypicality loss, which promotes each sentence in the database to have representative prototype close to it:
prrooo
Σ Σ
min ((r())) P&),
(4)
XEDXEX
where S is the total number of sentences in the data set.
Final loss The final loss function combines the above loss terms:
L Loee + a& + 3.oooooo
(5)
Empirically, coefficients values of 0.1 and B 1 used in this paper.
Remarks on Prototype Interpretability The diversity and prototypicality terms are designed for improving the interpretability. Here, to achieve good explanations, prototypes need to be different from each other to avoid redundancy, thus the diversity term. In addition, each input sentence needs to be mapped to a prototype that is similar enough to make the explanation convincing, thus the prototypicality term. These two loss terms can be considered regularization terms to serve interpretability purposes. Similar loss terms have been introduced other prototype based DNN models (Ming et al. 2019 Chen et al. 2019). We will later show in experiments that these two terms do not hurt the predictive performance. This can be explained by the recent research on Rashomon Set (Semenova al.
2019 Rudin 2019), that there exist many models with very similar performance. so one can add customized constraints to the model to achieve additional benefits, such as interpretability.
3.3 TRAINING
For the training of ProtoryNet, the adaptive moment estimation (ADAM) optimizer (Kingma Ba 2014) was employed. The learning rate was set to be Ie-4 and the exponential decay rates for the first and the second moment estimates were 0.9 and 0.999, respectively. Below are further details used for generating the results in this paper.
Differentiability Training of ProtoryNet requires the computation of the index where the simi- larity matrix $ maximum for the sparsity transformation in the prototype layer. This operation, unfortunately, is not differentiable and may lead to unexpected training behavior during auto- differentiation in deep learning packages. We get around this issue by the following approximation technique. Suppose the similarity matrix $= [
where &, E R elements indicate how similar the t-th sentence is each of the prototypes. If we let Softmax(-) to denote the softmax function, then for some large constant 7,
K is column vector whose
I [Softmax(y $i), Softmax(y sT)]
approximates the selection matrix whose element equals to at the position corresponding to where is the maximum for each column / and 0 elsewhere. Here, find 1 gives reasonable
5

---

approximation empirically. With the selection matrix, the sparsity transformation be approximated as follows without explicitly computing the maximum:
=
where O is the Hadamard product. Note that the softmax function is differentiable and, thus, is S.
Prototype Initialization The training of ProtoryNet can benefit from the initialization method described below. We first embed all sentences separately in the training data set. Then, in the embedding space, all sentences in the data set are clustered using the k-medoids clustering algorithm to categorize sentences by their semantic meaning. The medoids obtained from the k-medoids algorithm can be considered as the representative examples of each cluster and, hence, plausible candidates for prototypes. Thus, for the training of ProtoryNet, we use these medoids to initialize prototypes, which in turn accelerates the convergence.
Prototype Projection It should be noted that the numerical solutions for the prototypes are found in the embedding space. These numerical solutions and need to be deciphered. To this end, we project the prototypes to the closest sentence in the embedding space every 10 epochs during the training process, similar to the technique proposed in Ming et al. (2019): Chen et al. (2019)
automatically intelligible to human users
P&
arg min d(r(x)). x, x.EXO, VXWED
I E [1,
Sentiment Scores for Prototypes Once the training is done, ProtoryNet returns set of K proto- types. We then feed them back into the trained ProtoryNet one at a time, to evaluate the sentiment score of each individual prototype. These sentiment scores will later used to provide quantitative visualizations of how the tones and sentiments change within text data.
4 EXPERIMENTS
In this section, evaluate our model five different data sets. A vanilla LSTM method, state- of-the-art black-box box model (DistilBERT) Sanh et al. (2019)), and a state-of-the- prototype-base interpretable model (ProSeNet Ming al. (2019) are compared against our method. We also compare with classic non-neural baseline bag-of- of words, which provide explanations at word level.
Data set
DistiBBERT
Vanilla LSTM
ProSeNet
ProtoryNet
ProtoryNet (avg) Bag-of-words
IMDB
.923 (+.001)
.873 (+:007) .835 (+.008) .849 (+.002)
.801 (+.002)
.870 (+.004)
Amazon Reviews
.941 (+.004)
.871 (+.004 .840 (010 .882 (+.004
.860 (+.005)
.86 (+0005)
Yelp Reviews
.957 (+.003)
.897 (=.004) .688 108.-0008877777 (+.002)
.859 (#.004
.922 (+.010)
Rotten Tomatoes
.841 (11.001)
.751 (1.000)
.748 (1.006) .762 (L.007)
.656 (1.010)
.749 (1.005)
Hotel Reviews
.976 (1L.006)
.947 (1.004) ,909 (1.007) .949 (L.004)
.856 (1.006)
.944 (1.008)
Table 1: Performance of ProtoryNet in comparison with other benchmark models. The mean accuracy and the standard deviation from a 5-fold cross validation are reported in each case. Boldface denotes the best performing model, excluding the black box model, DistilBERT, which outperformed the others in all cases. Note each data set had balanced labels.
4.1 PREDICTION ACCURACY
Reported in Table I are the means and the standard deviations of accuracy evaluated from f-fold cross validations. It is clear that the state-of-the thrt black box model (DistilBERT) outperforms the interpretable models (ProSeNet and ProtoryNet) in all data sets used. Note, however, that DiRT was pre-trained on massive corpus of text data before being transferred to each specific data set. Hence, the performance metrics of DistilBERT should only be used for sanity check. Compared against the vanilla LSTM, the prediction performance of the interpretable models were on par. ProtoryNet is slightly better than Bfg-of-
average.
Between ProSeNet and ProtoryNet, ProtoryNet outperformed ProSeNet for all five cases overall. In particular, the performance difference was clearer when long text data were analyzed. In Table 2 we split each data set into short and long samples-ppragrapph that were less than 25 words were classified short samples, following the criterion used in the ProSeNet paper Ming et al. (011). As

---

% short reviews
ProSeNet (Ming et al.
2019
ProtoryNet (ours)
Data set
Short
Long
Short
Long
IMDB
0.09
0.883
(#00.16)
0.835 (+0.008)
0.865 (=:0.127)
0.849 (=00009)
Amazon
5.08
0.868 (=0.014)
0.833 (#0.009)
0.879 (#:0033)
0.882 (#0.004)
Yelp
9.22
0.879 (=00018)
0.859 (#0.016)
0.794 (=00005)
0.879 (=0.002)
Rotten Tomatoes
61.37
0.754 (=:0.009)
0.721 (4:0.015)
0.774
(#:0.006)
0.738
(#:0.015)
Hotel reviews
2.22
1.000 (==0.000)
0.904 (#:0.006)
1.000 (4:0.000)
0.948 (#:0.224)
Table 2: Comparison between ProSeNet and ProtoryNet on text data different lengths. In the table the means and the standard deviations of accuracy over 5-fold cross validation. The methods perform similarly on short reviews (S 25 words), while ProtoryNet performs better on long reviews all data sets. The second column represents the proportion of short reviews in each data set.
shown in the table, ProSeNet and ProtoryNet were on par when short proses were analyzed, while ProtoryNet was always better than ProSeNet when long paragraphs were concerned. In fact, this can be substantial advantage of ProtoryNet as instances with less than 25 words are quite rare in real-world data sets. Note, in Table more than 90% of instances are more than 25 words, with an exception of the Rotten Tomatoes data set.
Amazon rovlowS
0.8
In addition, we investigated how the number of prototypes, K, influences the performance of ProtoryNet. In Figure 3] the performance of Protory Net on the Amazon Review data is plotted with respect to different values of K. Other hyperpa- rameters controlled to be the same and the performance measures (accuracy) were averaged over 5-fold cross validation experiments (the whiskers the figure represent the standard deviation). Compared to the ProSeNet baseline, ProtoryNet Figure 3: Effect K on accuracy. reaches high accuracy much quicker with only few prototypes and shows a steady increase of performance afterwards. The same analyses on the other data sets are available in Appendix.
ProtoryNet PreSeNet
LTSM
Number of prototypes
Furthermore, we conducted an ablation study on the sparsity transformation. We measured the change in prediction accuracy when the sparsity transformation step had been removed and the dense similarity matrix S had been used directly. The result revealed that there was only small drop of accuracy (approx. 1%) caused by the sparsity transformation, while the benefit of interpretability was huge (see Appendix for the detail).
4.2 PROTOTYPE TRAJECTORIES
[SI] sandwiches giant, the atmosphere is good. [S2] service slow of the italian places this would go here first. [S3] had an italiam hoggie that was very good and yes they do have excellent meatballs
[S1] The food here is very good. [S2] did find the
Boo. [S2] What happened to this location. [S3] It used to happy
Awful everything. [S2] From the bad service, to the diluted soup, to lame- the chicken salad, to the PRE PRE-fabricated pizza. [S3] AVOID THIS PLACE
there while Steelers were festive. [S3] agree
[84] It just feels depressing and nearly everyone there is just doing the minimum get by, [S5] So sad
on, they are little dry. [S4] so get hot sauce, ask some mayo
chicken
ranch, oct. [S5] And was quite warm in there
0.6
06-
0.6
0.4
51
53
51
52
54
85
81
53
54
52
(b)
Figure 4: Prototype trajectories of two positive sentiment examples (a, b) and two negative samples (c, d). The corresponding prototypes are available in Appendix.
Prototypes and their trajectory play critical role in ProtoryNet. Here, show four examples of prototype trajectories in Figure 4 two for positive sentiment data and two for negative ones curated from the Yelp Review data. Observe that the trajectories be drastically different for the
7

---

same sentiment class. For example, review (c) starts off with negative sentence and changes the tone in the middle to positive and then ends with negative sentence while (d) maintains the negative tone from the beginning to the end. As such, interpretation of ProtoryNet can be more fine-graineds generating deeper insights to users.
4.3 THE EFFECT OF DIVERSITY AND PROTOPYPICALITY TERMS
The diversity and prototypicality terms are designed to improve interpretability since it forces each sentence to be mapped to prototype that close enough it and prototypes be sufficiently different from each other. Therefore, in this section, we examine whether the two terms serve such purposes, and meanwhile, whether these two terms hurt the predictive performance.
Euclidean distances between input sentences and their closest prototypes 20
Euclidean distances
Effect on Interpretability We plot two types of distances for hotel dataset. First, since the explanations are based on prototypes, we need the prototypes to be similar enough the input sentences mapped to them in or- der make the explanation convinc- ing. Thus we compute the Euclidean distances between each sentence and the prototypes they are mapped to. Second, we want prototypes to be at least dmin (in this experiment we choose dmin = 8) away from each other, since if prototypes are too
between prototypes
8
10
a-0,p-0
a--.1, p - 1e-
0-0,p-0 --0.p-tt
(a)
(b)
Figure 5: The effect of a, 3 on interpretability
similar, then similar sentences may be mapped to different prototypes, causing confusion to users. Therefore, we compute the Euclidean distances between each pair of prototypes. Figure shows, when setting a, 3 non-zero, the distances between sentences and their prototypes are closer and prototypes are sufficiently away from each other, compared to when a; B O, proving the effect of the diversity and prototypicality terms on interpretability.
ysaxis.ap
Effect on Accuracy We performed a sensitivity analysis understand the effect of the two terms on predictive perfor- mance. We trained ProtoryNet on the Amazon dataset with different combinations of 0, le-3, 1e-2___ 1e_1, and As seen in Figure our experiment
0,8/6
0.026
0,570
0.863
0.857
0, 1e le revealed that the performance of ProtoryNet is not sensitive to
1, 12
0.859
0.805
the selection of the parameters a and 3. Between a and B, Pro-
0.863
0.854
toryNet was more sensitive to 8, which may reveal the trade-off between the different loss terms. The results also illustrate how different loss terms impact the overall performance when
0.856
O
16-4
16-2
or 3 = O. The best performance was achieved when a and [ are to small values instead of O. A possible explanation would be that, having some constraints on the prototypes' diversity (Lav)) and their representativeness. (poro)) prevents overfitting as these terms "regulate" prototypes. Sensitivity analysis for other datasets are included in the Appendix.
Figure 6: Sensitivity analysis.
4.4 USER EVALUATION
The interpretability of ProtoryNet was further validated via survey conducted 111 individuals, among which 42 identified themselves non-technical users. Subjects were recruited through two different channels. Individuals from the authors' home institution holding master's degree or above having advanced knowledge of RNNs have been recruited as technical users. Non-technical users were recruited from Amazon Mechanical Turk. The summary statistics of the subjects as well as the survey design are disclosed in Appendix.
We first evaluated the interpretability of the explanation by testing whether the model-selected prototypes were indeed representative of the input text to the human users. We asked the users to choose the most appropriate prototype for a given sentence out of four options presented to them, one of which the actual prototype matched by the model, other two were randomly selected from the rest of the prototypes, and the other was "None the above." " We created 10 such questions by sampling reviews from the Yelp Review data set, each for ProtoryNet and ProSeNet. As reported in
8

---

Human-Model Agreement
Self-reported Interpretability
Model Diagnosing
148
ProtoryNet
0.6
ProSeNet
0.4
0.2
0.8
0.0
Technical p 0.025
Non-technical
Technical p 0.042
Non-technical
Technical
Non-tecbnical
p- 0.001
p 0.07
p 0.034
p 0.433
Figure 7: Interpretability of ProtoryNet assessed by human users. The p-values are evaluated for comparing the responses for ProtoryNet and ProSeNet technical users and non-technical users, respectively.
Figure a. ProtoryNet showed a more significant agreement between the model-selected prototype and the prototype that the human users found the most appropriate. For both technical users and non-technical users, ProtoryNet was significantly better than ProSeNet, as was validated by the t-test. The difference between technical users and non-technical users was insignificant, suggesting that non-technical users can comprehend prototypes equally well as technical users.
The survey also included self-report questions to assess how easy it was for them to select prototype in a score ranging between (very difficult) and (very easy). As reported in Figure 7p. subjects found that ProtoryNet was casier interpret in general, and the improvement in interpretaiblity was more significant for technical users. Again, the difference between technical users and non-technical users insignificant.
Finally, we measured how easily the users can learn to interpret the results ProtoryNet. For this, each subject randomly assigned to either ProtoryNet ProSeNet and trained on how the model that they are assigned to makes predictions. Then, their proficiency was measured by showing them three examples on which the model had made an incorrect prediction and asking them diagnose the problem by pointing out an inappropriately matched prototype. The problematic prototype (i.e., the "correct answer" for the survey question) was determined via discussion among the authors, which later turned out to be aligned with the consensus in the survey responses as well. As in Figure 7p. the both subject groups were more accurate diagnosing ProtoryNet in general. An explanation this should be that ProtoryNet uses shorter prototypes than ProSeNet and, thus, is easier to comprehend. notice that the while technical users find ProtoryNet easier to debug, such difference was not significant for non-technical users. In fact, there and non-technicas users when they use ProSeNet since it was almost equally difficult to these two groups of users.
significant difference between technical users
5 CONCLUSION
In this paper, we introduced novel idea of prototype trajectory in RNNs. Our model, ProtoryNet, utilizes the prototype trajectories map text data into a sequence of prototypical sentences, illuminating the underlying dynamics of semantics within the text data. We showed that ProtoryNet could achieve predictive performance higher than the current state-of-the-art. In addition, sentence level prototype trajectory allowed fine-grained analysis of paragraphs. Moreover, the survey result suggested that ProtoryNet provided more intuitive prototypes than the state-of saate-of-the-an method and that the novice users were able to interpret ProtoryNet equally well as the expert users.
Our immediate future work would be to apply ProtoryNet to other types of sequence data, such as medical data, longitudinal data, etc. In more theoretical context, it would be interesting to mathematically formalize some the well-established requirements to be a prototype in the linguistics literature. For example, Panther and Köpcke (Panther & Köpcke 2008) assert several conditions that a prototype must possess-a prototypical sentence is an affirmative declarative sentence; the subject in the nominative case: the verb in prototype is in the active voice and in the indicative mood; to list a few. Albeit non- non-trive trivial, the mathematical translation of such conditions should bring more interpretability and, perhaps, better performance of ProtoryNet.

---

REFERENCES
William Alston. Philosophy of language. Prentice Hall, 1964.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. 2014.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems, pp. 8928 8939, 2019.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pp. 3504-3512, 2016.
James M Cupello and David Mishelevich. Managing prototype knowledge/expent system projects. Communications of the ACM, 31(5):534-550, 1988.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6309-6317, 2019.
Richard Fikes and Tom Kehler. The role of frame-based representation in reasoning. Communications of the ACM, 28(9):904-920, 1985.
Tian Guo, Tao Lin, and Yao Lu. An interpretable LSTM neural network for autoregressive exogenous 08.54.555,, 2018.
model. arXiv preprint
Aya Abdelsalam Ismail, Mohamed Gunady, Luiz Pessoa, Hector Corrada Bravo, and Soheil Feizi. Input-cell attention reduces vamishing saliency of recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 10813-10823, 2019.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint arXiv: 1902. 10186, 2019.
Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren. Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models. arXiv preprint arXiv:1911.06194, 2019.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint 17X.v:1506.00077 020088 2015.
Been Kim, Cynthia Rudin, and Julie Shah. The Bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in Neural Information Processing Systems, pp. 1952-1960, 2014.
Diederik P Kingma and Jimmy Ba. ADAM: A method for stochastic optimization. arXiv preprint 1:Xiv:4112.6980 2014.
Scott M Lundberg and Su-In Lee. unified approach interpreting model predictions. In Advances in neural information processing systems, pp. 4765 2017.
Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 903-913, 2019.
W James Murdoch, Peter J Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. arXiv preprint arXiv:1801.05444 2018.
Panther and Klaus-Michael Köpcke. A prototype approach to sentences and sentence types. Annual Review of Cognitive Linguistics, 6(1):83-112, 2008.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014.
10

---

Nils Reimers and Iryna Gurevych. Sentence- bert: Sentence embeddings using Siamese BERT- networks. arXiv preprint arXiv:1908. 10084, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, and Phil Blunsom. Reasoning about entailment with neural attention. arXiv preprint arXiv: 06664, 2015.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-22 215,, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint urXivv1990.000 108, 2019.
Lesia Semenova, Cynthia Rudin, and Ronald Parr. A study rashomon curves and volumes: new perspective generalization and model simplicity in machine learning. arXiv preprint arXiv: 011559908.0115, 2019.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. arXiv preprint urXiv:1704.000 724.02685, 2017.
Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Plister, and Alexander M Rush. LSTMVis: tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE Transactions on Visualization and Computer Graphics, 24(1):667-67) 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXiv preprint arXiv: 1003::1773301666 2017.
Michael Tsang, Youbang Sun, Dongxu Ren, and Yan Liu. Can i trust you more? model-agnostic hierarchical explanations. arXiv preprint urXiv:1812.0448 2018
Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang. MDNet: semantically and visually interpretable medical image diagnosis network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6428-6436, 2017.
APPENDIX
6.1
REPRODUCIBILITY
6.1.1
DATA SETS
IMDB Movie Reviews The IMDB Movie Reivews data set is a standard benchmark data set for binary sentiment classification and available at atthtts:: /ttt:ps://a..stanoor.....
sta.ford, edw.eraadddd/aaaaaaaaaaaaaa data/ seraa/eeniine imett/] The data set is comprised of 25,000 movie reviews, divided equally into 12,500 positive and 12,500 negative reviews. The total of 332,541 sentences are found in the data set. All of those reviews samples were included in experiments.
Amazon Product Reviews The Amazon Product Review data set is publicly available on Kaggle: our experiment, took random samples of similar size to the IMDB data set-30,00 reviews were randomly selected from the original Amazon data set, among which 15,429 were positive and 14,572 were negative. The total of 192,709 709 sentences were found.
https: /wwww kaggle, www./ottt/tt///...oo wellinnnggerrrr /aa////viiineeieseeeeeeeee
Yelp Reviews The Yelp Reviews data set was obtained from http::tt//tt httt::://goo.3/yyyC. The data set is comprised of 555,000 Yelp review samples and their corresponding labels. The authors of the data set has binarized the sentiment scores by assuming and stars as a negative sentiment and 3 and 4 stars positive sentiment. The total of 30,000 review samples sampling. in which the number positive reviews were 14,389 and the number of negative reviews were 15,611.
extracted via random
11

---

Rotten Tomatoes The Rotten Tomatoes Movie Review data set is corpus of movie reviews used for sentiment analysis and available at ttttttpa://gtttiaa Eotten-tomatoss dases-ddasss.. The total of 30,000 reviews were randomly selected, in which the proportion between the positive reviews and the negative reviews was exactly 1:1.
com wow//icolas--eevaaaaa....oa
Hotel Reviews The Hotel Reviews data set is comprised of 20,000 review samples evaluat- ing 1,000 hotels and is available on Kaggle: https://wwws /www kaggle com/ datafininttt hatel-revies In this paper, we assumed positive sentiment for reviews of and 5 star ratings and a negative sentiment for reviews of 1 and 2 stars. Reviews with 3 stars were ignored. This assignment yields 17,746 positive reviews and 2,254 negative ones. To balance out the data set, we randomly picked 2,254 positive reviews to make them equal, making the total of 4,508 reviews used in our experiments.
6.1.2 MODELS
Vanilla LSTM We used 300-dimensional GloVe word embeddings Pennnigton et al. (2014) to encode words in sentences. An LSTM model with 2 hidden layers of size 128 each was used. The final prediction was made by a fully connected layer size 256. A dropout layer of the rate 0.5 was used immediately before the fully connected layer.
DistilBERT DistilBERT Sanh et al. (2019) considered as light-weigh version of the state-of- the-art BERT model with smaller, faster, and less expensive deployment time and resources. In our experiments, pre-trained DistilBERT model was transferred to each target data set.
ProSeNet ProSeNet Ming al. (2019) a state-of-the-ae prototype-based interpretable RNN. For the implementation of ProSeNet, we used an LSTM layer with 2 hidden layers of size 128 and the dropout rate 0.5 for the sequence encoder. This the same configuration as the ProtoryNet's RNN layer. We fixed K 200 for all experiments for both ProSeNet and ProtoryNet, with exception of Figure
For fair comparison, used the fixed constant K 200 for both ProtoryNet and ProSeNet. In addition, the LSTM layer in ProtoryNet was implemented to have the same architecture as the baseline methods eliminate the bias.
We used TensorFlow v1. to implement ProtoryNet and the other benchmark models including stan- dard LSTM and ProSeNet. For DistilBERT, used an implementation that available in the Hug- ging Face Transformers Library (https://gg tub.. com haggg//uuggiiin www.ggggwg..g.oogoooooooooooaeee faaf/aaaaaasss formers). which implemented in PyTorch and TensorFlow 2.0.
For pre- re-prrcessing, the period ('"), the question mark ('?'), and the exclamation mark (!!) were used delimiters define the boundary between sentences. All words were then converted the lowercase and punctuations were removed using the definition in st stringg punct uat ion constant in Python 3.5. In all experiments, we used pre-trained BERT-based language model with mean-tokens pooling Reimers & Gurevych (2019) to convert the raw sentence data to sentence embeddings.
6.2 ABLATION STUDY
6.2.1
SPARSITY TRANSFORMATION
In this paper, the sparsity transformation from Š to S was used to enhance the interpretability of the model. However, the effect of the sparsity transformation on the model performance has not been investigated in the main text. Hence, we compare ProtoryNet's performance with and without the sparsity transformation.
As reported in Table ProtoryNet without the sparsity transformation exhibits a better performance than when the sparsity transformation was used. Although the margin was narrow, such a trend was statistically significant. This should be an inevitable cost paid in exchange of the enhanced interpretability, future investigation may include representing sentence with more than a single prototype, to balance between the interpretability and the increased accuracy.
"htps:/wwww.teessoBooooogg
12

---

Data set
Dense
Sparse
IMDB
0.861 (110.002)
0.849 (1:0.002)
Amazon Reviews
0.893 (#0.015)
0.882 (#04004)
Yelp Reviews
0.885 (=10.002)
0.872 (+0.0)
Rotten Tomatoes
0.796 (=0.003)
0.762 0+.00)
Hotel Reviews
0.944 (+.0.003)
0.949 (+00.044)
Performance comparison between non-sparse $ and sparse S as the input to the LSTM layer. The mean accuracy and the standard deviation from 5-fold cross validation are reported for each case. Boldface was used to indicate the one with a better performance.
Table 3:
6.2.2 DISTANCE METRICS
One of the design decisions we made for ProtoryNet architecture was between the cosine distance and the Euclidean distance metries. However,as reported in Table a comparative study revealed that there was no significant difference in performance.
Data
Cosine
Euclidean
IMDB
0.879 (#0.004)
0.849 (#0..02)
Amazon Reviews
0.881 (=0.003)
0.882 (=0.0044
Yelp Reviews
0.879 (+05)
0.872 (=0.002)
Rotten Tomatoes
0.760 0.004)
0.762 (+0.003)
Hotel Reviews
0.946 (++0.008)
0.949 (+0.004)
Table 4: Performance comparison between the cosine distance and the Euclidean distance. The mean accuracy and the standard deviation from 5-fold cross validation are reported for each case. Boldface was used to indicate the with better performance.
6.2.3 INFLUENCE THE NUMBER OF PROTOTYPES ON ACCURACY
Here, we provide additional results to show how the number of prototypes K impacts the model performance.
6.3
SENSITIVITY ANALYSIS
We provide sensitivity analysis on all datasets except the one in the main paper. The sensitivity analysis studies how the predictive performance varies with different values of a and
6.3.1
LIST OF PROTOTYPES SECTION
Figure 10 shows the prototypes and their sentiment scores used in the main Section 4.2 and the main text Figure Note that several sentences could be mapped to one prototype.
13

---

IMDB reviews
Hotal reviews
0.8
Accuracy (%)
0.8
Accuracy
0.7
0.7
0.6
ProtoryNet ProSeNet
ProtoryNet
ProSeNet
0.5
LTSM
0.5
LTSM
20
100
400
20
100
400
Number of prototypes
Number prototypes
Rotten Tomato reviews
Yelp reviews
0.9
Accuracy (96)
0.70
Accuracy (96)
0.8
0.7
0.60
ProtoryNet
Pyet
ProSeNet
ProSeNet
LTSM
0.5
LTSM
0.50
1
20
100
400
20
100
400
Number of prototypes
Number of prototypes
Figure 8: Effect on the model accuracy.
14

---

X-axis: beta, ylaxis.alhaa
X-axis beta, y-axis.alpha
0.842
0.831
0.826
0.937
0.937
0.937
0.843
0.836
0.83
0.825
0.825
1e-1
0.937
0.337
0.937
0.938
0.377
0.839
0.835
0.83
0.823
0.83
0.E
1o-2
0.937
0.936
0.938
0.936
0337
0.936
0.847
0.844
0.832
0.829
0.825
0.8
1o-3
0.938
0.94
0.938
0.938
0:935
0.933
10-3
16-2
1o-4
(a) IMDB
(b) Hotel reviews
X-axis: beta, y-axis alpha
axle: beto, axle:3lpha
0.7G7
0.755
0.755
0.725
0.713
0.871
0.87
2.863
0.864
0.867
0.865
0.7
0.872
0.758
0.756
0.753
0.721
0.713
0.868
0.158
0./54
0.758
0.725
0.712
0.875
0.873
0.868
0.883
0.659
0.868
0.864
0.759
0.756
0.747
0.727
0.714
1e5
0.872
0.87
0.864
0.886
0.861
0.062
0.860
0.759
0.754
0.75
0.726
0.718
0.874
0.863
0.864
0.886
0.861
16-5
-04
16-3
16-2
16.6
1e.4
1a.3
16.2
(d) Yelp reviews
(c) Rotten Tomatoes
Figure 9; Effect of a, on the model accuracy
15

---

Sentences
Prototypes
Prototype
Sentiment changes
[S1]The food here very good.
Great menu, surroundings, good service excellent food
0.9080044
[S2]i did find the service tad of the italian places
My only complaint would that little small can get crowded busy, that happens go the other Japanese restaurant right around the corner
0.5125591
this would go first.
8.-
0.6
02-
[S3] had italian hoggie that very good and yes they do exce llent meatballs
Appetizers Bbq shrimp great bread
0.8834794
0.0
Crawfish ceviche was flavorful and spicy
(a)
Sentences
Prototypes
Prototype score
Sentiment changes
[he] sandwiches are giant, the atmosphere is good.
Great menu, nice surroundings, good service and excellent food
0.9080044
[S2]] was there while the Steelers were on, so it was festive
With having good experience previously, decided to give Men's Wearhouse and Jos
0.68709606
0.5
[S3] agree they are little dry
Not impressed whatsoever
0.16809905
0.4
[54] So get hot sauce, ask for
The rice and beans bland, but request they give spicy sauce that tastes much like
0.25679278
0.2
some mayo ranch, etc.
52
54
55
habenero sauce to me
[S5] And it was quite warm in there
This place great
0.853906
(b)
Sentences
Prototypes
Prototype score score
Sentiment changes
Boo.
Terrible service.
0.04091281
What happened to this location.
Uh oh
0.65373635
[53]
happy
Service was friendly and around great experience.
0.89761466
[54] It just depressing nearly everyone there just doing the bare minimum to get by.
just spent that was totally wrong, really put a sour taste in my mouth will not be a returning customer.
0.10304821
0.4
51
54
[S5] So sad
Terrible service.
0.04091281
Sentences
Prototypes
Prototype score
Sentiment changes
[S1] Awful everythin...
Terrible service.
0,04091281
From the bad service, the diluted soup, the lame-no chicken the chicken salad, the PRE-
The order not made correctly, cheese the whopper with cheese, the fries cold and soggy, the chicken strips were likely picked up from the floor.
0.10424823
0.5
fabricated pizza.
0.
AVOID THIS PLACE
won't be back.
0.03598875
(d)
Figure 10: The prototypes and sentiment scores used in Figure
16

---

6.3.2 SURVEY QUESTIONS
Figures below show few examples of the survey questions we used for the user evaluation study,
For the prototype selection, we created 10 questions each, for ProSeNet and ProtoryNet. Here we only show one example in Figure II
Figure 12 and Figure 13 show how we educated the subjects about how ProtoryNet or ProSeNet work.
For diagnosing the ProSeNet and ProtoryNet, we create 3 questions for each model. We show one example for each model Figure 14 and Figure 15
Select the sentence with the similar sentimental semanties the target sentence
Target Sentence:
was delicious
They are great
B: Needless to say, did not sign up for any of the plans and returned
C: Great location hut low on amenities
None above
Figure 11: Prototype selection question.
17

---

Now we present to you model ProloryNet thai bused un the sitilarity belween seulences and "prololyypcaa" sentences to make prodiction. A pcototype seutence is sentonee that selected by the model to copcesent group of scutenses with simila: meaningo.
herween (1 and represent whether sentiment positive negative. larger values (closer indicate positive sentiment and smaller values (close 0) indicate more negative sentiment.
ProteryNet lirst mups each seateace in a paragraph to the mest similar protolypical sentence and then based the irajectory of the sentimenr in paragrahh, the model deternines overall sentiment parngraph
[Example] Here's review of four sentences
[sI] food was delicious at this pace. [s2] But the service super low and we waited for more than half' an hour for our desert, [s3] When
we aekod the waitcess about it, she was incredibly zude us. [54] We will not go back again.
The model finds the following prototypen for each sentence.
Prototype for [s1]: great food (sontiment: 0.9)
Protodype for [s2]: there always long
food Conntiment 0.7)
Prototype (>3): service very focurimmat:: 0.05)
Prototype pe [s4] Overall, it worth (xommmmnn:0.1)
We visually observe bow the sentiment changes. Seatiment drops Irom 0.9 very Jow values.
sentiment score of sentences
prototype forst sl
prototype of s2
prototype of s3
prototype of st
Figure 12: Education material for ProtoryNet
18

---

Input:
excellent food extremely clean the staff is friendly and efficient really good atmosphere
Prediction: Positive(1.00) Explanation:
(0.86)
EXCELLENT FOOD NICE decor little pricey for the proportions the salmon WAS AMAZING -> Positive(1.58)
(0.68) GREAT FOOD SERVICE and views hard to beat this is our FAVORITE <other> restaurant in the valley -> Positive(1.23)
(0.50) GREAT FOOD GREAT SERVICE vietnamese <other> rolls unbelievable GREAT
food just what the neighborhood needed -> Positive(2.82)
Similarity between the input and each prototype For example, 0.86 is the similarity between the input and prototype
Confidence prototype (how much model believes prototype is positive negative)
The total score 0.86 1.58 0.68 1.23 0.5 *2.82 3.6052 >0 So the sentiment of the review positive
Figure 13: Education material for ProSeNet
Example
True Sentiment: Negative. Model prediction: Positive
Review:
[s1] Don't go. [s2] got more problems and sounds on my car after spent $800 there. [s3]
Unbelievable!
Explanations
[s1] prototype: Definitoly won't be going back to this location (sentiment: 0.005)
[s2] >> prototype: I'd be willing bet they get SO MANY complaint letters they just can't keep
up(sentiment: 0.198)
[s3] >> Prototype: but eh (sentiment: 0.683)
Can you identify which protolype caused the incorrect prediction of the input?
A: prototype
B: prototype
C: prototype
D: can't decide
Figure 14: Diagnosis question for Protory model
19

---

Example
True Sentiment: Negative. Model prediction: Positive
Input: Don't go. I got more problems and sounds on my car after spent $800 there. Unbelicvable! Explanations
(0.732) Prototype 1: Kuhn's automatically receive for being open 24 hours. Beyond that, customer service has been great, the shelves are stocked well, prepared food and deli items are better for you than fast food with better ingredients at a better value! When I'm in there, this is my go-to shopping spot.
Positive (0.784)
(0.718) Prototype 2: This Starbucks is teeny-tiny! Seating inside is VERY limited. This is Starbucks to grab and go and continue your shopping at the Waterfront. Baristas are friendly and fast. N4g4tive(0.445)
(0.715) Prototype 3: Exceeded my expectations! had the fried chicken. It was tender and not greasy. The yams were tasty. Sweet but not overbcaring, will definitcly visit again when I'm in the arca! Negative (0.751)
Can you identify which prototype caused the incorrect prediction of the input?
protolype
protorype 2
prototype
I can't decide
Figure 15: Diagnosis question for ProSeNet model
20


Reviews:
[{"review_id": "vpx0SChGmqn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of explainable AI by trying to build models which are inherently interpretable (as opposed to post-hoc methods that interpret black-box models after being trained).\n\nIn particular, they focus on a recent paper on prototype-based networks, and a model called prosenet. They introduce an extension, protorynet, which fares better on longer documents by applying prototypes to each sentence (rather than the whole document), leading to a \"prototype trajectory\", showing how the prediction evolves over the course of the sentence. \n\nThey show improvements in prediction accuracy across 5 datasets, analyse the hyperparameters and conduct a human study.\n\nStrengths:\nI really like the concept of prototype networks, as it is simple and I could see it being accessible to non-technical users. The authors are to commended for undertaking a proper evaluation, including human experiments, which are often painful to conduct but quite helpful.\n\nThe proposed extension is straightforward and easy to understand. As someone unfamiliar with the original paper from Ming et al, I found it easy to get up to speed. The authors are also good at plainly describing the details of their approach, and running the requisite ablation studies (e.g. between sigmoid/exponential). \n\nWeaknesses:\n1. The paper's prior work section is incomplete, and makes factually incorrect statements about the state of the field. In particular, the authors focus on attention-based techniques, ignoring the considerable amount of work on other approaches, and incorrectly saying that other approaches \"often turn out to be gibberish\". Attention is a particularly strange subset of interpretations to focus on, given that the community has recently started to argue about whether attention is an explanation at all [7]\n\nIn general, I would suggest the authors look at papers from the “Interpretability and Analysis of Models for NLP” track at ACL and the blackboxNLP workshop at ACL. For concrete starting points, I'll restrict myself to ICLR papers, dating back to 2017 [2-5]. General attribution methods, such as integrated gradients [6] are also pertinent.\n\nI don't think the paper can be published without a reasonable related work section, hence the \"clear reject\" score. If this were fixed, I'd upgrade to a \"weak reject\", i.e. 5.\n\n2. I am concerned that this model may be outperformed by simple, bag of words approaches. For IMDB, the original paper [1] has a variety of results hitting 88% accuracy, while the results reported for protorynet are 85%. If a bag of words model outperforms protorynet, that makes it a less desirable model. It is possible that preprocessing differences account for this, though.\n\nCan the authors give some clarity on this, for IMDB as well as the other datasets? Ideally, there would be an additional column in the results table providing results for a simple, non-neural, baseline.\n\n3. I worry that a lot of the added accuracy does not help the model's accuracy. In particular, the ablation charts in Figures 5 and 8 show that when alpha=beta=0, the model's accuracy is within ~.2% of the best accuracy, so do we need those additional loss terms? I also suspect that averaging the outputs, rather than feeding them through an RNN would be comparably accurate, and simpler. \n\n4. While I appreciate the human studies, the confidence bars are quite wide, so that most of the findings are not statistically significant.\n\n5. For the model diagnosis human experiment, the users were only shown three examples. How were these examples chosen? That is not very many, so I worry that either those examples were chosen to be ones where protorynet was better. Ideally they would be chosen randomly, subject to some reasonable criteria.\n\nNitpicks:\n- Distillbert is an odd SOTA to choose, as it is designed to have fewer parameters. Something like RoBERTA would likely have stronger results, and be more representative of SOTA.\n\n[1] https://ai.stanford.edu/~amaas/papers/wvSent_acl2011.pdf\n[2] https://arxiv.org/abs/1911.06194\n[3] https://arxiv.org/abs/1812.04801\n[4] https://arxiv.org/abs/1801.05453\n[5] https://arxiv.org/abs/1801.05453\n[6] https://arxiv.org/abs/1703.01365\n[7] https://arxiv.org/abs/1902.10186", "extra_scores": {}, "extra_text": {}}, {"review_id": "oK5nuwj43Qe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose ProtoryNet,  a prototype-based model for paragraph classification that associates each sentence in the paragraph with a relevant prototypical sentence from the training data. The idea is interesting and the ability to decompose sentiment scores over each sentence + find prototypes for each helps to build user understanding of the model prediction. Thank you to the authors for the submission.\n\nHowever, I have two concerns---baseline comparisons and model details---that prevent me from assigning a higher rating.\n\n**Baseline comparisons**\n\n1. Across 5 sentiment classification datasets, the authors find that ProtoryNet substantially underperforms a standard BERT model, in some cases obtaining more than double the error. This seems like a substantial price to pay the ability to associate each sentence with a prototype. The authors write that \"Note, however, that DistilBERT was pre-trained on a massive corpus of text data... Hence, [it] should only be used for [a] sanity check\". However, ProtoryNet also seems to build on top of standard pre-trained BERT embeddings, which have also been derived from a massive corpus of text data (and actually, the comparison should favor ProtoryNet, since DistilBERT is a smaller model than the standard BERT model). Could the authors elaborate on why they believe that this comparison is unfair? If it is unfair, the authors should perform a comparison that is as similar as possible to ProtoryNet but without the prototype parts, i.e., train a model on top of the same BERT embeddings and see how that performs.\n\n2. Related to the above question, it is common to fine-tune BERT models on the dataset of interest. Was this done here for DistilBERT? What about for ProtoryNet? And if not, why not?\n\n**Model details**\n\n3. It was difficult to follow all of the model details; perhaps consider reorganizing and clarifying the writing. For example, it was unclear how the prototypes are actually chosen until late in the paper, whereas it should have been explained in S3.1. The notation in S3.1 has a few minor errors. For example, if the entire sentence is encoded as $\\mathbb{R}^V$ then it seems like $V$ is not just the size of the vocabulary, but the size of the vocabulary to the power of the length of the sentence? Also, how were the hyperparameter values and coefficient values chosen? Is prototype projection also done at the end of training?\n\n4. There are many modeling decisions that seem somewhat ad-hoc or non-standard. It seems like it might be possible to simplify the model significantly, or if not, it would be nice for the effects of these decisions to be better studied. For example: (a) mean-squared error is used even on binary classification problems; (b) the loss function is complicated by diversity and prototypicality terms, but the sensitivity analysis reveals that the accuracies are basically indistinguishable even when we completely remove those terms; (c) the sparsity transformation was approximated by a softmax that seems basically indistinguishable from a step function since $\\gamma \\geq 10^6$, so does it actually matter? (Note that ReLUs are also not differentiable.) (d) How useful is the LSTM at the end, if it generally goes over only ~4 sentences? \n\n**Update**\n\nThank you to the authors for the revisions, and great to know that the experimental results have improved significantly. In the absence of an updated manuscript, it is difficult to update my score appropriately, so I will leave it as it currently is. However, I think the work is promising and that an updated manuscript that incorporates the new experimental results and more carefully teases apart the contributions of the different components (as the authors have started to do in this rebuttal period) would be impactful. Thank you to the authors again for all of their hard work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "C6c9EC6_jdM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents ProtoryNet, a framework for text data that classifies and explains the prototypes' results.   The key concept, that is the novelty of the work, is that this framework is based on sentence prototypes, called prototype trajectory in the paper. In particular, instead of working at the entity of the text level, the text is split into sentences and each sentence is analyzed by itself.  The structure of the framework is composed of a layer that encodes a text's sequences, followed by a prototype layer in which is computed the similarity among each sentence and the prototype trajectories. At this point, the sentences are represented in one-hot encoding: for each sentence, there is a bunch of zero and then a one for the most similar sentence prototype. This representation is used for the classification of the sentence, done using an LSTM structure. In this setting, the interpretation is given by exploiting the prototypes matched for the text under analysis. \n\nThe idea presented in the paper is really interesting: it allows for an interpretation based on prototypes that is simple and understandable while achieving acceptable prediction performance.\n\nPros\nIt is well written and easy to read. There is only a repetition of a “the” at the end of page 7.\nThe idea of the trajectory prototypes is quite interesting, especially because it can be employed for other kinds of sequence data.\nThe explanations obtained are compelling: the results obtained from the human evaluation showed excellent results, especially in the context of local explanations for non-experts.\n I appreciated the presentation of the framework using a picture due to the complicated structure.\n\nCons\nA few words more about the diversity and the prototypically would have been useful in the objective functions. The reader is able to get a general idea, but maybe in the appendix, there could be something more to understand the claims fully.\nThe prediction performance of the model is fine, but not excellent. In my opinion, some experiments more about the prototype initialization would have been helpful in understanding if it can be a possible source of errors. \nThe evaluation of the explanations is only w.r.t. ProSeNet. However, there are other methods to compare with, such as LIME and SHAP, to name agnostic methods, but also attention-based explanations, such as LRP (Layerwise-Relevance-Propagation), NeuroX and Integrated Gradients (that are not cited even in the related work section).\nWhat about the robustness of the explanations? Similar sentences are going to be represented by the same prototype or by prototypes that are similar? An analysis of the robustness of the explanations would be useful (fidelity, hit, and similar metrics could be used, but also methods such as ROAR-RemOve And Retrain or KAR-Keep and Retrain)\nWhat about other classificators? The last part of the framework uses a LSTM to predict the sentiment of the sentence. However, due to the structure based on similarity, other classificators such as logistic regression or decision tree may be tested in the same fashion of shapelet-based classifiers. The explanation may also benefit from the use of such methods due to the additional interpretable information they provide.\n\n[LIME] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). \" Why should I trust you?\" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).\n[SHAP] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. In Advances in neural information processing systems (pp. 4765-4774).\n[LRP] Patil, A., Wadekar, A., Gupta, T., Vijan, R., & Kazi, F. (2019, July). Explainable LSTM Model for Anomaly Detection in HDFS Log File using Layerwise Relevance Propagation. In 2019 IEEE Bombay Section Signature Conference (IBSSC) (pp. 1-6). IEEE.\n[LRP] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.\n[INTGRAD] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365.\n[SHAPELET] Lines, J., Davis, L. M., Hills, J., & Bagnall, A. (2012, August). A shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 289-297).", "extra_scores": {}, "extra_text": {}}, {"review_id": "4JFWVhHweNO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper presents an RNN sequence classifying model that generates a prototype for each sentence in a paragraph. The generated prototypes help explain the model's prediction. The method embeds each sentence, matches to prototypes, then runs through an LSTM before making a prediction. Experiments found improved accuracy compared to a previous model that generates only one prototype for a paragraph. A user evaluation also found improvement in interpretability.\n\nStrengths:\n-  The paper is well written and easy to understand.\n- Generating prototypes is a promising direction for improving the interpretability of RNNs and other neural nets models. The idea of generating a prototype trajectory for sentences in a paragraph is interesting and novel to my knowledge.\n - The architecture and training methods are technically sound.\n- The experiments show positive improvement in prediction accuracy.\n\nConcern on user evaluation:\n- There is some improvement but the error bars are large, it is not clear if the differences are statistically significant.\n- With a prototype for each sentence, users will need to read more. So a more fine-grained explanation will increase cognitive load. User evaluation can potentially investigate this.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce an RNN model, ProtoryNet, which uses trajectories of sentence protoypes to illuminate the semantics of text data.

Good points were brought up and addressed in discussion, which have improved the paper - including a helpful suggestion from Rev 3 to fine-tune BERT sentence embeddings in ProtoryNet, which led to significant performance gains.

Unfortunately the tone of discussion with one reviewer slipped below the respectful standards to which we aspire, but rest assured that only substantive points on the paper were considered.

Reviewers were split but in discussion converged to leaning against acceptance, allowing the authors to reflect on, and incorporate new results carefully in an updated manuscript.
Decision: Reject