Interpretable Sequence Classification Via Prototype Trajectory

1 INTRODUCTION Recurrent neural networks ( RNN ) have been widely adopted in natural language processing . RNNs achieve the state-ofthe-art performance by utilizing the contextual information in a “ memory ” mechanism modeled via hidden/cell states . Albeit the benefit , however , the memory mechanism obstructs the interpretation of model decisions : as hidden states are carried over time , various pieces of information get intertwined across time steps , making RNN models a “ black box ” inherently . The black box nature of RNNs has motivated a body of research works aiming to achieve the interpretability . One approach is to leverage certain architecture design in the DNNs such as Attention-based methods . As will be discussed in Section 2 , the attention-based approaches ( Karpathy et al. , 2015 ; Strobelt et al. , 2017 ; Choi et al. , 2016 ; Guo et al. , 2018 ) visualize the RNN using the attention mechanism , which weighs the importance of each hidden state element . However , while a few of them could be quite illuminative , the attention weights are , in fact , not always intelligible . Rather , they often turn out to be a gibberish collection of numbers that does not possess much sensical interpretations . In fact , recent research has been considering attention weights as not explanations ( Jain & Wallace , 2019 ) . Furthermore , the analysis of attention weights requires a certain level of understanding of how RNNs work in theory . Hence , a novice user may find it difficult to understand and , thus , the broader use in real-world applications might not be so feasible . The other is prototype-based approaches ( Ming et al. , 2019 ) , which use prototypes to explain the decision more intuitively . The process is analogous to how , for example , human doctors and judges make decisions on a new case by referring to similar previous cases : for a given sequence , a prototypebased approach looks up a few representative examples , or prototypes , from the data set and deduces a decision . From the interpretability standpoint , such prototypes then provide intuitive clues and evidences of how the model has reached a conclusion in a form that even a layperson can understand . However , the existing prototype-based methods find the prototypes at the whole-paragraph level , making it difficult to break down the analysis at the individual sentence level , e.g. , the connections and flows of individual sentences constituting a paragraph . Moreover , there may not be a suitable prototype when the length of a sequence is too large ( e.g . a long paragraph ) , as longer sequences have greater degrees of freedom and , thus , harder to find a matching prototype , as evidenced later in Section 4 . Here , we advocate the idea that the sentence level prototyping ( as opposed to the paragraph level prototyping in the previous literature ) produce more desirable outcomes , namely better interpretability and higher prediction accuracy . We propose a novel architecture , called ProtoryNet , in which we introduce a new concept of the prototype trajectory . Given one or more paragraphs , ProtoryNet looks up the nearest prototype for each sentence and computes the proximity . The prototype proximity values are then fed into an RNN backbone , which then captures the latent patterns across sentences by means of the trajectory of prototypes . The prototype trajectory , therefore , illuminates the semantic structure of a text sequence and the logical flow therein , and , hence , provides highly intuitive , useful interpretation of how the model has predicted an output , as can be witnessed in Figure 1 . In fact , the prototype theory in modern linguistics supplies a strong justification for the proposed idea of prototype trajectory . In the prototype theory , linguists view “ a sentence as the smallest linguistic unit that can be used to perform a complete action , ” ( Alston , 1964 ) and analyze texts with individual sentences as building blocks . Linguists assume that the sentences of a category are distributed along a continuum : at one extreme of this continuum are sentences having a maximal number of common properties ; while on the other extreme are sentences that have only one or very few of such properties ( Panther & Köpcke , 2008 ) . Here , the “ ideal ” sentence that possesses the maximally shared common properties can be considered as a prototypical sentence , or a prototype of the category . Thus , in some sense , this paper takes a meaningful first step towards mathematically formalizing the prototype theory in modern linguistics and its analysis methods by incorporating the above view in a computational framework and emulating how linguists analyze a text . As such , ProtoryNet permits a fine-grained understanding of sequence data alongside an intuitive explanation of the dynamics of the subsequences . In addition , since the technical details are hidden in the prototypes , a non-technical user can comprehend the interpretation . However , when necessary , technical users , i.e. , the ones that are more knowledgeable about RNNs , can still look at the coefficients in RNN , similar to how the attention approaches visualize RNNs , as the proximity vectors feeding the RNN backbone are essentially one-hot encoded ( i.e. , zero everywhere except the k-th position for prototype k ) , making it convenient to trace how coefficients are related to each prototype . 2 RELATED WORK . In addition to model-agnostic black-box explainers such as LIME ( Ribeiro et al. , 2016 ) and SHAP ( Lundberg & Lee , 2017 ) , various post hoc explanation methods have been proposed for DNN models , such as Integrated Gradients ( Sundararajan et al. , 2017 ) , DeepLift ( Shrikumar et al. , 2017 ) and NeuroX ( Dalvi et al. , 2019 ) . Specifically , to understand RNN models , Tsang et al . ( 2018 ) proposes a hierarchical explanations for neural networks to capture interactions and Jin et al . ( 2019 ) adapts the idea to text classification to quantify the importance of each word and phrase . For sentiment analysis , Murdoch et al . ( 2018 ) proposes contextual decomposition method for analyzing individual predictions made by standard LSTMs , and the method is able to reliably identify words and phrases of contrasting sentiment , and how they are combined to yield the LSTMs final prediction . In addition to the external explanation methods , the prior efforts to bring interpretability to RNNs can be categorized as attention-based and prototype-based approaches . Bahdanau , Cho , and Bengio ( Bahdanau et al. , 2014 ) proposed an encoder-decoder type machine translation algorithm , in which they implemented an attention mechanism in the decoder . By the means of the alignment probabilities and the association energy , reflecting the importance of a given word in predicting a translated word , they let the attention mechanism to weigh which part of the source sentence the model needs to pay attention to . This not only improved the performance of the model by relieving the burden of the encoder having to compress all the information about the source sentence into a fixed-length vector , but also inherently visualized how the translation was conducted through the alignment matrix ( e.g. , Figure 3 of Bahdanau et al . ( 2014 ) ) . Similarly , Rocktäschel et al . ( 2015 ) analyzed word-to-word attention weights for achieving insights into how a long short term memory ( LSTM ) classifier reasons about entailment . Zhang et al . ( 2017 ) proposed a language model to read and explore discriminative image feature descriptions from reports to learn a direct mapping between lexical components and image pixels via attention . Similar strategies can be found in a number of other works ( e.g . Ismail et al . ( 2019 ) ; Choi et al . ( 2016 ) ) that employ the idea of the attention mechanism . However , the attention-based approaches are mostly intended for expert users . Many non-technical users in the real world , who lack basic knowledge of how RNNs work ( or even neural networks in general ) , may find them difficult to understand . Instead , prototype-based approaches argue that the intuitiveness of interpretation can be significantly enhanced by visualizing the reasoning process in terms of prototypes . In fact , prototype-based reasoning has a long history as a fundamental interpretability mechanism in traditional models ( Cupello & Mishelevich , 1988 ; Fikes & Kehler , 1985 ; Kim et al. , 2014 ) . More recently , the idea of prototype-based interpretation attracted several authors in neural network research . For instance , Chen et al . ( 2019 ) incorporated the concept of prototypes to convolutional neural networks . A prototype layer was added after convolutional layers to compare the convolution responses at different locations with prototypes . From this , users can understand , for example , a bird is classified as a ‘ red-bellied woodpecker ’ because it has the typical prominent red tint at the belly and the top of its head as well as the black and white stripes on its wings . More closely related to the present work , ProSeNet ( Ming et al. , 2019 ) added a prototype layer in RNN . ProSeNet computed the similarities between an input sequence ( usually a short prose ) and prototypes and produced the final prediction as a linear combination of the similarities . Despite the built-in interpretability of ProSeNet , however , an issue might arise when the sequence was too long . The original paper ( Ming et al. , 2019 ) validates ProSeNet only on paragraphs shorter than 25 words . However , it is easily fathomable that ProSeNet may fail to assimilate a long paragraph data , due to large degrees of freedom that complicate matching of a prototypical example . This may render some practical concerns . For instance , in sentiment classification , even if a paragraph is classified as “ negative , ” it could consist of several twists of sentiments along sentences ( e.g. , sarcastic use of positive proses ) . With an increased length , such kinds of twists can get harder to be represented with a prototype , thus making the interpretation difficult and the explanation less credible . This claim is further supported by findings in modern linguistics , which suggests that sentences , instead of paragraphs , should be regarded as the basic elements for text analysis ( Panther & Köpcke , 2008 ) . Nonetheless , we share our view with ProSeNet in that the prototype based method is worth paying attention to . The benefit of prototype-based reasoning resides in the fact that it hides technical details by encapsulating them with prototypical examples while being still tractable numerically when desired . Hence , novice users can understand how the reasoning was achieved in RNNs so long as they can comprehend the prototypes , lowering the barrier for those numerous non-technical users who may use RNN-based applications in the real world . On the other hand , numerical weights assigned to prototypes alongside their association with the “ nuts and bolts ” of RNNs still allow experts to perform in-depth analyses of how a model has drawn a prediction . 3 METHOD . 3.1 THE PROTORYNET ARCHITECTURE . Suppose a data set D = { ( X ( i ) , y ( i ) ) : i = 1 , . . . , N } of size N , comprised of text sequences X ( i ) and the corresponding labels y ( i ) . Here , note that the superscript ( i ) may be dropped for notational convenience hereinafter , unless necessary . Each instance X can be understood as a sequence of sentences xt ∈ RV at t-th position , yielding the representation X = ( xt ) Tt=1 , where V is the size of vocabulary and T : = |X| is the number of sentences in the sequence X. y ∈ RC is a multi-hot encoded vector representing the class labels associated with the sequence X , i.e. , the c-th element yc of y equals 1 if the label c is associated with X or 0 otherwise . C is the total number of classes . ProtoryNet interfaces with text data via a sentence encoder ( Figure 2a ) modeled as a mapping r : RV → RJ , where J is the dimension of sentence encoding specified by the user . That is , the encoder takes each sentence xt ∈ X and produces a sentence embedding et = r ( xt ) . In this paper , the development of the encoder r is beyond the scope of this paper and , hence , we employ a pre-trained BERT-based sentence encoder with mean-tokens pooling ( Reimers & Gurevych , 2019 ) , where J = 768 by default . The sentence embeddings et are fed into the prototype layer ( Figure 2b ) , in which a set of trainable prototypes P = { pk ∈ RJ : k = 1 , . . . , K } are compared with et , where K : = |P| is the number of prototypes specified by the user . Then , given a distance metric d : RJ → R+ , the proximity st , k of the sentence embedding et to a given prototype pk is measured as st , k : = s ( et , pk ) = exp ( −d ( et , pk ) ψ2 ) , where ψ ∈ R is a user-specified constant . Between two popular choices for the distance metric d , namely the cosine distance and the Euclidean distance , we find that there is no significant difference between the two . Hence , we use the Euclidean distance in our experiments for convenience . Note here that the intermediate throughput of the prototype layer is the similarity matrix S̃ = [ st , k ] of the size T ×K , associating the t-th sentence with the k-th prototype . The rows of the similarity matrix S̃ then constitute the input to the LSTM backbone at time step t ( Figure 2c ) , which then finally produces an output prediction . Here , the sparsity transformation is performed on the similarity matrix S̃ before feeding it into the LSTM backbone by setting each row to zero except for the position where st , k is the maximum . That is , each row of the transformed similarity matrix S would be of the same topology as the one-hot encoded vector , whose elements equal st , k∗ at k∗ : = argmaxk st , k and 0 otherwise . The sparsity transformation of S̃ to S enhances the interpretability of the architecture , by enforcing each sentence be matched with no more than a single prototype and , thus , disentangling the information . This is accomplished only at a small cost of accuracy , as observed from an ablation study in Appendix . Motivating Example The text data in Figure 2 exemplifies the use of ProtoryNet for sentiment analysis ( text classification ) . In this example , the task is to predict whether the review of a restaurant is positive or not . The input text data X is comprised of T = 4 sentences , in this particular case , and the label y is the binary sentiment label of the review , either “ positive ” ( [ 1 , 0 ] ) or “ negative ” ( [ 0 , 1 ] ) . To this , ProtoryNet converts the text data into sentence embeddings , each of which are then matched with the closest prototype . Observe , in the figure , that the prototypes that ProtoryNet produced are , indeed , morphosyntactically equivalent to the corresponding input sentences , well-exemplifying them semantically . The one-hot-like similarity vectors between the sentences and the prototypes are then fed into the LSTM backbone , which captures the patterns and trends in the trajectory of prototypes and , finally , predicts the final sentiment label , which , in this case , is “ negative. ” 3.2 OBJECTIVE FUNCTIONS The training objectives of ProtoryNet entail four different terms aiming to achieve both the prediction accuracy and the interpretability . Below are the details of their definitions . Accuracy The accuracy loss is defined as the square loss between the predicted value and the ground truth label , promoting the model to make accurate predictions : Lacc ( D ) : = 1 N N∑ i=1 ∥∥∥y ( i ) − ŷ ( i ) ∥∥∥2 . ( 1 ) Diversity To ensure diverse and non-overlapping prototypes , we define the diversity loss term added to enforce the minimum mutual distance δ among the prototypes : dmin : = min k1 , k2 d ( pk1 , pk2 ) , ( 2 ) Ldiv ( D ) : = σ ( η ( δ − dmin ) ) , ( 3 ) where σ ( · ) is the sigmoid function and η is a smoothing constant , which we set η = 1 empirically . The constant δ ∈ R+∗ is a positive real number defined by the user , to enforce the minimum separation among prototypes . Hence , when the distances among the prototypes do not meet the minimum separation requirement i.e. , dmin < δ , the η ( δ − dmin ) term will have some positive value , making the diversity loss term Ldiv active ; on the other hand , when the minimum separation requirement is met and thus , dmin > δ , then the sigmoid function will pull the loss term to zero . Note that a smaller η will make such a transition by the sigmoid function smoother . Prototypicality The accuracy and the diversity terms alone , it is observable a prominent tendency of prototypes diverging away from the sentences during training . Such a behavior introduces overfitting , in which prototypes become less generalizable , as the prototypes lose their representativity of a category . Hence , we introduce the prototypicality loss , which promotes each sentence in the database to have a representative prototype close to it : Lproto = 1 S ∑ X∈D ∑ xt∈X min k d ( r ( xt ) , pk ) , ( 4 ) where S is the total number of sentences in the data set . Final loss The final loss function combines the above loss terms : L = Lacc + αLdiv + βLproto ( 5 ) Empirically , coefficients values of α = 0.1 and β = 1e−4 are used in this paper . Remarks on Prototype Interpretability The diversity and prototypicality terms are designed for improving the interpretability . Here , to achieve good explanations , prototypes need to be different from each other to avoid redundancy , thus the diversity term . In addition , each input sentence needs to be mapped to a prototype that is similar enough to make the explanation convincing , thus the prototypicality term . These two loss terms can be considered regularization terms to serve interpretability purposes . Similar loss terms have been introduced in other prototype based DNN models ( Ming et al. , 2019 ; Chen et al. , 2019 ) . We will later show in experiments that these two terms do not hurt the predictive performance . This can be explained by the recent research on Rashomon Set ( Semenova et al. , 2019 ; Rudin , 2019 ) , that there exist many models with very similar performance , so one can add customized constraints to the model to achieve additional benefits , such as interpretability .

Summary: this paper presents an RNN sequence classifying model that generates a prototype for each sentence in a paragraph. The generated prototypes help explain the model's prediction. The method embeds each sentence, matches to prototypes, then runs through an LSTM before making a prediction. Experiments found improved accuracy compared to a previous model that generates only one prototype for a paragraph. A user evaluation also found improvement in interpretability.