Interpretable Sequence Classification Via Prototype Trajectory

1 INTRODUCTION Recurrent neural networks ( RNN ) have been widely adopted in natural language processing . RNNs achieve the state-ofthe-art performance by utilizing the contextual information in a “ memory ” mechanism modeled via hidden/cell states . Albeit the benefit , however , the memory mechanism obstructs the interpretation of model decisions : as hidden states are carried over time , various pieces of information get intertwined across time steps , making RNN models a “ black box ” inherently . The black box nature of RNNs has motivated a body of research works aiming to achieve the interpretability . One approach is to leverage certain architecture design in the DNNs such as Attention-based methods . As will be discussed in Section 2 , the attention-based approaches ( Karpathy et al. , 2015 ; Strobelt et al. , 2017 ; Choi et al. , 2016 ; Guo et al. , 2018 ) visualize the RNN using the attention mechanism , which weighs the importance of each hidden state element . However , while a few of them could be quite illuminative , the attention weights are , in fact , not always intelligible . Rather , they often turn out to be a gibberish collection of numbers that does not possess much sensical interpretations . In fact , recent research has been considering attention weights as not explanations ( Jain & Wallace , 2019 ) . Furthermore , the analysis of attention weights requires a certain level of understanding of how RNNs work in theory . Hence , a novice user may find it difficult to understand and , thus , the broader use in real-world applications might not be so feasible . The other is prototype-based approaches ( Ming et al. , 2019 ) , which use prototypes to explain the decision more intuitively . The process is analogous to how , for example , human doctors and judges make decisions on a new case by referring to similar previous cases : for a given sequence , a prototypebased approach looks up a few representative examples , or prototypes , from the data set and deduces a decision . From the interpretability standpoint , such prototypes then provide intuitive clues and evidences of how the model has reached a conclusion in a form that even a layperson can understand . However , the existing prototype-based methods find the prototypes at the whole-paragraph level , making it difficult to break down the analysis at the individual sentence level , e.g. , the connections and flows of individual sentences constituting a paragraph . Moreover , there may not be a suitable prototype when the length of a sequence is too large ( e.g . a long paragraph ) , as longer sequences have greater degrees of freedom and , thus , harder to find a matching prototype , as evidenced later in Section 4 . Here , we advocate the idea that the sentence level prototyping ( as opposed to the paragraph level prototyping in the previous literature ) produce more desirable outcomes , namely better interpretability and higher prediction accuracy . We propose a novel architecture , called ProtoryNet , in which we introduce a new concept of the prototype trajectory . Given one or more paragraphs , ProtoryNet looks up the nearest prototype for each sentence and computes the proximity . The prototype proximity values are then fed into an RNN backbone , which then captures the latent patterns across sentences by means of the trajectory of prototypes . The prototype trajectory , therefore , illuminates the semantic structure of a text sequence and the logical flow therein , and , hence , provides highly intuitive , useful interpretation of how the model has predicted an output , as can be witnessed in Figure 1 . In fact , the prototype theory in modern linguistics supplies a strong justification for the proposed idea of prototype trajectory . In the prototype theory , linguists view “ a sentence as the smallest linguistic unit that can be used to perform a complete action , ” ( Alston , 1964 ) and analyze texts with individual sentences as building blocks . Linguists assume that the sentences of a category are distributed along a continuum : at one extreme of this continuum are sentences having a maximal number of common properties ; while on the other extreme are sentences that have only one or very few of such properties ( Panther & Köpcke , 2008 ) . Here , the “ ideal ” sentence that possesses the maximally shared common properties can be considered as a prototypical sentence , or a prototype of the category . Thus , in some sense , this paper takes a meaningful first step towards mathematically formalizing the prototype theory in modern linguistics and its analysis methods by incorporating the above view in a computational framework and emulating how linguists analyze a text . As such , ProtoryNet permits a fine-grained understanding of sequence data alongside an intuitive explanation of the dynamics of the subsequences . In addition , since the technical details are hidden in the prototypes , a non-technical user can comprehend the interpretation . However , when necessary , technical users , i.e. , the ones that are more knowledgeable about RNNs , can still look at the coefficients in RNN , similar to how the attention approaches visualize RNNs , as the proximity vectors feeding the RNN backbone are essentially one-hot encoded ( i.e. , zero everywhere except the k-th position for prototype k ) , making it convenient to trace how coefficients are related to each prototype . 2 RELATED WORK . In addition to model-agnostic black-box explainers such as LIME ( Ribeiro et al. , 2016 ) and SHAP ( Lundberg & Lee , 2017 ) , various post hoc explanation methods have been proposed for DNN models , such as Integrated Gradients ( Sundararajan et al. , 2017 ) , DeepLift ( Shrikumar et al. , 2017 ) and NeuroX ( Dalvi et al. , 2019 ) . Specifically , to understand RNN models , Tsang et al . ( 2018 ) proposes a hierarchical explanations for neural networks to capture interactions and Jin et al . ( 2019 ) adapts the idea to text classification to quantify the importance of each word and phrase . For sentiment analysis , Murdoch et al . ( 2018 ) proposes contextual decomposition method for analyzing individual predictions made by standard LSTMs , and the method is able to reliably identify words and phrases of contrasting sentiment , and how they are combined to yield the LSTMs final prediction . In addition to the external explanation methods , the prior efforts to bring interpretability to RNNs can be categorized as attention-based and prototype-based approaches . Bahdanau , Cho , and Bengio ( Bahdanau et al. , 2014 ) proposed an encoder-decoder type machine translation algorithm , in which they implemented an attention mechanism in the decoder . By the means of the alignment probabilities and the association energy , reflecting the importance of a given word in predicting a translated word , they let the attention mechanism to weigh which part of the source sentence the model needs to pay attention to . This not only improved the performance of the model by relieving the burden of the encoder having to compress all the information about the source sentence into a fixed-length vector , but also inherently visualized how the translation was conducted through the alignment matrix ( e.g. , Figure 3 of Bahdanau et al . ( 2014 ) ) . Similarly , Rocktäschel et al . ( 2015 ) analyzed word-to-word attention weights for achieving insights into how a long short term memory ( LSTM ) classifier reasons about entailment . Zhang et al . ( 2017 ) proposed a language model to read and explore discriminative image feature descriptions from reports to learn a direct mapping between lexical components and image pixels via attention . Similar strategies can be found in a number of other works ( e.g . Ismail et al . ( 2019 ) ; Choi et al . ( 2016 ) ) that employ the idea of the attention mechanism . However , the attention-based approaches are mostly intended for expert users . Many non-technical users in the real world , who lack basic knowledge of how RNNs work ( or even neural networks in general ) , may find them difficult to understand . Instead , prototype-based approaches argue that the intuitiveness of interpretation can be significantly enhanced by visualizing the reasoning process in terms of prototypes . In fact , prototype-based reasoning has a long history as a fundamental interpretability mechanism in traditional models ( Cupello & Mishelevich , 1988 ; Fikes & Kehler , 1985 ; Kim et al. , 2014 ) . More recently , the idea of prototype-based interpretation attracted several authors in neural network research . For instance , Chen et al . ( 2019 ) incorporated the concept of prototypes to convolutional neural networks . A prototype layer was added after convolutional layers to compare the convolution responses at different locations with prototypes . From this , users can understand , for example , a bird is classified as a ‘ red-bellied woodpecker ’ because it has the typical prominent red tint at the belly and the top of its head as well as the black and white stripes on its wings . More closely related to the present work , ProSeNet ( Ming et al. , 2019 ) added a prototype layer in RNN . ProSeNet computed the similarities between an input sequence ( usually a short prose ) and prototypes and produced the final prediction as a linear combination of the similarities . Despite the built-in interpretability of ProSeNet , however , an issue might arise when the sequence was too long . The original paper ( Ming et al. , 2019 ) validates ProSeNet only on paragraphs shorter than 25 words . However , it is easily fathomable that ProSeNet may fail to assimilate a long paragraph data , due to large degrees of freedom that complicate matching of a prototypical example . This may render some practical concerns . For instance , in sentiment classification , even if a paragraph is classified as “ negative , ” it could consist of several twists of sentiments along sentences ( e.g. , sarcastic use of positive proses ) . With an increased length , such kinds of twists can get harder to be represented with a prototype , thus making the interpretation difficult and the explanation less credible . This claim is further supported by findings in modern linguistics , which suggests that sentences , instead of paragraphs , should be regarded as the basic elements for text analysis ( Panther & Köpcke , 2008 ) . Nonetheless , we share our view with ProSeNet in that the prototype based method is worth paying attention to . The benefit of prototype-based reasoning resides in the fact that it hides technical details by encapsulating them with prototypical examples while being still tractable numerically when desired . Hence , novice users can understand how the reasoning was achieved in RNNs so long as they can comprehend the prototypes , lowering the barrier for those numerous non-technical users who may use RNN-based applications in the real world . On the other hand , numerical weights assigned to prototypes alongside their association with the “ nuts and bolts ” of RNNs still allow experts to perform in-depth analyses of how a model has drawn a prediction . 3 METHOD . 3.1 THE PROTORYNET ARCHITECTURE . Suppose a data set D = { ( X ( i ) , y ( i ) ) : i = 1 , . . . , N } of size N , comprised of text sequences X ( i ) and the corresponding labels y ( i ) . Here , note that the superscript ( i ) may be dropped for notational convenience hereinafter , unless necessary . Each instance X can be understood as a sequence of sentences xt ∈ RV at t-th position , yielding the representation X = ( xt ) Tt=1 , where V is the size of vocabulary and T : = |X| is the number of sentences in the sequence X. y ∈ RC is a multi-hot encoded vector representing the class labels associated with the sequence X , i.e. , the c-th element yc of y equals 1 if the label c is associated with X or 0 otherwise . C is the total number of classes . ProtoryNet interfaces with text data via a sentence encoder ( Figure 2a ) modeled as a mapping r : RV → RJ , where J is the dimension of sentence encoding specified by the user . That is , the encoder takes each sentence xt ∈ X and produces a sentence embedding et = r ( xt ) . In this paper , the development of the encoder r is beyond the scope of this paper and , hence , we employ a pre-trained BERT-based sentence encoder with mean-tokens pooling ( Reimers & Gurevych , 2019 ) , where J = 768 by default . The sentence embeddings et are fed into the prototype layer ( Figure 2b ) , in which a set of trainable prototypes P = { pk ∈ RJ : k = 1 , . . . , K } are compared with et , where K : = |P| is the number of prototypes specified by the user . Then , given a distance metric d : RJ → R+ , the proximity st , k of the sentence embedding et to a given prototype pk is measured as st , k : = s ( et , pk ) = exp ( −d ( et , pk ) ψ2 ) , where ψ ∈ R is a user-specified constant . Between two popular choices for the distance metric d , namely the cosine distance and the Euclidean distance , we find that there is no significant difference between the two . Hence , we use the Euclidean distance in our experiments for convenience . Note here that the intermediate throughput of the prototype layer is the similarity matrix S̃ = [ st , k ] of the size T ×K , associating the t-th sentence with the k-th prototype . The rows of the similarity matrix S̃ then constitute the input to the LSTM backbone at time step t ( Figure 2c ) , which then finally produces an output prediction . Here , the sparsity transformation is performed on the similarity matrix S̃ before feeding it into the LSTM backbone by setting each row to zero except for the position where st , k is the maximum . That is , each row of the transformed similarity matrix S would be of the same topology as the one-hot encoded vector , whose elements equal st , k∗ at k∗ : = argmaxk st , k and 0 otherwise . The sparsity transformation of S̃ to S enhances the interpretability of the architecture , by enforcing each sentence be matched with no more than a single prototype and , thus , disentangling the information . This is accomplished only at a small cost of accuracy , as observed from an ablation study in Appendix . Motivating Example The text data in Figure 2 exemplifies the use of ProtoryNet for sentiment analysis ( text classification ) . In this example , the task is to predict whether the review of a restaurant is positive or not . The input text data X is comprised of T = 4 sentences , in this particular case , and the label y is the binary sentiment label of the review , either “ positive ” ( [ 1 , 0 ] ) or “ negative ” ( [ 0 , 1 ] ) . To this , ProtoryNet converts the text data into sentence embeddings , each of which are then matched with the closest prototype . Observe , in the figure , that the prototypes that ProtoryNet produced are , indeed , morphosyntactically equivalent to the corresponding input sentences , well-exemplifying them semantically . The one-hot-like similarity vectors between the sentences and the prototypes are then fed into the LSTM backbone , which captures the patterns and trends in the trajectory of prototypes and , finally , predicts the final sentiment label , which , in this case , is “ negative. ” 3.2 OBJECTIVE FUNCTIONS The training objectives of ProtoryNet entail four different terms aiming to achieve both the prediction accuracy and the interpretability . Below are the details of their definitions . Accuracy The accuracy loss is defined as the square loss between the predicted value and the ground truth label , promoting the model to make accurate predictions : Lacc ( D ) : = 1 N N∑ i=1 ∥∥∥y ( i ) − ŷ ( i ) ∥∥∥2 . ( 1 ) Diversity To ensure diverse and non-overlapping prototypes , we define the diversity loss term added to enforce the minimum mutual distance δ among the prototypes : dmin : = min k1 , k2 d ( pk1 , pk2 ) , ( 2 ) Ldiv ( D ) : = σ ( η ( δ − dmin ) ) , ( 3 ) where σ ( · ) is the sigmoid function and η is a smoothing constant , which we set η = 1 empirically . The constant δ ∈ R+∗ is a positive real number defined by the user , to enforce the minimum separation among prototypes . Hence , when the distances among the prototypes do not meet the minimum separation requirement i.e. , dmin < δ , the η ( δ − dmin ) term will have some positive value , making the diversity loss term Ldiv active ; on the other hand , when the minimum separation requirement is met and thus , dmin > δ , then the sigmoid function will pull the loss term to zero . Note that a smaller η will make such a transition by the sigmoid function smoother . Prototypicality The accuracy and the diversity terms alone , it is observable a prominent tendency of prototypes diverging away from the sentences during training . Such a behavior introduces overfitting , in which prototypes become less generalizable , as the prototypes lose their representativity of a category . Hence , we introduce the prototypicality loss , which promotes each sentence in the database to have a representative prototype close to it : Lproto = 1 S ∑ X∈D ∑ xt∈X min k d ( r ( xt ) , pk ) , ( 4 ) where S is the total number of sentences in the data set . Final loss The final loss function combines the above loss terms : L = Lacc + αLdiv + βLproto ( 5 ) Empirically , coefficients values of α = 0.1 and β = 1e−4 are used in this paper . Remarks on Prototype Interpretability The diversity and prototypicality terms are designed for improving the interpretability . Here , to achieve good explanations , prototypes need to be different from each other to avoid redundancy , thus the diversity term . In addition , each input sentence needs to be mapped to a prototype that is similar enough to make the explanation convincing , thus the prototypicality term . These two loss terms can be considered regularization terms to serve interpretability purposes . Similar loss terms have been introduced in other prototype based DNN models ( Ming et al. , 2019 ; Chen et al. , 2019 ) . We will later show in experiments that these two terms do not hurt the predictive performance . This can be explained by the recent research on Rashomon Set ( Semenova et al. , 2019 ; Rudin , 2019 ) , that there exist many models with very similar performance , so one can add customized constraints to the model to achieve additional benefits , such as interpretability .

Summary: This paper presents ProtoryNet, a framework for text data that classifies and explains the prototypes' results.   The key concept, that is the novelty of the work, is that this framework is based on sentence prototypes, called prototype trajectory in the paper. In particular, instead of working at the entity of the text level, the text is split into sentences and each sentence is analyzed by itself.  The structure of the framework is composed of a layer that encodes a text's sequences, followed by a prototype layer in which is computed the similarity among each sentence and the prototype trajectories. At this point, the sentences are represented in one-hot encoding: for each sentence, there is a bunch of zero and then a one for the most similar sentence prototype. This representation is used for the classification of the sentence, done using an LSTM structure. In this setting, the interpretation is given by exploiting the prototypes matched for the text under analysis.