Interactively Generating Explanations for Transformer Language Models

1 INTRODUCTION . Transformer language models ( LMs ) are ubiquitous in NLP today but also notoriously opaque . Therefore , it is not surprising that a growing body of work aims to interpret them : Recent evaluations of approaches of saliency methods ( Ding & Koehn , 2021 ) and instance attribution methods Pezeshkpour et al . ( 2021 ) find that , while intriguing , for the same outputs , different methods assign importance to different inputs . Furthermore , they are usually employed post hoc , thus possibly encouraging reporting bias ( Gordon & Van Durme , 2013 ) . The black-box character of LMs becomes especially problematic , as the data to train them might be unfiltered and contain ( human ) bias . As a result , ethical concerns about these models arise , which can have a substantial negative impact on society as they get increasingly integrated into our lives ( Bender et al. , 2021 ) . Here , we focus on providing case-based reasoning explanations during the inference process , directly outputting the LM ’ s predictions . In doing so , we avoid the problems mentioned above of post hoc explanations and help to reduce the issue of ( human ) bias . To increase the interpretability of the model , we enhance the transformer architecture with a prototype layer and propose PrototypicalTransformer Explanation ( Proto-Trex ) Networks . Proto-Trex networks provide an explanation as a prototypical example for a specific model prediction , which is similar to ( training- ) samples with the corresponding label . Our experimental results demonstrate that Proto-Trex networks perform on par with noninterpretable baselines , e.g . BERT ( Devlin et al. , 2019 ) and GPT ( Radford et al. , 2019 ) . In terms of explanations , we show promising results with learned prototypes providing helpful explanations for the user to understand better the LMs decision-making , which , in turn , increases trust and reliability . To further enhance the prototypical network with human supervision , we propose an interactive learning setting ( iProto-Trex ) that allows users of any knowledge to give feedback and improve the model ( Ribeiro et al. , 2016 ) , moving beyond a purely data-driven approach . To summarize , our contributions are as follows : We ( i ) introduce prototype networks for transformer LMs that provide explanations and ( ii ) show that they are on par with non-interpretable baselines on classification tasks on different architectures . Furthermore , to improve prototype networks , we ( iii ) provide a novel interactive prototype learning setup accounting for user feedback certainty . We proceed as follows . We start by briefly reviewing related work of interpretability in NLP . Then we introduce Proto-Trex networks , including our novel interactive learning setup combining explanatory interactive learning with prototype networks . Before concluding , we discuss faithful explanations and touch upon the results of our experimental evaluation . 2 TOWARDS THE EXPLAINABILITY OF TRANSFORMER LANGUAGE MODELS . To open the black-box of transformers , we use i/Proto-Trex networks built upon post hoc interpretation methods , case-based reasoning approaches , and explanatory interactive learning . Post hoc interpretability . Various ( post hoc ) interpretability methods focus on different parts of the transformer architecture . Atanasova et al . ( 2020 ) and Belinkov & Glass ( 2019 ) provide overviews of this fast developing field . Generally , there are methods that analyze word representations ( Voita et al. , 2019a ) , the attention distribution throughout the model ( Jain & Wallace , 2019 ; Wiegreffe & Pinter , 2019 ) , and the ( attention and classification ) model heads ( Voita et al. , 2019b ; Geva et al. , 2021 ) . Other approaches such as Geva et al . ( 2020 ) focus on the feed-forward layers . Gradientbased approaches , such as Sundararajan et al . ( 2017 ) and Smilkov et al . ( 2017 ) can generally be used to trace gradients , while influence functions ( Koh & Liang , 2017 ) trace model parameter changes throughout a LM ( Han et al. , 2020 ) . While backtracking all the model weights might be possible , such explanations can only tell us which part of the input they are looking at ( Chen et al. , 2019 ) and humans are ill-equipped to interpret them ( Stammer et al. , 2021 ) . So instead , we aim for more intuitive and sparse explanations : well-descriptive but short sequences as prototypes . These are not only able to reveal the parts of the input it is looking at but also show prototypical cases similar to those parts . Case-based Reasoning in Deep Neural Networks . Our work relates most closely to previous work demonstrating the benefits of prototype networks in Computer Vision ( Li et al. , 2018 ; Chen et al. , 2019 ) as well as for sequential data by combining them with RNNs ( Hase et al. , 2019 ; Ming et al. , 2019 ) . More precisely , in contrast to post hoc interpretation methods , prototype networks employ explanations in the , e.g . classification process , i.e . classify a sample by comparing several parts – which could be words of a sentence– to ( learned ) prototypical parts from other samples for a given class . If things look similar , they are classified similarly . Humans behave very similarly , which is called case-based reasoning ( Clifton & Frohnsdorff , 2001 ) . The prototypical parts provided by the network help the user to understand the classification . Tab . 1 illustrates the benefit of prototypical explanations . One can observe that the post hoc explanation can leave the user clueless , while the prototypical explanations can help better understand the network ’ s decision . Accordingly , prototypes are an additional method in the interpretability toolbox , extending current post hoc methods . As described , the reasoning process of prototypical networks is qualitatively similar to that of humans ( Clifton & Frohnsdorff , 2001 ; Chen et al. , 2019 ) . In previous approaches , however , the training of such networks was entirely data-driven . Therefore , like Chen et al . ( 2019 ) , in our work , we do not focus on quantifying unit interpretability of prototypical networks , but extend learning the reasoning process of our network by Explanatory Interactive Learning ( Schramowski et al. , 2020 ) . Explanatory Interactive Learning ( XIL ) incorporates explanatory supervision in the learning process by involving the human user . More precisely , human users can ask the model to explain a prediction . They can then respond by correcting the model if necessary , providing a slightly improved –but not necessarily optimal– feedback on the explanations ( Stammer et al. , 2021 ) . Interactive learning and , in particular , explanatory interactive learning showed not only to increase the explanation quality of deep black-box models but also the trust of the user ( Ross et al. , 2017 ; Selvaraju et al. , 2019 ; Teso & Kersting , 2019 ; Schramowski et al. , 2020 ) . In contrast to previous ( post hoc ) XIL methods , we avoid tracing gradients . That makes our approach particularly efficient , as we only change prototypes ( cf . Fig 4 ) . 3 PROTO-TREX : PROTOTYPE LEARNING FOR TRANSFORMER LMS . The prototype network ( Proto-Trex ) architecture builds upon large-scale transformer LMs , summarized in Fig . 2 . In the following , we describe the general idea of prototypical networks and our transformer-based prototype architecture and its modules in more detail . 3.1 PROTO-TREX NETWORKS . General Idea . Proto-Trex is a prototype network , where prototypes are representative observations in the training set . The classification is based on the neighboring prototypes , i.e . an input query is classified as positive when the network thinks it looks similar to a certain ( near ) positive prototype within the training set . In more detail , the reasoning and classification processes use the same module , i.e . , they use the similarity between query and prototypes . As both reasoning and classification use the same module ( in parallel ) , we consequently have an interpretable model . In particular , ( interpretable ) prototypical explanations help verify Right for the Right Reasons ( Ross et al. , 2017 ) . Architecture . Specifically , the model , in our case the pre-trained transformer f , creates a context embedding e from the input sequence x . This embedding contains useful features for the prediction and is then passed on to the prototype layer . This layer computes the similarity between the embedding e and each of the trainable prototypes pj from the set of prototypes P . The prototypes are learned during the training process and represent prototypical ( sub- ) sequences in the training data . We use transformer models both on the sentence-level ( a ) and the word-level ( b ) , resulting in a single input representation for the sentence-level transformer and l for word-level . To compare the word-level embeddings ei with the prototypes , we have to patch them ( Z = { patch ( ei ) } ) into subsequences according to the desired prototype length k. The resulting similarity values are passed on to the final linear layer g with weight matrix wg and no bias , mapping from m ( the number of prototypes ) to the number of classes . Finally , the output values are passed on to a softmax producing class probabilities and predictions t . 3.2 PROTO-TREX LOSS . Optimization of prototype networks aims at maximizing both performance and interpretability . To this end , our Proto-Trex loss L combines performance and interpretability with prior knowledge of the prototype network ’ s structure . Let us illustrate this for the word-level case ( the sentence-level case results from e=z ) : L : = min P , wg 1 n n∑ i=1 CE ( ti , yi ) + λ1Clst ( z , p ) + λ2Sep ( z , p ) + λ3Distr ( z , p ) + λ4Divers ( p ) + λ5||wg|| ( 1 ) where λi weights the influence of the different terms and n is the number of training examples . The first term is the cross-entropy ( CE ) loss optimizing the predictive power of the network . The second term ( Clst ) clusters the prototypes w.r.t . the training examples of the same class , maximizing similarity to them ( we rewrite maximization as minimization terms ) , and the separation loss minimizes the similarity to other-class instances : Clst ( z , p ) : = − 1 n n∑ i=1 min j : pj∈Pyi min z∈Z sim ( z , pj ) ; Sep ( z , p ) : = 1 n n∑ i=1 min j : pj /∈Pyi min z∈Z sim ( z , pj ) ( 2 ) Together , Clst and Sep push each prototype to focus more on training examples from the same class and less on training examples from other classes . Both are motivated by ProtoPNet ( Chen et al. , 2019 ) . To get prototypes that are distributed well in the embedding space , we introduce two additional losses , a distribution loss ( Li et al. , 2018 ) , assuring that a prototype is nearby each training example , and a diversity loss ( Ming et al. , 2019 ) : Distr ( z , p ) : = − 1 m m∑ j=1 min i∈ [ 1 , n ] min z∈Z sim ( z , pj ) ; Divers ( p ) : = 1 m m∑ ȷ̂=1 min j : pj∈P sim ( pȷ̂ , pj ) ( 3 ) In contrast to the other terms , the diversity loss does not compute similarities between embeddings and prototypes but between prototypes themselves . It is another way of distributing prototypes in the embedding space as it maximizes the distance between prototypes , preventing them from staying at the same –not necessarily optimal– location . This is especially helpful in the case of multiple prototypes for a single class , encouraging them to represent different facets of the class . If they are otherwise too similar , no information is gained and resulting in redundant prototypes – together with the class-specific loss encouraged by the cluster loss ( Clst ) and the separation loss ( Sep ) , this helps to compute prototypes that focus solely on their class . Otherwise , we can get ambiguous prototypes leading to negative reasoning . Also , we clamp the weights of the classification layer with min ( wg , 0 ) to avoid negative reasoning ( Chen et al. , 2019 ) . Finally , the last term of the ProtoTrex loss ( Eq . 1 ) is an L1-regularization term of the last layer ( g ) , which prevents the network from overfitting or relying too much on a single prototype .

Summary: This paper proposes Proto-Trex model to increase the interpretability of the text classification systems. The proposed model mainly adds a bunch of prototype layers to learn the similarity between the query and prototypes. They also propose an extension caleld iProto-Trex to interactively learn from users' feedbacks. Experimental results and example cases show that the Proto-Trex could give comparable classification accuracy compared to plain classification model (w/o explanation) and could provide reasonable explanations.