Interactively Generating Explanations for Transformer Language Models

1 INTRODUCTION . Transformer language models ( LMs ) are ubiquitous in NLP today but also notoriously opaque . Therefore , it is not surprising that a growing body of work aims to interpret them : Recent evaluations of approaches of saliency methods ( Ding & Koehn , 2021 ) and instance attribution methods Pezeshkpour et al . ( 2021 ) find that , while intriguing , for the same outputs , different methods assign importance to different inputs . Furthermore , they are usually employed post hoc , thus possibly encouraging reporting bias ( Gordon & Van Durme , 2013 ) . The black-box character of LMs becomes especially problematic , as the data to train them might be unfiltered and contain ( human ) bias . As a result , ethical concerns about these models arise , which can have a substantial negative impact on society as they get increasingly integrated into our lives ( Bender et al. , 2021 ) . Here , we focus on providing case-based reasoning explanations during the inference process , directly outputting the LM ’ s predictions . In doing so , we avoid the problems mentioned above of post hoc explanations and help to reduce the issue of ( human ) bias . To increase the interpretability of the model , we enhance the transformer architecture with a prototype layer and propose PrototypicalTransformer Explanation ( Proto-Trex ) Networks . Proto-Trex networks provide an explanation as a prototypical example for a specific model prediction , which is similar to ( training- ) samples with the corresponding label . Our experimental results demonstrate that Proto-Trex networks perform on par with noninterpretable baselines , e.g . BERT ( Devlin et al. , 2019 ) and GPT ( Radford et al. , 2019 ) . In terms of explanations , we show promising results with learned prototypes providing helpful explanations for the user to understand better the LMs decision-making , which , in turn , increases trust and reliability . To further enhance the prototypical network with human supervision , we propose an interactive learning setting ( iProto-Trex ) that allows users of any knowledge to give feedback and improve the model ( Ribeiro et al. , 2016 ) , moving beyond a purely data-driven approach . To summarize , our contributions are as follows : We ( i ) introduce prototype networks for transformer LMs that provide explanations and ( ii ) show that they are on par with non-interpretable baselines on classification tasks on different architectures . Furthermore , to improve prototype networks , we ( iii ) provide a novel interactive prototype learning setup accounting for user feedback certainty . We proceed as follows . We start by briefly reviewing related work of interpretability in NLP . Then we introduce Proto-Trex networks , including our novel interactive learning setup combining explanatory interactive learning with prototype networks . Before concluding , we discuss faithful explanations and touch upon the results of our experimental evaluation . 2 TOWARDS THE EXPLAINABILITY OF TRANSFORMER LANGUAGE MODELS . To open the black-box of transformers , we use i/Proto-Trex networks built upon post hoc interpretation methods , case-based reasoning approaches , and explanatory interactive learning . Post hoc interpretability . Various ( post hoc ) interpretability methods focus on different parts of the transformer architecture . Atanasova et al . ( 2020 ) and Belinkov & Glass ( 2019 ) provide overviews of this fast developing field . Generally , there are methods that analyze word representations ( Voita et al. , 2019a ) , the attention distribution throughout the model ( Jain & Wallace , 2019 ; Wiegreffe & Pinter , 2019 ) , and the ( attention and classification ) model heads ( Voita et al. , 2019b ; Geva et al. , 2021 ) . Other approaches such as Geva et al . ( 2020 ) focus on the feed-forward layers . Gradientbased approaches , such as Sundararajan et al . ( 2017 ) and Smilkov et al . ( 2017 ) can generally be used to trace gradients , while influence functions ( Koh & Liang , 2017 ) trace model parameter changes throughout a LM ( Han et al. , 2020 ) . While backtracking all the model weights might be possible , such explanations can only tell us which part of the input they are looking at ( Chen et al. , 2019 ) and humans are ill-equipped to interpret them ( Stammer et al. , 2021 ) . So instead , we aim for more intuitive and sparse explanations : well-descriptive but short sequences as prototypes . These are not only able to reveal the parts of the input it is looking at but also show prototypical cases similar to those parts . Case-based Reasoning in Deep Neural Networks . Our work relates most closely to previous work demonstrating the benefits of prototype networks in Computer Vision ( Li et al. , 2018 ; Chen et al. , 2019 ) as well as for sequential data by combining them with RNNs ( Hase et al. , 2019 ; Ming et al. , 2019 ) . More precisely , in contrast to post hoc interpretation methods , prototype networks employ explanations in the , e.g . classification process , i.e . classify a sample by comparing several parts – which could be words of a sentence– to ( learned ) prototypical parts from other samples for a given class . If things look similar , they are classified similarly . Humans behave very similarly , which is called case-based reasoning ( Clifton & Frohnsdorff , 2001 ) . The prototypical parts provided by the network help the user to understand the classification . Tab . 1 illustrates the benefit of prototypical explanations . One can observe that the post hoc explanation can leave the user clueless , while the prototypical explanations can help better understand the network ’ s decision . Accordingly , prototypes are an additional method in the interpretability toolbox , extending current post hoc methods . As described , the reasoning process of prototypical networks is qualitatively similar to that of humans ( Clifton & Frohnsdorff , 2001 ; Chen et al. , 2019 ) . In previous approaches , however , the training of such networks was entirely data-driven . Therefore , like Chen et al . ( 2019 ) , in our work , we do not focus on quantifying unit interpretability of prototypical networks , but extend learning the reasoning process of our network by Explanatory Interactive Learning ( Schramowski et al. , 2020 ) . Explanatory Interactive Learning ( XIL ) incorporates explanatory supervision in the learning process by involving the human user . More precisely , human users can ask the model to explain a prediction . They can then respond by correcting the model if necessary , providing a slightly improved –but not necessarily optimal– feedback on the explanations ( Stammer et al. , 2021 ) . Interactive learning and , in particular , explanatory interactive learning showed not only to increase the explanation quality of deep black-box models but also the trust of the user ( Ross et al. , 2017 ; Selvaraju et al. , 2019 ; Teso & Kersting , 2019 ; Schramowski et al. , 2020 ) . In contrast to previous ( post hoc ) XIL methods , we avoid tracing gradients . That makes our approach particularly efficient , as we only change prototypes ( cf . Fig 4 ) . 3 PROTO-TREX : PROTOTYPE LEARNING FOR TRANSFORMER LMS . The prototype network ( Proto-Trex ) architecture builds upon large-scale transformer LMs , summarized in Fig . 2 . In the following , we describe the general idea of prototypical networks and our transformer-based prototype architecture and its modules in more detail . 3.1 PROTO-TREX NETWORKS . General Idea . Proto-Trex is a prototype network , where prototypes are representative observations in the training set . The classification is based on the neighboring prototypes , i.e . an input query is classified as positive when the network thinks it looks similar to a certain ( near ) positive prototype within the training set . In more detail , the reasoning and classification processes use the same module , i.e . , they use the similarity between query and prototypes . As both reasoning and classification use the same module ( in parallel ) , we consequently have an interpretable model . In particular , ( interpretable ) prototypical explanations help verify Right for the Right Reasons ( Ross et al. , 2017 ) . Architecture . Specifically , the model , in our case the pre-trained transformer f , creates a context embedding e from the input sequence x . This embedding contains useful features for the prediction and is then passed on to the prototype layer . This layer computes the similarity between the embedding e and each of the trainable prototypes pj from the set of prototypes P . The prototypes are learned during the training process and represent prototypical ( sub- ) sequences in the training data . We use transformer models both on the sentence-level ( a ) and the word-level ( b ) , resulting in a single input representation for the sentence-level transformer and l for word-level . To compare the word-level embeddings ei with the prototypes , we have to patch them ( Z = { patch ( ei ) } ) into subsequences according to the desired prototype length k. The resulting similarity values are passed on to the final linear layer g with weight matrix wg and no bias , mapping from m ( the number of prototypes ) to the number of classes . Finally , the output values are passed on to a softmax producing class probabilities and predictions t . 3.2 PROTO-TREX LOSS . Optimization of prototype networks aims at maximizing both performance and interpretability . To this end , our Proto-Trex loss L combines performance and interpretability with prior knowledge of the prototype network ’ s structure . Let us illustrate this for the word-level case ( the sentence-level case results from e=z ) : L : = min P , wg 1 n n∑ i=1 CE ( ti , yi ) + λ1Clst ( z , p ) + λ2Sep ( z , p ) + λ3Distr ( z , p ) + λ4Divers ( p ) + λ5||wg|| ( 1 ) where λi weights the influence of the different terms and n is the number of training examples . The first term is the cross-entropy ( CE ) loss optimizing the predictive power of the network . The second term ( Clst ) clusters the prototypes w.r.t . the training examples of the same class , maximizing similarity to them ( we rewrite maximization as minimization terms ) , and the separation loss minimizes the similarity to other-class instances : Clst ( z , p ) : = − 1 n n∑ i=1 min j : pj∈Pyi min z∈Z sim ( z , pj ) ; Sep ( z , p ) : = 1 n n∑ i=1 min j : pj /∈Pyi min z∈Z sim ( z , pj ) ( 2 ) Together , Clst and Sep push each prototype to focus more on training examples from the same class and less on training examples from other classes . Both are motivated by ProtoPNet ( Chen et al. , 2019 ) . To get prototypes that are distributed well in the embedding space , we introduce two additional losses , a distribution loss ( Li et al. , 2018 ) , assuring that a prototype is nearby each training example , and a diversity loss ( Ming et al. , 2019 ) : Distr ( z , p ) : = − 1 m m∑ j=1 min i∈ [ 1 , n ] min z∈Z sim ( z , pj ) ; Divers ( p ) : = 1 m m∑ ȷ̂=1 min j : pj∈P sim ( pȷ̂ , pj ) ( 3 ) In contrast to the other terms , the diversity loss does not compute similarities between embeddings and prototypes but between prototypes themselves . It is another way of distributing prototypes in the embedding space as it maximizes the distance between prototypes , preventing them from staying at the same –not necessarily optimal– location . This is especially helpful in the case of multiple prototypes for a single class , encouraging them to represent different facets of the class . If they are otherwise too similar , no information is gained and resulting in redundant prototypes – together with the class-specific loss encouraged by the cluster loss ( Clst ) and the separation loss ( Sep ) , this helps to compute prototypes that focus solely on their class . Otherwise , we can get ambiguous prototypes leading to negative reasoning . Also , we clamp the weights of the classification layer with min ( wg , 0 ) to avoid negative reasoning ( Chen et al. , 2019 ) . Finally , the last term of the ProtoTrex loss ( Eq . 1 ) is an L1-regularization term of the last layer ( g ) , which prevents the network from overfitting or relying too much on a single prototype .

Summary: This paper introduces a method for improving interpretability of black box transformer based text classifiers. The approach is based on “case based reasoning” where the network classifies an input by comparing it against a library of learnt prototypes (each with a corresponding label) and classifying the input based on a weighted similarity with the prototypes. Thus, in a sense it is more interpretable than an end-to-end classifier since a user can directly look at the similarity scores with the prototypes to understand the labeling decision. The approach also promises to be more “faithful” than post-hoc interpretability methods since the predictions are based directly on similarity scores. The paper also shows that end-users of the system can interact with it by either editing prototypes (if they are experts) or providing weak feedback about which prototypes are good, and this feedback can be integrated to re-learn better prototypes. From experiments we see that their approach is competitive with standard end-to-end finetuning while being more interpretable. Based on sufficiency and comprehensiveness scores, we see some evidence that the explanations from prototypes are faithful, though it is unclear if these scores are competitive.