Interpretations are useful: penalizing explanations to align neural networks with prior knowledge

For an explanation of a deep learning model to be effective , it must provide both insight into a model and suggest a corresponding action in order to achieve some objective . Too often , the litany of proposed explainable deep learning methods stop at the first step , providing practitioners with insight into a model , but no way to act on it . In this paper , we propose contextual decomposition explanation penalization ( CDEP ) , a method which enables practitioners to leverage existing explanation methods in order to increase the predictive accuracy of deep learning models . In particular , when shown that a model has incorrectly assigned importance to some features , CDEP enables practitioners to correct these errors by directly regularizing the provided explanations . Using explanations provided by contextual decomposition ( CD ) ( Murdoch et al. , 2018 ) , we demonstrate the ability of our method to increase performance on an array of toy and real datasets . 1 INTRODUCTION . In recent years , neural networks have demonstrated strong predictive performance across a wide variety of settings . However , in order to achieve that accuracy , they sometimes latch onto spurious correlations , leading to undesirable behavior as a result of dataset bias ( Winkler et al. , 2019 ) , racial and ethnic stereotypes ( Garg et al. , 2018 ) , or simply overfitting . While recent work into explaining neural network predictions ( Murdoch et al. , 2019 ; Doshi-Velez & Kim , 2017 ) has demonstrated an ability to uncover the relationships learned by a model , it is still unclear how to actually alter the model in order to remove incorrect , or undesirable , relationships . We introduce contextual decomposition explanation penalization ( CDEP ) , a method which leverages existing explanation techniques for neural networks in order to prevent a model from learning unwanted relationships and ultimately improve predictive accuracy . Given particular importance scores , CDEP works by allowing the user to directly penalize importances of certain features , or interactions . This forces the neural network to not only produce the correct prediction , but also the correct explanation for that prediction . While we focus on contextual decomposition ( CD ) ( Murdoch et al. , 2018 ; Singh et al. , 2018 ) , which allows the penalization of both feature importances and interactions , CDEP can be readily adapted for existing interpretation techniques , as long as they are differentiable . Moreover , CDEP is a general technique , which can be applied to arbitrary neural network architectures , and is orders of magnitude faster and more memory efficient than recent gradient-based methods , allowing its use on meaningful datasets . In order to demonstrate the effectiveness of CDEP , we conducted experiments across a wide variety of tasks . In the prediction of skin cancer from images , CDEP improves the prediction of a classifier by teaching it to ignore spurious confounding variables present in the training data . In a colored MNIST task , CDEP allows the network to focus on a digit ’ s shape rather than its color ( with no extra human annotation needed ) . Finally , a toy example using text classification shows how the penalization can help a network avoid a bias towards particular words , such as those involving gender . 2 BACKGROUND . Explanation methods Many methods have been developed to help explain the learned relationships contained in a DNN . For local , or prediction-level , explanation , most prior work has focused on assigning importance to individual features , such as pixels in an image or words in a document . There are several methods that give feature-level importance for different architectures . They can be categorized as gradient-based ( Springenberg et al. , 2014 ; Sundararajan et al. , 2017 ; Selvaraju et al. , 2016 ; Baehrens et al. , 2010 ; Rieger & Hansen , 2019 ) , decomposition-based ( Murdoch & Szlam , 2017 ; Shrikumar et al. , 2016 ; Bach et al. , 2015 ) and others ( Dabkowski & Gal , 2017 ; Fong & Vedaldi , 2017 ; Ribeiro et al. , 2016 ; Zintgraf et al. , 2017 ) , with many similarities among the methods ( Ancona et al. , 2018 ; Lundberg & Lee , 2017 ) . However , many of these methods have thus far been poorly evaluated ( Adebayo et al. , 2018 ; Nie et al. , 2018 ) , casting doubt on their usefulness . Another line of work , which we build upon , has focused on uncovering interactions between features , in addition to feature importances , ( Murdoch et al. , 2018 ) , and using those interactions to create a hierarchy of features displaying the model ’ s prediction process ( Singh et al. , 2018 ) . Uses of explanation methods While much work has been put into developing methods for explaining DNNs , relatively little work has explored the potential to use these explanations to help build a better model . Some recent work proposes forcing models to attend to regions of the input which are known to be important ( Burns et al. , 2018 ; Mitsuhara et al. , 2019 ) , although it is important to note that attention is often not the same as explanation ( Jain & Wallace , 2019 ) . An alternative line of work proposes penalizing the gradients of a neural network to match human-provided binary annotations and shows the possibility to improve performance ( Ross et al. , 2017 ) and adversarial robustness ( Ross & Doshi-Velez , 2018 ) . Two recent papers extend these ideas by penalizing attributions for natural language models ( Liu & Avci , 2019 ) and penalizing a modified gradient-based score to produce smooth attributions ( Erion et al. , 2019 ) . Predating deep learning , ( Zaidan et al. , 2007 ) consider the use of “ annotator rationales ” in sentiment analysis to train support vector machines . This work on annotator rationales was recently extended to show improved explanations ( not accuracy ) in CNNs ( Strout et al. , 2019 ) . Other ways to constrain DNNs While we focus on the use of explanations to constrain the relationships learned by neural networks , other approaches for constraining neural networks have also been proposed . A computationally intensive alternative is to augment the dataset in order to prevent the model from learning undesirable relationships , through domain knowledge ( Bolukbasi et al. , 2016 ) , projecting out superficial statistics ( Wang et al. , 2019 ) or dramatically altering training images ( Geirhos et al. , 2018 ) . However , these processes are often not feasible , either due to their computational cost or the difficulty of constructing such an augmented data set . Adversarial training has also been explored ( Zhang & Zhu , 2019 ) . These techniques are generally limited , as they are often tied to particular datasets , and do not provide a clear link between learning about a model ’ s learned relationships through explanations , and subsequently correcting them . 3 METHODS . We now introduce CDEP , which penalizes the explanations of a neural network in order to align with prior knowledge about why a model should make a prediction . To do so , for each data point it penalizes the CD scores of features , or groups of features , which a user does not want the model to learn to be important . While we focus on CD scores , which allow the penalization of interactions between features in addition to features themselves , this approach readily generalizes to other interpretation techniques , so long as they are differentiable . 3.1 AUGMENTING THE LOSS FUNCTION . Given a particular classification task , we want to teach a model to not only produce the correct prediction , but also to arrive at the prediction for the correct reasons . That is , we want the model to be right for the right reasons , where the right reasons are provided by the user and are datasetdependent . To accomplish this , CDEP modifies the objective function used to train a neural network , as displayed in Eq 1 . In addition to the standard prediction loss L , which teaches the model to produce the correct predictions , CDEP adds an explanation error Lexpl , which teaches the model to produce the correct explanations for its predictions . In place of the prediction and labels fθ ( X ) , y , used in the prediction error L , the explanation error Lexpl uses the explanations produced by an interpretation method explθ ( X ) , along with targets provided by the user explX . As is common with penalization , the two losses are weighted by a hyperparameter λ ∈ R : θ̂ = argmin θ L ( fθ ( X ) , y ) ︸ ︷︷ ︸ Prediction error +λLexpl ( explθ ( X ) , explX ) ︸ ︷︷ ︸ Explanation error ( 1 ) The precise meanings of explX depend on the context . For example , in the skin cancer image classification task described in Section 4 , many of the benign skin images contain band-aids , but none of the malignant images . To force the model to ignore the band-aids in making their prediction , in each image explθ ( X ) denotes the importance score of the band-aid and explX would be zero . These and more examples are further explored in Section 4 . 3.2 CONTEXTUAL DECOMPOSITION ( CD ) . In this work , we use the CD score as the explanation function . In contrast to other interpretation methods , which focus on feature importances , CD also captures interactions between features . CD was originally designed for LSTMs ( Murdoch et al. , 2018 ) and subsequently extended to convolutional neural networks and arbitrary DNNs ( Singh et al. , 2018 ) . For a given DNN f ( x ) , one can represent its output as a SoftMax operation applied to logits g ( x ) . These logits , in turn , are the composition of L layers gi , such as convolutional operations or ReLU non-linearities . f ( x ) = SoftMax ( g ( x ) ) = SoftMax ( gL ( gL−1 ( ... ( g2 ( g1 ( x ) ) ) ) ) ) ( 2 ) Given a group of features { xj } j∈S , the CD algorithm , gCD ( x ) , decomposes the logits g ( x ) into a sum of two terms , β ( x ) and γ ( x ) . β ( x ) is the importance score of the feature group { xj } j∈S , and γ ( x ) captures contributions to g ( x ) not included in β ( x ) . The decomposition is computed by iteratively applying decompositions gCDi ( x ) for each of the layers gi ( x ) . gCD ( x ) = gCDL ( g CD L−1 ( ... ( g CD 2 ( g CD 1 ( x ) ) ) ) ) ) = ( β ( x ) , γ ( x ) ) ( 3 ) β ( x ) + γ ( x ) = g ( x ) ( 4 ) 3.3 CDEP OBJECTIVE FUNCTION . We now substitute the above CD scores into the generic equation in Eq 1 to arrive at the method used in this paper . While we use CD for the explanation method explθ ( X ) , other explanation methods could be readily substituted at this stage . In order to convert CD scores to probabilities , we apply a SoftMax operation to gCD ( x ) , allowing for easier comparison with the user-provided labels explX . We collect from the user , for each input xi , a collection of feature groups xi , S , xi ∈ Rd , S ⊆ { 1 , ... , d } , along with explanation target values explxi , S , and use the ‖ · ‖1 loss for Lexpl . θ̂ = argmin θ ∑ i ∑ c − yi , c log fθ ( xi ) c︸ ︷︷ ︸ Classification error +λ ∑ i ∑ S ||β ( xi , S ) − explxi , S ||1︸ ︷︷ ︸ Explanation error ( 5 ) In the above , i indexes each individual example in the dataset , S indexes a subset of the features for which we penalize their explanations , and c sums over each class . Updating the model parameters in accordance with this formulation ensures that the model not only predicts the right output but also does so for the right ( aligned with prior knowledge ) reasons .

Summary: The paper presents a way of using generated explanations of model predictions to help prevent a model from learning "unwanted" relationships between features and class labels. This idea was implemented with a particular explanation generation method from prior work, called contextual decomposition (CD). For a given feature, the corresponding CD can be used to measure its importance. The proposed learning objective in this work optimizes not only the cross entropy loss, but also the difference between the CD score of a given feature and its explanation target value. Experiments show that this new learning algorithm can largely improve the classification performance.