Introduction
One of the recent challenges in machine learning (ML) is interpreting the predictions made by models, especially deep neural networks. Understanding models is not only beneficial, but necessary for wide-spread adoption of more complex (and potentially more accurate) ML models. From healthcare to financial domains, regulatory agencies mandate entities to provide explanations for their decisions BIBREF0 . Hence, most machine learning progress made in those areas is hindered by a lack of model explainability – causing practitioners to resort to simpler, potentially low-performance models. To supply for this demand, there has been many attempts for model interpretation in recent years for tree-based algorithms BIBREF1 and deep learning algorithms BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . On the other hand, the amount of research focusing on explainable natural language processing (NLP) models BIBREF8 , BIBREF9 , BIBREF10 is modest as opposed to image explanation techniques.
Inherent problems in data emerge in a trained model in several ways. Model explanations can show that the model is not inline with human judgment or domain expertise. A canonical example is model unfairness, which stems from biases in the training data. Fairness in ML models rightfully came under heavy scrutiny in recent years BIBREF11 , BIBREF12 , BIBREF13 . Some examples include sentiment analysis models weighing negatively for inputs containing identity terms such as “jew” and “black”, and hate speech classifiers leaning to predict any sentence containing “islam” as toxic BIBREF14 . If employed, explanation techniques help divulge these issues, but fail to offer a remedy. For instance, the sentence “I am gay” receives a high score on a toxicity model as seen in Table TABREF1 . The Integrated Gradients BIBREF4 explanation method attributes the majority of this decision to the word “gay.” However, none of the explanations methods suggest next steps to fix the issue. Instead, researchers try to reduce biases indirectly by mostly adding more data BIBREF12 , BIBREF15 , using unbiased word vectors BIBREF16 , or directly optimizing for a fairness proxy with adversarial training BIBREF17 , BIBREF11 . These methods either offer to collect more data, which is costly in many cases, or make a tradeoff between original task performance and fairness.
In this paper, we attempt to enable injecting priors through model explanations to rectify issues in trained models. We demonstrate our approach on two problems in text classification settings: (1) model biases towards protected identity groups; (2) low classification performance due to lack of data. The core idea is to add INLINEFORM0 distance between Path Integrated Gradients attributions for pre-selected tokens and a target attribution value in the objective function as a loss term. For model fairness, we impose the loss on keywords identifying protected groups with target attribution of 0, so the trained model is penalized for attributing model decisions to those keywords. Our main intuition is that undesirable correlations between toxicity labels and instances of identity terms cause the model to learn unfair biases which can be corrected by incorporating priors on these identity terms. Moreover, our approach allows practitioners to impose priors in the other direction to tackle the problem of training a classifier when there is only a small amount of data. As shown in our experiments, by setting a positive target attribution for known toxic words , one can improve the performance of a toxicity classifier in a scarce data regime.
We validate our approach on the Wikipedia toxic comments dataset BIBREF18 . Our fairness experiments show that the classifiers trained with our method achieve the same performance, if not better, on the original task, while improving AUC and fairness metrics on a synthetic, unbiased dataset. Models trained with our technique also show lower attributions to identity terms on average. Our technique produces much better word vectors as a by-product when compared to the baseline. Lastly, by setting an attribution target of 1 on toxic words, a classifier trained with our objective function achieves better performance when only a subset of the data is present.
Feature Attribution
In this section, we give formal definitions of feature attribution and a primer on [Path] Integrated Gradients (IG), which is the basis for our method.
Definition 2.1 Given a function INLINEFORM0 that represents a model, and an input INLINEFORM1 . An attribution of the prediction at input INLINEFORM2 is a vector INLINEFORM3 and INLINEFORM4 is defined as the attribution of INLINEFORM5 .
Feature attribution methods have been studied to understand the contribution of each input feature to the output prediction score. This contribution, then, can further be used to interpret model decisions. Linear models are considered to be more desirable because of their implicit interpretability, where feature attribution is the product of the feature value and the coefficient. To some, non-linear models such as gradient boosting trees and neural networks are less favorable due to the fact that they do not enjoy such transparent contribution of each feature and are harder to interpret BIBREF19 .
Despite the complexity of these models, prior work has been able to extract attributions with gradient based methods BIBREF3 , Shapley values from game theory (SHAP) BIBREF2 , or other similar methods BIBREF5 , BIBREF20 . Some of these attributions methods, for example Path Intergrated Gradients and SHAP, not only follow Definition SECREF3 , but also satisfy axioms or properties that resemble linear models. One of these axioms is completeness, which postulates that the sum of attributions should be equal to the difference between uncertainty and model output.
Integrated Gradients
Integrated Gradients BIBREF4 is a model attribution technique applicable to all models that have differentiable inputs w.r.t. outputs. IG produces feature attributions relative to an uninformative baseline. This baseline input is designed to produce a high-entropy prediction representing uncertainty. IG, then, interpolates the baseline towards the actual input, with the prediction moving from uncertainty to certainty in the process. Building on the notion that the gradient of a function, INLINEFORM0 , with respect to input can characterize sensitivity of INLINEFORM1 for each input dimension, IG simply aggregates the gradients of INLINEFORM2 with respect to the input along this path using a path integral. The crux of using path integral rather than overall gradient at the input is that INLINEFORM3 's gradients might have been saturated around the input and integrating over a path alleviates this phenomenon. Even though there can be infinitely many paths from a baseline to input point, Integrated Gradients takes the straight path between the two. We give the formal definition from the original paper in SECREF4 .
Definition 2.2 Given an input INLINEFORM0 and baseline INLINEFORM1 , the integrated gradient along the INLINEFORM2 dimension is defined as follows. DISPLAYFORM0
where INLINEFORM0 represents the gradient of INLINEFORM1 along the INLINEFORM2 dimension at INLINEFORM3 .
In the NLP setting, INLINEFORM0 is the concatenated embedding of the input sequence. The attribution of each token is the sum of the attributions of its embedding.
There are other explainability methods that attribute a model's decision to its features, but we chose IG in this framework due to several of its characteristics. First, it is both theoretically justified BIBREF4 and proven to be effective in NLP-related tasks BIBREF21 . Second, the IG formula in SECREF4 is differentiable everywhere with respect to model parameters. Lastly, it is lightweight in terms of implementation and execution complexity.
Incorporating Priors
Problems in data manifest themselves in a trained model's performance on classification or fairness metrics. Traditionally, model deficiencies were addressed by providing priors through extensive feature engineering and collecting more data. Recently, attributions help uncover deficiencies causing models to perform poorly, but do not offer actionability.
To this end, we propose to add an extra term to the objective function to penalize the INLINEFORM0 distance between model attributions on certain features and target attribution values. This modification allows model practitioners to inject priors. For example, consider a model that tends to predict every sentence containing “gay” as toxic in a comment moderation system. Penalizing non-zero attributions on the tokens identifying protected groups would force the model to focus more on the context words rather than mere existence of certain tokens.
We give the formal definition of the new objective function that incorporates priors as the follows:
Definition 3.1 Given a vector INLINEFORM0 of size INLINEFORM1 , where INLINEFORM2 is the length of the input sequence and INLINEFORM3 is the attribution target value for the INLINEFORM4 th token in the input sequence. The prior loss for a scalar output is defined as: DISPLAYFORM0
where INLINEFORM0 refers to attribution of the INLINEFORM1 th token as in Definition SECREF3 .
For a multi-class problem, we train our model with the following joint objective, DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the attribution and attribution target for class INLINEFORM2 , INLINEFORM3 is the hyperparameter that controls the stength of the prior loss and INLINEFORM4 is the cross-entropy loss defined as follows: DISPLAYFORM0
where INLINEFORM0 is an indicator vector of the ground truth label and INLINEFORM1 is the posterior probability of class INLINEFORM2 .
The joint objective function is differentiable w.r.t. model parameters when attribution is calculated through Equation EQREF5 and can be trained with most off-the-shelf optimizers. The proposed objective is not dataset-dependent and is applicable to different problem settings such as sentiment classification, abuse detection, etc. It only requires users to specify the target attribution value for tokens of interest in the corpus. We illustrate the effectiveness of our method by applying it to a toxic comment classification problem. In the next section, we first show how we set the target attribution value for identity terms to remove unintended biases while retaining the same performance on the original task. Then, using the same technique, we show how to set target attribution for toxic words to improve classifier performance in a scarce data setting.
Experiments
experiments
Discussion and Related Work
relatedwork
Conclusion and Future Work
In this paper, we proposed actionability on model explanations that enable ML practitioners to enforce priors on their model. We apply this technique to model fairness in toxic comment classification. Our method incorporates Path Integrated Gradients attributions into the objective function with the aim of stopping the classifier from carrying along false positive bias from the data by punishing it when it focuses on identity words.
Our experiments indicate that the models trained jointly with cross-entropy and prior loss do not suffer a performance drop on the original task, while achieving a better performance in fairness metrics on the template-based dataset. Applying model attribution as a fine-tuning step on a trained classifier makes it converge to a more debiased classifier in just a few epochs. Additionally, we show that model can be also forced to focus on pre-determined tokens.
There are several avenues we can explore as future research. Our technique can be applied to implement a more robust model by penalizing the attributions falling outside of tokens annotated to be relevant to the predicted class. Another avenue is to incorporate different model attribution strategies such as DeepLRP BIBREF5 into the objective function. Finally, it would be worthwhile to invest in a technique to extract problematic terms from the model automatically rather than providing prescribed identity or toxic terms.
Acknowledgments
We thank Salem Haykal, Ankur Taly, Diego Garcia-Olano, Raz Mathias, and Mukund Sundararajan for their valuable feedback and insightful discussions.

Question: Which datasets do they use?
Answer:  Wikipedia toxic comments