Feature Attribution	In this section, we give formal definitions of feature attribution and a primer on [Path] Integrated Gradients (IG), which is the basis for our method. Definition 2.1 Given a function INLINEFORM0 that represents a model, and an input INLINEFORM1 . An attribution of the prediction at input INLINEFORM2 is a vector INLINEFORM3 and INLINEFORM4 is defined as the attribution of INLINEFORM5 . Feature attribution methods have been studied to understand the contribution of each input feature to the output prediction score. This contribution, then, can further be used to interpret model decisions. Linear models are considered to be more desirable because of their implicit interpretability, where feature attribution is the product of the feature value and the coefficient. To some, non-linear models such as gradient boosting trees and neural networks are less favorable due to the fact that they do not enjoy such transparent contribution of each feature and are harder to interpret  . Despite the complexity of these models, prior work has been able to extract attributions with gradient based methods  , Shapley values from game theory (SHAP)  , or other similar methods  ,  .

The core idea is to add INLINEFORM0 distance between Path Integrated Gradients attributions for pre-selected tokens and a target attribution value in the objective function as a loss term. For model fairness, we impose the loss on keywords identifying protected groups with target attribution of 0, so the trained model is penalized for attributing model decisions to those keywords. Our main intuition is that undesirable correlations between toxicity labels and instances of identity terms cause the model to learn unfair biases which can be corrected by incorporating priors on these identity terms. Moreover, our approach allows practitioners to impose priors in the other direction to tackle the problem of training a classifier when there is only a small amount of data. As shown in our experiments, by setting a positive target attribution for known toxic words , one can improve the performance of a toxicity classifier in a scarce data regime. We validate our approach on the Wikipedia toxic comments dataset  . Our fairness experiments show that the classifiers trained with our method achieve the same performance, if not better, on the original task, while improving AUC and fairness metrics on a synthetic, unbiased dataset. Models trained with our technique also show lower attributions to identity terms on average.

Conclusion and Future Work	In this paper, we proposed actionability on model explanations that enable ML practitioners to enforce priors on their model. We apply this technique to model fairness in toxic comment classification. Our method incorporates Path Integrated Gradients attributions into the objective function with the aim of stopping the classifier from carrying along false positive bias from the data by punishing it when it focuses on identity words. Our experiments indicate that the models trained jointly with cross-entropy and prior loss do not suffer a performance drop on the original task, while achieving a better performance in fairness metrics on the template-based dataset. Applying model attribution as a fine-tuning step on a trained classifier makes it converge to a more debiased classifier in just a few epochs. Additionally, we show that model can be also forced to focus on pre-determined tokens. There are several avenues we can explore as future research. Our technique can be applied to implement a more robust model by penalizing the attributions falling outside of tokens annotated to be relevant to the predicted class. Another avenue is to incorporate different model attribution strategies such as DeepLRP  into the objective function. Finally, it would be worthwhile to invest in a technique to extract problematic terms from the model automatically rather than providing prescribed identity or toxic terms.

Some examples include sentiment analysis models weighing negatively for inputs containing identity terms such as “jew” and “black”, and hate speech classifiers leaning to predict any sentence containing “islam” as toxic  . If employed, explanation techniques help divulge these issues, but fail to offer a remedy. For instance, the sentence “I am gay” receives a high score on a toxicity model as seen in Table TABREF1 . The Integrated Gradients  explanation method attributes the majority of this decision to the word “gay.” However, none of the explanations methods suggest next steps to fix the issue. Instead, researchers try to reduce biases indirectly by mostly adding more data  ,  , using unbiased word vectors  , or directly optimizing for a fairness proxy with adversarial training  ,  . These methods either offer to collect more data, which is costly in many cases, or make a tradeoff between original task performance and fairness. In this paper, we attempt to enable injecting priors through model explanations to rectify issues in trained models. We demonstrate our approach on two problems in text classification settings: (1) model biases towards protected identity groups; (2) low classification performance due to lack of data.

In the next section, we first show how we set the target attribution value for identity terms to remove unintended biases while retaining the same performance on the original task. Then, using the same technique, we show how to set target attribution for toxic words to improve classifier performance in a scarce data setting.

Integrated Gradients	Integrated Gradients  is a model attribution technique applicable to all models that have differentiable inputs w.r.t. outputs. IG produces feature attributions relative to an uninformative baseline. This baseline input is designed to produce a high-entropy prediction representing uncertainty. IG, then, interpolates the baseline towards the actual input, with the prediction moving from uncertainty to certainty in the process. Building on the notion that the gradient of a function, INLINEFORM0 , with respect to input can characterize sensitivity of INLINEFORM1 for each input dimension, IG simply aggregates the gradients of INLINEFORM2 with respect to the input along this path using a path integral. The crux of using path integral rather than overall gradient at the input is that INLINEFORM3 's gradients might have been saturated around the input and integrating over a path alleviates this phenomenon. Even though there can be infinitely many paths from a baseline to input point, Integrated Gradients takes the straight path between the two. We give the formal definition from the original paper in SECREF4 . Definition 2.2 Given an input INLINEFORM0 and baseline INLINEFORM1 , the integrated gradient along the INLINEFORM2 dimension is defined as follows.

Some of these attributions methods, for example Path Intergrated Gradients and SHAP, not only follow Definition SECREF3 , but also satisfy axioms or properties that resemble linear models. One of these axioms is completeness, which postulates that the sum of attributions should be equal to the difference between uncertainty and model output.

The prior loss for a scalar output is defined as: DISPLAYFORM0  where INLINEFORM0 refers to attribution of the INLINEFORM1 th token as in Definition SECREF3 . For a multi-class problem, we train our model with the following joint objective, DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are the attribution and attribution target for class INLINEFORM2 , INLINEFORM3 is the hyperparameter that controls the stength of the prior loss and INLINEFORM4 is the cross-entropy loss defined as follows: DISPLAYFORM0  where INLINEFORM0 is an indicator vector of the ground truth label and INLINEFORM1 is the posterior probability of class INLINEFORM2 . The joint objective function is differentiable w.r.t. model parameters when attribution is calculated through Equation EQREF5 and can be trained with most off-the-shelf optimizers. The proposed objective is not dataset-dependent and is applicable to different problem settings such as sentiment classification, abuse detection, etc. It only requires users to specify the target attribution value for tokens of interest in the corpus. We illustrate the effectiveness of our method by applying it to a toxic comment classification problem.

Incorporating Priors	Problems in data manifest themselves in a trained model's performance on classification or fairness metrics. Traditionally, model deficiencies were addressed by providing priors through extensive feature engineering and collecting more data. Recently, attributions help uncover deficiencies causing models to perform poorly, but do not offer actionability. To this end, we propose to add an extra term to the objective function to penalize the INLINEFORM0 distance between model attributions on certain features and target attribution values. This modification allows model practitioners to inject priors. For example, consider a model that tends to predict every sentence containing “gay” as toxic in a comment moderation system. Penalizing non-zero attributions on the tokens identifying protected groups would force the model to focus more on the context words rather than mere existence of certain tokens. We give the formal definition of the new objective function that incorporates priors as the follows: Definition 3.1 Given a vector INLINEFORM0 of size INLINEFORM1 , where INLINEFORM2 is the length of the input sequence and INLINEFORM3 is the attribution target value for the INLINEFORM4 th token in the input sequence.

Introduction	One of the recent challenges in machine learning (ML) is interpreting the predictions made by models, especially deep neural networks. Understanding models is not only beneficial, but necessary for wide-spread adoption of more complex (and potentially more accurate) ML models. From healthcare to financial domains, regulatory agencies mandate entities to provide explanations for their decisions  . Hence, most machine learning progress made in those areas is hindered by a lack of model explainability – causing practitioners to resort to simpler, potentially low-performance models. To supply for this demand, there has been many attempts for model interpretation in recent years for tree-based algorithms  and deep learning algorithms  ,  ,  ,  ,  ,  . On the other hand, the amount of research focusing on explainable natural language processing (NLP) models  ,  ,  is modest as opposed to image explanation techniques. Inherent problems in data emerge in a trained model in several ways. Model explanations can show that the model is not inline with human judgment or domain expertise. A canonical example is model unfairness, which stems from biases in the training data. Fairness in ML models rightfully came under heavy scrutiny in recent years  ,  ,  .

Acknowledgments	We thank Salem Haykal, Ankur Taly, Diego Garcia-Olano, Raz Mathias, and Mukund Sundararajan for their valuable feedback and insightful discussions.

Discussion and Related Work	relatedwork

Experiments	experiments

DISPLAYFORM0  where INLINEFORM0 represents the gradient of INLINEFORM1 along the INLINEFORM2 dimension at INLINEFORM3 . In the NLP setting, INLINEFORM0 is the concatenated embedding of the input sequence. The attribution of each token is the sum of the attributions of its embedding. There are other explainability methods that attribute a model's decision to its features, but we chose IG in this framework due to several of its characteristics. First, it is both theoretically justified  and proven to be effective in NLP-related tasks  . Second, the IG formula in SECREF4 is differentiable everywhere with respect to model parameters. Lastly, it is lightweight in terms of implementation and execution complexity.

Our technique produces much better word vectors as a by-product when compared to the baseline. Lastly, by setting an attribution target of 1 on toxic words, a classifier trained with our objective function achieves better performance when only a subset of the data is present.

Incorporating Priors with Feature Attribution on Text Classification	Feature attribution methods, proposed recently, help users interpret the predictions of complex models. Our approach integrates feature attributions into the objective function to allow machine learning practitioners to incorporate priors in model building. To demonstrate the effectiveness our technique, we apply it to two tasks: (1) mitigating unintended bias in text classifiers by neutralizing identity terms; (2) improving classifier performance in a scarce data setting by forcing the model to focus on toxic terms. Our approach adds an L2 distance loss between feature attributions and task-specific prior values to the objective. Our experiments show that i) a classifier trained with our technique reduces undesired model biases without a trade off on the original task; ii) incorporating priors helps model performance in scarce data settings.

Question: Which datasets do they use?
Answer: Wikipedia toxic comments