SlowGuess/NAACL_2022_OCR huggingface dataset:
Esma Balkir, Isar Nejadgholi, Kathleen C. Fraser, and Svetlana Kiritchenko
{Esma.Balkir, Isar.Nejadgholi, Kathleen.Fraser, Svetlana.Kiritchenko}@nrc-cnrc.gc.ca
We present a novel feature attribution method for explaining text classifiers, and analyze it in the context of hate speech detection. Although feature attribution models usually provide a single importance score for each token, we instead provide two complementary and theoretically-grounded scores - necessity and sufficiency - resulting in more informative explanations. We propose a transparent method that calculates these values by generating explicit perturbations of the input text, allowing the importance scores themselves to be explainable. We employ our method to explain the predictions of different hate speech detection models on the same set of curated examples from a test suite, and show that different values of necessity and sufficiency for identity terms correspond to different kinds of false positive errors, exposing sources of classifier bias against marginalized groups.
Explainability in AI (XAI) is critical in reaching various objectives during a system's development and deployment, including debugging the system, ensuring its fairness, safety and security, and understanding and appealing its decisions by end-users (Vaughan and Wallach, 2021; Luo et al., 2021).
A popular class of local explanation techniques is feature attribution methods, where the aim is to provide scores for each feature according to how important that feature is for the classifier decision for a given input. From an intuitive perspective, one issue with feature attribution scores is that it is not always clear how to interpret the assigned importance in operational terms. Specifically, saying that a feature is 'important' might translate to two different predictions. The first interpretation is that if an important feature value is changed, then the prediction will change. The second interpretation is that, as long as the feature remains, the prediction
will not change. The former interpretation corresponds to the necessity of the feature value, while the latter corresponds to its sufficiency.
To further illustrate the difference between necessity and sufficiency, we take an example from hate speech detection. Consider the utterance "I hate women". For a perfect model, the token 'women' should have low sufficiency for a positive prediction, since merely the mention of this identity group should not trigger a hateful prediction. However, this token should have fairly high necessity, since a target identity is required for an abusive utterance to count as hate speech (e.g., "I hate oranges" should not be classified as hate speech). In this paper, we develop a method to estimate the necessity and sufficiency of each word in the input, as explanations for a binary text classifier's decisions.
Model-agnostic feature attribution methods like ours often perturb the input to be explained, obtain the predictions of the model for the perturbed instances, and aggregate the results to make conclusions about which input features are more influential on the model decision. When applying these methods to textual data, it is common to either drop the chosen tokens, or replace them with the mask token for those models that have been trained by fine-tuning a masked language model such as BERT (Devlin et al., 2019). However, deleting tokens raises the possibility that a large portion of the perturbed examples are not fluent, and lie well outside the data manifold. Replacing some tokens with the mask token partially remedies this issue, however it raises others. Firstly, the explanation method ceases to be truly model-agnostic. Secondly, a masked sentence is in-distribution for the pre-trained model but out-of-distribution for the fine-tuned model, because the learned manifolds deviate from those formed during pre-training in the fine-tuning step.
To avoid these problems we use a generative model to replace tokens with most probable  $n$ -
Human Language Technologies, pages 2672 - 2686
grams. Generating perturbations in this way ensures that the perturbed instances are close to the true data manifold. It also provides an additional layer of transparency to the user, so they can decide whether to trust the explanation by checking how reasonable the perturbed examples seem.
Although supervised discriminative models rely fundamentally on correlations within the dataset, different models might rely on different correlations more or less depending on model architecture and biases, training methods, and other idiosyncrasies. To capture the distinction between correlations in the data and the direct causes of the prediction, we turn to the notion of interventions from causal inference (Pearl, 2009). Previous work employing causal definitions of necessity and sufficiency for XAI have assumed tabular data with binary or numerical features. The situation in NLP is much more complex, since each feature is a word in context, and we have no concept of 'flipping' or 'increasing' feature values (as in binary data and numerical data, respectively). Instead, our method generates perturbations of the input text that have high probability of being fluent while minimizing the probability that the generated text will also be a direct cause of the prediction we aim to explain.
As our application domain we choose hate speech detection, a prominent NLP task with significant social outcomes (Fortuna and Nunes, 2018; Kiritchenko et al., 2021). It has been shown that contemporary hate speech classifiers tend to learn spurious correlations, including those between identity terms and the positive (hate) class, which can result in further discrimination of already marginalized groups (Dixon et al., 2018; Park et al., 2018; Garg et al., 2019). We apply our explainability metrics to test classifiers' fairness towards identity-based groups (e.g., women, Muslims). We show how necessity and sufficiency metrics calculated for identity terms over hateful sentences can explain the classifier's behaviour on non-hateful statements, highlighting classifiers' tendencies to over-rely on the presence of identity terms or to ignore the characteristics of the object of abuse (e.g., protected identity groups vs. non-human entities).
- We present the first methodology for calculating necessity and sufficiency metrics for text data as a feature attribution method. Arguably, this dual explainability measure is more informative and allows for deeper insights into a
model's inner workings than traditional single metrics.
- We use a generative model for producing input perturbations to avoid the out-of-distribution prediction issues that emerge with token deletion and masking techniques.
- To evaluate the new methodology, we apply it to the task of explaining hate speech classification, and demonstrate that it can detect and explain biases in hate speech classifiers.
We make the implementation code freely available to researchers to facilitate further advancement of explainability techniques for NLP.
Explanations are often categorized as to whether they are for an individual prediction (local) or for the model reasoning as a whole (global), and whether the explanation generation is a part of the prediction process (self-explaining) or generated through additional post-processing (post-hoc) (Guidotti et al., 2018; Adadi and Berrada, 2018). The necessity and sufficiency explanations presented here belong to the class of local explanation methods, as do most of the XAI methods applied to NLP data (Danilevsky et al., 2020). It is also a post-hoc method to the degree that it is entirely model-agnostic: all it requires is binary predictions on provided inputs.
There are a few classes of popular techniques for explaining natural language processing models (see Danilevsky et al. (2020) for a survey). One approach is feature attribution methods that allocate importance scores to each feature. These can be architecture-specific (Bahdanau et al., 2015; Sundararajan et al., 2017), or model-agnostic (Ribeiro et al., 2016; Lundberg and Lee, 2017).
Another approach is counterfactual explanations, which provide similar examples to the input in order to show what kinds of small differences affect the prediction of the model (Wu et al., 2021; Kaushik et al., 2021; Ribeiro et al., 2020; Ross et al., 2020). These contrastive examples are related to the concept of counterfactual reasoning from the causality literature, that formalizes the question: "Would the outcome have happened if this event had not occurred?" in order
<https://github.com/esmab/necessity-sufficiency
to determine whether the event was a cause of the observed outcome (Pearl, 2009). Counterfactual explanation methods are often targeted at certain semantic or syntactic phenomena such as negation (Kaushik et al., 2021) or swapping objects and subjects (Zhang et al., 2019), and hence do not guarantee that the counterfactuals cover the data distribution around the input text well.
In this work, we combine methods from feature attribution and counterfactual generation models. This allows us to calculate scores that capture local feature importance, and provide counterfactual examples as justification for the assigned scores.
Necessity and sufficiency. These are two notions from causal analysis that capture what one intuitively expects a true cause of an event to exhibit (Pearl, 2009; Halpern, 2016). Several works have recently suggested applying necessity and sufficiency to explain model predictions. Mothilal et al. (2021) used the actual causality framework of Halpern (2016) to calculate necessity and sufficiency scores for tabular data. Galhotra et al. (2021) suggested an approach to capture the notions of necessity and sufficiency from probabilistic causal models (Pearl, 2009). Watson et al. (2021) presented a different method for quantifying necessity and sufficiency over subsets of features. We follow the framework of probabilistic causal models, and adopt the definitions from Galhotra et al. (2021). In NLP explanations, necessity and sufficiency have been used for evaluating rationales (Zaidan et al., 2007; DeYoung et al., 2020; Mathew et al., 2021) $^{2}$ , however to the best of our knowledge, this is the first work to explore their usage for estimating feature attribution scores.
The out-of-distribution problem in feature attribution models. Virtually all model-agnostic feature attribution models calculate importance scores by perturbing input features and assign importance according to which feature changes the outcome the most. However, an issue has been raised that these perturbed inputs are no longer drawn from the data distribution that the model would naturally encounter for a given task (Fong and Vedaldi, 2017; Chang et al., 2018; Hooker et al., 2019; Janzing et al., 2020; Hase et al., 2021). This is problematic because then, any change in the model predictions could be caused by the distribution shift rather than
the removal of feature values (Hooker et al., 2019). Recently, Hase et al. (2021) have argued that the problem is due to social misalignment (Jacovi and Goldberg, 2021), where the information communicated by the model differs in non-intuitive ways from the information people expect.
One solution to address these issues is to calculate importance scores by marginalizing over counterfactuals that respect the data distribution. Kim et al. (2020) and Harbecke and Alt (2020) adopted this approach and targeted text data specifically by marginalizing over infills generated by BERT. In our preliminary experiments, this resulted in the model putting an overwhelmingly high probability mass to one or few very common words, making the generated perturbations relatively non-diverse.<sup>3</sup> As Pham et al. (2021) also pointed out, BERT is very good at guessing the masked word, doing so correctly about half of the time. This behaviour results in assigning low importance to highly predictable words regardless of their true importance.
For this reason, we choose to use a generative language model to infill masked sections with  $n$ -grams. Our mask-and-infill approach is similar to that of Wu et al. (2021) and Ross et al. (2021), who used fine-tuned causal language models to infill masked sections of text with variable length sequences. Ross et al. also used the contrasting label to condition the generative model. However, both these works aim to find counterfactual examples as explanations, while we marginalize over them to calculate necessity and sufficiency of each token.
A central idea in causal inference is that of intervention, where a random variable is intervened on and set to a certain value. The intuition is that, if a random variable is the cause of another, then intervening on the first one should affect the other, whereas if they are correlated by other means then the intervention should not have an effect.
Necessity. Let  $X \gets a$  denote that the random variable  $X$  has been intervened so that  $X = a$ . When talking about a feature vector  $x$ , we will denote by  $x_{i \gets a}$  that we intervene on the  $i$ th feature value and set it to  $a$ . For an input with features  $x$  where  $x_{i} = a$ , the necessity of  $x_{i} = a$  for the
2The term comprehensiveness is often used instead of necessity in this context.
3Making the softmax scores more distributed across the vocabulary results in unpredictably disfluent infills.
model prediction  $f(x) = y$  is defined as:
$$
N _ {x _ {i}, y} = P _ {c \sim \mathcal {D} _ {n} (x)} (f (c _ {i \leftarrow a ^ {\prime}}) = y ^ {\prime} | c _ {i} = a, f (c) = y)
$$
where  $a'$  is an alternative feature value such that  $a' \neq a$  and  $y'$  is an alternate outcome such that  $y' \neq y$ .  $\mathcal{D}_n(x)$  is a distribution that covers the neighborhood of  $x$ , and can be defined according to the data and the implementation. In words,  $x_i = a$  has high necessity for the prediction  $y$  if, for those points in the neighborhood of  $x$  that also have the value  $a$  for the  $i$ th feature and the same model prediction  $y$ , changing the  $i$ th feature value from  $a$  to  $a'$  changes the prediction from  $y$  to  $y'$  with high probability.
Sufficiency. The sufficiency of  $x_{i} = a$  for the model prediction  $f(x) = y$  is defined as:
$$
S _ {x _ {i}, y} = P _ {c \sim \mathcal {D} _ {s} (x)} (f (c _ {i \leftarrow a}) = y | c _ {i} = a ^ {\prime}, f (c) = y ^ {\prime})
$$
This means that if  $x_{i} = a$  has high sufficiency for the outcome  $y$ , then for inputs in the neighborhood of  $x$  that differ in the  $i$ th feature value, changing  $i$ th feature value to that of  $a$  will flip the prediction to  $f(x) = y$ .
Interventions. Previous works applying notions of necessity and sufficiency from causal inference to XAI assume tabular data. This makes it relatively straightforward to apply these measures to the features since  $a$  it is clear how to assess and compare the  $i$ th feature of each input and  $b$  there is little ambiguity in how to change one feature value to another. Both these are issues for NLP data, where each feature is a token in the context of the wider text.
We argue that the replacements should reflect the likelihood of natural data, but should still be distinct from purely observational correlations in task-specific aspects. To achieve this balance, we sample the replacement values  $a'$  conditioned both on the other parts of the text and on the opposite class  $y'$ . If there are two features  $x_i = a$  and  $x_j = b$  that are both correlated with the outcome  $y$ , the intervention  $x_{i\leftarrow a'}$ , where  $a'$  is sampled in this way results in  $a'$  being still plausible with respect to the context  $x_j = b$ , but removes the potential indirect effect that  $x_j = b$  causes  $x_i = a$ , which causes  $f(x) = y$ . This allows us to distinguish which of the correlated features the model relies on more for a given prediction.
Estimation. The formulae for necessity and sufficiency suggest a naive implementation of sampling first from the neighborhood of the input, picking those samples that conform to the conditions, and intervening on the feature of interest and marginalizing over the model predictions to calculate the final value. To perform these steps for each token in a sentence is prohibitively expensive. We therefore perform interventions on subsets of tokens at once, so that one perturbation can be used in the necessity and sufficiency estimation of multiple tokens.
We estimate the necessity of a token by perturbing subsets of tokens containing the given token and calculating the average change in model prediction, weighted according to the size of the subset. For calculating necessity, we marginalize over  $f(c_{i \leftarrow a'})$  where  $c = x_{j_1 \leftarrow b_1, \dots, j_k \leftarrow b_k}$  for a random subset of features  $x_{j_1}, \dots, x_{j_k}$ , not including the original feature  $i$ . This means that in our implementation,  $\mathcal{D}_n(x)$  is an interventional distribution around  $x$  rather than an observational one. In practice, we estimate a simplified version of this value where we do not explicitly condition on  $f(c) = y$  in order to perform the estimation efficiently.
We consider the instances where only one or a few tokens are perturbed to have higher probability in  $\mathcal{D}_n(x)$ . As such, the weight assigned to a sample with  $k$  perturbed tokens is proportional to  $1 / k$ . This means that the difference between the original and the perturbed instance is attributed to each perturbed token equally.
For estimating sufficiency we take the dual approach. We perturb subsets of tokens excluding the target token, and calculate the difference between the weighted average of the model predictions and the baseline prediction. Here too,  $\mathcal{D}_s(x)$  is an interventional distribution where each sample  $c = x_{i\leftarrow a',j_1\leftarrow b_1,\dots j_k\leftarrow b_k}$  for the focus feature  $x_{i}$  and a subset of other features  $x_{j_1},\dots ,x_{j_k}$ . Even though we do not explicitly condition on  $f(c) = y'$ ,  $\mathcal{D}_s(x)$  is biased towards such  $c$  because the interventions are conditioned on  $y^\prime$ . For a sequence of length  $n$ , the weight assigned to an instance where  $k$  tokens are perturbed is  $1 / (n - k)$ . This means that for an perturbed example that contains only a single token from the original instance, the difference from the baseline will be attributed entirely to that token, whereas if there is  $k$  original tokens, the attribution is shared between them. Note that  $\mathcal{D}_s(x)$  still assigns a higher probability mass to instances closer to  $x$ , but is less peaked than  $\mathcal{D}_n(x)$ .
Figure 1: An illustration of how necessity and sufficiency are calculated for a chosen token "women" in the input "I hate women" that the model classifies as hateful. In the MASKING step, the subsets of tokens are masked. For the necessity calculation the masked tokens always include the focus word, and for sufficiency they always exclude it. In the INFILLING step, the generative language model is used to infill the masked sections with  $n$ -grams of various lengths. These are then passed to the classifier. The necessity is the proportion of instances where changing the token changes the prediction, and sufficiency is the proportion of instances where changing other tokens does not change the original prediction. The infills are real examples generated by our method.
4 Explaining Hate Speech Models
For tasks with very skewed class distributions such as those for binary hate speech classification, it is not intuitive to ask for explanations for the majority class predictions: it is difficult to answer why a regular utterance such as "I would like some coffee." is not hate speech. This echoes the argument of Miller (2019) that humans demand explanations only for selective and surprising aspects of an occurrence. We assume that there exists a majority, 'neutral' class, and aim to provide explanations only for the minority, 'positive' class.
At the core of our approach is sampling replacements  $x_{i} \gets a'$  that are interventional with respect to the task, but also have high likelihood in the context of other tokens. We implement such a perturbation model for explaining positive predictions of hate speech classifiers, and explore what information necessity and sufficiency provide for this task through quantitative and qualitative analysis. Our implementation is task specific, since we sample from the data labelled as 'neutral' only; however, it is not dataset specific in that it allows comparing hate speech classifiers that are trained on different datasets. Figure 1 illustrates the calculations of
necessity and sufficiency for an example classifier.
4.1 Generating perturbations
We use the infilling language model (ILM) of Donahue et al. (2020) to generate the perturbed examples. This model fine-tunes GPT-2 (Radford et al., 2019) to allow infilling masked sections of text with  $n$ -grams, where the length of the infill varies from 1 to 7 tokens.
We fine-tune the ILM on training data that is labelled 'neutral', which is the opposite of the model prediction we aim to explain. Training the perturbation model only on the neutral examples allows us to distinguish direct causes of the model prediction from correlations in data. To see this, consider the case where we would like to determine whether a given identity mention is sufficient for a positive prediction. If the ILM captures the entire data distribution rather than just the neutral class, then a large chunk of the perturbed instances might still be hateful utterances. In those cases a good model should predict the positive class even though the occurrence of the identity term is arguably not the direct cause of the prediction. Rather, the presence of the identity term causes the ILM to generate a hateful infill, which then causes the positive pre
Functionality	Example Test Case	Gold Label	n
Explicit	F1: Expression of strong negative emotions	“I hate [IDENTITY].”	hateful	40
“You [IDENTITY] make me sick.”
F2: Description using very negative attributes	“[IDENTITY] are disgusting.”	hateful	40
“All [IDENTITY] like them are vile.”
F3: Dehumanisation	“[IDENTITY] are scum.”	hateful	40
“[IDENTITY] are a pest.”
Non-hate grp. ident.	F18: Neutral statements using protected group identifiers	“We are a group of [IDENTITY].”	non-hate	36
“I live with two [IDENTITY].”
F19: Positive statements using protected group identifiers	“I love [IDENTITY].”	non-hate	54
“[IDENTITY] are great.”
Abuse against non-prot. targets	F22: Abuse targeted at objects	“I really can’t stand cauliflower.”	non-hate	65
"All cocktails like these are vile”
F23: Abuse targeted at individuals (not as member of a prot. group)	“You make me sick.”	non-hate	65
“I will hurt you.”
F24: Abuse targeted at non-protected groups (e.g. professions)	“Artists are parasites to our society.”	non-hate	62
“Death to all capitalists.”
Table 1: Test cases of interest from HateCheck (Röttger et al., 2021). We hypothesize that for identity terms in explicit test cases, if sufficiency is high then the model will have a high error rate on the test cases that capture non-hate group identity mentions, and if necessity is low then the model will have a high error rate on the test cases that cover abuse against non-protected targets.
diction. Using the non-hateful data distribution to train the infilling model helps avoid such cases, and enables the method to attribute importance to a token only when the classifier relies on it directly.
To train our infilling language model, we choose four widely-used datasets for hate speech and abusive language detection that are from various sources such as Twitter (Founta et al., 2018), Reddit (Vidgen et al., 2021), Wikipedia comments (Wulczyn et al., 2017) and news article comments (Borkan et al., 2019). For all these datasets, we fine-tune our generative model on benign instances from the training sets. Details for training and datasets can be found in Appendix A.