SlowGuess/NAACL_2022_OCR huggingface dataset:
For our experiments, we focus on investigating the necessity and sufficiency of identity mentions for a sentence to be detected as hate speech, since a known bias in many hate speech detection models is that mere mentions of identity terms result in false positive predictions (Dixon et al., 2018).
For a set of instances that contain mentions of identity terms we leverage HateCheck (Röttger et al., 2021), which is a suite of functional tests targeted at diagnosing weaknesses of hate speech classifiers. Tests are constructed from hand-crafted
templates, where the target is picked from a predefined set of protected groups.
To evaluate our explanation method, we train six BERT classifiers on three different datasets (Founta et al., 2018; Vidgen et al., 2021; Davidson et al., 2017), and obtain the necessity and sufficiency of the identity terms on explicitly hateful test cases in HateCheck that target women and Muslims. Details for the datasets and classifiers can be found in Appendix B. We train classifiers on both 'hate speech' labels and on more general 'abusive' language labels in order to observe the effects of this distinction on the necessity and sufficiency values for identity terms. The difference between the two is that abusive language does not need to target a protected group identity. Accordingly, our first hypothesis is:
Hypothesis 1: We will observe lower necessity for the identity terms for those models that are trained on the 'abuse' labels compared to the ones that are trained on the 'hate' labels.
We further hypothesize that different necessity and sufficiency scores for identity mentions in explicitly hateful test cases indicate different biases, and correlate with how well a model does in the Hate-Check functionalities that aim to capture those biases (see Table 1 for the test cases), leading to our
Figure 2: Visualization for necessity and sufficiency scores on an example HateCheck test case targeting women and Muslims for the classifier Founta2018-hate. Darker shades correspond to higher values.
second and third hypotheses:
Hypothesis 2: If a model has high sufficiency scores for identity terms in explicit expressions of hate (functionalities  $\mathbf{F1}$ ,  $\mathbf{F2}$  and  $\mathbf{F3}$ ), this should indicate that the model is over-sensitive to identity terms. Therefore, we expect it have increased error rate in  $\mathbf{F18}$  and  $\mathbf{F19}$ , which consist of neutral or positive statements that mention identity terms.
Hypothesis 3: If the necessity scores for identity terms are low in explicit expressions of hate, we can conclude under-reliance on the identity terms, and over-reliance on the context. Consequently, we expect that the model will perform poorly on F22, F23 and F24, which capture abuse not targeted at protected identity groups.
We obtain the average necessity and sufficiency values for explicitly hateful test cases targeting women and Muslims for each of the classifiers. We calculate necessity and sufficiency by masking a subset of the tokens and using our fine-tuned language model to generate infillings. If multiple consecutive tokens are chosen, we aggregate them to a single mask instance to be inflated. We choose the number of perturbations for each example so that the expected number of perturbations for each token is 100. The necessity and sufficiency scores are only calculated for test cases that a classifier returns a correct prediction, since we only aim to explain positive predictions. The results can be found in Table 2. Table 3 presents the proportions of test cases classified as hateful/abusive by each of the six classifiers on the non-hateful statements that mention identity terms (F18 and F19) and abusive utterances not targeting protected identity groups (F22, F23, and F24). We report the results where necessity and sufficiency are calculated with masking rather than perturbing the chosen tokens in Appendix C.
As baselines, we calculate the average importance of the tokens corresponding to target groups
with SHAP $^4$  and LIME $^5$ . For both of these methods, we use the default parameters for textual data. As with the calculation of necessity and sufficiency, we only include the attribution scores for test cases on which the classifier correctly predicts the positive class. These results can be found in Table 4.
An example necessity and sufficiency attribution is given in Figure 2. It shows that for this input, the token 'Muslims' is more sufficient compared to 'women', and the token 'disgust' is more necessary in the context of 'women' than that of 'Muslims'.
According to our first hypothesis, we expect the models that were trained on the abuse versions of each dataset to have lower necessity for identity terms compared to those that have been trained on hate labels. Indeed, in Table 2 we observe this pattern for all models and targets except Davidson2017 for the target women. This correctly suggests that identity terms are necessary for a comment to be hate speech, but not for it to be abusive.
The results also clearly support our second hypothesis that if an identity mention has high sufficiency on explicit examples for a given model, then this model is over-sensitive to the identity term. Comparing the sufficiency of women and Muslims in Table 2 illustrates this difference: for all models except Davidson2017-abuse sufficiency is high for Muslims and significantly lower for women. Accordingly, all models except Davidson2017-abuse display a large difference between their error rates on neutral or positive mentions for women and Muslims in Table 3 (F18, F19). That is, the mere occurrence of the word "Muslims" is sufficient for the classifiers to classify a text as hate speech, even if the text is neutral. Furthermore within each group, higher sufficiency values correspond to higher error rates in functionalities F18, F19. Vidgen2021-hate
4https://github.com/slundberg/shap
<sup>5</sup>https://github.com/marcotcr/lime
women	Muslims
necc	suff	necc	suff
Founta2018-hate	0.82 ±0.18	0.29 ±0.1	0.89 ±0.16	0.81 ±0.08
Founta2018-abuse	0.54 ±0.17	0.34 ±0.1	0.65 ±0.21	0.82 ±0.06
Davidson2017-hate	0.58 ±0.09	0.21 ±0.06	0.91 ±0.12	0.74 ±0.09
Davidson2017-abuse	0.82 ±0.14	0.43 ±0.13	0.83 ±0.13	0.41 ±0.14
Vidgen2021-hate	0.96 ±0.02	0.71 ±0.17	0.97 ±0.03	0.88 ±0.13
Vidgen2021-abuse	0.82 ±0.14	0.64 ±0.14	0.82 ±0.15	0.88 ±0.07
Table 2: The mean and standard deviation of necessity and sufficiency scores for target tokens in explicitly hateful cases of HateCheck (F1, F2, and F3) targeting women or Muslims for the three classifiers trained on hate, and three classifiers trained on abuse labels.
Neutral/supportive group identity mention (F18, F19)	Abuse against non-protected targets (F22, F23, F24)
women	Muslims	group	individual	object
Founta2018-hate	0.02	0.78	0.19	0.15	0.05
Founta2018-abuse	0.02	0.78	0.45	0.72	0.37
Davidson2017-hate	0.02	0.78	0.37	0.18	0.02
Davidson2017-abuse	0.31	0.22	0.26	0.28	0.14
Vidgen2021-hate	0.36	0.82	0.02	0.00	0.00
Vidgen2021-abuse	0.42	0.96	0.40	0.61	0.00
Table 3: Proportions of test cases classified as hateful/abusive for different non-hateful HateCheck functionalities and targets.
and Vidgen2021-abuse display the highest sufficiency for women, and correspondingly have the highest error rates on these test cases for women. Davidson2017-abuse has the lowest sufficiency for Muslims, and the lowest error rate for this target.
Our third hypothesis is that low necessity for identity terms will be correlated with positive predictions for abusive instances that do not target a protected identity. In Table 2, the lowest necessity for both target groups are observed with Founta2018-abuse. Indeed, this model has the highest rate of positive (abuse) predictions on all functionalities that test for abuse against non-protected targets in Table 3. The false positives in the test cases that target objects is much higher than the corresponding errors for the other models, indicating that Founta2018-abuse is indeed over-sensitive to abusive contexts, and does not consider the target of the abuse to be a necessary feature for the classification. On the other hand, the classifier trained on Vidgen2021-hate shows the highest necessity values for both targets, and the lowest error rates on F22, F23, F24.
6.1 Comparison of Average SHAP and LIME Values with Necessity and Sufficiency
The average SHAP and LIME values for the two targets are presented in Table 4. While Founta-abuse and Davidson-abuse get very similar SHAP scores
for the target Muslims, Founta2018-abuse has high sufficiency for this token while Davidson2017-abuse has high necessity. These two classifiers have very different false-positive rates for test instances that are non-abusive mentions of this target as reported in Table 3, and hence can be observed to be biased against this group to a different extent. This distinction is clearly captured with the necessity and sufficiency scores, but not with SHAP.
LIME scores seem even less consistent with the false-positive rates in Table 3 than SHAP. For example, Davidson2017-hate has a very high false-positive rate for neutral/supportive mentions of the target Muslims, however the average LIME score for this model and target group is in low negatives. This means that LIME was unable to capture the biases of the model against this target group.
For the target women, LIME gives very similar average importance to the target tokens for Founta2018-abuse and Vidgen2021-hate, however all of the other metrics we calculate highlight significant differences. Indeed, we can observe in Table 3 that Founta2018-abuse has high false-positive rates for abuse against non-protected targets, but low false-positive rates for neutral/supportive mentions of the target women, whereas Vidgen2021-hate shows the exact opposite pattern. LIME scores seem unable to distinguish this important difference, unlike our necessity and sufficiency scores.
women	Muslims
SHAP	LIME	SHAP	LIME
Founta2018-hate	0.36 ±0.24	0.018 ±0.045	0.84 ±0.2	0.013 ±0.036
Founta2018-abuse	-0.07 ±0.21	0.011 ±0.02	0.39 ±0.25	0.042 ±0.037
Davidson2017-hate	0.01 ±0.02	-0.021 ±0.018	0.89 ±0.15	-0.002 ±0.057
Davidson2017-abuse	0.42 ±0.19	0.001 ±0.086	0.37 ±0.2	-0.032 ±0.092
Vidgen2021-hate	0.89 ±0.17	0.012 ±0.077	0.95 ±0.11	0.042 ±0.049
Vidgen2021-abuse	0.66 ±0.23	0.045 ±0.088	0.75 ±0.22	0.087 ±0.062
Table 4: The mean and standard deviation of SHAP and LIME scores for target tokens in explicitly hateful cases of HateCheck (F1, F2, and F3) targeting women or Muslims for the three classifiers trained on hate, and three classifiers trained on abuse labels.
Examining the generated perturbations reveals that the majority of the time, the generator successfully replaces the original snippet with non-offensive text. There are minimal repetitions of the same infill, indicating that the perturbations are sufficiently diverse. The quality of the perturbations is mixed, and tends to decrease with the number of infills. Our method can be used with any generator that can model the data distribution conditioned on the label. For our current model we prioritize diversity over fluency, and leave improvements on this aspect for future work.
Inspecting the generated perturbations reveal that there are often instances that combine a few HateCheck functionalities. For example, as a perturbation for "I hate [IDENTITY]" model generates "This is a subreddit I hate.", combining shuffling the word order with changing the target. For the template "[IDENTITY] are the dumbest people.", one of the generated counterfactuals is "I don't believe that American people are the dumbest people.", combining negation and phrasing.
Interestingly, many infills for the identity terms do not clearly correspond to either object, individuals or non-protected groups, which are the three cases that HateCheck checks for. Instead, our generative model infills it with clauses such as "I hate that I feel like I have to do these things", actions such as "I hate crying," or types of events like "I hate surprises." This illustrates that our relatively non-constrained generation of counterfactuals provides better coverage of potential replacements, and provides a good complement to manual checks.
This work is a step towards more informative and transparent feature attribution metrics for explaining text classifiers. We argue that standard token
importance metrics can be ambiguous in terms of what 'importance' means. Instead, we adapt the theoretically-grounded concepts of necessity and sufficiency to explain text classifiers. Besides being more informative, the process of generating these two metrics is intuitive and can be explained to lay people in terms of "how much the perturbations in input change the output of the classifier". Moreover, the input perturbations can be presented to the users, leading to a transparent and understandable explainability framework.
Considering the complexities of perturbing textual features, we introduced a practical implementation to compute the necessity and sufficiency of the input tokens. Taking hate speech detection as an example application, we showed that sufficiency and necessity can be used to explain the expected differences between a classifier that is intended to detect identity-based hate speech and those trained for detecting general abuse. We also leveraged these metrics to explain the observed over-sensitivity and under-sensitivity to mentions of target groups, issues that are tightly related to fairness in hate speech detection. While the current work focused on binary hate speech detection for English-language social media posts, in future work, we will explore the effectiveness of these metrics in generating explanations for other applications and languages. We will also explore how the new metrics can improve the debugging of the models or communicating the model's decision-making process to the end-users.
The proposed method has benefits and risks that should be considered from an ethics perspective.
One principle of ethical AI is transparency, and we have developed this method with the goal of improving transparency for system developers, end users, and other stakeholders to better understand
the inner workings of complex NLP systems. In the application domain of hate speech detection, we demonstrated how necessity and sufficiency scores might be used to diagnose possible classification biases against identity groups, who are frequently subjects of online abuse. This can help in addressing the known issue of over-sensitivity to identity terms, ensuring that benign conversations around issues concerning marginalized groups are not misclassified as hate speech.
However, there are also potential risks. We make use of existing datasets and thus our analysis is limited by those data: they were collected from public, online platforms without user's explicit consent, and may not accurately represent speakers from all demographic groups, they are only in English, and they may be biased towards or against certain topics of conversation. The data and analysis are also limited to the English language. Training language models on user data also has privacy implications, as the language model may then re-generate user text when deployed.
While transparency and explainability are seen as desirable properties, they can also expose AI systems to malicious attacks. In the context of hate speech, our explainability metrics could potentially be used to identify and then exploit system vulnerabilities.
Finally, our approach requires the use of large language models, which are computationally expensive to train and can reflect the biases of their training data. Our method of generating multiple counterfactual examples per word, rather than simply removing or masking that word, also increases the computational resources required.
Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6:52138-52160.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In *Companion Proceedings of the 2019 World Wide Web Conference*, pages 491-500.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. 2018. Explaining image classifiers by counterfactual generation. In Proceedings of the International Conference on Learning Representations.
Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020. A survey of the state of explainable AI for natural language processing. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 447-459, Suzhou, China. Association for Computational Linguistics.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT).
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67-73.
Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492-2501.
Ruth C Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3437.
Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1-30.
Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of Twitter abusive behavior. In Proceedings of the Twelfth International AAAI Conference on Web and Social Media.
Sainyam Galhotra, Romila Pradhan, and Babak Salimi. 2021. Explaining black-box algorithms using probabilistic contrastive counterfactuals. In Proceedings of the 2021 International Conference on Management of Data, pages 577-590.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 219-226.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1-42.
Joseph Y. Halpern. 2016. Actual Causality. MIT Press.
David Harbecke and Christoph Alt. 2020. Considering likelihood in NLP classification explanations with occlusion and language modeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 111-117.
Peter Hase, Harry Xie, and Mohit Bansal. 2021. The out-of-distribution problem in explainability and search methods for feature importance explanations. Advances in Neural Information Processing Systems, 34.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. Advances in Neural Information Processing Systems, 32:9737-9748.
Alon Jacovi and Yoav Goldberg. 2021. Aligning faithful interpretations with their social attribution. Transactions of the Association for Computational Linguistics, 9:294-310.
Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. 2020. Feature relevance quantification in explainable AI: A causal problem. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 2907-2916. PMLR.
Divyansh Kaushik, Amrith Setlur, Eduard H Hovy, and Zachary Chase Lipton. 2021. Explaining the efficacy of counterfactually augmented data. In Proceedings of the International Conference on Learning Representations.
Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. 2020. Interpretation of NLP models through input marginalization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3154-3167, Online. Association for Computational Linguistics.
Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C Fraser. 2021. Confronting abusive language online:
A survey from the ethical and human rights perspective. Journal of Artificial Intelligence Research, 71:431-478.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765-4774. Curran Associates, Inc.
Siwen Luo, Hamish Ivison, Caren Han, and Josiah Poon. 2021. Local interpretations for explainable natural language processing: A survey. arXiv preprint arXiv:2103.11072.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14867-14875.
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38.
Ramaravind Mothilal, Divyat Mahajan, Chenhao Tan, and Amit Sharma. 2021. Towards unifying feature attribution and counterfactual explanations: Different means to the same end. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 652-663.
Isar Nejadgholi and Svetlana Kiritchenko. 2020. On cross-dataset generalization in automatic detection of online abuse. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 173-183.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2799-2804, Brussels, Belgium.
Judea Pearl. 2009. Causality. Cambridge University Press.
Thang M Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Double trouble: How to not explain a text classifier's decisions using counterfactuals synthesized by masked language models? arXiv preprint arXiv:2110.11929.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144.