Concept-Based Explanations to Test for False Causal Relationships Learned by Abusive Language Classifiers
Year: 2023

Abstract:
Classifiers tend to learn a false causal relationship between an over-represented concept and a label, which can result in over-reliance on the concept and compromised classification accuracy. It is imperative to have methods in place that can compare different models and identify over-reliances on specific concepts. We consider three well-known abusive language classifiers trained on large English datasets and focus on the concept of negative emotions, which is an important signal but should not be learned as a sufficient feature for the label of abuse. Motivated by the definition of global sufficiency, we first examine the unwanted dependencies learned by the classifiers by assessing their accuracy on a challenge set across all decision thresholds. Further, recognizing that a challenge set might not always be available, we introduce concept-based explanation metrics to assess the influence of the concept on the labels. These explanations allow us to compare classifiers regarding the degree of false global sufficiency they have learned between a concept and a label. Content Warning: This paper presents examples that may be offensive or upsetting.

text: The 7th Workshop on Online Abuse and Harms (WOAH), pages 138–149 July 13, 2023 c©2023 Association for Computational Linguistics
Content Warning: This paper presents examples that may be offensive or upsetting.

heading: 1 Introduction
text: In various natural language classification tasks, particularly in abusive language detection, certain concepts are known to be strong signals for the label of interest. These concepts are often overrepresented in the respective class of the training set, making them susceptible to being learned as potential causes for the label. Consequently, the classifier over-relies on these concepts and ignores the broader context, leading to reduced generalizability (Yin and Zubiaga, 2021). Hence, to ensure models are robust and reliable, it is crucial to develop methods that can detect these over-reliances in various natural language classification tasks.
In the context of abusive language detection, we consider the concept of negative emotions. The
presence of an expression associated with negative emotion is an important signal for detecting abusive language and has been used in feature-based systems before (Chiril et al., 2022; Fortuna and Nunes, 2018). Crucially, in some examples, negative emotion words might be the cause for the abusive label, i.e., the sentence might not be abusive if the negative emotion word is replaced with other words (e.g., I know these people. They are disgusting). However, at the global level, the relationship between negative emotion and abusive language is a strong correlation, not causation, as it is neither globally necessary nor globally sufficient for the label of abuse.1 Negative emotions are not globally necessary for the label of abuse because there are abusive sentences that do not contain any negative emotion words (e.g., offensive jokes, stereotyping and microaggressions). Also, words evoking negative emotions are not globally sufficient for a sentence to be abusive when interpreted in a broader context (e.g., We should admit that in our society, they are oppressed.). But, an end-to-end model might learn that negative emotion in a sentence is globally sufficient for that sentence to be abusive. Such a classifier will struggle in classifying nonabusive sentences that contain negative emotion
1Phenomenon P is globally sufficient for the Phenomenon Q, if whenever P happens, Q happens too. P is globally necessary for Q if whenever Q happens, P happens, too (Zaeem and Komeili, 2021).
138
words leading to a lack of generalizability. An example of such a case is shown in Figure 1. Specifically, classifiers’ over-reliance on the negative emotion signal can inadvertently discriminate against marginalized groups since their communications (e.g., discussing their experiences of discrimination and marginalization) can contain negative emotion words and, therefore can be wrongly considered abusive.
We explore a scenario where a user, aware of the importance of negative emotions for their use case, wants to evaluate and compare a set of trained models. Their goal is to identify and eliminate those models that are more prone to generating inaccurate results due to an overemphasis on negative emotions as primary indicators. For that, we use concept-based explanations to test if a model has learned a false global causal relationship between a user-identified concept and a label where the true relationship is a correlation. Note that global causal relationships explain the model’s output across an entire dataset, as opposed to local causal explanations, which concern the dependency of an individual prediction on a specific input feature.
Concept-based explanations are a class of explainability methods that provide global explanations at the level of human-understandable concepts (Yeh et al., 2022). While local explanations help the users understand the model’s reasoning for an individual decision with respect to the input features, global explanations are critical in comparing the processes learned by models and selecting the one that best suits the needs of a use case (Balkir et al., 2022; Burkart and Huber, 2021). Global explanations might be obtained at the level of input features through aggregating local explanations (Lundberg et al., 2020). Alternatively, global-bydesign methods (e.g., probing classifiers (Conneau et al., 2018)) can be used to gain insights at higher levels of abstractions, such as linguistic properties or human-defined concepts.2
Similar to most feature importance explainability methods (e.g, Ribeiro et al. (2016); Lundberg and Lee (2017a)), concept-based explanations are originally designed to measure the importance of a concept. The intuitive meaning of importance usually refers to correlation, and it can be interpreted differently based on two notions of causality: necessity and sufficiency (Galhotra et al., 2021). Local
2Here, we use the term “feature” to refer to the latent representations of a semantic concept learned by a classifier.
explainability methods usually focus on features that are of high local necessity or high local sufficiency for the label (Watson et al., 2021; Balkır et al., 2022; Joshi et al., 2022), thus considered important by human users. However, at the global level, all features must be interpreted in a larger context for accurate decision-making. We aim to determine if concept-based explanations can be utilized to evaluate whether a trained binary classifier for abusive language detection has learned a false global sufficiency relationship between the label and the concept of negative emotion. Our code and data are available at https://github.com/Isa rNejad/Global-Sufficiency/tree/main. Our main contributions are:
• We formalize the issue of over-reliance on a concept as falsely learned global sufficiency. For the task of an abusive language classifier, we consider concepts related to negative emotion as being important but not globally sufficient for the label of abuse. We discuss how learning these concepts as globally sufficient results in compromised classification accuracies.
• Based on our formalization of false global sufficiency, as a baseline method, we measure the over-reliance of models on a human-defined concept using an unseen challenge set that contains the concept in both classes. Recognizing that various classifiers may have a distinct range of optimal decision thresholds, we assess the overreliance on a concept across all possible decision thresholds and show that one of the classifiers over-relies on emotion-related concepts significantly more than the other two classifiers.
• Taking the challenge set approach as a baseline for comparison, we propose novel concept-based explanation metrics, demonstrating that similar conclusions about the degree of false global sufficiency can be drawn using these metrics. Building on previous work, we modify the TCAV procedure to measure not only the feature’s importance but also the extent of its impact on the label. We conclude that a concept-based method is preferable as it eliminates the need for manual data curation.

heading: 2 Concept-Based Explanations
text: Concept-based explanations evaluate the model’s decision-making mechanism at the level of a
human-defined concept expected to be important for the task (Koh et al., 2020). Specifically, we use the Testing Concept Activation Vectors (TCAV) method to measure the influence of a humandefined concept on the model’s predictions (Kim et al., 2018). The idea of TCAV is based on the observation that human-understandable concepts can be encoded as meaningful and insightful information in the linear vector space of trained neural networks (Mikolov et al., 2013). A Concept Activation Vector (CAV), which represents the concept in the embedding space, is a vector normal to a hyperplane that separates concept and non-concept examples. Such a hyperplane is obtained by training a linear binary classifier to separate the representations of concept and non-concept examples in the embedding space.
Although TCAV can be applied to all neural network classifiers, for simplicity we limit our experiments to binary RoBERTa-based abusive language classifiers. We choose the RoBerta-based models for their superior performance in processing social media data compared to other base language models (Liu et al., 2019). The concept, C, is defined by NC concept examples. Also, NR random examples are used to define non-concept examples. The RoBERTa representations for all these examples are calculated using femb, which maps an input text to its [CLS] token representation. Then, P number of CAVs, υpC , are generated, each through training a linear classifier that separates a sub-sample (with size Nc) of concept examples from a sub-sample of random examples (with size Nr) in the RoBERTa embedding space. The conceptual sensitivity of a label to the CAV, υpC , at input x can be computed as the directional derivative SC,p(x):
SC,p(x) = lim ϵ→0
h(femb(x)+ϵυ p C)−h(femb(x)) ϵ
= ▽h(femb(x)).υpC (1) where h maps the RoBERTa representation to the logit value of the class of interest.
In this work, we use two metrics to specify the influence of the concept on the model’s prediction. First, we calculate TCAVdir, the fraction of inputs in a set of input examples X , for which the directional derivative SC,p(x) is positive, i.e.:
TCAV C,pdir = |x ∈ X : SC,p(x) > 0|
|X| (2)
TCAVdir indicates the fraction of input examples for which the prediction scores of the model
increase if the input representation is infinitesimally moved towards the concept representation. This metric has been widely used to identify if the label has learned the concept as an important signal for the label (Yeh et al., 2020).
Besides the widely used metric of TCAVdir (referred to as TCAV score in previous work), we introduce a new metric, TCAVmag, which considers the size of the directional derivatives, and measures the magnitude of the influence of the concept on the label for the positive directional derivatives:
TCAV C,pmag =
∑ x∈X,SC,p(x)>0 SC,p(x)
|X| (3)
We demonstrate in our results that TCAVmag can be an indicator of the over-reliance of the label on the concept. When calculated for all CAVs, Equations 2 and 3 generate two distributions of scores with size P for the concept C. Using a t-test, these distributions are compared with the distributions of TCAVdir and TCAVmag calculated for random examples to check for statistical significance (Kim et al., 2018).

heading: 3 False Global Sufficiency
text: Phenomenon P is considered globally sufficient for phenomenon Q (P ⇒ Q) if, whenever P occurs, Q also occurs (Zaeem and Komeili, 2021). In other words, global sufficiency refers to the extent to which a concept can explain the model’s output across all instances in a held-out dataset, as opposed to the more studied topic of local sufficiency, which concerns the stability of an individual prediction for a given feature in perturbed contexts (Balkır et al., 2022).
In a real-world setting, it is very unlikely that any single concept is truly sufficient for the label at a global level. In a binary classifier, a concept C is falsely learned as sufficient for the positive label if all inputs containing C are classified as positive by the classifier, regardless of context. This undesired dependency of the label on the concept suggests that the model has failed to learn how the concept interacts with context to influence the label. While this issue is closely related to spurious correlation, we use the term false global sufficiency because spurious correlation typically implies that the feature is irrelevant to the label, and a correlation is learned due to a confounding factor. In contrast, we consider the cases where the feature is relevant and important but not globally sufficient.
To make this clearer, consider the case of abusive language detection and the concept of negative emotions; if the mere presence of negative emotions in a sentence always guarantees the prediction of the positive label (abuse), then the model has learned a false sufficiency relation between the concept and the label. It over-relies on this feature and ignores the context.
To quantify falsely learned global sufficiency, we consider two scenarios: 1) where a balanced challenge set is available, which contains C in all of its examples (both classes), and 2) where no challenge set is available. For the first scenario we use the traditional approach of assessing accuracy of the classifier on a held-out test set. This approach provides a baseline in our evaluations. For the second scenario, we propose concept-based explanation metrics and compare them with the baselines obtained with the challenge sets.

heading: 3.1 Quantifying the Falsely Learned Global Sufficiency with a Challenge Set
text: Based on our definition of global sufficiency, one way to assess a model’s over-reliance on a concept is to evaluate its performance on a held-out challenge set, F, containing both positive and negative examples of the concept of interest (Yin and Zubiaga, 2021). For simplicity, we assume that this challenge set consists of equal numbers of positive and negative examples. If a model learns a high global sufficiency between the concept C and the label of abuse, all examples in both positive and negative classes of a challenge set F will be labeled as abusive. However, if the model interprets the concept in context, only the positive examples of F will receive the abusive label. This indicates that in
cases where the decision threshold of the classifier clearly separates the probability distributions of the two classes, the model has learned a low global sufficiency between the concept and the label.
However, when comparing different classifiers, it is important to note that a reliable classifier should perform well (high precision and high recall) over a broad range of decision thresholds. This is because different applications may require different thresholds depending on the desired tradeoff between precision and recall. For example, a classifier used to moderate social media content may need to prioritize precision over recall, which could mean using a high threshold to avoid false positives. On the other hand, a classifier used to detect all instances of abusive language may need to prioritize recall over precision, which would mean using a lower threshold to catch as many instances of abuse as possible, even if it means tolerating more false positives. Therefore, a classifier that is reliable over a wide range of decision thresholds can be more effective in different use cases, making it more practical and adaptable.
Figure 2 demonstrates two hypothetical cases for the distribution of probabilities that the classifiers might generate for the challenge set F. A classifier that learned low global sufficiency between C and the positive label generates easily separable distributions of probabilities for the positive and negative examples of F. In other words, for a large range of decision thresholds, the two classes of F are separable, and high accuracy is achieved. Conversely, the classifier that has learned high global sufficiency between C and the positive label assigns a similar distribution of probabilities to both negative and positive examples. The two classes
of F are hardly separable, and for a wide range of thresholds, the accuracy is low. Note that in order for this classifier to be accurate, it requires a careful adjustment of the decision threshold with a labeled dataset. However, this process can be very costly.
Based on this discussion, we argue that AUC_Challenge, the area under the curve of accuracy vs threshold, is a quantitative indicator of the separability of two classes of F for all decision thresholds. According to our definition above, global sufficiency is negatively correlated with the separability of these classes. Therefore, False_Suff , described in Equation 4, is a quantitative metric that can be used to compare the degree of sufficiency learned by the classifiers based on F:
False_Suff = 1−AUC_Challenge (4)

heading: 3.2 Quantifying the Falsely Learned Global Sufficiency with Concept-Based Explanations
text: The practical application of the method detailed in Section 3.1 can be limited due to the necessity of creating a custom challenge set. In this section, we use concept-based explanation to measure the falsely learned global sufficiency in a scenario where a challenge set is not available, but a lexicon representing the concept of interest exists. Following the approach of Nejadgholi et al. (2022a), we employ short templates and the concept lexicon to generate unlabeled concept examples. Then, we utilize the method described in Section 2 to compute two metrics: TCAVdir and TCAVmag. If the TCAVdir value for the concept significantly deviates from that of random concepts, it indicates that the classifier has learned an association between the label and the concept. A significant difference in TCAVmag compared to random concepts suggests a strong influence of the concept on the label, potentially causing the classifier to disregard the context when the concept is present. While the absolute values of these metrics might not be definitive, we show that they can be used to compare various classifiers in terms of the degree of global sufficiency they have learned for a concept.
4 Sufficiency of the Concept of Describing Protected Groups with Negative Emotion
In this section, we evaluate the metrics introduced in Section 3 in explaining the extent of the falsely learned sufficiency between a human-defined concept and the positive label of the classifiers. We
specifically consider the concept of describing a protected group with negative emotion words and refer to it as DesNegEm for brevity. We chose this concept because it is tightly related to hate speech and is expected to be important for more general definitions of harmful language, such as toxic, abusive or offensive. Still, it is not a sufficient concept for these labels and has to be interpreted in the broader context (as shown by examples in Table 1).
We consider three RoBERTa-based binary classifiers, publically available and trained with large English datasets. The models are trained for general definitions of abusive language, toxicity or offensive language. We refer to these classifiers by their training datasets: Jigsaw, Civil Comments (or Civil for brevity) and TweetEval. These models are described in detail in Appendix A
Quantifying Sufficiency with a Challenge Set: To calculate the metric described in Section 3.1, we first use the HateCheck (Röttger et al., 2021) test cases to build a challenge set for the concept of DesNegEm. For that, we use the F2 and F21 functionalities of HateCheck, i.e., the hateful and non-hateful examples that include this concept (Table 1). Figure 3 shows the distribution of probabilities that the three classifiers generate for this challenge set. We observe that, for a large range of decision thresholds, all three classifiers label the majority of the examples of both classes of the challenge set with a positive label. In other words, all three classifiers have learned a high sufficiency between DesNegEm and the label of abuse. However, the extent of the learned sufficiency is different among the classifiers. The TweetEval classifier makes the least differentiation between the two classes and generates similar distributions of probabilities for negative and positive examples with the DesNegEm concept. Because of this overlap between probability distributions of positive and negative classes, the accuracy of this classifier is low over all ranges of thresholds, as shown in Fig-
ure 4. The false sufficiency learned by the Jigsaw and the Civil Comments classifiers is less extreme, and Jigsaw makes the most differentiation between the two classes.
This observation can be quantified with the False_Suff metric (Equation 4) using the area under the curves in Figure 4. We obtain False_Suff of 0.41, 0.40, and 0.50 for Civil, Jigsaw and TweetEval, respectively. This metric shows a higher falsely learned sufficiency score for TweetEval than the two other classifiers, as expected. Based on these observations, we expect TCAV metrics to show lower scores for Civil and Jigsaw than the TweetEval classifier.
Global Sufficiency with Concept-Based Explanations: Here, we use the results obtained with the challenge set as a baseline to evaluate the TCAVbased metrics. Concept examples are generated using the template ‘<protected_group> are <emotion_word>.’, where <protected_group> is one of the protected groups women, trans people, gay people, black people, disabled people, Muslims and immigrants as identified by Röttger et al. (2021). For <emotion_word>, we use the disgust and anger categories of the NRC Emotion Intensity Lexicon (NRC-EIL) (Mohammad, 2018). We use the NLTK package3 to filter out words other than adjectives, past tense verbs and past participles, and also remove the words with emotion intensity lower than
3https://www.nltk.org/
0.5. After these steps, we are left with 368 concept words. We calculate the TCAVdir and TCAVmag scores for the concept of DesNegEm and compare those to the metrics calculated for random concepts with t-test for statistical significance. For random concepts, the concept examples are random tweets collected with stop words. In our implementation of the TCAV procedure, NR = 1000, Nc = 50, Nr = 200 and NC = 386 (number of filtered lexicon words). For input examples, X , we use 2000 tweets collected with stop words.
As presented in Table 2, for the Civil classifier, TCAVdir is not significantly different from the random concept, indicating that the concept information might not always be encoded as a coherent concept in the embedding space of this classifier. However, TCAVmag is significantly higher than random, indicating that when the information is encoded well, the presence of this concept has a significant influence on the label of abuse. The other two classifiers have learned a strong association between the concept and the label, i.e., when the concept is added to a neutral context, the likelihood of the positive label increases. However, only in the case of the TweetEval classifier, TCAVmag is significantly different from the random concepts, indicating a strong influence of the concept on the label, which might override the context. Therefore, for TweetEval the distribution of generated probabilities is mostly determined by the concept, not the context (similar distributions are obtained for the positive and negative examples of the challenge set). The other two classifiers consider the context to some extent and generate relatively different distributions of probabilities for the two classes.
Discussion: For all classifiers, the presence of the concept describing a protected group with negative emotion words is a strong signal for the label of abuse. All classifiers struggle in considering the broader contexts in sentences such as ‘It is not acceptable to say <protected_group> are disgusting.’ Among the three classifiers, TweetEval has learned a higher degree of sufficiency, leading to its worse performance on a challenge set containing this con-
cept. The TCAV metrics can be used to compare the classifiers regarding the false sufficiency relationships they have learned. These metrics provide similar insights to what is learned from assessing global sufficiency with a challenge set.

heading: 5 Global Sufficiency of Fine-Grained Negative Emotions Concepts
text: In the previous section, we considered the concept of describing protected groups with negative emotions, which is tightly related to hate speech, and thus prone to be mistakenly learned as sufficient for the label of abuse. In this section, we test our proposed method for a less obvious case by disentangling the concept of emotions and hate speech. We focus on the concept of describing a (non-protected) group of people with negative emotions, which differs from the previous section in 1) removing the protected groups and replacing them with unprotected groups and 2) breaking down the emotion concept to more fine-grained levels.
For fine-grained emotion concepts, we first develop a compact challenge set, examples of which are presented in Table 3. Since we consider nonprotected groups in this challenge set, the examples are labeled as abusive/non-abusive as opposed to hateful/non-hateful in HateCheck (shown in Table 1). We assess the sufficiency of these concepts with the challenge set first and then compare the results to those of the proposed concept-based explanation metrics. Our goal is to investigate if the findings for the broad concept of describing protected groups with negative emotions can also be replicated at a more nuanced level of emotional granularity. We analyze the models for fine-grained categories of negative emotions, identified by Mohammad (2018), namely disgust, anger, sadness, and fear. Similar pre-processing steps to what was described in Section 4 were performed to filter the lexicon in each category of emotions.
For the challenge set, we write five abusive and five non-abusive example templates for each
emotion. Then we generate 40 abusive and 40 non-abusive examples by replacing <group> with one of the terms Canadians, Chinese people, doctors, teachers, school children, football players, my neighbours, and men to represent non-protected groups.4 Full list of examples of this challenge set is available in our GitHub repository mentioned in Section 1. Equivalently, for the TCAV procedure for concept templates, we use ‘They are <emotion_word>’, instead of ‘<protected_group> are <emotion word>’, which we used in Section 4.

heading: 5.1 Results
text: We first compare the three classifiers in handling negative emotions by investigating the results they produce for the challenge set. The FalseSuff scores in Table 5 show that TweetEval has learned the highest sufficiency between these concepts and the label of abuse and therefore achieves the lowest separability between the positive and negative classes of the challenge set. To further clarify this we show the accuracy vs threshold curve for the disgust category of the challenge set in Figure 5. We observe that TweetEval only reaches high accuracies for a small range of thresholds, i.e, it generates a similar distribution of probabilities for the positive and negative classes that contain the emotion of disgust. On the other hand, Jigsaw has learned the least global sufficiency and reaches high accuracy over a wide range of thresholds.
Then we turn to the TCAV scores shown in Table 4. First, TCAVdir shows that the Civil Comments classifier is not significantly sensitive to negative emotions, i.e., the feature of negative emotions is not fully learned as a coherent feature by this classifier. TweetEval, on the other hand, shows significant TCAVdir and TCAVmag scores, indicating that this classifier is not only sensitive to these concepts but the influence of the concept on
4Though nationality may be considered a protected characteristic in some contexts, we include “Canadian” and “Chinese” here since nationality was not included in HateCheck and therefore not covered in the previous section.
the label is also significantly high. Jigsaw is the classifier that has learned the dependency between negative emotions and the label of abuse and therefore is sensitive to it (as indicated by TCAVdir), but the magnitude of the influence of concept on the label is not significantly high, and the concept is interpreted in the larger context. Interestingly, the magnitude of the influence of disgust and anger is higher than fear and sadness for all classifiers, stating a higher association of disgust and anger with abusive language. These results are in line with conclusions drawn from assessing global sufficiency with a challenge set.

heading: 6 Related Works
text: Most of the explainability works in NLP focus on feature importance methods to measure the importance of an input feature for the prediction at the
local level (Bahdanau et al., 2015; Sundararajan et al., 2017; Ribeiro et al., 2016; Lundberg and Lee, 2017b). However, recent works highlight that models should be assessed beyond feature importance criteria and that the reasoning behind the model’s decisions should be investigated through explainability methods. Some examples of such explainability methods include counterfactual reasoning (Wu et al., 2021; Kaushik et al., 2021; Ribeiro et al., 2020; Ross et al., 2020) or necessity and sufficiency metrics (Balkır et al., 2022; Joshi et al., 2022). Also, there is a need to compare various classifiers at the global level. Although local explanations can be aggregated to generate global explanations, they are usually obtained through costly interventions and are not practical to be applied on a large scale. For global explanations, a popular approach is to train probing classifiers (Conneau et al., 2018). However, probes only identify whether a classifier has learned a feature but stay silent about whether the feature is used in predictions (Belinkov, 2022; Tenney et al., 2019; Rogers et al., 2020). Amnesic probing is an extension of probing classifiers that identifies whether removing a feature influences the model’s predictions, which relates to the notion of the global necessity of a human-understandable concept for a prediction (Ravfogel et al., 2020; Elazar et al., 2021). Our work, on the other hand, focuses on the global sufficiency of concepts. While probing classifiers are applied to linguistic properties such as POS
tagging, which are necessary for accurate language processing, we focus on human-defined semantic concepts that are known to be important for the label and test if they have been falsely learned as a sufficient cause for the label.
Concept-based explanations have been introduced in computer vision and are mostly used to explain image classification models (Graziani et al., 2018; Ghorbani et al., 2019; Yeh et al., 2020). In NLP, concept-based explanations were used to measure the sensitivity of an abusive language classifier to the emerging concept of COVID-related antiAsian hate speech (Nejadgholi et al., 2022b), to assess the fairness of abusive language classifiers in using the concept of sentiment (Nejadgholi et al., 2022a), and to explain a text classifier with reference to the concepts identified through topic modelling (Yeh et al., 2020). To the best of our knowledge, our work is the first that uses concept-based explanations to assess the sufficiency of humandefined concepts in text classification.

heading: 7 Conclusion
text: Concept-based explanations can assess the influence of a concept on a model’s predictions. We used two metrics based on the TCAV method: the TCAV direction score identifies whether the classifier has learned an association between a concept and a label, and the TCAV magnitude score measures the extent of the influence of the concept on the label. We showed that the best-performing abusive language classifiers learned that negative emotion is associated with abuse (positive direction) but did not over-rely on this concept (low magnitude); that is, they did not overestimate the global sufficiency of that concept.
Our method can potentially be used for other NLP classification tasks. This approach is suitable for tasks where certain concepts are closely related to the label, but not enough to make a definitive determination. For example, in sentiment analysis, the price of products may have a strong connection to negative sentiment, but is insufficient to
determine it. Further research should explore how concept-based explanations can help identify cases where certain concepts are relied upon too heavily in abusive language detection or other NLP classification tasks.

heading: 8 Limitations
text: Our work has limitations. First, we use the TCAV framework, which assumes that concepts are encoded in the linear space of semantic representations. However, recent works show that in some cases, linear discriminants are not enough to define the semantic representations of concepts in the embedding spaces (Koh et al., 2020). Future work should consider nonlinear discriminants to accurately represent concepts in the hidden layers of NLP neural networks.
In this study, we used simple challenge sets to obtain a baseline for assessing the effectiveness of concept-based explanations in measuring false global sufficiency. Future work should focus on curating challenge sets by annotating user-generated data for the label and the concepts, in order to achieve a stronger baseline.
Our work is limited to pre-defined concepts and requires human input to define the concepts with examples. However, defining concepts in TCAV is less restrictive than pre-defining features in other explainability methods, in that concepts are abstract ideas that can be defined without requiring in-depth knowledge of the model’s inner workings or the specific features it is using. This allows for a more flexible approach where users can test the model regarding their concept of interest.
Our method can only be applied to concepts that are known to be important for the classifier and are prone to being over-represented in training sets. It’s important to check this condition independently before using our metrics. In cases where this condition does not hold true, the metrics we use in our work may be interpreted differently and may not be reliable indicators of global sufficiency. Also, we only considered two variations of emotion-related concepts. Other variations such as expression of negative emotions by the writer of the post should be investigated in future work.
Further, our metrics are limited to cases where different classifiers are being compared since the most important information is in the relative value of the metrics. Our metrics should not be used as absolute scores for testing a classifier.
Testing a classifier for false causal relationships is most valuable for detecting the potential flaws of the models. If our metrics do not reveal a false relationship between the concept and the label, that should not be interpreted as an indicator of a flawless model.
Ethical Statement
As with most AI technology, this approach can be used adversely to exploit the system’s vulnerabilities and produce toxic texts that would be undetectable by the studied classifier. Specifically, for methods that require access to the model’s inner layers, care should be taken so that only trusted parties could gain such access. The obtained knowledge should only be used for model transparency purposes, and the security concerns should be adequately addressed.
Regarding environmental concerns, contemporary NLP systems based on pre-trained large language models, such as RoBERTa, require significant computational resources to train and fine-tune. Larger training datasets, used for fine-tuning, usually result in better classification performance but also an even higher computational cost. To lower the cost of this study and its negative impact on the environment, we chose to use existing, publicly available classification models.

heading: A Models
text: We include the following publicly available abusive language classification models in this study:
• Jigsaw5: a RoBERTa-based binary toxicity classifier fine-tuned on the combination of two datasets created by Jigsaw and used in
5https://huggingface.co/SkolkovoInsti tute/roberta_toxicity_classifier/tree/ma in
Kaggle competitions on toxicity prediction in 2018-2020. The first dataset, Wikipedia Toxic Comments (Wulczyn et al., 2017), includes 160K comments from Wikipedia talk pages. The second dataset, Civil Comments (Borkan et al., 2019), comprises over 1.8M online comments from news websites. Both datasets are annotated for toxicity (and its subtypes) by crowd-sourcing. The model creators report the AUC of 0.98 and F1-score of 0.76 on the Wikipedia Toxic Comments test set. The model is released under CC BY-NC-SA 4.0.
• Civil Comments6 (Hanu and Unitary team, 2020): a multi-class RoBERTa-based model fine-tuned on the Civil Comments dataset to predict toxicity and six toxicity subtypes (severe toxicity, obscene, threat, insult, identity attack, and sexual explicit). A part of the dataset is annotated for identity groups targeted in toxic comments. The prediction model is trained to optimize the outcome fairness for the groups in addition to the overall accuracy. This is achieved through the loss function that combines the weighted loss functions for two tasks, toxicity prediction and identity prediction (Hanu, 2020).
• TweetEval7 (Barbieri et al., 2020): a RoBERTa-based binary classifier to detect offensive language, released as part of the TweetEval evaluation benchmark. The model was trained on 58M tweets and then finetuned on the Offensive Language Identification Dataset (OLID) (Zampieri et al., 2019). The OLID training set comprises about 12K tweets. The model achieved the macroaveraged F1-score of 77.1 on the OLID test set.
6https://huggingface.co/unitary/unbia sed-toxic-roberta
7https://huggingface.co/cardiffnlp/tw itter-roberta-base-offensive

heading: Future Work
text: Future work should consider nonlinear discriminants to accurately represent concepts in the hidden layers of NLP neural networks. In this study, we used simple challenge sets to obtain a baseline for assessing the effectiveness of concept-based explanations in measuring false global sufficiency. Future work should focus on curating challenge sets by annotating user-generated data for the label and the concepts, in order to achieve a stronger baseline. Our work is limited to pre-defined concepts and requires human input to define the concepts with examples. However, defining concepts in TCAV is less restrictive than pre-defining features in other explainability methods, in that concepts are abstract ideas that can be defined without requiring in-depth knowledge of the model’s inner workings or the specific features it is using. This allows for a more flexible approach where users can test the model regarding their concept of interest. Our method can only be applied to concepts that are known to be important for the classifier and are prone to being over-represented in training sets. It’s important to check this condition independently before using our metrics. In cases where this condition does not hold true, the metrics we use in our work may be interpreted differently and may not be reliable indicators of global sufficiency. Also, we only considered two variations of emotion-related concepts. Other variations such as expression of negative emotions by the writer of the post should be investigated in future work. Further, our metrics are limited to cases where different classifiers are being compared since the most important information is in the relative value of the metrics. Our metrics should not be used as absolute scores for testing a classifier. Testing a classifier for false causal relationships is most valuable for detecting the potential flaws of the models. If our metrics do not reveal a false relationship between the concept and the label, that should not be interpreted as an indicator of a flawless model