SlowGuess/NAACL_2022_OCR huggingface dataset:
Rohit Sridhar
School of Interactive Computing Georgia Institute of Technology rsridhar37@gatech.edu
Diyi Yang
School of Interactive Computing Georgia Institute of Technology dyang888@gatech.edu
Warning: This paper contains content that is offensive and may be upsetting.
Biased or toxic speech can be harmful to various demographic groups. Therefore, it is not only important for models to detect these speech, but to also output explanations of why a given text is toxic. Previous literature has mostly focused on classifying and detecting toxic speech, and existing efforts on explaining stereotypes in toxic speech mainly use standard text generation approaches, resulting in generic and repetitive explanations. Building on these prior works, we introduce a novel knowledge-informed encoder-decoder framework to utilize multiple knowledge sources to generate implications of biased text. Experiments show that our knowledge informed models outperform prior state-of-the-art models significantly, and can generate detailed explanations of stereotypes in toxic speech compared to baselines, both quantitatively and qualitatively.
The toxic speech detection and classification problem has seen increasing interest in recent years. However, it is not only important for AI agents to recognize and classify toxic speech, but to also explain why it is toxic. For instance, debiasing methods that use information about toxic language may benefit from additional information given by detailed explanations of toxicity in text (Ma et al., 2020). Furthermore, detailed explanations of toxicity may facilitate human interaction with toxicity detection systems (Rosenfeld and Richardson, 2019). They can also help humans who work with toxicity classifiers use more information about the input when making decisions about toxic speech. To elucidate, consider the following offensive joke: "What type of punch do you use against a kindergartener? A sandy-hook." While the literal text is not toxic, the implied meaning is offensive, particularly to those affected by school shootings. An AI
agent capable of generating the implied meaning could thus provide additional information to downstream actors. Note that, we use the term biased and toxic interchangeably in this work.
Existing work largely addresses the problem of detecting and classifying toxic speech (Waseem and Hovy, 2016; Founta et al., 2018; Davidson et al., 2017). As mentioned earlier, explanations of toxicity can help with downstream tasks such as debiasing or decision making by humans, thus there has been increasing demand for explainable machine learning classifiers (Ribeiro et al., 2016; Došilović et al., 2018). Recent work around explainable toxicity classification introduced Social Bias Frames (Sap et al., 2020), a formal framework which combines explanations of toxicity along with toxicity classifications along multiple dimensions. However the explanations generated from the current state-of-the-art methods tend to be generic, without much detail. For instance, explanations may focus on certain toxic components of the input but ignore others, or include irrelevant stereotypes about the minority group affected.
To fill this gap, our work proposes to leverage different types of knowledge to provide rich context and background for toxicity explanation. Specifically, we introduce a novel framework to utilize three distinct knowledge sources. Prior work (Yu et al., 2022) divides knowledge broadly into internal and external knowledge, where internal knowledge is knowledge embedded in the input text, and external knowledge is derived from sources outside the input. Building upon these, we leverage expert knowledge that comes from high-quality expert annotations of the input, and explicit knowledge from knowledge graphs and bases, as such symbolic knowledge can provide relevant information to the output text (Yu et al., 2022; Mou et al., 2016). While knowledge graphs and bases deterministically retrieve and restructure knowledge from raw text sources, large pretrained generative models are
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811 - 826 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: MixGEN takes 3 types of knowledge sources and outputs an implied explanation for the input post.
found to be effective in outputting useful knowledge in a probabilistic manner, complementing the expert and explicit knowledge (Razniewski et al., 2021). To this end, we also include implicit knowledge models which source knowledge from large pretrained text generation models. We further build a family of mixture models, MIXGEN, to synthesize knowledge from all three sources, as shown in Figure 1. To sum up, our contributions are twofold: (1) We leverage three different sources of knowledge, and further combine them using simple yet effective mixture models to explain toxic text. (2) We show that our models outperform prior state-of-the-art baselines and generate more detailed explanations.
Prior work on knowledge enhanced text generation (Yu et al., 2020) can be viewed across two different knowledge sources.
2.1 Internal Knowledge
Internal knowledge includes knowledge that is available within the input. For instance, Latent Dirichlet Allocation (LDA) (Blei et al., 2003) can learn topics from inputs, which can then be incorporated into text generation models (Cao et al., 2015; Guo et al., 2020). Keywords may also be extracted from input text using techniques like TF-IDF, PMI or independent classifiers. In one such work, Song et al. (2019) extract emotion oriented keywords using an independent emotion classifier to enhance dialogue generation. Similarly, Mou et al. (2016) use PMI to find relevant keywords for short text conversation. Forbes et al. (2020) develop conceptual formalisms that rely on annotations about the input to generate text. In this work, we denote the use of independent annotations on input to enhance
text generation as expert knowledge, as such annotations often come from human experts.
2.2 External Knowledge
Knowledge graphs and bases are commonly used as a form of external knowledge. Zhang et al. (2019) use knowledge graph embeddings to model conversation flow, while Guan et al. (2020) use triples extracted from knowledge graphs to enhance story generation. Finally, Lian et al. (2019) develop probabilistic mechanisms to select knowledge from knowledge bases for response generation. Knowledge from conventional sources are deterministically created, in that they simply restructure raw text and store them in a knowledge base or graph. We refer to this type of external knowledge as explicit knowledge. On the other hand, there has been increasing interest in the use of large pretrained generative models as a source of knowledge. Heinzerling and Inui (2021) argue that large pretrained models can in fact serve as knowledge bases, while Davison et al. (2019) argue that pretrained models can accurately assess the validity of knowledge mined from raw text. While pretrained models are also trained on raw texts, similar to knowledge bases and graphs, they draw from this knowledge probabilistically and thus are a distinct approach. We denote this type of external knowledge as implicit knowledge.
2.3 Toxic Text Understanding
Prior work around toxicity understanding mainly focuses on detection (Schmidt and Wiegand, 2017). Early approaches include using n-grams (Waseem and Hovy, 2016; Sood et al., 2012; Perera and Fernando, 2021) as well as word clustering (Xiang et al., 2012; Zhong et al., 2016). Recently, knowledge enhanced approaches have also been used for toxicity detection. For instance, Dinakar et al. (2012) use ConceptNet to detect anti-LGBT bullying. The use of meta-information, such as information about the user (Dadvar et al., 2012), has proven to be useful, depending on the type of information used. Sap et al. (2020) use Social Bias Frames to produce both toxicity classifications and explanations of toxicity. Similarly, our approach attempts to explain toxicity by leveraging different sources of knowledge to provide more context and grounding for the models to generate explanations. Different from many prior works, we synthesize these diverse knowledge sources in a unified framework to utilize the unique contribution from each
individual knowledge source.
3 Knowledge Enhanced MIXGEN
This section presents our selected three different types of knowledge—expert knowledge, explicit knowledge and implicit knowledge, and our MIX-GEN models for toxicity explanation.
3.1 Expert Knowledge
Expert knowledge is sourced from annotations of the input. For instance, in the Social Bias Frames dataset (Sap et al., 2020), such expert knowledge include human judgements towards the lewdness, offensiveness, intent to offend, and group targeted categories. This type of expert knowledge provides useful insights and heuristics for the toxicity explanation task, if they are available.
We incorporate expert knowledge into the generation process using the join embedding technique (Pryzant et al., 2020) along with toxicity classification models. The join embedding architecture uses attention weights from the toxicity classifiers to inform the text generation model about parts of the input post relevant to toxicity classification, thus providing a heuristic for the related toxicity explanation task. Formal details of the architecture can be found in Appendix A.1.
We denote these models with the naming convention, "EXPERT [FEATURE]", where "[FEATURE]" is the categorical variable we use for the join embedding. We replace "[FEATURE]" with "ALL" when we train on all features.
3.2 Explicit Knowledge
Explicit knowledge is sourced from some knowledge base or graph. Common sources include ConceptNET, DBpedia, WikiData, etc. (Auer et al., 2007; Vrandecic and Krötzsch, 2014). We opt to use ConceptNet (Speer et al., 2017), since it contains commonsense knowledge (Liu and Singh, 2004). Commonsense knowledge incorporates everyday concepts, especially knowledge regarding social groups and situations.
Following Chang et al. (2020), given a BART model and an input, we extract ranked triples related to the input post and keep the top  $k$  triples per post where we vary the  $k \in \{3, 5, 10, 15, 20, 25\}$ . We experiment with both concatenation and attention based methods to incorporate the top  $k$  triples, but settle on a concatenation based approach due to its simplicity and the lack of performance gains
from the attention based approach. Results and analysis for both the concatenation and attention based approaches are provided in Appendix 12. We denote these models with the naming convention, "EXPLICIT (K)", where K denotes the number of triples used.
3.3 Implicit Knowledge
Implicit knowledge is obtained from some text generator, such as a large pretrained generative model. Prior work such as Heinzerling and Inui (2021), argue that large pretrained models can in fact serve as knowledge bases. Implicit knowledge grants models a probabilistic view of external raw text sources related to a given scenario or input, since generative models tend to generate based on statistical correlations found in their training corpora (Razniewski et al., 2021).
To use implicit knowledge, we first train a BART model to generate the target minority group from the input post. Following Sheng et al. (2019), we use the predicted target minority corresponding to each input post and a set of prompts to induce biased prompt completions from GPT models. We may use multiple prompts per input post, where the number of prompt completions generated is governed by a hyperparameter,  $k$ . Then we train an independent BART model to generate these biased prompt completions, given the origin input post. This BART model is then retrained to produce the implied stereotype, given the input post. Again, we provide a formal description in Appendix A.3.
We denote these models with "IMPLICIT [GPT|GPT-2] (K)", where "GPT" and "GPT-2" correspond to the model used for prompt completion, while  $\kappa$  corresponds to the number of biased prompt completions generated per input post.
3.4 MixGEN Models
We introduce a simple and effective approach to combine all three knowledge sources as input for our MIXGEN family of models, and generate the final stereotype by integrating complementary knowledge from these sources. In our design, we take inspiration from Mixture of Experts models (Masoudnia and Ebrahimpour, 2012), which combine base expert model outputs into a final output using a gating mechanism. Here, we rely on attention mechanisms over the knowledge informed model outputs to serve as the gating mechanism.
We build two variants. The first variant is called MIXGEN CONCAT which uses concatenation to
Type	Description	Pros	Cons
Expert Knowledge	Expert judgements from an-notations on the input.	·Easy-to-use.
·High quality and accurate knowledge.	·Sparse and hard to obtain.
·Difficult to get a lot of diverse knowledge.
Explicit Knowledge	External knowledge sourced from knowledge bases and graphs. Restructured, deterministic interpretations of raw text sources.	·Many knowledge bases and graphs exist.
·Easy to query due to symbolic representation.
·Explainable since knowledge source is known.	·Fixed knowledge, thus limited retrieval diversity.
·Explicitly constructed, thus may not be complete.
Implicit Knowledge	External knowledge sourced from large pretrained models. Probabilistically generated from raw text sources	·Easy to retrieval.
·Probabilistic, thus increasing diversity in retrieval.	·Low explainability.
·Low quality, since the knowledge is implicitly learned.
Table 1: Some pros and cons about different types of knowledge used for toxicity explanation.
Figure 2: The MIXGEN model takes in the output of multiple trained knowledge models, concatenates them with separator tokens and uses the concatenation as input to a BART model to output the explanation. We test with six models, two from each knowledge source.
combine outputs from the knowledge informed models, as shown in Figure 2. The second variant is called MIXGEN MULTIVIEW which uses views to perform self attention over outputs of the knowledge informed models (Chen and Yang, 2020). Since the BART model already uses self attention over input tokens, we experiment with the MultiView architecture to see whether the additional self attention mechanisms of the MultiView model causes changes in performance.
For MIXGEN CONCAT, suppose we have  $k$  trained models,  $M_1, \ldots, M_k$ , each trained to produce the implied stereotype given the input post, and each informed by one of the aforementioned knowledge types. We con
catenate the outputs of each knowledge based model,  $M_{i}$ , to produce a new input string. Thus if each model  $M_{i}$  outputs "s[OUT_I]", we get the following concatenated input string: "s[OUT_1][SEP]s[OUT_2]...[SEP]s[OUT_K]". Now, let  $M$  be a standard pretrained BART model. We train model  $M$  to produce the implied stereotype using "s[OUT_1][SEP]s[OUT_2]...[SEP]s[OUT_K]" as input. Note that the knowledge based models,  $M_{1},\ldots ,M_{k}$  are fixed when training  $M$ . Model  $M$  serves as the final MIXGEN CONCAT model.
MIXGEN MULTIVIEW uses the MultiView architecture proposed by Chen and Yang (2020). In this case, the outputs of  $M_1, \ldots, M_k$  are treated as separate views. If each model  $M_i$  outputs "s[OUT_I]" given the input post, then for each model  $M_i$  we configure the corresponding view as the string "v1i s[OUT_I][SEP]...v2i s[OUT_I]v3i... [SEP]s[OUT_K]", where v1i, v2i, and v3i are view tokens. Here, v1i is always the first token in the view string and v2i and v3i surround  $M_i$ 's output. We configure  $k$  such views (one for each model) and pass each into the BART MultiView model as a set of views corresponding to the original input post. The BART MultiView model is then trained to produce the corresponding output stereotype. For details on the MultiView architecture, please see Chen and Yang (2020).
We utilize the BART encoder decoder framework throughout (Lewis et al., 2020). We use batch gradient descent when training. For a batch  $B$  with padded input sequences  $X_{i}$  of length  $N_{s}$  and corresponding padded target sequences  $Y_{i}$  of length  $N_{t}$ , along with knowledge  $K_{i}$  from some knowledge
Dataset	# Posts	Annotations/Post
SBIC Train	35933	3.14
SBIC Dev	4680	3.58
SBIC Test	4705	3.72
Implicit Hate Train	5722	1
Implicit Hate Test	636	1
source, we minimize cross entropy loss:
$$
\begin{array}{l} \mathcal {L} = - \frac {1}{| B | N _ {t}}. \\ \sum_ {i = 1} ^ {| B |} \sum_ {j = 1} ^ {N _ {t}} \log p \left(Y _ {i j} \mid Y _ {i (1: j - 1)}, X _ {i}, K _ {i}\right) \tag {1} \\ \end{array}
$$
We conduct our experiments on the SBIC dataset (Sap et al., 2020) and the Implicit Hate dataset (ElSherief et al., 2021). The SBIC dataset contains an input post, toxicity annotations and free text annotations of the implied stereotype. We work with the input post and the the implied stereotype. The Implicit Hate dataset (ElSherief et al., 2021) contains free text annotations of the implied stereotype. Dataset statistics are provided in Table 2.
We compare our models with BART, and state of the art baselines from Sap et al. (2020):
- GPT: Following Sap et al. (2020), we train the GPT pretrained model from huggingface to generate the toxicity classifications, the Target Minority, and the Implied Stereotype as a string, when prompted with the input post.
- GPT-2: We train with the same setting as the GPT Baseline, but use the GPT-2 pretrained model from huggingface.
- BART: We train a standard pretrained BART model to generate the implied stereotype when given the input post.
We use BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004) and BERTScore (Zhang* et al., 2020) to evaluate our models and take the maximum score for each hypothesis over all of the corresponding references. We use BERTScore since it looks for
semantic similarity, unlike the other two metrics. While we acknowledge it as a limitation, we ultimately do not use human evaluation for multiple reasons. First, the generated stereotypes are minimal in length compared to other text generation tasks. Moreover, we perform manual analyses of the results in Section 5. Second, we are following precedent set by prior work (Sap et al., 2020). Finally, we want to minimize annotator exposure to harmful content.
4.4 Results on SBIC
Our results on both dev and test are described in Table 3. Here we focus on dev since both sets of results track similar trends. We observe that the MIXGEN models outperform all other models. After MIXGEN, the model using Implicit Knowledge sources perform best. These are followed by the model using Explicit Knowledge, in turn followed by model using Expert Knowledge.
Both models with explicit and with implicit knowledge outperform expert language models. The models using implicit knowledge tend to perform best overall (BLEU: from 0.650 to 0.683, ROUGE-L: from 0.624 to 0.659, BERTScore: from 0.759 to 0.800). This is likely because implicit knowledge is less structured and hence easier to induce bias from (Petroni et al., 2019). On the other hand, while our source (ConceptNET) for explicit knowledge may be biased (Mehrabi et al., 2021), retrieved stereotypes are often mixed with general, unbiased facts.
The MIXGEN models outperform every other model. This makes intuitive sense since MIXGEN synthesizes multiple types of knowledge. Unexpectedly, MIXGEN MULTIVIEW model does not improve performance (the absolute difference is within 0.002 across all scores) over MIXGEN CONCAT. This is likely due to the fact that the MultiView model was intended to capture meta-sequences in text (Chen and Yang, 2020), whereas in our setting the input is not sequential. We also note that the MIXGEN models perform better than the source models, despite their input being sourced from the source models. Thus, despite differing performance, models from different knowledge sources are likely providing some distinct and complementary information.
4.5 Results on Implicit Hate Speech Corpus
Results on the implicit hate corpus (ElSherief et al., 2021) are given in Table 4. All models (including
Model	dev	test
BLEU	ROUGE-L	BERTScore	BLEU	ROUGE-L	BERTScore
GPT	0.597	0.579	0.712	0.591	0.574	0.713
GPT-2	0.617	0.601	0.733	0.620	0.605	0.741
BART Base	0.495	0.467	0.624	0.505	0.476	0.638
EXPERT (GROUP)	0.630*	0.604*	0.765*	0.637**	0.608*	0.776*
EXPLICIT (20)	0.650**	0.624**	0.770**	0.645**	0.617**	0.773**
IMPLICIT GPT-2 (15)	0.683**	0.659**	0.800**	0.689**	0.662**	0.811**
MIXGEN CONCAT	0.692**	0.665**	0.807**	0.696**	0.669**	0.817**
MIXGEN MULTIVIEW	0.691**	0.664**	0.806**	0.694**	0.666**	0.816**
Table 3: We report performance of baseline models (first three rows) and our representative models from each knowledge source. A superscript of * indicates statistically significant (p-value  $< 0.05$ ) improvements over the GPT and BART baselines, while a superscript of ** indicates statistically significant improvements over all baselines. We use Wilcoxon's signed rank test with a one-sided alternative hypothesis to compute p values (Wilcoxon, 1992).
Model	BLEU	ROUGE-L	BERTS.
BART Base	0.460	0.323	0.909
Exp. (Grp)	0.404	0.234	0.894
Expl. (20)	0.463*	0.327*	0.909
Impl. GPT-2 (15)	0.463*	0.331*	0.910*
MixGEN C	0.467*	0.340*	0.912*
MixGEN MV	0.459	0.325*	0.910*
Table 4: This table shows the performance of our models on the implicit hate corpus (ElSherief et al., 2021). BERTS. stands for BERTScore. A superscript of * indicates statistically significant (p value  $< 0.05$ ) improvements over the BART baseline.
bases) generally perform worse than they do on the SBIC dataset. This is likely because the implicit hate corpus contains one reference per post, in contrast to the SBIC dataset (see Table 2). While the Expert knowledge model performs worse than the baseline, the other models perform slightly better. This is likely because the Expert model relies on toxicity classifications, which weren't available in the implicit hate corpus. We believe our models can be generalized to text generation tasks on other datasets, but they likely need multiple reference points where the implicit hate corpus only has one.
5 Error Analysis and Ablation Studies
We perform analyses and ablation studies on model results on the SBIC dataset. We do not perform these on the implicit hate dataset, since we have too few references per example. Examples of the error and challenge types below are given in Table 5. An additional full set of examples for each error and challenge type is given in Appendix 13.
Figure 3: Distribution on the four error types across knowledge types and baselines.
We categorize the types of errors made by models on a small sample of 200 examples from the dev set and provide the distributions in Figure 3. We provide the error categories below.
1. Non-Existent Stereotype: Model generates a stereotype when the reference stereotype is an empty string.
2. Ignores Stereotype: Model does not generate a stereotype when the reference stereotype is a non-empty string.
3. Incorrect Target Minority: Model uses the incorrect target minority.
4. Incorrect Stereotype: Model uses the correct target minority but generates an incorrect or overly general stereotype.
Figure 4: Distribution of challenges across knowledge types and baselines. We leave out cases of where models detect non-existent stereotypes from challenge type 2.
The baseline GPT models tend to make more errors of every type, except that the expert model makes more errors of type 4. The expert model likely focuses on tokens that trigger toxicity classifications, which makes it less likely to focus on other relevant tokens. In the second example of Table 5, the token "black" may be triggering the Expert Model, causing an error of type 4.
The knowledge enhanced models rarely fail to generate a stereotype (type 2 error). On the other hand, they often detect non-existent stereotypes (type 1 error), but less often than the baselines. In the third example of Table 5, "contraceptive" may be incorrectly triggering the implicit and explicit models, while the expert model is not triggered.
5.2 Challenges in Stereotype Generation