SlowGuess/NAACL_2022_OCR huggingface dataset:
We categorize the various challenges faced by our text generation models and provide distributions in Figure 4 and counts in Appendix 15. We analyze the same sample of 200 examples from Section 5.1.
1. Misundersands Post: The model fundamentally misunderstands the post and generates an irrelevant stereotype as a result.
2. High Sensitivity: The model is highly sensitive to trigger words, which causes it to detect non-existent stereotypes or generate stereotypes based on the triggers alone.
3. Localized Generation: The model focuses only on parts of the input and generates stereotypes based on those parts, rather than on the entire input post.
4. Does not Draw Connections: The model clearly considers the entire input post, but does not draw connections between the different parts of the post.
5. Misundersands Sarcasm or Irony: The model takes a more literal interpretation of a sarcastic or ironic post, causing it to output text that has the opposite meaning of the reference stereotype.
6. Ignores Stereotype: The model does not generate a stereotype despite a non-empty reference stereotype.
Interestingly, the MIXGEN model tends to misunderstand sarcasm and irony at a slightly higher rate than the other knowledge model types. In the fourth example of 5, MIXGEN and the Implicit Model take literal interpretations of the input post. The Implicit Model type has difficulty with drawing connections over the input (challenge type 4). An example is given in the 6th row of Table 5, where the model does not draw a connection between the target minority and the stereotypes present.
5.3 How MIXGEN Synthesizes Knowledge
Table 5 provides examples of MIXGEN synthesizing knowledge across sources. In the third example, MIXGEN produces an empty string even though two of the more reliable sources (explicit and implicit knowledge) produce non empty strings. In the fifth example, MIXGEN clearly combines parts of all the knowledge sources, while in the sixth example MIXGEN produces a more accurate stereotype than any of the other models. Thus, it seems that MIXGEN does not simply attempt to copy the correct model, but actually possess a deeper understanding of the knowledge types it synthesizes.
5.4 Implicit Knowledge Ablation Study
Table 6 contains results of ablations on the Implicit Knowledge models. We vary the amount  $(k)$  and source (GPT and GPT-2) of implicit knowledge. The model which sources knowledge using GPT only outperforms the model sourcing knowledge using GPT-2 when  $k = 3$ . When  $k = 15$ , the latter model sourcing GPT-2 knowledge outperforms the model sourcing knowledge from GPT. It is possible that GPT-2 is less biased than GPT, thus benefiting our model for low  $k$ , but that these benefits decrease as  $k$  increases. With greater  $k$ , both models have a greater chance of exhibiting bias.
Input Post	Target Stereotype	Expert	Explicit	Implicit	MIXGEN
what's the best part about being a midget pedophile? your dick can fit.	can fit inside a per-son sexually	mentally dis-abled folks are incompetent (3,1)	midgets are gross (4,2)	midgets have small penises (-,-)	midgets have small penises (-,-)
what's the most perfect thing? a rainbow, because it has no black on it.	black people are worth less	black people are criminals (4,2)	black people are inferior (-,-)	black people are inferior (-,-)	black people are inferior (-,-)
when we finally get the male contraceptive pill, it should be red.	“”	“” (-,-)	women are promiscuous (1,2)	women are promiscuous (1,2)	“” (-,-)
what do a dull blade and a lethargic emo have in common? they both don't cut like they used to	mentally ill people cut themselves	makes light of child molestation (3,1)	mentally disabled folks aren't smart (4,1)	emos don't cut (4,5)	physically disabled folks are useless (3,5)
what do you call two anorexic lesbians with yeast infections: double quarter pounder with cheese	lesbians are dirty	lesbians can't drive (4,2)	anorexics are gross (4,3)	anorexics are gross (4,3)	lesbians are gross (-,-)
what's can a white man say that a black man can't? “thanks offi-cer!” and “hey dad!”	black folks are criminals,black fathers are not present in childrens lives	black people are criminals (-,-)	black people don't have fathers (-,-)	black people can't be trusted (4,4)	black men abandon their kids (-,-)
Table 5: Examples of generation from MIXGEN. The Error Type and the Challenge Type are given in parenthesis, " (error type number, challenge type number.)", with a “(-,-)” signifying no error.
Model	dev	test
GPT	0.712	0.713
GPT-2	0.733	0.741
BART Base	0.624	0.638
IMPLICIT GPT (3)	0.787	0.796
IMPLICIT GPT (15)	0.796	0.807
IMPLICIT GPT-2 (3)	0.783	0.795
IMPLICIT GPT-2 (15)	0.800	0.811
5.5 MixGEN CONCAT Ablation Study
Table 6: BERTScores of baseline models and the implicit knowledge models.
Model	dev	test
GPT	0.712	0.713
GPT-2	0.733	0.741
BART Base	0.624	0.638
MixGEN CONCAT (3)	0.803	0.814
MixGEN CONCAT (6)	0.807	0.817
MixGEN CONCAT (9)	0.799	0.804
Table 7: BERTScores of baselines and our MIXGEN CONCAT models. The value in parentheses indicates the number of knowledge informed models used.
In Table 7, we look at ablations on the number of knowledge informed models, for the MIXGEN CONCAT model. Let  $k$  be the number of knowledge informed models. The variant using  $k = 6$  models performs best. Since the models variants
within each knowledge type only vary by some hyperparameter, information gain probably saturates as the number of models increases; however performance decreases when  $k = 9$ . Since we tend to include models with lower performance for larger  $k$ , worse performing models are likely counter productive.
We discuss limitations of our study here. Per the discussion in Subsection 4.3, we do not perform human evaluation of our results for multiple reasons. We do believe this is a limitation, and think future work may benefit from some form of human evaluation, while mitigating some of the concerns mentioned in Subsection 4.3.
The MIXGEN model requires significant computational power. One needs to train models across knowledge sources, and then train the MIXGEN model itself. Future work may alleviate this burden by considering end to end solutions, or more efficient knowledge retrieval techniques.
Future work could perform an "in the wild" analysis, produced by procuring random comments from the internet and running our proposed models on these comments, to determine how effective the models might be in a real world setting. Further ablations may also provide insight. For instance, with respect to our implicit knowledge models, it
may be helpful to remove BART altogether and instead use GPT for predicting the target minority, pretraining on implicit knowledge and for generating the final stereotype. Finally, it may be helpful to standardize the incorporation of knowledge into the models, such that the models using different knowledge types may be directly compared.
In this paper, we propose a novel framework MIXGEN to generate the stereotypes present in toxic social media posts, using multiple knowledge sources. We categorized three different sources of knowledge and synthesize the sources of knowledge using the MIXGEN models. While the knowledge models perform as well as baselines, models built on different knowledge types vary in strengths and weaknesses. For instance, the expert model suffers from high sensitivity to trigger words, while the implicit models may not draw connections over complex inputs. The MIXGEN models takes this into account and minimizes the number of examples on which it has errors and/or faces challenges. We conclude that mixture and ensemble methods as simple as concatenation can leverage the complementary nature of distinct knowledge sources to produce high quality text generations.
Models such as the one proposed in this paper, which output toxicity classifications of text or speech and reasoning behind such classifications should be used with care. Considerations of algorithmic fairness should be taken into account (Corbett-Davies et al., 2017), as well as cultural differences (Oliva et al., 2020) and racial biases (Xia et al., 2020) which can lead to misclassifications of offensiveness. Care should be taken to avoid political bias in training datasets, when training these models for deployment purposes (Wich et al., 2020). Finally, concerns about censorship should be taken seriously (Heins, 2014).
Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In *The Semantic Web*, pages 722-735, Berlin, Heidelberg. Springer Berlin Heidelberg.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan.
2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3(null):993-1022.
Ziqiang Cao, Sujian Li, Yang Liu, Wenjie Li, and Heng Ji. 2015. A novel neural topic model and its supervised extension. In AAAI.
Ting-Yun Chang, Yang Liu, Karthik Gopalakrishnan, Behnam Hedayatnia, Pei Zhou, and Dilek Hakkani-Tur. 2020. Incorporating commonsense knowledge graph in pretrained models for social commonsense tasks. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 74-79, Online. Association for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118, Online. Association for Computational Linguistics.
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 797-806, New York, NY, USA. Association for Computing Machinery.
M. Dadvar, F.M.G. de Jong, R. Ordelman, and D. Trieschnigg. 2012. Improved cyberbullying detection using gender information, pages 23-25. Universiteit Gent, Belgium. Reporting year: 2012.
Thomas Davidson, Dana Warmsley, M. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173-1178, Hong Kong, China. Association for Computational Linguistics.
Karthik Dinakar, Birago Jones, Catherine Havasi, Henry Lieberman, and Rosalind Picard. 2012. Common sense reasoning for detection, prevention, and mitigation of cyberbullying. ACM Trans. Interact. Intell. Syst., 2(3).
Filip Karlo Dosilović, Mario Brčić, and Nikica Hlupić. 2018. Explainable artificial intelligence: A survey. In 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pages 0210-0215.
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653-670, Online. Association for Computational Linguistics.
Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. Proceedings of the International AAAI Conference on Web and Social Media, 12(1).
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation. Transactions of the Association for Computational Linguistics, 8:93-108.
Dandan Guo, Bo Chen, Ruiying Lu, and Mingyuan Zhou. 2020. Recurrent hierarchical topic-guided neural language models.
M. Heins. 2014. The brave new world of social media censorship. Harvard Law Review, 127:325-330.
Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1772-1791, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. CoRR, abs/1902.04911.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In ACL 2004.
H. Liu and P. Singh. 2004. Conceptnet: A practical commonsense reasoning toolkit. BT Technology Journal, 22(4):211-226.
Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. 2020. PowerTransformer: Unsupervised controllable revision for biased language correction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7426-7441, Online. Association for Computational Linguistics.
Saeed Masoudnia and Reza Ebrahimpour. 2012. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275-293.
Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5016-5033, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lili Mou, Yiping Song, Rui Yan, Ge Li, L. Zhang, and Zhi Jin. 2016. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. In COLING.
Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2020. Fighting hate speech, silencing drag queens? artificial intelligence in content moderation and risks to lgbtq voices online. Sexuality & Culture, 25:700-732.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Patrick von Platen. 2020. How to generate text: using different decoding methods for language generation with transformers. https://huggingface.co/blog/how-to-generate. Accessed: 2020-06-27.
Andrea Perera and Pumudu Fernando. 2021. Accurate cyberbullying detection and prevention on social media. Procedia Computer Science, 181:605-611. CENTERIS 2020 - International Conference on ENTERPRISE Information Systems / ProjMAN 2020 - International Conference on Project MANagement / HCist 2020 - International Conference on Health and Social Care Information Systems and Technologies 2020, CENTERIS/ProjMAN/HCist 2020.
Fabio Petroni, Tim Rocktäschel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? CoRR, abs/1909.01066.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, S. Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically neutralizing subjective bias in text. In AAAI.
Simon Razniewski, Andrew Yates, Nora Kassner, and Gerhard Weikum. 2021. Language models as or for knowledge bases.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 1135-1144, New York, NY, USA. Association for Computing Machinery.
Robyn Speer. 2019. Relations in conceptnet5. https://github.com/commonsense/conceptnet5/wiki/Relations. Accessed: 2020-06-20.
A. Rosenfeld and A. Richardson. 2019. Explainability in human-agent systems. Autonomous Agents and Multi-Agent Systems, pages 1-33.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490, Online. Association for Computational Linguistics.
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1-10, Valencia, Spain. Association for Computational Linguistics.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In EMNLP (short).
Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, and Xuanjing Huang. 2019. Generating responses with a specific emotion in dialog. In ACL.
Sara Owsley Sood, Judd Antin, and Elizabeth F. Churchill. 2012. Using crowdsourcing to improve profanity detection. In AAAI Spring Symposium: Wisdom of the Crowd.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 4444-4451. AAAI Press.
Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In NAACL.
Maximilian Wich, Jan Bauer, and Georg Groh. 2020. Impact of politically biased data on hate speech classification. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 54-64, Online. Association for Computational Linguistics.
Frank Wilcoxon. 1992. Individual Comparisons by Ranking Methods, pages 196-202. Springer New York, New York, NY.
Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 7-14, Online. Association for Computational Linguistics.
Guang Xiang, Bin Fan, Ling Wang, Jason Hong, and Carolyn Rose. 2012. Detecting offensive tweets via topical feature discovery over a large scale twitter corpus. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM '12, page 1980-1984, New York, NY, USA. Association for Computing Machinery.
W. Yu, Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2020. A survey of knowledge-enhanced text generation. ArXiv, abs/2010.04389.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation. ACM Comput. Surv. Just Accepted.
Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2019. Conversation generation with concept flow. CoRR, abs/1911.02707.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
Haoti Zhong, Hao Li, Anna Squicciarini, Sarah Rajt-majer, Christopher Griffin, David Miller, and Cornelia Caragea. 2016. Content-driven detection of cyberbullying on the instagram social network. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16, page 3952-3958. AAAI Press.
A Formal Details for Knowledge Incorporation
A.1 Expert Knowledge
A.1.1 Incorporating Expert Knowledge using the Join Embedding Technique
Given the BERT classifier with  $m$  attention heads, an input with length  $n$ , and a BART model with hidden dimension  $d$ , we pass the input to the BERT classifier to retrieve the attention head outputs of the last layer, namely  $a_1, \ldots, a_m$ , each in  $\mathbb{R}^n$ . We
also pass the input to the BART model and retrieve the BART encoded input, namely  $H \in \mathbb{R}^{n \times d}$ . Let  $v_{1}, \ldots, v_{m}$  in  $\mathbb{R}^{d}$  be trainable weight vectors. For the  $j$ -th row vector,  $h_{j}$  of  $H$ , we compute an enriched hidden state  $h_{j}'$  as follows:
$$
\left(h _ {j} ^ {\prime}\right) ^ {T} = h _ {j} ^ {T} + \sum_ {i = 1} ^ {m} a _ {i j} v _ {i} \tag {2}
$$
We then pass the enriched hidden state through the BART decoder to generate the output stereotype. To combine knowledge from multiple variables, we sum the enriched hidden state in Equation (2) over each variable.
A.2 Explicit Knowledge
A.2.1 Incorporating Explicit Knowledge using Concatenation
In this section, we provide formal details on how we incorporate explicit knowledge from ConceptNET using concatenation. In order to retrieve the top  $k$  triples (varying  $k \in \{3,5,10,15,20,25\}$ ), we first extract nouns, verbs, and adjectives from the input as our query tokens. We then query ConceptNet for triples associated with the query tokens and sort the triples by the product of the query's IDF weight and the triple's edge weight.
We then translate the triples into English (Robyn Speer, 2019). For example, if the entities "car" and "vehicle" are connected by the edge relation "IsA", the translation would be "Car is a vehicle". We concatenate the translations to the input post to form a new input. Formally, let the input be " $s_{\mathrm{post}}$ " and " $s_{[\mathrm{trp\_i}]}$ " be the sentence derived from the  $i$ -th triple. The modified input is then " $s_{\mathrm{post}}[SEP]s_{[\mathrm{trp\_1}]}[SEP]\cdots[SEP]s_{[\mathrm{trp\_k}]}$ ". We then pass the modified to the BART model which generates the implied stereotype.
The concatenation based approach allows the model to encode the external knowledge into its own embedding space.
A.2.2 Incorporating Explicit Knowledge using Attention
In this section, we provide formal details on how we incorporate explicit knowledge from ConceptNET using attention and the fusion layer described in Chang et al. (2020). This is an alternative method we tried in addition to the method described in Section A.2.1.
We first accumulate the top  $k$  triples (varying  $k\in \{5,10,20\}$ ) associated with a post, using the
method described in Appendix A.2.1. We then concatenate the numberbatch embeddings (each of dimension  $p$ ) of the two entities in each of the  $k$  triples vertically to produce a vector in  $\mathbb{R}^{2p}$ . We then horizontally stack the concatenations to produce a matrix,  $H_{KG} \in \mathbb{R}^{k \times 2p}$ . The encoded input generated by the BART model can be represented by the matrix  $H_B \in \mathbb{R}^{n \times d}$ , where  $n$  is the input length and  $d$  is the hidden size of the BART model. We compute knowledge aware attention over the input as follows:
$$
Q = H _ {B} * W _ {1} + b _ {1} ^ {T} \tag {3}
$$
$$
K = H _ {K G} * W _ {2} + b _ {2} ^ {T} \tag {4}
$$
$$
V = H _ {K G} * W _ {3} + b _ {3} ^ {T} \tag {5}
$$
$$
H _ {K G} ^ {B} = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d}} V\right) \tag {6}
$$
where  $W_{1}\in \mathbb{R}^{d\times d},W_{2},W_{3}\in \mathbb{R}^{2p\times d}$  and  $b_{1},b_{2},b_{3}\in \mathbb{R}^{d}$ . The knowledge aware matrix is  $H_{KG}^{B}\in \mathbb{R}^{n\times d}$ . Finally, we concatenate the original encoded input and the knowledge aware matrix,  $H_{KG}^{B}$  and perform an affine transformation:
$$
H ^ {\prime} = \left(H _ {B} \oplus H _ {K G} ^ {B}\right) * W _ {4} + b _ {4} ^ {T} \tag {7}
$$