SlowGuess/ACL_2020_OCR huggingface dataset:
We evaluate performance of our models in the following ways. For classification, we report precision, recall, and  $F_{1}$  scores of the positive class. Following previous generative inference work (Sap et al., 2019b), we use automated metrics to evaluate model generations. We use BLEU-2 and RougeL ( $F_{1}$ ) scores to capture word overlap between the generated inference and the references, which captures quality of generation (Galley et al., 2015; Hashimoto et al., 2019). We additionally compute word mover's distance (WMD; Kusner et al., 2015), which uses distributed word representations to measure similarity between the generated and target text.[11]
As each post can contain multiple annotations, we define a training instance as containing one post-group-statement triple (along with the five categorical annotations). We then split our dataset into train/dev./test (75:12.5:12.5), ensuring that no post is present in multiple splits. For evaluation (dev., test), we combine the categorical variables by averaging their binarized values and re-binarizing using a .5 threshold, and compare the generated
inferences (hypotheses) to all targeted groups and implied statements (references).
All experiments are carried out using Hugging-Face's Transformers library.12 We tune hyperparameters on the dev. set, and report performance for the best performing setting (according to average  $F_{1}$ ). We train or finetune our models using a batch size of 4, a learning rate of  $5 \times 10^{-6}$  for GPT and  $10^{-5}$  for GPT2 (both with linear warm up), and consider training for  $e \in \{1,2,5\}$  epochs.
Listed in Tables 4 and 5, our modelling results indicate that making inferences about social biases in language is challenging for these models.
Classification Shown in Table 4, models perform well on higher-level variables such as offensiveness and lewdness, despite the latter being heavily skewed. We hypothesize that correctly predicting lewdness might require more lexical matching (e.g., detecting words with sexual connotations).
Whether a group is targeted is slightly less easy for models to predict, and whether the language is in-group is even more challenging, with most of the models defaulting to never predicting it. This highly skewed category poses a challenge for all models, likely due to subtlety of the task and the lack of positive instances.
$\mathrm{SBF - GPT_2}$  gdy is the only model that predicts positive values for in-group language, for which it benefits from constrained decoding with a  $1.9\%$  improvement in  $F_{1}$  score (we show results with all constrained decoding variants in Table 7 in the appendix).
$^{12}$ https://github.com/huggingface/transformers
<sup>9</sup>We only use the possible assignments in the same forward pass; we do not use assignments from different samples.
10 In practice, as seen in Tables 4, 5, and 7, this only slightly improves predictions.
<sup>11</sup>We use GloVe trained on CommonCrawl, as part of the SpaCy en_core_web_md package.
5483
post	predicted group	predicted implication	reference groups	reference implications
(a) All-female casts bother me but I have literally never noticed the hun-dreds of movies with 0 female char-acters I have already watched	—	—	women	women aren't good actors; women can't be in leading roles
(b) As expected, when the terrorist group Hamas won the election in Gaza it was the usual one man, one vote, one time, Islamist election. #Islam	muslim folks	muslims are terrorists	muslim folks; arabic folks	bad people; islam promotes control-ling governments; muslims are au-thoritarians; not fair
(c) "Black guy in class: *attempts to throw a paper ball into the trash* *misses* Mrs. Mundy: "You're a disgrace to your race, Marcus"	black folks	black people are trash	black folks	are good at basketball; black men are defined by athletic skill.; black men should be good at basketball.; do not take things seriously
(d) How can a Libtard speak truth to power when the only thing they vomit from their mouths is lies?	liberals	liberals stupid	liberals	liberals are incompetent; liberals are stupid; liberals only speak lies
(e) I hate fat bitches	overweight/fat folks	fat people are ugly	overweight/fat folks	are not pleasant people; fat folks are all the same; fat folks are less than others; not likable
Table 6: Examples of SBF-GPT $_2$ -gdy-constr model predictions. The model struggles to pick up on subtle biases (a), and tends to generate generic stereotypes rather than implications that are entailed by the post (b, c).
Generation When evaluating our models on the generation tasks (i.e., targeted group and implied statement), we find that no one model outperforms others across all metrics (Table 5).
Overall, models do well at generating the targeted groups, likely because of the more limited generation space (there are only 1.4k possible groups in SBIC). Conversely, for implied statement generation (where output space is much larger), model performance is slightly worse.
Similar to the classification tasks, SBF-GPT $_2$ -gdy shows a slight increase in RougeL score when using constrained decoding, but we see a slight drop in BLEU scores.
Error analysis Since small differences in automated evaluation metrics for text generation sometimes only weakly correlate with human judgments (Liu et al., 2016), we manually perform an error analysis on a manually selected set of generated development-set examples from the SBF-GPT $_2$ -gdy-constr model (Table 6). Overall, the model seems to struggle with generating textual implications that are relevant to the post, instead generating very generic stereotypes about the demographic groups (e.g., in examples b and c). The model generates the correct stereotypes when there is high lexical overlap with the post (e.g., examples d and e). This is in line with previous research showing that large language models rely on correlational patterns in data (Sap et al., 2019c; Sakaguchi et al., 2020).
Bias and toxicity detection Detection of hateful, abusive, or other toxic language has received increased attention recently (Schmidt and Wiegand, 2017), and most dataset creation work has cast this detection problem as binary classification (Waseem and Hovy, 2016; Davidson et al., 2017; Founta et al., 2018). Moving beyond a single binary label, Wulczyn et al. (2017) and the PerspectiveAPI use a set of binary variables to annotate Wikipedia comments for several toxicity-related categories (e.g., identity attack, profanity). Similarly, Zampieri et al. (2019) hierarchically annotate a dataset of tweets with offensiveness and whether a group or individual is targeted. Most related to our work, Ousidhoum et al. (2019) create a multilingual dataset of 13k tweets annotated for five different emotion- and toxicity-related aspects, including a 16-class variable representing social groups targeted. In comparison, SOCIAL BIAS FRAMES not only captures binary toxicity and hierarchical information about whether a group is targeted, but also free-text implications about 1.4k different targeted groups and the implied harm behind statements.
Similar in spirit to this paper, recent work has tackled more subtle bias in language, such as microaggressions (Breitfeller et al., 2019) and condescension (Wang and Potts, 2019). These types of biases are in line with the biases covered by Social Bias Frames, but more narrowly scoped.
Inference about social dynamics Various work has tackled the task of making inferences about power and social dynamics. Particularly, previous work has analyzed power dynamics about specific entities, either in conversation settings (Prabhakaran et al., 2014; Danescu-Niculescu-Mizil et al., 2012) or in narrative text (Sap et al., 2017; Field et al., 2019; Antoniak et al., 2019). Additionally, recent work in commonsense inference has focused on mental states of participants of a situation (e.g., Rashkin et al., 2018; Sap et al., 2019b). In contrast to reasoning about particular individuals, our work focuses on biased implications of social and demographic groups as a whole.
Risks in deployment Automatic detection of offensiveness or reasoning about harmful implications of language should be done with care. When deploying such algorithms, ethical aspects should be considered including which performance metric should be optimized (Corbett-Davies et al., 2017), as well as the fairness of the model on speech by different demographic groups or in different varieties of English (Mitchell et al., 2019). Additionally, deployment of such technology should discuss potential nefarious side effects, such as censorship (Ullmann and Tomalin, 2019) and dialect-based racial bias (Sap et al., 2019a; Davidson et al., 2019). Finally, offensiveness could be paired with promotions of positive online interactions, such as emphasis of community standards (Does et al., 2011) or counter-speech (Chung et al., 2019; Qian et al., 2019).
Risks in annotation Recent work has highlighted various negative side effects caused by annotating potentially abusive or harmful content (e.g., acute stress; Roberts, 2016). We mitigated these by limiting the number of posts that one worker could annotate in one day, paying workers above minimum wage ($7-12), and providing crisis management resources to our annotators. [13] Additionally, we acknowledge the implications of using data available on public forums for research (Zimmer, 2018) and urge researchers and practitioners to respect the privacy of the authors of posts in SBIC (Ayers et al., 2018).
To help machines reason about and account for societal biases, we introduce SOCIAL BIAS FRAMES, a new structured commonsense formalism that distills knowledge about the biased implications of language. Our frames combine categorical knowledge about the offensiveness, intent, and targets of statements, as well as free-text inferences about which groups are targeted and biased implications or stereotypes. We collect a new dataset of 150k annotations on social media posts using a new crowdsourcing framework and establish baseline performance of models built on top of large pretrained language models. We show that while classifying the offensiveness of statements is easier, current models struggle to generate relevant social bias inferences, especially when implications have low lexical overlap with posts. This indicates that more sophisticated models are required for SOCIAL BIAS FRAMES inferences.
We thank the anonymous reviewers for their insightful comments. Additionally, we are grateful to Hannah Rashkin, Lucy Lin, Jesse Dodge, Hao Peng, and other members of the UW NLP community for their helpful comments on the project. This research was supported in part by NSF (IIS-1524371, IIS-1714566), DARPA under the CwC program through the ARO (W911NF-15-1-0543), and DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031).
Maria Antoniak, David Mimno, and Karen Levy. 2019. Narrative paths and negotiation of power in birth stories. In *CSCW*.
John W Ayers, Theodore L Caputi, Camille Nebeker, and Mark Dredze. 2018. Don't quote me: reverse identification of research participants in social media studies. NPJ digital medicine, 1(1):1-2.
Maria Becker, Michael Staniek, Vivi Nastase, and Anette Frank. 2017. Enriching argumentative texts with implicit knowledge. In NLDB.
Jeanette Bicknell. 2007. What is offensive about offensive jokes? Philosophy Today, 51(4):458-465.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: a case study of African-American English. In EMNLP.
13We direct workers to the Crisis Text Line (https:// www.crisistextline.org/).
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for automatic knowledge graph construction. In ACL.
Lorraine Bowman-Grieve. 2009. Exploring "Stormfront": a virtual community of the radical right. Studies in conflict & terrorism, 32(11):989-1007.
Luke M Breitbartler, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: a case for locating elusive phenomena in social media posts. In *EMNLP*.
Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics, pages 160-169. IEEE.
Southern Poverty Law Center. 2012. Misogyny: the sites. Intelligence Report, 145.
Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN - COUNTER NArratives through nichesourcing: a multilingual dataset of responses to fight online hate speech. In ACL.
Raphael Cohen-Almagor. 2014. Countering hate on the internet. Annual review of law and ethics, 22:431-443.
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In KDD.
Adam M Croom. 2011. Slurs. Language Sciences, 33(3):343-358.
Helen L Daly. 2018. On insults. Journal of the American Philosophical Association, 4(4):510-524.
Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg. 2012. Echoes of power: language effects and power differences in social interaction. In WWW.
Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In *Abusive Language Workshop*.
Thomas Davidson, Dana Warmsley, Michael W Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM.
Serena Does, Belle Derks, and Naomi Ellemers. 2011. Thou shalt not discriminate: how emphasizing moral ideals rather than obligations increases whites' support for social equality. Journal of Experimental Social Psychology, 47(3):562-571.
Marta Dynel. 2015. The landscape of impoliteness research. Journal of Politeness Research, 11(2):383.
Connie C Eble. 1996. Slang & sociability: in-group language among college students. Univ of North Carolina Press.
Anjalie Field, Gayatri Bhat, and Yulia Tsvetkov. 2019. Contextual affective analysis: a case study of people portrayals in online #MeToo stories. In ICWSM.
Charles J Fillmore and Collin F Baker. 2001. Frame semantics for text understanding. In Proceedings of WordNet and Other Lexical Resources Workshop, NAACL.
Jon Fingas. 2017. Reddit bans misogynist community as part of anti-violence crackdown. https://www.engadget.com/2017/11/08/reddit-bans-misogynist-community-in-anti-violence-crackdown/. Accessed: 2019-12-06.
Susan T Fiske. 1993. Controlling other people: the impact of power on stereotyping. American psychologist, 48(6):621-628.
Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of Twitter abusive behavior. In ICWSM.
Yiannis Gabriel. 1998. An introduction to the social psychology of insults in organizations. Human Relations, 51(11):1329-1354.
Adam D Galinsky, Cynthia S Wang, Jennifer A Whitson, Eric M Anicich, Kurt Hugenberg, and Galen V Bodenhausen. 2013. The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling. *Psychol. Sci.*, 24(10):2020-2029.
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and William B. Dolan. 2015. deltaBLEU: a discriminative metric for generation tasks with intrinsically diverse targets. In ACL.
Ona de Gibert, Naiara Pérez, Aitor García-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. In *Abusive Language Workshop at EMNLP*.
Gil Greengross and Geoffrey F Miller. 2008. Dissing oneself versus dissing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Deprecating humor. Evolutionary Psychology, 6(3).
Shirley Gregor and Izak Benbasat. 1999. Explanations from intelligent systems: Theoretical foundations and implications for practice. MIS quarterly, pages 497-530.
Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? empirical analysis and detecting attributes of convincingness in web argumentation. In EMNLP, pages 1214-1223.
Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In *NAACL-HLT*.
Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In ACL, pages 539-545.
Amanda Hess. 2016. The far right has a new digital safe space. https://www.nytimes.com/2016/11/30/arts/the-far-right-has-a-new-digital-safe-space.html. Accessed: 2019-12-06.
Gabriele Kasper. 1990. Linguistic politeness: current research issues. Journal of Pragmatics, 14(2):193-218.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: a conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1-10. ACM.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In ICML, pages 957-966.
Robin Lakoff. 1973. Language and woman's place. Language in society, 2(1):45-79.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. In ACL.
Marie-Catherine de Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301-333.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In FAccT.
An T Nguyen, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Elizabeth Tate, Byron C Wallace, and Matthew Lease. 2018. Believe it or not: designing a human-AI partnership for mixed-initiative fact-checking. In The 31st Annual ACM Symposium on User Interface Software and Technology, pages 189-199. ACM.
Conor J O'Dea, Stuart S Miller, Emma B Andres, Madelyn H Ray, Derrick F Till, and Donald A Saucier. 2015. Out of bounds: Factors affecting the perceived offensiveness of racial slurs. Language Sciences, 52:155-164.
Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and Multi-Aspect hate speech analysis. In EMNLP.
Gonçalo Pereira, Rui Prada, and Pedro A Santos. 2016. Integrating social power into the decision-making of cognitive agents. Artificial Intelligence, 241:1-44.
Vinodkumar Prabhakaran, Prabhakaran Vinodkumar, and Rambow Owen. 2014. Predicting power relations between participants in written dialog from a single thread. In ACL.
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. A benchmark dataset for learning to intervene in online hate speech. In EMNLP.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Unpublished.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Unpublished.
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2mind: commonsense inference on events, intents, and reactions. In ACL.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should I trust you?": Explaining the predictions of any classifier. In KDD.
Sarah T Roberts. 2016. Commercial content moderation: digital laborers' dirty work. In Safiya Umoja Noble and Brendesha M Tynes, editors, The Intersectional Internet: Race, Sex, Class and Culture Online, Media Studies Publications. Peter Lang Publishing.
Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In NLP 4 CMC Workshop.
RWJF. 2017. Discrimination in america: experiences and views. https://www.rwjf.org/en/library/research/2017/10/discrimination-in-america--experiences-and-views.html. Accessed: 2019-11-5.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: an adversarial winograd schema challenge at scale. In AAAI.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019a. The risk of racial bias in hate speech detection. In ACL.
Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019b. ATOMIC: an atlas of machine commonsense for if-then reasoning. In AAAI.
Maarten Sap, Marcella Cindy Prasetio, Ariel Holtzman, Hannah Rashkin, and Yejin Choi. 2017. Connotation frames of power and agency in modern films. In EMNLP.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019c. Social IQa: commonsense reasoning about social interactions. In EMNLP.
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Workshop on NLP for Social Media at EACL.
Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in ConceptNet 5. In LREC.
Whitney Strub. 2008. The clearly obscene and the queerly obscene: heteronormativity and obscenity in cold war los angeles. American Quarterly, 60(2):373-398.
Stefanie Ullmann and Marcus Tomalin. 2019. Quarantining online hate speech: technical and ethical perspectives. Ethics and Information Technology.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
James Vincent. 2016. Twitter taught Microsoft's AI chatbot to be a racist asshole in less than a day. https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist. Accessed: 2019-10-26.
Andrew J Vonasch and Roy F Baumeister. 2017. Unjustified side effects were strongly intended: taboo tradeoffs and the side-effect effect. Journal of Experimental Social Psychology, 68:83-92.
Zijian Wang and Christopher Potts. 2019. TalkDown: a corpus for condescension detection in context. In EMNLP.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In *NAACL Student Research Workshop*.
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: personal attacks seen at scale. In WWW.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In *NAACL*.
Michael Zimmer. 2018. Addressing conceptual gaps in big data research ethics: an application of contextual integrity. Social Media + Society, 4(2).
model	offensive 42.2% pos. (dev.) F1	pr.	rec.	intent 44.8% pos. (dev.) F1	pr.	rec.	lewd 3.0% pos. (dev.) F1	pr.	rec.	group 66.6% pos. (dev.) F1	pr.	rec.	in-group 5.1% pos. (dev.) F1	pr.	rec.
dev.	SBF-GPT1-gdy	75.2	88.3	65.5	74.4	89.8	63.6	75.2	78.2	72.5	62.3	74.6	53.4	-	-	-
''-constr	75.2	88.3	65.5	74.4	89.8	63.6	75.2	78.2	72.5	62.3	74.6	53.4	-	-	-
SBF-GPT2-gdy	77.2	88.3	68.6	76.3	89.5	66.5	77.6	81.2	74.3	66.9	67.9	65.8	24.0	85.7	14.0
''-constr	77.2	88.3	68.6	76.3	89.5	66.5	77.6	81.2	74.3	66.9	67.9	65.8	25.9	63.6	16.3
SBF-GPT2-smp	80.5	84.3	76.9	75.3	89.9	64.7	78.6	80.6	76.6	66.0	67.6	64.5	-	-	-
''-constr	80.4	84.3	76.8	75.3	89.9	64.7	78.5	80.6	76.5	66.0	67.6	64.5	-	-	-
test	SBF-GPT2-gdy	78.8	89.8	70.2	78.6	90.8	69.2	80.7	84.5	77.3	69.9	70.5	69.4	-	-	-
''-constr	78.8	89.8	70.2	78.6	90.8	69.2	80.7	84.5	77.3	69.9	70.5	69.4	-	-	-