SlowGuess/ACL_2020_OCR huggingface dataset:
SOCIAL BIAS FRAMES: Reasoning about Social and Power Implications of Language
Maarten Sap† Saadia Gabriel†‡ Lianhui Qin†‡ Dan Jurafsky° Noah A. Smith†‡ Yejin Choi†‡
†Paul G. Allen School of Computer Science & Engineering, University of Washington
$^{\ddagger}$ Allen Institute for Artificial Intelligence
$^{\circ}$ Linguistics & Computer Science Departments, Stanford University
Warning: this paper contains content that may be offensive or upsetting.
Language has the power to reinforce stereotypes and project social biases onto others. At the core of the challenge is that it is rarely what is stated explicitly, but rather the implied meanings, that frame people's judgments about others. For example, given a statement that "we shouldn't lower our standards to hire more women," most listeners will infer the implicature intended by the speaker — that "women (candidates) are less qualified." Most semantic formalisms, to date, do not capture such pragmatic implications in which people express social biases and power differentials in language.
We introduce SOCIAL BIAS FRAMES, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others. In addition, we introduce the Social Bias Inference Corpus to support large-scale modelling and evaluation with 150k structured annotations of social media posts, covering over 34k implications about a thousand demographic groups.
We then establish baseline approaches that learn to recover SOCIAL BIAS FRAMES from unstructured text. We find that while state-of-the-art neural models are effective at high-level categorization of whether a given statement projects unwanted social bias  $(80\% F_{1})$  they are not effective at spelling out more detailed explanations in terms of SOCIAL BIAS FRAMES. Our study motivates future work that combines structured pragmatic inference with commonsense reasoning on social implications.
Language has enormous power to project social biases and reinforce stereotypes on people (Fiske,
Figure 1: Understanding and explaining why an arguably innocuous statement is potentially unjust requires reasoning about conversational implicatures and commonsense implications with respect to the underlying intent, offensiveness, and power differentials between different social groups. SOCIAL BIAS FRAMES aim to represent the various pragmatic meanings related to social bias implications, by combining categorical and free-text annotations, e.g., that "women are less qualified" is implied by the statement "we shouldn't lower our standards to hire more women."
1993). The way such biases are projected is rarely in what is stated explicitly, but in all the implied layers of meanings that frame and influence people's judgments about others. For example, on hearing a statement that an all-Muslim movie was a "box office bomb", most people can instantly
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
post	off?	int?	lewd?	group?	targeted group	implied statement	ingroup?
Kendrick just dropped the coldest video of all fucking time	no	no	no
What a fucking stupid bitch shut the fuck up already	yes	yes	no
need to fuck sum bad, I got some pressure built up :P	no	no	yes
I take a bitch home give her long dick	yes	yes	yes	no
You annoyed me every day for months you're a fucking moron	yes	yes	no	no
My problem with Korean artists: I dont know how to pronounce your name I can't hanggul to save my life	yes	no	no	yes	Korean folks	have weird names	no
I'll continue this thread till non-lifting fags stop lifting like idiots at the gym... Which is never.	yes	yes	no	yes	gay men	are weak and not manly	no
I thought drugs were the only things black people could shoot up Boy was I wrong	yes	yes	no	yes	Black folks	do drugs kill people commit shootings	no
Table 1: Examples of inference tuples in SBIC. The types of inferences captured by SOCIAL BIAS FRAMES cover (potentially subtle) offensive implications about various demographic groups.
recognize the implied demonizing stereotype that "Muslims are terrorists" (Figure 1). Understanding these biases with accurate underlying explanations is necessary for AI systems to adequately interact in the social world (Pereira et al., 2016), and failure to do so can result in the deployment of harmful technologies (e.g., conversational AI systems turning sexist and racist; Vincent, 2016).
Most previous approaches to understanding the implied harm in statements have cast this task as a simple toxicity classification (e.g., Waseem and Hovy, 2016; Founta et al., 2018; Davidson et al., 2017). However, simple classifications run the risk of discriminating against minority groups, due to high variation and identity-based biases in annotations (e.g., which cause models to learn associations between dialect and toxicity; Sap et al., 2019a; Davidson et al., 2019). In addition, detailed explanations are much more informative for people to understand and reason about why a statement is potentially harmful against other people (Gregor and Benbasat, 1999; Ribeiro et al., 2016).
Thus, we propose SOCIAL BIAS FRAMES, a novel conceptual formalism that aims to model pragmatic frames in which people project social biases and stereotypes on others. Compared to semantic frames (Fillmore and Baker, 2001), the meanings projected by pragmatic frames are richer, and thus cannot be easily formalized using only categorical labels. Therefore, as illustrated in Figure 1, our formalism combines hierarchical categories of biased implications such
as intent and offensiveness with implicatures described in free-form text such as groups referenced and implied statements. In addition, we introduce SBIC, $^{1}$  a new corpus collected using a novel crowdsourcing framework. SBIC supports large-scale learning and evaluation with over 150k structured annotations of social media posts, spanning over 34k implications about a thousand demographic groups.
We then establish baseline approaches that learn to recover SOCIAL BIAS FRAMES from unstructured text. We find that while state-of-the-art neural models are effective at making high-level categorization of whether a given statement projects unwanted social bias  $(80\% F_1)$ , they are not effective at spelling out more detailed explanations by accurately decoding SOCIAL BIAS FRAMES. Our study motivates future research that combines structured pragmatic inference with commonsense reasoning on social implications.
Important implications of this study. We recognize that studying SOCIAL BIAS FRAMES necessarily requires us to confront online content that may be offensive or disturbing (see §7 for further discussion on the ethical implications of this study). However, deliberate avoidance does not eliminate such problems. Therefore, the important premise we take in this study is that assessing social media content through the lens of SOCIAL
$^{1}$ SBIC: Social Bias Inference Corpus, available at http://tinyurl.com/social-bias-frames.
BIAS FRAMES is important for automatic flagging or AI-augmented writing interfaces, where potentially harmful online content can be analyzed with detailed explanations for users or moderators to consider and verify. In addition, the collective analysis over large corpora can also be insightful for educating people on reducing unconscious biases in their language.
2 SOCIAL BIAS FRAMES Definition
To better enable models to account for socially biased implications of language, $^{2}$  we design a new pragmatic formalism that distinguishes several related but distinct inferences, shown in Figure 1. Given a natural language utterance, henceforth, post, we collect both categorical as well as free text inferences (described below), inspired by recent efforts in free-text annotations of commonsense knowledge (e.g., Speer and Havasi, 2012; Rashkin et al., 2018; Sap et al., 2019b) and argumentation (Habernal and Gurevych, 2016; Becker et al., 2017). The free-text explanations are crucial to our formalism, as they can both increase trust in predictions made by the machine (Kulesza et al., 2012; Bussone et al., 2015; Nguyen et al., 2018) and encourage a poster's empathy towards a targeted group, thereby combating biases (Cohen-Almagor, 2014).
We base our initial frame design on social science literature of pragmatics (Lakoff, 1973; de Marneffé et al., 2012) and impoliteness (Kasper, 1990; Gabriel, 1998; Dynel, 2015; Vonasch and Baumeister, 2017). We then refine the frame structure (including number of possible answers to questions) based on the annotator (dis)agreement in multiple pilot studies. We describe each of the included variables below.
Offensiveness is our main categorical annotation, and denotes the overall rudeness, disrespect, or toxicity of a post. We consider whether a post could be considered "offensive to anyone", as previous work has shown this to have higher recall (Sap et al., 2019a). This is a categorical variable with three possible answers (yes, maybe, no).
Intent to offend captures whether the perceived motivation of the author was to offend, which is key to understanding how it is received (Kasper,
1990; Dynel, 2015), yet distinct from offensiveness (Gabriel, 1998; Daly, 2018). This is a categorical variable with four possible answers (yes, probably, probably not, no).
Lewd or sexual references are a key subcategory of what constitutes potentially offensive material in many cultures, especially in the United States (Strub, 2008). This is a categorical variable with three possible answers (yes, maybe, no).
Group implications are distinguished from individual-only attacks or insults that do not invoke power dynamics between groups (e.g., "F*ck you" vs. "F*ck you, f*ggot"). This is a categorical variable with two possible answers: individual-only (no), group targeted (yes).
Targeted group describes the social or demographic group that is referenced or targeted by the post. Here we collect free-text answers, but provide a seed list of demographic or social groups to encourage consistency.
Implied statement represents the power dynamic or stereotype that is referenced in the post. We collect free-text answers in the form of simple Hearst-like patterns (e.g., "women are ADJ", "gay men VBP"; Hearst, 1992).
In-group language aims to capture whether the author of a post may be a member of the same social/demographic group that is targeted, as speaker identity changes how a statement is perceived (O'Dea et al., 2015). Specifically, in-group language (words or phrases that (re)establish belonging to a social group; Eble, 1996) can change the perceived offensiveness of a statement, such as reclaimed slurs (Croom, 2011; Galinsky et al., 2013) or self-deprecating language (Greengross and Miller, 2008). Note that we do not attempt to categorize the identity of the speaker. This variable takes three possible values (yes, maybe, no).
3 Collecting Nuanced Annotations
To create SBIC, we design a crowdsourcing framework to distill the biased implications of posts at a large scale.
We draw from various sources of potentially biased online content, shown in Table 2, to select
2In this work, we employ the U.S. sociocultural lens when discussing bias and power dynamics among demographic groups.
type	source	# posts
Reddit	r/darkJokes	10,095
r/meanJokes	3,483
r/offensiveJokes	356
Microaggressions	2,011
subtotal	15,945
Twitter	Founta et al. (2018)	11,864
Davidson et al. (2017)	3,008
Waseem and Hovy (2016)	1,816
subtotal	16,688
Hate Sites	Gab	3,715
Stormfront	4,016
Banned Reddit	4,308
subtotal	12,039
SBIC	total # posts	44,671
Table 2: Breakdown of origins of posts in SBIC. Microaggressions are drawn from the Reddit corpus introduced by Breitbart et al. (2019), and Banned Reddit include r/Incels and r/MensRights.
posts to annotate. Since online toxicity can be relatively scarce (Founta et al., 2018),<sup>3</sup> we start by annotating English Reddit posts, specifically three intentionally offensive subReddit and a corpus of potential microaggressions from Breitbart et al. (2019). By nature, the three offensive subreddits are very likely to have harmful implications, as posts are often made with intents to deride adversity or social inequality (Bicknell, 2007). Microaggressions, on the other hand, are likely to contain subtle biased implications—a natural fit for SOCIAL BIAS FRAMES.
In addition, we include posts from three existing English Twitter datasets annotated for toxic or abusive language, filtering out @-replies, retweets, and links. We mainly annotate tweets released by Founta et al. (2018), who use a bootstrapping approach to sample potentially offensive tweets. We also include tweets from Waseem and Hovy (2016) and Davidson et al. (2017), who collect datasets of tweets containing racist or sexist hashtags and slurs, respectively.
Finally, we include posts from known English hate communities: Stormfront (de Gibert
Figure 2: Snippets of the annotation task used to collect SBIC. Lewdness, group implication, and in-group language questions are omitted for brevity but shown in larger format in Figure 4 (Appendix).
et al., 2018) and Gab, which are both documented white-supremacist and neo-nazi communities (Bowman-Grieve, 2009; Hess, 2016), and two English subreddits that were banned for inciting violence against women (r/Incels and r/MensRights; Fingas, 2017; Center, 2012).
3.2 Annotation Task Design
We design a hierarchical annotation framework to collect biased implications of a given post (snippet shown in Figure 2) on Amazon Mechanical Turk (MTurk). The full task is shown in the appendix (Figure 4).
For each post, workers indicate whether the post is offensive, whether the intent was to offend, and whether it contains lewd or sexual content. Only if annotators indicate potential offensiveness do they answer the group implication question. If the post targets or references a group or demographic, workers select or write which one(s); per selected group, they then write two to four stereotypes. Finally, workers are asked whether they think the speaker is part of one of the minority groups referenced by the post.
We collect three annotations per post, and restrict our worker pool to the U.S. and Canada. We ask workers to optionally provide coarse-grained demographic information.
4https://files.pushshift.io/gab/ GABPOSTS_CORPUS.xz
This study was approved by our institutional review board.
${}^{3}$  Founta et al. (2018) find that the prevalence of toxic content online is  $< 4\%$  .
total # tuples	147,139
# unique	posts	44,671
groups	1,414
implications	32,028
post-group	48,923
post-group-implication	87,942
group-implication	34,333
skews(% pos.)	offensive	44.8%
intent	43.4%
lewd	7.9%
group targeted	50.9%
in-group	4.6%
Table 3: Statistics of the SBIC dataset. Skews indicate the number of times a worker annotated a post as offensive, etc.
Annotator demographics In our final annotations, our worker pool was relatively gender-balanced and age-balanced (55% women, 42% men, <1% non-binary; 36±10 years old), but racially skewed (82% White, 4% Asian, 4% Hispanic, 4% Black).
Annotator agreement Overall, the annotations in SBIC showed  $82.4\%$  pairwise agreement and Krippendorf's  $\alpha = 0.45$  on average, which is substantially higher than previous work in toxic language detection (e.g.,  $\alpha = 0.22$  in Ross et al., 2017). Broken down by each categorical question, workers agreed on a post being offensive at a rate of  $76\%$  (Krippendorf's  $\alpha = 0.51$ ), its intent being to offend at  $75\%$  ( $\alpha = 0.46$ ), and it having group implications at  $74\%$  ( $\alpha = 0.48$ ). For categorizing posts as lewd, workers agreed substantially ( $94\%$ ,  $\alpha = 0.62$ ). However, flagging potential in-group speech had lower agreement, likely because this is a very nuanced annotation, and because highly skewed categories (only  $5\%$  "yes"; see Table 3) lead to low  $\alpha$ s (here,  $\alpha = 0.17$  with agreement  $94\%$ ).<sup>6</sup> Finally, workers agreed on the exact same targeted group  $80.2\%$  of the time ( $\alpha = 0.50$ ).
3.3 SBIC Description
After data collection, SBIC contains 150k structured inference tuples, covering 34k free text group-implication pairs (see Table 3). We show example inference tuples in Table 1.
Figure 3: Breakdown of targeted group categories by domains. We show percentages within domains for the top three most represented identities, namely gender/sexuality (e.g., women, LGBTQ), race/ethnicity (e.g., Black, Latinx, and Asian), and culture/origin (e.g., Muslim, Jewish).
Additionally, we show a breakdown of the types of targeted groups in Figure 3. While SBIC covers a variety of types of biases, gender-based, race-based, and culture-based biases are the most represented, which parallels the types of discrimination happening in the real world (RWJF, 2017).
We find that our dataset is predominantly written in White-aligned English (78% of posts), as measured by a lexical dialect detector by Blodgett et al. (2016), with  $< 10\%$  of posts having indicators of African-American English. We caution researchers to consider the potential for dialect- or identity-based biases in labelling (Davidson et al., 2019; Sap et al., 2019a) before deploying technology based on SBIC (see Section 7).
4 Social Bias Inference
Given a post, we establish baseline performance of models at inferring SOCIAL BIAS FRAMES. An ideal model should be able to both generate the implied power dynamics in textual form, as well as classify the post's offensiveness and other categorical variables. Satisfying these conditions, we use the OpenAI-GPT transformer networks (Vaswani et al., 2017; Radford et al., 2018, 2019) as a basis for our experiments, given their recent successes at
Given our data selection process, we expect the rate of in-group posts to be very low (see §3.3).
model	offensive 42.2% pos. (dev.) F1 pr. rec.	intent 44.8% pos (dev.) F1 pr. rec.	lewd 3.0% pos (dev.) F1 pr. rec.	group 66.6% pos (dev.) F1 pr. rec.	in-group 5.1% pos (dev.) F1 pr. rec.
dev.	SBF-GPT1-gdy	75.2	88.3	65.5	74.4	89.8	63.6	75.2	78.2	72.5	62.3	74.6	53.4	-	-	-
SBF-GPT2-gdy	77.2	88.3	68.6	76.3	89.5	66.5	77.6	81.2	74.3	66.9	67.9	65.8	24.0	85.7	14.0
SBF-GPT2-smp	80.5	84.3	76.9	75.3	89.9	64.7	78.6	80.6	76.6	66.0	67.6	64.5	-	-	-
test	SBF-GPT2-gdy	78.8	89.8	70.2	78.6	90.8	69.2	80.7	84.5	77.3	69.9	70.5	69.4	-	-	-
Table 4: Experimental results (\%) of various models on the classification tasks (gdy: argmax, smp: sampling). Some models did not predict the positive class for "in-group language," their performance is denoted by " $-$ ". We bold the  $F_{1}$  scores of the best performing model(s) on the development set. For easier interpretation, we also report the percentage of instances in the positive class in the development set.
classification, commonsense generation, and conditional generation (Bosselut et al., 2019; Keskar et al., 2019).
Training We cast our frame prediction task as a hybrid classification and language generation task, where we linearize the variables following the frame hierarchy. At training time, our model takes as input a sequence of  $N$  tokens:
$$
\begin{array}{l} \mathbf {x} = \left\{\left[ \mathrm {S T R} \right], w _ {1}, w _ {2}, \dots , w _ {n}, \left[ \mathrm {S E P} \right], \right. \\ w _ {[ \text {l e w d} ]}, w _ {[ \text {o f f} ]}, w _ {[ \text {i n t} ]}, w _ {[ \text {g r p} ]}, [ \text {S E P} ], \\ w _ {[ \mathbf {G} ] _ {1}}, w _ {[ \mathbf {G} ] _ {2}}, \dots , [ \mathrm {S E P} ], \\ w _ {[ \mathbf {S} ] _ {1}}, w _ {[ \mathbf {S} ] _ {2}}, \dots , [ \mathbf {S E P} ], \\ w _ {[ \text {i n g} ]}, [ \text {E N D} ] \} \quad (1) \\ \end{array}
$$
where [STR] is our start token,  $w_{1:n}$  is the sequence of tokens in a post,  $w_{[\mathsf{G}]_i}$  the tokens representing the group, and  $w_{[\mathsf{S}]_i}$  the implied statement. We add two task-specific vocabulary items for each of our five classification tasks ( $w_{[lewd]}, w_{[off]}, w_{[int]}, w_{[grp]}, w_{[ing]}$ ), each representing the negative and positive values of the class (e.g., for offensiveness,  $[\text{offY}]$  and  $[\text{offN}]$ ).
The model relies on a stack of transformer blocks of multi-headed attention and fully connected layers to encode the input tokens (for a detailed modelling description, see Radford et al., 2018, 2019). Since GPT is a forward-only language model, the attention is only computed over preceding tokens. At the last layer, the model projects the embedding into a vocabulary-sized vector, which is turned into a probability distribution over the vocabulary using a softmax layer.
We minimize the cross-entropy of the contextual probability of the correct token in our full linearized frame objective (of length  $N$ ):
$$
\mathcal {L} = - \frac {1}{N} \sum_ {i} \log p _ {\mathrm {G P T}} \left(w _ {i} \mid w _ {0: i - 1}\right)
$$
During training, no loss is incurred for lower-level variables with no values, i.e., variables that cannot take values due to earlier variable values (e.g., there is no targeted group for posts marked as non-offensive).
In our experiments we use pretrained versions of OpenAI's GPT and GPT2 (Radford et al., 2018, 2019) for our model variants, named SBF-GPT $_1$  and SBF-GPT $_2$ , respectively. While their architectures are similar (stack of Transformers), GPT was trained on a large corpus of fiction books, whereas GPT2 was trained on 40Gbs of English web text.
Inference We frame our inference task as a conditional language generation task. Conditioned on the post, we generate tokens one-by-one either by greedily selecting the most probable one, or by sampling from the next word distribution, and appending the selected token to the output. We stop when the [END] token is generated, at which point our entire frame is predicted. For greedy decoding, we only generate our frames once, but for sampling, we repeat the generation procedure to yield ten candidate frame predictions and choose the highest scoring one under our model.
In contrast to training time, where all inputs are consistent with our frames' structure, at test time, our model can sometimes predict combinations of variables that are inconsistent with the constraints of the frame (e.g., predicting a post to be inoffensive, but still predict it to be offensive to a group). To mitigate this issue, we also experiment with a constrained decoding algorithm (denoted "constr") that considers various global assignments of
<sup>7</sup>We linearize following the order in which variables were annotated (see Figure 4). Future work could explore alternate orderings.
<sup>8</sup>We binarize our categorical annotations, assigning 1 to "yes," "probably," and "maybe," and 0 to all other values.
group targeted	implied statement
BLEU	Rouge-L	WMD	BLEU	Rouge-L	WMD
dev.	SBF-GPT1-gdy	69.9	60.3	1.01	49.9	40.2	2.97
SBF-GPT1-gdy-constr	69.2	64.7	1.05	49.0	42.8	3.02
SBF-GPT2-gdy	74.2	64.6	0.90	49.8	41.4	2.96
SBF-GPT2-gdy-constr	73.4	68.2	0.89	49.6	43.5	2.96
SBF-GPT2-smp	83.2	33.7	0.62	44.3	17.8	3.31
SBF-GPT2-smp-constr	83.0	33.7	0.63	44.1	17.9	3.31
test	SBF-GPT2-gdy	77.0	71.3	0.76	52.2	46.5	2.81
SBF-GPT2-gdy-constr	77.9	68.7	0.74	52.6	44.9	2.79
Table 5: Automatic evaluation of various models on the generation task. We bold the scores of the best performing model(s) on the development set. Higher is better for BLEU and ROUGE scores, and lower is better for WMD.
variables. Specifically, after greedy decoding, we recompute the probabilities of each of the categorical variables, and search for the most probable assignment given the generated text candidate and variable probabilities.[9] This can allow variables to be assigned an alternative value that is more globally optimal.[10]