Venue: ACL 2020

# Social Bias Frames:

Reasoning about Social and Power Implications of Language

 Maarten Sap\({}^{\dagger}\)   Saadia Gabriel\({}^{\dagger\ddagger}\)   Lianhui Qin\({}^{\dagger\ddagger}\)

**Dan Jurafskyo**  Noah A. Smith\({}^{\dagger\ddagger}\)   Yejin Choi\({}^{\dagger\ddagger}\)

\({}^{\dagger}\)Paul G. Allen School of Computer Science & Engineering, University of Washington

\({}^{\ddagger}\)Allen Institute for Artificial Intelligence

\({}^{\diamond}\)Linguistics & Computer Science Departments, Stanford University

###### Abstract

_Warning: this paper contains content that may be offensive or upsetting._

Language has the power to reinforce stereotypes and project social biases onto others. At the core of the challenge is that it is rarely what is stated explicitly, but rather the _implied_ meanings, that frame people's judgments about others. For example, given a statement that "we shouldn't lower our standards to hire more women," most listeners will infer the implicature intended by the speaker -- that "women (candidates) are less qualified." Most semantic formalisms, to date, do not capture such pragmatic implications in which people express social biases and power differentials in language.

We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others. In addition, we introduce the Social Bias Inference Corpus to support large-scale modelling and evaluation with 150k structured annotations of social media posts, covering over 34k implications about a thousand demographic groups.

We then establish baseline approaches that learn to recover Social Bias Frames from unstructured text. We find that while state-of-the-art neural models are effective at high-level categorization of whether a given statement projects unwanted social bias (80% \(F_{1}\)), they are not effective at spelling out more detailed explanations in terms of Social Bias Frames. Our study motivates future work that combines structured pragmatic inference with commonsense reasoning on social implications.

## 1 Introduction

Language has enormous power to project social biases and reinforce stereotypes on people [26]. The way such biases are projected is rarely in what is stated explicitly, but in all the implied layers of meanings that frame and influence people's judgments about others. For example, on hearing a statement that an all-Muslim movie was a "box office bomb", most people can instantly

Figure 1: Understanding and explaining why an arguably innocuous statement is potentially unjust requires reasoning about conversational implicatures and commonsense implications with respect to the underlying intent, offensiveness, and power differentials between different social groups. Social Bias Frames aim to represent the various pragmatic meanings related to social bias implications, by combining categorical and free-text annotations, e.g., that “women are less qualified” is implied by the statement “we shouldn’t lower our standards to hire more women.”

recognize the implied demonizing stereotype that "Muslims are terrorists" (Figure 1). Understanding these biases with accurate underlying explanations is necessary for AI systems to adequately interact in the social world Pereira et al. (2016), and failure to do so can result in the deployment of harmful technologies (e.g., conversational AI systems turning sexist and racist; Vincent (2016)).

Most previous approaches to understanding the implied harm in statements have cast this task as a simple _toxicity_ classification (e.g., Waseem and Hovy (2016); Founta et al. (2018); Davidson et al. (2017)). However, simple classifications run the risk of discriminating against minority groups, due to high variation and identity-based biases in annotations (e.g., which cause models to learn associations between dialect and toxicity; Sap et al. (2019); Davidson et al. (2019). In addition, detailed explanations are much more informative for people to understand and reason about _why_ a statement is potentially harmful against other people Gregor and Benbasat (1999); Ribeiro et al. (2016).

Thus, we propose Social Bias Frames, a novel conceptual formalism that aims to model pragmatic frames in which people project social biases and stereotypes on others. Compared to semantic frames Fillmore and Baker (2001), the meanings projected by pragmatic frames are richer, and thus cannot be easily formalized using only categorical labels. Therefore, as illustrated in Figure 1, our formalism combines hierarchical categories of biased implications such as _intent_ and _offensiveness_ with implicatures described in free-form text such as _groups referenced_ and _implied statements_. In addition, we introduce SBIC,1 a new corpus collected using a novel crowdsourcing framework. SBIC supports large-scale learning and evaluation with over 150k structured annotations of social media posts, spanning over 34k implications about a thousand demographic groups.

Footnote 1: SBIC: Social Bias Inference Corpus, available at [http://tinyurl.com/social-bias-frames](http://tinyurl.com/social-bias-frames).

We then establish baseline approaches that learn to recover Social Bias Frames from unstructured text. We find that while state-of-the-art neural models are effective at making high-level categorization of whether a given statement projects unwanted social bias (80% \(F_{1}\)), they are not effective at spelling out more detailed explanations by accurately decoding Social Bias Frames. Our study motivates future research that combines structured pragmatic inference with commonsense reasoning on social implications.

Important implications of this study.We recognize that studying Social Bias Frames necessarily requires us to confront online content that may be offensive or disturbing (see SS7 for further discussion on the ethical implications of this study). However, deliberate avoidance does not eliminate such problems. Therefore, the important premise we take in this study is that assessing social media content through the lens of Social

\begin{table}
\begin{tabular}{l c c c c c l r} post & off? & int? & lewd? & group? & targeted & implied statement & ingroup? \\  & & & & & group & & \\ Kendrick just dropped the coldest video of all fucking time What a fucking stupid bitch shut the fuck up already need to fuck sum bad , I got some pressure built up !P I take a bitch home give her long dick You annoyed me every day for months you’re a fucking moron My problem with Korean artists: I don know how to pronounce your name I can’t hangspul to save my life I’ll continue this thread till non-lifting fags stop lifting like idiots at the gym... Which is never. I thought drugs were the only things black people could shoot up Boy was I wrong & yes & no & yes & Korean & have weird names & no \\ \end{tabular}
\end{table}
Table 1: Examples of inference tuples in SBIC. The types of inferences captured by Social Bias Frames cover (potentially subtle) offensive implications about various demographic groups.

 

Bias Frames is important for automatic flagging or AI-augmented writing interfaces, where potentially harmful online content can be analyzed with detailed explanations for users or moderators to consider and verify. In addition, the collective analysis over large corpora can also be insightful for educating people on reducing unconscious biases in their language.

## 2 Social Bias Frames Definition

To better enable models to account for socially biased implications of language,2 we design a new pragmatic formalism that distinguishes several related but distinct inferences, shown in Figure 1. Given a natural language utterance, henceforth, _post_, we collect both categorical as well as free text inferences (described below), inspired by recent efforts in free-text annotations of commonsense knowledge [e.g., 8, 9] and argumentation [4, 5]. The free-text explanations are crucial to our formalism, as they can both increase trust in predictions made by the machine [30, 6, 7] and encourage a poster's empathy towards a targeted group, thereby combating biases [1].

Footnote 2: In this work, we employ the U.S. sociocultural lens when discussing bias and power dynamics among demographic groups.

We base our initial frame design on social science literature of pragmatics [10, 11, 12] and impoliteness [13, 14]. We then refine the frame structure (including number of possible answers to questions) based on the annotator (dis)agreement in multiple pilot studies. We describe each of the included variables below.

Offensivenessis our main categorical annotation, and denotes the overall rudeness, disrespect, or toxicity of a post. We consider whether a post could be considered "offensive to anyone", as previous work has shown this to have higher recall [9]. This is a categorical variable with three possible answers (_yes_, _maybe_, _no_).

Intent to offendcaptures whether the perceived motivation of the author was to offend, which is key to understanding how it is received [13, 12], yet distinct from offensiveness [13, 14]. This is a categorical variable with four possible answers (_yes_, _probably_, _probably not_, _no_).

Lewdorsexual references are a key subcategory of what constitutes potentially offensive material in many cultures, especially in the United States [26]. This is a categorical variable with three possible answers (_yes_, _maybe_, _no_).

Group implicationsare distinguished from individual-only attacks or insults that do not invoke power dynamics between groups (e.g., "F*ck you" vs. "F*ck you, f*ggot"). This is a categorical variable with two possible answers: individual-only (_no_), group targeted (_yes_).

Targeted groupdescribes the social or demographic group that is referenced or targeted by the post. Here we collect _free-text answers_, but provide a seed list of demographic or social groups to encourage consistency.

Implied statementrepresents the power dynamic or stereotype that is referenced in the post. We collect _free-text answers_ in the form of simple Hearst-like patterns (e.g., "_women are_ ADJ", "_gap men VBP_"; Hearst, 1992).

In-group languageaims to capture whether the author of a post may be a member of the same social/demographic group that is targeted, as speaker identity changes how a statement is perceived [10]. Specifically, in-group language (words or phrases that (re)establish belonging to a social group; Eble, 1996) can change the perceived offensiveness of a statement, such as reclaimed slurs [10, 11] or self-deprecating language [12]. Note that we do not attempt to categorize the identity of the speaker. This variable takes three possible values (_yes_, _maybe_, _no_).

## 3 Collecting Nuanced Annotations

To create SBIC, we design a crowdsourcing framework to distill the biased implications of posts at a large scale.

### Data Selection

We draw from various sources of potentially biased online content, shown in Table 2, to select 

posts to annotate. Since online toxicity can be relatively scarce (Founta et al., 2018),3 we start by annotating English Reddit posts, specifically three intentionally offensive subReddits and a corpus of potential microaggressions from Breitfeller et al. (2019). By nature, the three offensive subreddits are very likely to have harmful implications, as posts are often made with intents to deride adversity or social inequality (Bicknell, 2007). Microagressions, on the other hand, are likely to contain subtle biased implications--a natural fit for Social Bias Frames.

Footnote 3: Founta et al. (2018) find that the prevalence of toxic content online is \(<\)4%.

In addition, we include posts from three existing English Twitter datasets annotated for toxic or abusive language, filtering out @-replies, retweets, and links. We mainly annotate tweets released by Founta et al. (2018), who use a bootstrapping approach to sample potentially offensive tweets. We also include tweets from Waseem and Hovy (2016) and Davidson et al. (2017), who collect datasets of tweets containing racist or sexist hashtags and slurs, respectively.

Finally, we include posts from known English hate communities: Stormfront (de Gibert et al., 2018) and Gab,4 which are both documented white-supremacist and neo-nazi communities (Bowman-Grieve, 2009; Hess, 2016), and two English subreddits that were banned for inciting violence against women (r/Incels and r/MensRights; Fiingsas, 2017; Center, 2012).

Footnote 4: [https://files.pushshift.io/gab/](https://files.pushshift.io/gab/) GABBOSTS_CORPUS.xz

### Annotation Task Design

We design a hierarchical annotation framework to collect biased implications of a given post (snippet shown in Figure 2) on Amazon Mechanical Turk (MTurk). The full task is shown in the appendix (Figure 4).

For each post, workers indicate whether the post is offensive, whether the intent was to offend, and whether it contains lewd or sexual content. Only if annotators indicate potential offensiveness do they answer the group implication question. If the post targets or references a group or demographic, workers select or write which one(s); per selected group, they then write two to four stereotypes. Finally, workers are asked whether they think the speaker is part of one of the minority groups referenced by the post.

We collect three annotations per post, and restrict our worker pool to the U.S. and Canada. We ask workers to optionally provide coarse-grained demographic information.5

\begin{table}
\begin{tabular}{l l c} \hline \hline type & source & \# posts \\ \cline{3-3}  & r/darkJokes & 10,095 \\  & r/meanJokes & 3,483 \\ Reddit & r/offensiveJokes & 356 \\  & Microaggressions & 2,011 \\ \cline{2-3}  & _subtotal_ & _15,945_ \\ \hline Twitter & Founta et al. (2018) & 11,864 \\  & Davidson et al. (2017) & 3,008 \\  & Waseem and Hovy (2016) & 1,816 \\ \cline{2-3}  & _subtotal_ & _16,688_ \\ \hline Hate Sites & Gab & 3,715 \\  & Stormfront & 4,016 \\  & Banned Reddits & 4,308 \\ \cline{2-3}  & _subtotal_ & _12,039_ \\ \hline \hline SBIC & **total \# posts** & 44,671 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Breakdown of origins of posts in SBIC. Microagressions are drawn from the Reddit corpus introduced by Breitfeller et al. (2019), and Banned Reddits include r/Incels and r/MensRights.

Figure 2: Snippet of the annotation task used to collect SBIC. Lewdness, group implication, and in-group language questions are omitted for brevity but shown in larger format in Figure 4 (Appendix).

 

Annotator demographicsIn our final annotations, our worker pool was relatively gender-balanced and age-balanced (55% women, 42% men, \(<\)1% non-binary; 36\(\pm\)10 years old), but racially skewed (82% White, 4% Asian, 4% Hispanic, 4% Black).

Annotator agreementOverall, the annotations in SBIC showed 82.4% pairwise agreement and Krippendorf's \(\alpha\)=0.45 on average, which is substantially higher than previous work in toxic language detection (e.g., \(\alpha\)=0.22 in Ross et al., 2017). Broken down by each categorical question, workers agreed on a post being offensive at a rate of 76% (Krippendorf's \(\alpha\)=0.51), its intent being to offend at 75% (\(\alpha\)=0.46), and it having group implications at 74% (\(\alpha\)=0.48). For categorizing posts as lewd, workers agreed substantially (94%, \(\alpha\)=0.62). However, flagging potential incongru speech had lower agreement, likely because this is a very nuanced annotation, and because highly skewed categories (only 5% "yes"; see Table 3) lead to low \(\alpha\)s (here, \(\alpha\)=0.17 with agreement 94%).6 Finally, workers agreed on the exact same targeted group 80.2% of the time (\(\alpha\)=0.50).

Footnote 6: Given our data selection process, we expect the rate of in-group posts to be very low (see §3.3).

### Sbic Description

After data collection, SBIC contains 150k structured inference tuples, covering 34k free text group-implication pairs (see Table 3). We show example inference tuples in Table 1.

Additionally, we show a breakdown of the types of targeted groups in Figure 3. While SBIC covers a variety of types of biases, gender-based, race-based, and culture-based biases are the most represented, which parallels the types of discrimination happening in the real world (RWJF, 2017).

We find that our dataset is predominantly written in White-aligned English (78% of posts), as measured by a lexical dialect detector by Blodgett et al. (2016), with \(<\)10% of posts having indicators of African-American English. We caution researchers to consider the potential for dialect- or identity-based biases in labelling Davidson et al. (2019); Sap et al. (2019) before deploying technology based on SBIC (see Section 7).

## 4 Social Bias Inference

Given a post, we establish baseline performance of models at inferring Social Bias Frames. An ideal model should be able to both _generate_ the implied power dynamics in textual form, as well as _classify_ the post's offensiveness and other categorical variables. Satifying these conditions, we use the OpenAI-GPT transformer networks Vaswani et al. (2017); Radford et al. (2018, 2019) as a basis for our experiments, given their recent successes at

Figure 3: Breakdown of targeted group categories by domains. We show percentages within domains for the top three most represented identities, namely gender/sexuality (e.g., women, LGBTQ), race/ethnicity (e.g., Black, Latinx, and Asian), and culture/origin (e.g., Muslim, Jewish).

\begin{table}
\begin{tabular}{l l r} \hline total \# tuples & & 147,139 \\ \hline  & posts & 44,671 \\  & groups & 1,414 \\ \multirow{2}{*}{\# unique} & implications & 32,028 \\ \cline{2-3}  & post-group & 48,923 \\  & post-group-implication & 87,942 \\  & group-implementation & 34,333 \\ \hline  & offensive & 44.8\% \\ \multirow{2}{*}{skews (\% pos.)} & intent & 43.4\% \\  & lewd & 7.9\% \\ \multirow{2}{*}{(\% pos.)} & group targeted & 50.9\% \\ \multicolumn{2}{r}{in-group} & 4.6\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics of the SBIC dataset. Skews indicate the number of times a worker annotated a post as offensive, etc.

 

classification, commonsense generation, and conditional generation Bosselut et al. (2019); Keskar et al. (2019).

TrainingWe cast our frame prediction task as a hybrid classification and language generation task, where we linearize the variables following the frame hierarchy.7 At training time, our model takes as input a sequence of \(N\) tokens:

Footnote 7: We linearize following the order in which variables were annotated (see Figure 4). Future work could explore alternate orderings.

\[{\bf x}=\{[{\tt STR}],w_{1},w_{2},...,w_{n},[{\tt SEP}],\\ w_{[{\tt leaf}]},w_{[{\tt off}]},w_{[{\tt int}]},w_{[{\tt grp}]},[{ \tt SSP}],\\ w_{[{\tt g}_{1}]},w_{[{\tt g}_{2}]},...,[{\tt SSP}],\\ w_{[{\tt s}]_{1}},w_{[{\tt S}]_{2}},...,[{\tt SSP}],\\ w_{[{\tt sing}]},[{\tt END}]\} \tag{1}\]

where \([{\tt STR}]\) is our start token, \(w_{1:n}\) is the sequence of tokens in a post, \(w_{[{\tt G}_{i}]}\) the tokens representing the group, and \(w_{[{\tt S}_{i}]}\) the implied statement. We add two task-specific vocabulary items for each of our five classification tasks (\(w_{[{\tt leaf}]}\), \(w_{[{\tt off}]}\), \(w_{[{\tt int}]}\), \(w_{[{\tt grip}]}\), \(w_{[{\tt lag}]}\)), each representing the negative and positive values of the class (e.g., for offensiveness, [offY] and [offN]).8

Footnote 8: We binarize our categorical annotations, assigning 1 to “yes,” probably,” and “maybe,”, and 0 to all other values.

The model relies on a stack of transformer blocks of multi-headed attention and fully connected layers to encode the input tokens (for a detailed modelling description, see Radford et al., 2018, 2019). Since GPT is a forward-only language model, the attention is only computed over preceding tokens. At the last layer, the model projects the embedding into a vocabulary-sized vector, which is turned into a probability distribution over the vocabulary using a softmax layer.

We minimize the cross-entropy of the contextual probability of the correct token in our full linearized frame objective (of length \(N\)):

\[\mathcal{L}=-\frac{1}{N}\sum_{i}\log p_{\text{GPT}}(w_{i}\mid w_{0:i-1})\]

During training, no loss is incurred for lower-level variables with no values, i.e., variables that cannot take values due to earlier variable values (e.g., there is no targeted group for posts marked as non-offensive).

In our experiments we use pretrained versions of OpenAI's GPT and GPT2Radford et al. (2018, 2019) for our model variants, named SBF-GPT\({}_{1}\) and SBF-GPT\({}_{2}\), respectively. While their architectures are similar (stack of Transformers), GPT was trained on a large corpus of fiction books, whereas GPT2 was trained on 40Gbbs of English web text.

InferenceWe frame our inference task as a conditional language generation task. Conditioned on the post, we generate tokens one-by-one either by greedily selecting the most probable one, or by sampling from the next word distribution, and appending the selected token to the output. We stop when the \([{\tt END}]\) token is generated, at which point our entire frame is predicted. For greedy decoding, we only generate our frames once, but for sampling, we repeat the generation procedure to yield ten candidate frame predictions and choose the highest scoring one under our model.

In contrast to training time, where all inputs are consistent with our frames' structure, at test time, our model can sometimes predict combinations of variables that are inconsistent with the constraints of the frame (e.g., predicting a post to be inoffensive, but still predict it to be offensive to a group). To mitigate this issue, we also experiment with a constrained decoding algorithm (denoted "constr") that considers various global assignments of

\begin{table}
\begin{tabular}{c c|c c|c c c|c c|c c c|c c c} \hline \hline  & \multicolumn{3}{c|}{offensive} & \multicolumn{3}{c|}{intent} & \multicolumn{3}{c|}{lewd} & \multicolumn{3}{c|}{group} & \multicolumn{3}{c}{in-group} \\ model & \multicolumn{3}{c|}{42.2\% _pos. (dev.)_} & \multicolumn{3}{c|}{_44.8\% pos (dev.)_} & \multicolumn{3}{c|}{_3.0\% pos (dev.)_} & \multicolumn{3}{c|}{_66.6\% pos (dev.)_} & \multicolumn{3}{c}{_5.1\% pos (dev.)_} \\  & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. \\ \hline \multirow{3}{*}{dev.} & SBF-GPT\({}_{1}\)-gdy & 75.2 & 88.3 & 65.5 & 74.4 & 89.8 & 63.6 & 75.2 & 78.2 & 72.5 & 62.3 & 74.6 & 53.4 & – & – & – \\  & SBF-GPT\({}_{2}\)-gdy & 77.2 & 88.3 & 68.6 & **76.3** & 89.5 & 66.5 & 77.6 & 81.2 & 74.3 & **66.9** & 67.9 & 65.8 & **24.0** & 85.7 & 14.0 \\  & SBF-GPT\({}_{2}\)-smp & **80.5** & 84.3 & 76.9 & 75.3 & 89.9 & 64.7 & **78.6** & 80.6 & 76.6 & 66.0 & 67.6 & 64.5 & – & – & – \\ \hline test & SBF-GPT\({}_{2}\)-gdy & 78.8 & 89.8 & 70.2 & 78.6 & 90.8 & 69.2 & 80.7 & 84.5 & 77.3 & 69.9 & 70.5 & 69.4 & – & – & – \\ \hline \hline \end{tabular}
\end{table}
Table 4: Experimental results (%) of various models on the classification tasks (gdy: argmax, smp: sampling). Some models did not predict the positive class for “in-group language,” their performance is denoted by “–”. We bold the \(F_{1}\) scores of the best performing model(s) on the development set. For easier interpretation, we also report the percentage of instances in the positive class in the development set.

 

variables. Specifically, after greedy decoding, we recompute the probabilities of each of the categorical variables, and search for the most probable assignment given the generated text candidate and variable probabilities.9 This can allow variables to be assigned an alternative value that is more globally optimal.10

Footnote 9: We only use the possible assignments in the same forward pass; we do not use assignments from different samples.

Footnote 10: In practice, as seen in Tables 4, 5, and 7, this only slightly improves predictions.

### Evaluation

We evaluate performance of our models in the following ways. For classification, we report precision, recall, and scores of the positive class. Following previous generative inference work Sap et al. (2019), we use automated metrics to evaluate model generations. We use BLEU-2 and RougeL (\(F_{1}\)) scores to capture word overlap between the generated inference and the references, which captures quality of generation Galley et al. (2015); Hashimoto et al. (2019). We additionally compute word mover's distance (WMD; Kusner et al. (2015), which uses distributed word representations to measure similarity between the generated and target text.11

Footnote 11: We use GloVe trained on CommonCrawl, as part of the SpaCy encorewebmd package.

### Training Details

As each post can contain multiple annotations, we define a training instance as containing one post-group-statement triple (along with the five categorical annotations). We then split our dataset into train/dev./test (75:12.5:12.5), ensuring that no post is present in multiple splits. For evaluation (dev., test), we combine the categorical variables by averaging their binarized values and re-binarizing using a .5 threshold, and compare the generated inferences (hypotheses) to all targeted groups and implied statements (references).

All experiments are carried out using HuggingFace's Transformers library.12 We tune hyperparameters on the dev. set, and report performance for the best performing setting (according to average \(F_{1}\)). We train or finetune our models using a batch size of 4, a learning rate of \(5\times 10^{-6}\) for GPT and \(10^{-5}\) for GPT2 (both with linear warm up), and consider training for \(e\in\{1,2,5\}\) epochs.

Footnote 12: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

## 5 Results

Listed in Tables 4 and 5, our modelling results indicate that making inferences about social biases in language is challenging for these models.

ClassificationShown in Table 4, models perform well on higher-level variables such as offensiveness and lewdness, despite the latter being heavily skewed. We hypothesize that correctly predicting lewdness might require more lexical matching (e.g., detecting words with sexual connotations).

Whether a group is targeted is slightly less easy for models to predict, and whether the language is in-group is even more challenging, with most of the models defaulting to never predicting it. This highly skewed category poses a challenge for all models, likely due to subtlety of the task and the lack of positive instances.

SBF-GPT\({}_{2}\)-gdy is the only model that predicts positive values for in-group language, for which it benefits from constrained decoding with a 1.9% improvement in \(F_{1}\) score (we show results with all constrained decoding variants in Table 7 in the appendix).

\begin{table}
\begin{tabular}{c c|c c c|c c c} \hline \hline  & & \multicolumn{3}{c|}{group targeted} & \multicolumn{3}{c}{implied statement} \\  & & BLEU & Rouge-L & WMD & BLEU & Rouge-L & WMD \\ \hline \multirow{8}{*}{dev.} & SBF-GPT\({}_{1}\)-gdy & 69.9 & 60.3 & 1.01 & **49.9** & 40.2 & 2.97 \\  & SBF-GPT\({}_{1}\)-gdy-constr & 69.2 & 64.7 & 1.05 & 49.0 & 42.8 & 3.02 \\  & SBF-GPT\({}_{2}\)-gdy & 74.2 & 64.6 & 0.90 & 49.8 & 41.4 & **2.96** \\  & SBF-GPT\({}_{2}\)-gdsr-constr & 73.4 & **68.2** & 0.89 & 49.6 & **43.5** & **2.96** \\  & SBP-GPT\({}_{2}\)-smp & **83.2** & 33.7 & **0.62** & 44.3 & 17.8 & 3.31 \\  & SBP-GPT\({}_{2}\)-smp-constr & 83.0 & 33.7 & 0.63 & 44.1 & 17.9 & 3.31 \\ \hline \multirow{2}{*}{test} & SBF-GPT\({}_{2}\)-gdy & 77.0 & 71.3 & 0.76 & 52.2 & 46.5 & 2.81 \\  & SBF-GPT\({}_{2}\)-gdy-constr & 77.9 & 68.7 & 0.74 & 52.6 & 44.9 & 2.79 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Automatic evaluation of various models on the generation task. We bold the scores of the best performing model(s) on the development set. Higher is better for BLEU and ROUGE scores, and lower is better for WMD.

 

GenerationWhen evaluating our models on the generation tasks (i.e., targeted group and implied statement), we find that no one model outperforms others across all metrics (Table 5).

Overall, models do well at generating the targeted groups, likely because of the more limited generation space (there are only 1.4k possible groups in SBIC). Conversely, for implied statement generation (where output space is much larger), model performance is slightly worse.

Similar to the classification tasks, SBF-GPT\({}_{2}\)-gdy shows a slight increase in RougeL score when using constrained decoding, but we see a slight drop in BLEU scores.

Error analysisSince small differences in automated evaluation metrics for text generation sometimes only weakly correlate with human judgments Liu et al. (2016), we manually perform an error analysis on a manually selected set of generated development-set examples from the SBF-GPT\({}_{2}\)-gdy-constr model (Table 6). Overall, the model seems to struggle with generating textual implications that are relevant to the post, instead generating very generic stereotypes about the demographic groups (e.g., in examples b and c). The model generates the correct stereotypes when there is high lexical overlap with the post (e.g., examples d and e). This is in line with previous research showing that large language models rely on correlational patterns in data Sap et al. (2019c; Sakaguchi et al. (2020).

## 6 Related Work

Bias and toxicity detectionDetection of hateful, abusive, or other toxic language has received increased attention recently Schmidt and Wiegand (2017), and most dataset creation work has cast this detection problem as binary classification Waseem and Hovy (2016); Davidson et al. (2017); Founta et al. (2018). Moving beyond a single binary label, wulczyn et al. (2017) and the PerspectiveAPI use a set of binary variables to annotate Wikipedia comments for several toxicity-related categories (e.g., identity attack, profanity). Similarly, Zampieri et al. (2019) hierarchically annotate a dataset of tweets with offensiveness and whether a group or individual is targeted. Most related to our work, Ousidhoum et al. (2019) create a multilingual dataset of 13k tweets annotated for five different emotion- and toxicity-related aspects, including a 16-class variable representing social groups targeted. In comparison, Social Bias Frames not only captures binary toxicity and hierarchical information about whether a group is targeted, but also _free-text_ implications about 1.4k different targeted groups and the implied harm behind statements.

Similar in spirit to this paper, recent work has tackled more subtle bias in language, such as microaggression Breitfeller et al. (2019) and condensation Wang and Potts (2019). These types of biases are in line with the biases covered by Social Bias Frames, but more narrowly scoped.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline post & predicted group & predicted implication & reference groups & reference implications \\ \cline{2-6}  & All-female casts bother me but I have literally never noticed the hundreds of movies with 0 female characters I have already watched & — & women & women aren’t good actors; women can’t be in leading roles \\ \cline{2-6}  & As expected, when the terrorist group Hamas won the election in Gaza it was the usual one man, one vote, one time, Islamist election. & muslim folks, terrorists & are muslim folks, arabic folks & bad people; islam promotes controlling governments; muslims are authoratarians; not fair \\ \cline{2-6}  & “Black guy in class: ”attempts to throw a paper ball into the trash*” misses* Mrs. Mundy: ”You’re a disagree to your race, Marcus" & black folks are trash & black people & black folks are dead for basketball; black men are defined by athletic skill; black men should be good at basketball.; do not take things seriously & are defined by athletic skill; black men should be good at basketball.; do not take things seriously & men should be good at basketball.; does not take things seriously & men should be good at basketball.;; does not take things seriously & don not take things seriously \\ \cline{2-6}  & “How can a Libhard speak truth vomit from their mouths is lies? & liberals & liberals & stipapd & liberals are micropentier; liberals are stupid; liberals only speak lies \\ \cline{2-6}  & “overweight/fat folds & overweight/fat ugly & fat people are & overweight/fat folks & are not pleasant people; fat folks are all the same; fat folks are less than others; not likable \\ \hline \hline \end{tabular}
\end{table}
Table 6: Examples of SBF-GPT\({}_{2}\)-gdy-constr model predictions. The model struggles to pick up on subtle biases (a), and tends to generate generic stereotypes rather than implications that are entailed by the post (b, c).

 

Inference about social dynamicsVarious work has tackled the task of making inferences about power and social dynamics. Particularly, previous work has analyzed power dynamics about specific entities, either in conversation settings (Prabhakaran et al., 2014; Danescu-Niculescu-Mizil et al., 2012) or in narrative text (Sap et al., 2017; Field et al., 2019; Antoniak et al., 2019). Additionally, recent work in commonsense inference has focused on mental states of participants of a situation (e.g., Rashkin et al., 2018; Sap et al., 2019b). In contrast to reasoning about particular individuals, our work focuses on biased implications of social and demographic groups as a whole.

## 7 Ethical Considerations

Risks in deploymentAutomatic detection of offensiveness or reasoning about harmful implications of language should be done with care. When deploying such algorithms, ethical aspects should be considered including which performance metric should be optimized (Corbett-Davies et al., 2017), as well as the fairness of the model on speech by different demographic groups or in different varieties of English (Mitchell et al., 2019). Additionally, deployment of such technology should discuss potential nefarious side effects, such as censorship (Ullmann and Tomalin, 2019) and dialect-based racial bias (Sap et al., 2019a; Davidson et al., 2019). Finally, offensiveness could be paired with promotions of positive online interactions, such as emphasis of community standards (Does et al., 2011) or counterspeech (Chung et al., 2019; Qian et al., 2019).

Risks in annotationRecent work has highlighted various negative side effects caused by annotating potentially abusive or harmful content (e.g., acute stress; Roberts, 2016). We mitigated these by limiting the number of posts that one worker could annotate in one day, paying workers above minimum wage ($7-12), and providing crisis management resources to our annotators.13 Additionally, we acknowledge the implications of using data available on public forums for research (Zimmer, 2018) and urge researchers and practitioners to respect the privacy of the authors of posts in SBIC (Ayers et al., 2018).

Footnote 13: We direct workers to the Crisis Text Line ([https://www.crisistextline.org/](https://www.crisistextline.org/)).

## 8 Conclusion

To help machines reason about and account for societal biases, we introduce Social Bias Frames, a new structured commonsense formalism that distills knowledge about the biased implications of language. Our frames combine categorical knowledge about the offensiveness, intent, and targets of statements, as well as free-text inferences about which groups are targeted and biased implications or stereotypes. We collect a new dataset of 150k annotations on social media posts using a new crowdsourcing framework and establish baseline performance of models built on top of large pretrained language models. We show that while classifying the offensiveness of statements is easier, current models struggle to generate relevant social bias inferences, especially when implications have low lexical overlap with posts. This indicates that more sophisticated models are required for Social Bias Frames inferences.

## Acknowledgments

We thank the anonymous reviewers for their insightful comments. Additionally, we are grateful to Hannah Rashkin, Lucy Lin, Jesse Dodge, Hao Peng, and other members of the UW NLP community for their helpful comments on the project. This research was supported in part by NSF (IIS-1524371, IIS-1714566), DARPA under the CwC program through the ARO (W911NF-15-1-0543), and DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031).

## References

* Antoniak et al. (2019) Maria Antoniak, David Mimno, and Karen Levy. 2019. Narrative paths and negotiation of power in birth stories. In _CSCW_.
* Ayers et al. (2018) John W Ayers, Theodore L Caputi, Camille Nebeker, and Mark Dredze. 2018. Don't quote me: reverse identification of research participants in social media studies. _NPJ digital medicine_, 1(1):1-2.
* Becker et al. (2017) Maria Becker, Michael Staniek, Vivi Nastase, and Anette Frank. 2017. Enriching argumentative texts with implicit knowledge. In _NDB_.
* Bicknell (2007) Jeanette Bicknell. 2007. What is offensive about offensive jokes? _Philosophy Today_, 51(4):458-465.
* Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialactical variation in social media: a case study of African-American English. In _EMNLP_ 

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for automatic knowledge graph construction. In _ACL_. Bowman-Grieve (2009) Lorraine Bowman-Grieve. 2009. Exploring "Stormfront": a virtual community of the radical right. _Studies in conflict & terrorism_, 32(11):989-1007.
* Breitfeller et al. (2019) Luke M Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: a case for locating elusive phenomena in social media posts. In _EMNLP_.
* Bussone et al. (2015) Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In _2015 International Conference on Healthcare Informatics_, pages 160-169. IEEE.
* Center (2012) Southern Poverty Law Center. 2012. Misogyny: the sites. _Intelligence Report_, 145.
* COunter NAratives through nichesourcing: a multilingual dataset of responses to fight online hate speech. In _ACL_.
* Cohen-Almagor (2014) Raphael Cohen-Almagor. 2014. Countering hate on the internet. _Annual review of law and ethics_, 22:431-443.
* Corbett-Davies et al. (2017) Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In _KDD_.
* Croom (2011) Adam M Croom. 2011. Slurs. _Language Sciences_, 33(3):343-358.
* Daly (2018) Helen L Daly. 2018. On insults. _Journal of the American Philosophical Association_, 4(4):510-524.
* Danescu-Niculescu-Mizil et al. (2012) Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg. 2012. Echoes of power: language effects and power differences in social interaction. In _WWW_.
* Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In _Abustive Language Workshop_.
* Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael W Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In _ICWSM_.
* Does et al. (2011) Serena Does, Belle Derks, and Naomi Ellemers. 2011. Thou shalt not discriminate: how emphasizing moral ideals rather than obligations increases whites' support for social equality. _Journal of Experimental Social Psychology_, 47(3):562-571.
* Dynel (2015) Marta Dynel. 2015. The landscape of impoliteness research. _Journal of Politeness Research_, 11(2):383.
* Eble (1996) Connie C Eble. 1996. _Slang & sociability: in-group language among college students_. Univ of North Carolina Press.
* Field et al. (2019) Anjalie Field, Gayatri Bhat, and Yulia Tsvetkov. 2019. Contextual affective analysis: a case study of people portrayals in online #MeToo stories. In _ICWSM_.
* Fillmore and Baker (2001) Charles J Fillmore and Collin F Baker. 2001. Frame semantics for text understanding. In _Proceedings of WordNet and Other Lexical Resources Workshop, NAACL_.
* Fiagsa (2017) Jon Fiagsa. 2017. Reddit bans misogynist community as part of anti- violence crackdown. [https://www.engadget.com/2017/11/08/reddit-bans-misogynist-community-in-anti-violence-crackdown/](https://www.engadget.com/2017/11/08/reddit-bans-misogynist-community-in-anti-violence-crackdown/). Accessed: 2019-12-06.
* Fiske (1993) Susan T Fiske. 1993. Controlling other people. the impact of power on stereotyping. _American psychologist_, 48(6):621-628.
* Fontua et al. (2018) Antigoni-Maria Fontua, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of Twitter abusive behavior. In _ICWSM_.
* Gabriel (1998)Yiannis Gabriel. 1998. An introduction to the social psychology of insults in organizations. _Human Relations_, 51(11):1329-1354.
* Galinsky et al. (2013) Adam D Galinsky, Cynthia S Wang, Jennifer A Whitson, Eric M Aniciich, Kurt Hugenberg, and Galen V Bodenhausen. 2013. The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling. _Psychcol. Sci._, 24(10):2020-2029.
* Galley et al. (2015) Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and William B. Dolan. 2015. deltaBLEU: a discriminative metric for generation tasks with intrinsically diverse targets. In _ACL_.
* de Gibert et al. (2018) Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. In _Abusive Language Workshop at EMNLP_.
* Greengross and Miller (2008) Gil Greengross and Geoffrey F Miller. 2008. Dissing oneself versus Missing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Deprecating humor. _Evolutionary Psychology_, 6(3).
* Gregor and Benbasat (1999) Shirley Gregor and Izak Benbasat. 1999. Explanations from intelligent systems: Theoretical foundations and implications for practice. _MIS quarterly_, pages 497-530 

Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? empirical analysis and detecting attributes of convincineness in web argumentation. In _EMNLP_, pages 1214-1223.
* Hashimoto et al. (2019) Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In _NAACL-HLT_.
* Hearst (1992) Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In _ACL_, pages 539-545.
* Hess (2016) Amanda Hess. 2016. The far right has a new digital safe space. [https://www.nytimes.com/2016/11/30/arts/the-far-right-has-a-new-digital-safe-space.html](https://www.nytimes.com/2016/11/30/arts/the-far-right-has-a-new-digital-safe-space.html). Accessed: 2019-12-06.
* Kasper (1990) Gabriele Kasper. 1990. Linguistic politeness: current research issues. _Journal of Pragmatics_, 14(2):193-218.
* Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: a conditional transformer language model for controllable generation. _arXiv preprint arXiv:1909.05858_.
* Kulesza et al. (2012) Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_, pages 1-10. ACM.
* Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In _ICML_, pages 957-966.
* Lakoff (1973) Robin Lakoff. 1973. Language and woman's place. _Language in society_, 2(1):45-79.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. In _ACL_.
* de Marneffe et al. (2012) Marie-Catherine de Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. _Computational Linguistics_, 38(2):301-333.
* Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Rajii, and Timnit Gebru. 2019. Model cards for model reporting. In _FAccT_.
* Nguyen et al. (2018) An T Nguyen, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Elizabeth Tate, Byron C Wallace, and Matthew Lease. 2018. Believe it or not: designing a human-AI partnership for mixed-initiative fact-checking. In _The 31st Annual ACM Symposium on User Interface Software and Technology_, pages 189-199. ACM.
* O'Dea et al. (2015) Conor J O'Dea, Stuart S Miller, Emma B Andres, Madelyn H Ray, Derrick F Till, and Donald A Saucier. 2015. Out of bounds: Factors affecting the perceived offensiveness of racial slurs. _Language Sciences_, 52:155-164.
* Osudhoum et al. (2019) Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and Multi-Aspect hate speech analysis. In _EMNLP_.
* Pereira et al. (2016) Goncalo Pereira, Rui Prada, and Pedro A Santos. 2016. Integrating social power into the decision-making of cognitive agents. _Artificial Intelligence_, 241:1-44.
* Prabhakaran et al. (2014) Vinodkumar Prabhakaran, Prabhakaran Vinodkumar, and Rambow Owen. 2014. Predicting power relations between participants in written dialog from a single thread. In _ACL_.
* Qian et al. (2019) Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Bedding, and William Yang Wang. 2019. A benchmark dataset for learning to intervene in online hate speech. In _EMNLP_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Unpublished.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Unpublished.
* Rashkin et al. (2018) Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2mind: commonsense inference on events, intents, and reactions. In _ACL_. Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should I trust you?": Explaining the predictions of any classifier. In _KDD_.
* Roberts (2016) Sarah T Roberts. 2016. Commercial content moderation: digital laborers' dirty work. In Safiya Umoja Noble and Brendesha M Types, editors, _The Intersectional Internet: Race, Sex, Class and Culture Online_, Media Studies Publications. Peter Lang Publishing.
* Ross et al. (2017) Bjorn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In _NLP 4 CMC Workshop_.
* Williams (2017) RWJF. 2017. Discrimination in america: experiences and views. [https://www.rwjf.org/en/library/research/2017/10/discrimination-in-america--experiences-and-views.html](https://www.rwjf.org/en/library/research/2017/10/discrimination-in-america--experiences-and-views.html). Accessed: 2019-11-5.
* Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: an adversarial winograd schema challenge at scale. In _AAAI_ 

Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019a. The risk of racial bias in hate speech detection. In _ACL_.
* Sap et al. (2019b) Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019b. ATOMIC: an atlas of machine commonsense for if-then reasoning. In _AAAI_.
* Sap et al. (2017) Maarten Sap, Marcella Cindy Prasetio, Ariel Holtzman, Hannah Rashkin, and Yejin Choi. 2017. Connotation frames of power and agency in modern films. In _EMNLP_.
* Sap et al. (2019c) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019c. Social IQA: commonsense reasoning about social interactions. In _EMNLP_.
* Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In _Workshop on NLP for Social Media at EACL_.
* Speer and Havasi (2012) Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in ConceptNet 5. In _IREC_.
* Strub (2008) Whitney Strub. 2008. The clearly obscene and the query obscure: heteronormativity and obscenity in cold war los angeles. _American Quarterly_, 60(2):373-398.
* Ullmann and Tomalin (2019) Stefanie Ullmann and Marcus Tomalin. 2019. Quarantining online hate speech: technical and ethical perspectives. _Ethics and Information Technology_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NeurIPS_.
* Vincent (2016) James Vincent. 2016. Twitter taught Microsoft's AI chatbot to be a racist asshole in less than a day. [https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist](https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist). Accessed: 2019-10-26.
* Vonasch and Baumeister (2017) Andrew J Vonasch and Roy F Baumeister. 2017. Unjustified side effects were strongly intended: taboo tradeoffs and the side-effect effect. _Journal of Experimental Social Psychology_, 68:83-92.
* Wang and Potts (2019) Zijian Wang and Christopher Potts. 2019. TalkDown: a corpus for condescension detection in context. In _EMNLP_.
* Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In _NAACL Student Research Workshop_.
* Wolczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: personal attacks seen at scale. In _WWW_.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In _NACL_.
* Zimmer (2018) Michael Zimmer. 2018. Addressing conceptual gaps in big data research ethics: an application of contextual integrity. _Social Media + Society_, 4(2).

 



\begin{table}
\begin{tabular}{c c|c c|c c c|c c c|c c c|c c c} \hline \hline  & \multicolumn{3}{c|}{offensive} & \multicolumn{3}{c|}{intent} & \multicolumn{3}{c|}{lewd} & \multicolumn{3}{c|}{group} & \multicolumn{2}{c}{in-group} \\ model & \multicolumn{3}{c|}{_42.2\% pos. (dev.)_} & \multicolumn{3}{c|}{_44.8\% pos. (dev.)_} & \multicolumn{3}{c|}{_3.0\% pos. (dev.)_} & \multicolumn{3}{c|}{_66.6\% pos. (dev.)_} & \multicolumn{3}{c}{_5.1\% pos. (dev.)_} \\  & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. & \(F_{1}\) & pr. & rec. \\ \hline \multirow{7}{*}{**SBF-GPT\({}_{1}\)-gdy**} & SBF-GPT\({}_{1}\)-gdy & 75.2 & 88.3 & 65.5 & 74.4 & 89.8 & 63.6 & 75.2 & 78.2 & 72.5 & 62.3 & 74.6 & 53.4 & – & – & – \\  & \({}^{\prime\prime}\)-constr & 75.2 & 88.3 & 65.5 & 74.4 & 89.8 & 63.6 & 75.2 & 78.2 & 72.5 & 62.3 & 74.6 & 53.4 & – & – & – \\  & SBF-GPT\({}_{2}\)-gdy & 77.2 & 88.3 & 68.6 & **76.3** & 89.5 & 66.5 & 77.6 & 81.2 & 74.3 & **66.9** & 67.9 & 65.8 & 24.0 & 85.7 & 14.0 \\  & \({}^{\prime\prime}\)-constr & 77.2 & 88.3 & 68.6 & **76.3** & 89.5 & 66.5 & 77.6 & 81.2 & 74.3 & **66.9** & 67.9 & 65.8 & **25.9** & 63.6 & 16.3 \\  & SBF-GPT\({}_{2}\)-smp & **80.5** & 84.3 & 76.9 & 75.3 & 89.9 & 64.7 & **78.6** & 80.6 & 76.6 & 66.0 & 67.6 & 64.5 & – & – & – \\  & \({}^{\prime\prime}\)-constr & 80.4 & 84.3 & 76.8 & 75.3 & 89.9 & 64.7 & 78.5 & 80.6 & 76.5 & 66.0 & 67.6 & 64.5 & – & – & – \\ \cline{2-16}  & SBF-GPT\({}_{2}\)-gdy & 78.8 & 89.8 & 70.2 & 78.6 & 90.8 & 69.2 & 80.7 & 84.5 & 77.3 & 69.9 & 70.5 & 69.4 & – & – & – \\  & \({}^{\prime\prime}\)-constr & 78.8 & 89.8 & 70.2 & 78.6 & 90.8 & 69.2 & 80.7 & 84.5 & 77.3 & 69.9 & 70.5 & 69.4 & – & – & – \\ \hline \end{tabular}
\end{table}
Table 7: Full experimental results (%) of various models on the classification tasks (gdy: argmax, smp: sampling; constr: constrained decoding). Some models did not predict the positive class for “in-group language,” their performance is denoted by “–”. We bold the \(F_{1}\) scores of the best performing model(s) on the development set. For easier interpretation, we also report the percentage of instances in the positive class in the development set.



* [16] M. C. B. Wilson, R. J. S. Thorne, and D. W. T. Jones, _Theoretical Computer Science_**2004**, 375-398 (1999)
* [17] A. K. Wilson, _The Theory of Computation: An Introduction to the Theory of Computing_, Cambridge University Press, Cambridge, MA, USA, 1999.
* [18] G. E. Hinton, _A Practical Guide for Machine Learning_, Springer, New York, NY, 1999.
* [19] L. H. Friedman, _Computational Statistics_**199**, 1-12 (1992)
* [20] P. F. Hastie, _Random Forests: The Structure of a Neural Network_, Oxford University Press, Oxford, 1999.

[MISSING_PAGE_POST]