BiasX: “Thinking Slow” in Toxic Content Moderation with Explanations of Implied Social Biases
Authors: Yiming Zhang
Sravani Uttara Nanduri
Liwei Jiang
Tongshuang Wu
Maarten Sap
Venue: EMNLP 2023 Main (2023)
Keywords: Social biases
Toxicity moderation
Human-AI collaboration
Free-text explanations

Abstract: Toxicity annotators and content moderators often default to mental shortcuts when making decisions. This can lead to subtle toxicity being missed, and seemingly toxic but harmless content being over-detected. We introduce BiasX, a framework that enhances content moderation setups with free-text explanations of statements' implied social biases, and explore its effectiveness through a large-scale crowdsourced user study. We show that indeed, participants substantially benefit from explanations for correctly identifying subtly (non-)toxic content. The quality of explanations is critical: imperfect machine-generated explanations (+2.4% on hard toxic examples) help less compared to expert-written human explanations (+7.2%). Our results showcase the promise of using free-text explanations to encourage more thoughtful toxicity moderation.

BIASX: "Thinking Slow" in Toxic Content Moderation with Explanations of Implied Social Biases
Warning: content paper may be upsetting or offensive.
Yiming Zhang
Sravani Nanduri Liwei Jiang Tongshuang Wu Maarten Sap*
Carnegie Mellon University
University of Washington yimingz 06 cmu. edu, maartensap@cmu edu
Abstract
No, can you get one of the boys to carry that out?
It's too heavy for you.
Toxicity annotators content moderators of- ten default to mental shortcuts when making decisions. This can lead to subtle toxicity being missed, and seemingly toxic but harmless con- tent being over-detected. We introduce BIASX, a framework that assists content moderators with free-text explanations of statements' im- plied social biases, and explore its effectiveness through a large-scale user study. We show that participants indeed benefit substantially from explanations for correctly moderating subtly mion-)toxie content. The quality of explana- tions is critical: imperfect machine-genenae generated explanations (+2.4% on hard toxic examples) help less compared to expert-written human ex- planations (+7.2%). Our results showcase the promise using free-text explanations to en- courage more thoughtful toxicity moderation.!
"Thinking fast' no explanations
Targeted group:
Implies women are physically weak
:Allow
"Thinking slow" (BiasX)
Moderate
We propose the BIASX framework to help moderators think through the biased prejudiced im- plications of statements with free-text explanations, in contrast to most existing moderation paradigms which provide little to explanations.
Figure I:
Tversky and Kahneman (1974), mental shortcuts are heuristics people employ in decision-making when facing uncertainty or pressure. Exacerbated by an increasing time pressure to moderate con- tent (Roberts, 2019), the use of these heuristics by annotators can cascade to poor dataset quality and biased models (Malaviya et al., 2022).
1 Introduction
Content moderation and online hate speech de- tection frameworks, which aim to flag prejudiced, hateful, or otherwise toxic content, often fall prey to spurious correlations and lexical biases when de- ployed (Draws et al., 2021). For example, content moderators often assume that statements without hateful or profane words are not prejudiced or toxic (such as the subtly sexist statement in Figure 1), without deeper reasoning about potentially biased implications (Breitfeller al., 2019; Sap et al., 2022). In the meantime, they overflag benign posts such as those that contain expletives or in-group phrases used by minorities (Dixon et al., 2018; Sap et al., 2019). These biases present in content mod- eration can suppress harmless speech by and about minorities (Yasin, 2018), and risk hindering equi- table online experiences.
To mitigate such shortcuts, we introduce BIASX, framework to enhance content moderators' de- cision making with free-text explanations of a potentially toxic statement's targeled group and subtle biased prejudiced implication (Figure 1). We take inspiration from cognitive science's dual process theory (James et al., 1890), BIASX is meant to encourage more conscious reasoning about statements beyond what is written ("think- ing slow"; Kahneman, 2011) 1), to circumvent the mental shortcuts and cognitive heuristics resulting from automatic processing ("thinking fast") that often lead to drop in model and human perfor- mance alike (Malaviya al., 2022). 2 To this end, ground BIASX in SOCIAL BIAS FRAMES (Sap et al., 2020), linguistic framework that explicit spells out the biases and offensiveness implied- but not written in text.
major cause of such biases in moderation is the use of mental shortcuts by moderators. Defined by
Note, "thinking slow" refers a deeper and more thought- ful reasoning ahout statements and their implications, not necessarily slower in terms of reading or decision time.
OOu code is publicly available at https: //gi thub. com/ y 0mi wgthangggggggg.........nnggggg /blasx.

---

Via large-scale (NV > 450) user study, we eval- uate the usefulness of BIASX, and explore three primary research questions: (1) When do free-text explanations help improve the content moderation quality, and how? (2) Is the explanation format in BIASX effective? and (3) How might the qual- ity of the explanations affect their helpfulness? Our results show that BIASX indeed helps mod- erators better detect hard, subtly toxic instances, as reflected both in increased moderation perfor- mance and subjective feedback, demonstrating the promise of domain-specific free-text explanations. Notably, we also find that explanation quality matters: models sometimes miss the veiled biases that are present in text, making their explanations unhelpful or even counterproductive for users. Our findings serve as proof of concept in showing the promise of free-text explanations in improv- ing content moderation fairness, while highlighting the need for Al systems that are more capable of identifying and explaining subtle biases in text.
explanations of offensiveness, which has the poten- tial of communicating rich information to humans.
Implied Social Biases. To maximize its utility, we further design BIASX to optimize for content moderation, by grounding the explanation format in the established SOCIAL BIAS FRAMES (SBF; Sap et al., 2020) formalism. SBF distills biases and offensiveness that are implied in language, and its definition and demonstration of implied stereo- type naturally allows us for explaining subtly toxic statements. Specifically, for toxic posts, BIASX ex- planations take the same format as SOCIAL BIAS FRAMES, which spells out both the targeted group and the implied stereotype, as shown in Figure 1. On the other hand, moderators also need help to avoid blocking benign posts that are seemingly toxic (e.g.. positive posts with expletives or state- ments denouncing biases). To accommodate this need, we extend SOCIAL BIAS FRAMES-Style im- plications to provide explanations of why a post might be non-toxic. For non-toxic statement, the explanation acknowledges the (potential) aggres- siveness of the statement while noting the lack of toxicity: given the statement "This is fucking an- noying because it keeps raining in my country", BIASX could provide an explanation "Uses profan- ity without prejudice or hate".
2 Explaining (Nxiiitoxicity with BIASX
The goal of our work is help content modera- tors reason through whether statements could be subtly or implicitly biased, prejudiced, or offensive to help them explicitly flag microaggressions and social biases projected by statements which are often missed, and alleviate the over-moderation of deceivingly non-toxic statements (Dixon et al., 2018; Dinan et al., 2019; Sap et al., 2022). To do so, we propose BIASX, framework for assist- ing content moderators with free-text explanations of implied social biases. There are two primary design desiderata in the design of BIASX:
Experiment Design
To explore the following questions, we conduct a user study to measure the effectiveness of BIASX. Q.1 Does BIASX improve the content moderation quality, especially on challenging instances?
Q.2 Do BIASX explanations enable moderators to think carefully about moderation decisions?
Free-text explanations. Identifying and explain- ing implicit biases in online social interactions is difficult, as the underlying stereotypes rarely stated explicitly by definition; this is nonetheless important due the risk of harm to individu- als (Williams, 2020). Psychologists have argued that common types of explanation in literature, such as highlights and rationales (e.g., Lai et al., 2020; Vasconcelos et al., 2023) or classifier confi- dence scores (e.g., Bansal et al., 2021) are of lim- ited utility to humans (Miller, 2019). This moti- vates the need for explanations that go beyond what is written. Inspired by Gabriel et al. (2022) who use AI- Al-generated free-text explanations of an author's likely intent to help users identify misinformation in news headlines, we propose to focus on free-text
Q.3 Are higher quality explanation more effective?
To answer these questions, we design crowd- sourced user study that simulates real con- tent moderation environment: crowdworkers are asked to play the role of content moderators, and to judge the toxicity of series of 30 online posts, po- tentially with explanations from BIASX. Our study incorporates examples of varying difficulties and different forms of explanations as detailed below.
3.1 Experiment Setup
Conditions. Participants in diffeernt conditions have access to different kinds of explanation assis- tance. To answer Q.1 and Q.2, we set two base-
'A non-toxic statement by definition does not target any minority group, and we use "N/A" as a filler.
2

---

line conditions: (1) NO-EXPL, where participants make decisions without seeing any explanations; (2) LIGHT-EXPL, where we provide only the tar- geted group as the explanation. This can be con- sidered an ablation of BIASX with the detailed implied stereotype on toxic posts and justification on non-toxic posts removed, and helps us verify the effectiveness of our explanation format. Further, to answer Q.3, we add two BIASX conditions, with varying qualities of explanations following Bansal et al. (2021): (3) HUMAN-EXI with high quality explanations manually written by experts, and (4) MODEL-EXPL. with possibly imperfect machine- generated explanations.
Moderation labels. Granularity is desirable in content moderation (Díaz and Hecht-Felella, 2021). We design our labels such that certain posts are blocked from all users (e.g.. for inciting violence against marginalized groups), while others are pre- sented with warnings (c.g., for projecting a sub- tle stereotype). Loosely following the modera- tion options available to Reddit content moderators, provide participants with four options: Allow, Lenient, Moderate, and Block. They differ both in the severity of toxicity, and the corresponding effect (c.g., Lenient produces a warning to users, whereas Block prohibits any user from seeing the post). Mirroring most content moderation settings, the goal is for participants to correctly identify the right label for each post, following prescriptive paradigm of data labeling (Rottger et al., 2022). Appendix shows the label definitions.
Data selection. To better tease out when BIASX could be effective, we examine both easy and hard examples; consider example hard if it is likely to be mislabeled by an annotator using sim- ple heuristics, such as exclusively using swear- words as the indicator for toxicity. Specifically, we define the following three categories of exam- ples: simple examples which are easily flagged as toxic or non-toxic, hard-toxic examples that could potentially be overlooked due to the subtlety of the offensiveness, and hard-non-toxic examples that could be over-flagged as toxic despite being non- toxic. We drew 30 posts (10 from each category) from the SBIC dataset (Sap et al., 2020), which contains short social media posts paired with anno- tations of toxicity and SBF-implied stereotypes. To identify hard examples, we follow Han and Tsvetkov (2020) and use a fine-tuned DeBERTa toxicity classifier (He et al., 2021) to find mis- classified instances from the SBIC test set, which are likely harder than those correctly classified. Among these, two authors removed mislabeled ex- amples, and selected 20 that they agreed were hard but could be unambiguously labeled. The full list of examples can be found in Table 3.
3.2 Study Procedure
total of N=454 participants recruited from Ama- zon MTurk are randomly assigned to one of the four conditions, in which they provide labels for 30 selected examples. Upon completion, partic- ipants also complete a post-study survey which collects their demographics information and sub- jective feedback the moderation task and pro- vided explanations (if any). We report additional details on the user study in Appendix C.
Results and Discussion
We analyze the usefulness of BIASX, examining worker moderation accuracy (Figure 2a), efficiency (Figure 2b), and subjective feedback (Figure 3).
BIASX improves moderation quality, especially on hard-toxic examples. Shown in Figure 2a, we find that HUMAN-EXPL. leads to substantial gains in moderation accuracy over the No-ExPL baseline on both hard-toxic (+7.2%) and hard-non- toxic examples (+7.7%). a result further reflected in a +4.7% improvement overall. This indicates that the free-text explanations BIASX do encourage content moderators to think more thoroughly about the toxicity of posts beyond what is written.
Explanation generation. To generate explana- tions for MODEL-EXPL, we used 5 (Ouyang et al., 2022) prompted to generate SBF-style expla- nations with 3 toxic and non-toxic in-context examples from SBIC.5 For the HUMAN-EXPL con- dition, two authors wrote SBF-style explanations.
Illustrating this effect, we show an example of hard-toxic statement against transgender peo- ple in Figure 4A. While the majority of moder- ators (60.3%) in the No-EXPL condition failed flag this post, BIASX assistance in both MODEL-- EXPL (+20.5%) and HUMAN-EXPP (+18.4%) con- ditions substantially improved moderator perfor-
*While the easy vs. hard dichotomy is fuzzy practice, we nevertheless note that data selection has no bearing on the deployment of our framework.
"We use text-dar inc 0000 as the explanation gener- ation model. Further details can he found in Appendix A.I.
3

---

hard-toxic
hard-non-tosic
easy
No-Fxpl-
Light-iapl-
61.7
65.5
57.7
B1.5
24.5
Expe
46.5
Human-Expl-
66.5
51.3
62.7
54.4
L4.
40
60
90
60
80
40
60
10
Average annotator (4-way) accuracy (%).
Median labeling time
Figure 2: Accuracy and efficiency results for the user study across evaluation sets and conditions. Error bars
represent 95% confidence intervals.
Mental demand
After you strip off his makeup and clothes. biologically he's
Expl
Response
A
(No Expl: 39.7%)
Light: Expl
strongly disegree disagree neutral
Model-Expl
Targeted group: transgender people implies that transgender people are not garder just pretendinng.
(Uight Expl: 54.5%)
Human-Expl
agree strongly agree
(Model Expl: 60.2%)
100%
0%
100%
Torgeted transgender
Percentage participants
Impliss women roal womenn.
(Human-Expl: 58.1%)
Useful for subtle
Light Expl
Alabama where eharmony and ancestry are the same website
(No-Expl: 55.4%)
Model Expl
Human-Expl
[Light-Expl: 52.7%)
100%
0%
100%
largeted groupt N/A
Percentage participants
Makes joke about unrelated websites.
(Model-Expl: 39.8%)
Figure 3: User survey results on mental demand, and whether explanations are useful for subtle stereotypes.
Torgeted group: people from Alabama
that Alaboma theyr
(Human-Expl: 60.0%)
Figure 4: Explanations and worker performances for two examples in the hard-toxic set.
mance. The subjective feedback from moderators further corroborates this observation that BIASX explanations have potential (Figure 3): the major- ity of moderators agreed strongly agreed that the BIASX explanations made them more aware of subtle stereotypes (77.1% in MODEL-EXPL: 78.15 in HUMAN-EXPLL
the LIGHT-EXPL condition, compared to 41% in MODEL-EXPI, and in HUMAN-EXPI. This result suggests that providing the targeted group exclu- sively could mislead moderators without improving accuracy efficiency.
Such a tradeoff between speed and accuracy is often necessary to ensure fair and accurate modera- tion (Jiang et al., 2023), moderators very likely resort to heuristics when they do not spend enough time their decisions. Given the correlation be- tween increased use of heuristics and decreased time spent on task in content moderation (Malaviya et al., 2022), the increase in task time with BIASX explanations further suggests the promotion of de- liberate and thoughtful decision-making.
Our designed explanation format efficiently promotes more thorough decisions. While Esplanation-assisted framework improves moder- ators' accuracy and awareness of implied biases, it can slow down their labeling compared to the control condition without any explanation, due to the extra text to read.
We can quantify and analyze this trade-off by ex- amining lighter-weight explanation type (LIGHT- EXPL) with only the targeted group, reducing the amount text workers process. Figure 2b, we indeed see a sizable increase (4-5s) in labeling time for MODEL-EXPL. and HUMAN-EXPL. Interest- ingly, LIGHT-EXPL. shares a similar increase in labeling time (m4s). As LIGHT- ExPr. has brief cxplanations (1-2 words), this increase is unlikely be due reading, but rather points additional mental processing. This extra mental processing is further evident from users' subjective evaluation in Figure 3: 56% participants agreed or strongly agreed that the task was mentally demanding in
Explanation quality matters. Compared to expert-written explanations, the effect of model- generated explanations moderator performance is mixed. A key reason behind this mixed result that model explanations can be wrong, and thus mislead moderators. In Table 1, we compare the correctness of explanations to the accuracy of par- ticipants. On the hard toxic set, only 60% of model explanations are accurate, which leads to
"Binarizing instances with moderation labels Allow and Lenient as non-toxic, and Moderate and Block as toxic.
4

---

MODEL-EXS EXPL
HUMAN-EXT
Our less significant result hard-non-toxic exam- ples also sounds a cautionary note, and shows the need for investigating more careful definitions and frameworks around non-toxic examples (e.g.. by extending SOCIAL. BIAS FRAMES), or exploring alternative designs for their explanations.
Evaluation set
E
U
U
hard toxic
60.0
56.4
100.0
64.1
hard non-toxie
90.0
77.7
100.0
80.1
easy
100.0
98.0
100.0
97.0
overall
83.3
77.4
100.0
80.4
Table I: Binary accuracy explanations (E) and users (U) in -oDEL.EXXPL and HUMAN-EXPI. conditions. .on.iiiix...oniiinnnn.
Going from proof-of-concept to practical usage. we note two additional nuances that deserve careful consideration. On the one hand, our study shows that while explanations have benefits, they come at the cost of a sizable increase in labeling time. We argue for these high-staker tasks, the increase in labeling time and cost is justifiable to degree (echoing our intent of encouraging moderators "think slow"). However, could look more into potential ways to improve performance while reducing time through, e-g.. se- lectively introducing explanations on hard exam- ples (Lai et al., 2023). This approach could aid in scaling our framework for everyday use, where the delicate balance between swift annotation and care- ful moderation is more prominent. On the other hand, our study follows set of prescriptive mod- eration guidelines (Rottger et al., 2022), written based on the researchers' definitions of toxicity. While they are similar to actual platforms' terms of service and moderation rules, they may not reflect the norms of all online communities. Customized labeling might be essential to accommodate for platform needs. We are excited to see further explo- rations around and extensions of our framework.
56.4% worker accuracy, -7.7% drop from the HUMAN-EXPL condition where workers always have access to correct explanations. Figure 4B shows an example where the model explains an implicitly toxic statement as harmless and misleads content moderators (39.8% MODEL-EXPH vs. 55.4% in No-EXPL).
do hope future work
On a positive note, expert-written explanations substantially improve moderator performance over baselines, highlighting the potential of our frame- work with higher quality explanations and serving as proof-of-coneept of BIASX, while motivating future work to explore methods to generate higher- quality explanations.
5 Conclusion and Future Work
In this work, we propose BIASX, a collaborative framework that provides Al-generated explanations to assist users in content moderation, with the ob- jective of enabling moderators to think more thor- oughly about their decisions. In an online user study, we find that when shown explanations of sub- tle biases beyond what is written, humans bet- ter identify hard-toxic examples. The even greater gain in performance with expert-written explana- tions further highlights the potential of framing content moderation under the lens of human-AI collaborative decision making, while motivating future work to build AI systems more capable of identifying and explaining biases in text beyond what current state-of- the-r-- models can do.
Limitations, Ethical Considerations & Broader Impact
While our user study of toxic content moderation is limited to examples in English and to a US- centric perspective, hate speech is hardly a mono- lingual (Ross al., 2016) monocultural (Ma- ronikolakis al., 2022) issue, and future work can investigate the extension of BIASX to languages and communities beyond English. In addition, our study uses a fixed sample of 30 curated examples. The main reason for using a small set of represen- tative examples is that it enables us to conduct the user study with a large number of participants to demonstrate salient effects across groups of partic- ipants. Another reason for the fixed sampling is the difficulty of identifying high-quality examples and generating human explanations: toxicity labels and implication annotations in existing datasets are noisy. Additional research efforts into build-
Our research highlights the importance of adding explanations to help with difficult examples (sub- tle biases) in free text. Subsequent studies could investigate various forms of free-text explanations and objectives, e.g., reasoming about intent (Gabriel et al., 2022) or distilling possible harms to the tar- geted groups (e.g.. Cabrs: Zhou et al., 2023).
'GPT-4 (OpenAI, 2023) reports 80% accuracy on the hard toxic set, 20% improvement from GPT-3.5. While still not perfect, these explanations have potential to make BIASX more effective.
5

---

ing qualtty datasets in implicit hate speech could enable larger-scale explorations of model- assisted content moderation.
form" award "ContExTox: Context-Aware and Ex- plainable Toxicity Detection."
Pre-trained language models are rife with biases that are present in their training corpus (Gehman et al., 2020; Nadeem et al., 2020). This is in-part what motivates our work on a human-Al collabora- tive framework, as opposed to full delegation to a potentially biased Al model. That said, imperfect models can generate biased explanations, poten- tially confirming annotators' own biases. Although we show that generated explanations are empir- ically useful for annotators, future work should nevertheless investigate how to better debias these language models.
References
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Ray- mond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHll Conference on Human Factors in Computing Systems, pages 1-16.
Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in So- cial Media Posts. Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1664-1674, Hong Kong, China, Association for Computational Linguistics.
Just as communities have diverging norms, anno- tators have diverse identities and beliefs, which can shift their individual perception of toxicity (Rottger et al., 2022). Similar to Sap et al. (2022) and Pei and Jurgens (2023), we find annotator performance their demographics, specifically political orientation. As shown in Figure 9 (Appendix), a more liberal participant achieves higher labeling accuracies on hard-toxic, hard-non-toxie and easy examples than a more con- servative one. This result highlights that the design of moderation scheme should take into account the varying backgrounds of annotators, cover a broad spectrum of political views, and raises inter- esting questions about whether annotator variation can be mitigated by explanations, which future work should explore.
varies greatly depending
Ángel Díaz and Laura Hecht-Felella. 2021. Dou- ble Standards in Social Media Content Moderation. Technical report. Brennan Center for Justice.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix for dialogue safety: Robustness from adversarial human attack. In EMNLP-IJCNIL
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigat- ing unintended bias in text classification. In AFES, AIES '18.
Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju. and Nava Tintarev. 2021. A checklist to combat cog- nitive biases in crowdsourcing. In AAAI Conference on Human Computation & Crowdsourcing.
Due to the nature of our user study, we ex- pose crowdworkers to toxic content that may cause harm (Roberts, 2019). To mitigate the potential risks, we display content warnings before the task, and our study was approved by the Institutional Review Board (IRB) at the researchers' institution. Finally, we ensure that study participants are paid fair wages (> $10/hr). See Appendix C for further information regarding the user study.
Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert- Georg Lang. 2009, Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 4 41(4): 1499 1160.
Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. Misinfo reaction frames: Reasoning about readers' reactions to news headlines. In ACL.
Acknowledgments
We thank workers on Amazon Mturk who partic- ipated in our online user study for making our re- scarch possible. We thank Karen Zhou, people from various paper clinies and anonymous review- ers for insightful feedback and fruitful discussions. This research was supported in part by Meta Fun- damental AI Research Laboratories (FAIR) "Dyn- abench Data Collection and Benchmarking Plat-
Sam Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A Smith. 2020. Realtoxicityrropps:: Evaluating neural toxic degeneration in language models. In Findings of EMNLP.
Xiaochuang Han and Yulia Tsvetkov. 2020. Fortifying toxic speech detectors against veiled toxicity. EMNLP.
6

---

Pengcheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DEBERTA: DECODING-
Bjöm Ross, Michael Rist, Guillermo Carbonell, Ben- jamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016. Measuring the Reliability of Hate Speech An- notations: The Case of the European Refugee Crisis
ENHANCED BERT WITH DISENTANGLED AT-
TENTION. In International Conference on Learning Representations.
Paul Rottger. Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 175-190, Seattle, United States. Association for Computational Linguisties.
William James, Frederick Burkhardt, Fredson Bowers, and Ignas K Skrupskelis. 1890. The principles of psychology, volume 1. Macmillan London.
Jialun Aaron Jiang, Peipei Nie, Jed R. Brubaker, and Casey Fiesler. 2023. A Trade-oft oof off-cenntreed Frame- work of Content Moderation. ACM Transactions on Computer- Human Interaction, 30(1):3:1-3:34
Maarten Sap. Dallas Card, Saadia Gabriel. Yejin Choi, and Noah A Smith. 2019. The risk of racial bias hate speech detection. In ACI.
Daniel Kahneman. 2011. Thinking, fast and slow.
Vivian Lai, Han Liu, and Chenhao Tan. 2020. "Why 'Chicago' Deceptive?" towards building model- driven tutorials for humans. In Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems, CHI 20, pages 1-13, New York, NY, USA. Association for Computing Machinery.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky. Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. ACL..
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In NAACI..
Vivian Lai, Yiming Zhang, Chacha Chen, Q. Vera Liao, and Chenhao Tan. 2023. Selective Explanations: Leveraging Human Input to Align Explainable AI.
A. Tversky and D. Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science (New York, N.).J, 185(4157):1124-11)
Chaitanya Malaviya, Sudeep Bhatia, and Mark Yatskar. 2022. Cascading biases: Investigating the effect of heuristic annotation strategies on data and models. In EMNLP.
Helena Vasconcelos, Matthew Jörke, Madeleine Grunde- McLaughlin, Tobias Gerstenberg, Michael Bernstein, and Ranjay Krishna. 2023. Explanations Can Re- duce Overreliance on AI Systems During Decision- Making.
Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schuetze. 2022. Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments.
Monnica Williams. 2020. Microaggressions: Clari- fication, evidence, and impact. Perspectives on Psy- chological Science, 15(1):3 26.
Tim Miller. 2019, Explanation in artificial intelligence: Insights from the social sciences. Artificial intelli- gence.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of of-te-of-the-Are Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online. Association for Computational Linguistics.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models.
OpenAl. 2023. abs/2303.08774.
Gpt-4 technical report. ArXiv,
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll I.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Danyaal banned:
Yasin. 2018. Who is free speech for? http://wwww.ineexxcccnnssshh hwt.sshiippggg////////bleee bbaaned-whoo-i-ffe---peelll 2018-12-6.
Black and
Accessed:
Jiaxin Pei and David Jurgens. 2023, When Do Annota- tor Demographics Matter? Measuring the Influence of Annotator Demographies with the POPQUORN Dataset.
Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas David- son, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. 2023. Cobra frames: Contextual rea- soning about effects and harms of offensive state- ments. In Findings of ACL.
Sarah T Roberts. 2019. Behind the screen.
7

---

A Implementation Details
A.1 Explanation Generation with LLMs
We use large language models (Ouyang et al., 2022) to generate free-text explanations. Given a statement 8, we use a pattern F' to encode offensiveness of the statement W|otf, the light explanation Egroup and the full explanation ffll in the simple format below:
F(s) = {s, [SEP),
offensive: W[off]: [SEP],
Targeted group: Egroup [SEP],
Explanation: etalt..
where [SEP] is a newline character. While we do not provide the predicted offensiveness as a part explanation to assist humans, we nevertheless include it inside the prompt, so that the generation of group and explanation is conditioned on whether the given statement is offensive.
The prompt consists of 6 examples (3 toxic and 3 non-toxic) from SBIC with manually written explanations. During evaluation, we feed the prompt GPT-3.5 (Ouyang et al., 2022) and extract the targeted group and explanation from completion. We greedily decode the offensiveness token W[ll] and sample the targeted group egroup and explanation efull with a temperature of 0.3.
A.2 Hyperparameters for fine-tuning DeBERTa
Hyperparameter
value
Batch size
16
Learning rate
10-5
Training steps
8000
Optimizer
Adam
Table 2: DeBERTa fine-tuning hyperparameters.
We HugginglFace (Wolf al., 2020) to fine-tune pre-trained deberta-v3- model. The model achieves an FI score 87.5% on the SBIC set. For the fine-tuning of the DeBERTa classifier, we used the hyperparameters reported in Table 2. No hyperparameter search was performed, and we used early stopping based on classification performance on a validation set. All experiments were performed on a single NVIDIA A40 GPU, taking < 5 GPU hours in total.
Moderation Labels
Allow: the post is not offensive or prejudiced and anyone should be able view it.
Lenient:
the post might be aggressive, angry, or use profanity/swear words, but not offensive prejudiced and does not use slurs hateful ways.
Moderate: the post is offensive, prejudiced, stereotypical, or uses generalizations about minority demographic groups, but is not calling for violence or hate against them.
the post qualifies as hate speech, or encourages violence towards minority individuals or groups and should be blocked, such that no users will be able to view this content at all.
Block:
Human Evaluation
We obtained an Institutional Review Board (IRB) approval for our user study. Prior to the user study, we conduted a power analysis to determine the scale of the experiment. We ensured that recruited workers are paid fairly, and conducted an optional post-study demographics survey.
8

---

C.1 Power Analysis
We used Powwwer (Faul al., 2009) to conduct an a priori power analysis for one-way ANOVA. With the goal of having 80% power to detect a moderate effect size of 0.15 at a significance level of 0.05, we yield a target number of 492 participants.
C.2 MTurk Setup and Participant Compensation
Our study consists of a qualification stage and a task stage. During qualification, we deployed Human Intelligence Tasks (HITs) Amazon Mechanical Turk (MTurk) in which workers go through 4 rounds of training to familiarize with the task and the user interface. Then, workers asked to label two straightforward posts without assistance.
In both the qualification phase and the task phase, we use the following MTurk qualifications: HIT Approval Rate 2 98%, Number HITs Approved 2 5000, and location is US. Among the 731 workers who participated in the qualification phase, 603 passed, and the workers were paid a median hourly wage of $10.23/h. Among the workers passing qualification, 490 participated in the task phase, in which they were further paid a median hourly wage of $14.4/h. After filtering out workers who failed the qualification questions during the task stage, our user study has 454 remaining participants.
C.3 Human Evaluation User Interface
We provide comprehensive instructions for users to complete the task, demonstrated in Figure 6. Figure 7 shows the interface for one of 4 rounds of user training, and Figure 8 shows the labeling interface, both under the MODEL-EXPP condition.
C.4 Participant Demographics
In the post-study survey, we included optional demographics survey. Among users who self-identified gender, 53.4% were male, 46. 1 % were female and 0.4% were non-binary. The majority of participants identified as White (79.9%), 6.5% as Black/African American, 6.0% as Asian/Asian American, 3.6% as Hispanic/Latinx, 3.1% as Mixed/Other, 0.4% as Native Hawaiian/Pacific Islander, 0.2% as Middle Eastern and 0.2% as South Asian/Indian American. Most participants were aged 20-50 (72.6%).
9

---

Consent Form
Background research project
we're passionate about understanding how potentially toxic or disrespectful language stereotypes can be used against certain demograaphicsgroups of people (e.g., racism, sexism, etc.). Although there is direct benefit to you for participating, we very much appreciate your help identifying and explaining such language/stereotypes, since this is something computational models have no clue about. We do not agree with any of the content/stereolypes presented to you, but it's important that gather these annotations for research purposes.
At the
We want to see giving content moderators addeed support of a model's predictions can help make their work little simpler. We do not agree with of the content/sterealypec presented to you, but it's important that we gather these annotations these research purposes.
What will out of it: You will receive $3.5 for completing the entire task. Distributed form base payment plus bonus.
Data collection & sharing
During the project we will collect your Mechanical Turk ID, and may collect personal information about you such age, gender, and race/ethnicity. We will not ask you for your name, or any other identifiable information. The personal information and other data collected during this project will used primarily perform research for purposes described in the introduction above. Such information and data, or the results of the research may eventually used develop and improve commercial products, services or technologles. clicking the "OK" button below, you are attesting that you've time to read and consider this consent form are comfortable with what is being asked of you as participant. We will securely store the data on our servers. will not share your MTurk IDs outside the research team.
Contact information:
We will not ask you for your name, and data collected in this study will be made unidentifiable to the best of our extent. We will securely store the data and only share with qualified researchers (e.g., who want to further the study of hate speech detection), If you later decide that you do not want your responses included in this study, please email
servers
so we can exclude your work.
If you have questions about your rights as a research participant, or wish obtain Information, ask questions discuss concerns about this study with someone other than the researcher(s), please contact the
Cancel
OK
Figure 5: The consent page available to participants.
10

---

Welcome to the task!
Please read the instruction and finish the task carefully! We will be monitoring the quality of your result, and may reject your work your labels consistently disagree with the other annotators.
warning: posts
(uncensored) internel; while crucial study, endorse any stereotypes
offensiie/mmmrralluua material. You may find the upsatting!
TASK DESCRIPTION this HIT, you will act as content moderator, decide whether post should be visible public readers
Allow: the post is not oftensive or prajuciced and anyone should be able view it.
2. Lenient:
the post might be aggressive, angry, or use profanihylswaar words, but not offersive or prejudiced and does not use slurs in hateful ways. Users could choose to hide such gosts from their feed with toggle.
Moderate: the post is oftensive, prejudiced, stereotysical, uses generalizations about minority demographic groups, but calling violence hate against them. be given warning message
INSTRUCTION
nature past Examples include: offensive jokes,
viewing
microeggressions
stereotyping.
TRAINING
Block: post qualifies
speach, encourages violence towards minority able ('agol),
Progress:
individuals groups and should be blocked, such that
0/4
this content Examples include: hateful uses slurs inciting violence killings of minority etc.
LABEL
Progress:
0
these definitions, you labeling post
round. example
SURVEY
post given
Reset
Next
You're girl. You're expected where
How visible should this post be?
go through training phrase help you Then, you complete rounds labelings.
receive $3.5 for completing the
By leonsent
employee
family member compoyee, student involved in this particular research. Please proceed otherwise won't able proceed
Figure 6: The instructions page available to participants.
11

---

Labol the following!
Revicw the instructions!
What do you winning battle royale in China? Winner Winner Dog dinner
Targeted group: N/A
Explanation: Does not express prejudice or hate.
TRAINING
Progress:
visible should this
3/4
Allow
Lenient
Moderate
Block
Do you agree with the pravided explanation?
Progress:
32
SURVEY
You correctly lebeled the example Moderate!
Explanation:
prejudiced joke implying Chinese people dogs, therafore should moderated. the suggestion missed the group (Chinese) and failed idantify affensive stereatype, you should not always the explanations.
Figure 7: Example of training round under the MODEL-EXe EXPPL condition.
Label the following!
Review the instructionsl
INSTRUCTION
just coincidence
dangerous?
Targeted group: black people
TRAINING
Explanation: Implies stereotype that bleck people are more dangerous than other people.
Progress:
4/4
How visible should this post be?
LABEL
Progress:
Do agree with the provided explanation?
20 /22
SURVEY
Next
Figure 8: Example of labeling round under the MODEL-EXPL condition.
hard-non-toxic
90
85.0 85 87.7
political orientation
74.3
Conseriative Nectral
Accuracy (5)
70-
$4.5
$3.1
885
Strondly lberal
10
$3.5
SOL
145
Figure 9: Average human performance grouped by political orientation, with 95% confidence intervals reported as error
12

---

Warning: the table contains content that may be upsetting offensive.
Post
Classification
Your mom is Brazilian isn't she? Maybe she could come to the school sometime for gym and teach the class how to dance samba!
hard toxic
No, can you get one of the boys to carry that out? It's too heavy for you.
hard toxic
What's best way to wash down five hundred chocolates? Katrina.
hard toxic
After you strip off his makeup and clothes, biologically he's still man.
hard toxic
Bisexual is a term that gay guys in high school use when they want hold hands with girls and feel
hard toxic
like a normal person for change.
Is it just coincidence that black ice is the most dangerous?
hard toxic
alabama where charmony and ancestry are the same website.
hard toxic
When look at the Zimbawee's Mugabe, think "Ape" or "Gorilla".
hard toxic
Just how Americans draw themselves slimmer. Japan does the same, they make anime have giant eyes while in reality they can't '1 see what their drawing
hard toxic
You mean your boyfriend coded it up in week!
hard toxic
YALL MOTHERFUCKER NORMIES WILL GET SPOILERS ON SEASON 8 OF GAME OF THRONES IF YOU DONT LEAVE NOW
hard non-toxic
@nOiratre: hookup culture can suck my fucking toe
hard non-toxic
DO NOT GO AROUND MY FUCKING CHILD IF YOU DONT EXPECT TO HAVE FUCKING WORDS WITH ME. That is MY MINNNE.
hard non-toxic
The reason people become homebodies they are tired of being surrounded by idiots
hard non-toxic
Sick of seeing liberalism/cconseraaiism is cancer. No, extremism on both sides cancer and ur incompetence at realizing this is disgusting
hard non-toxic
HAVE THE WORST AUTUMN EVER IS FUCKING ANNOYING BECAUSE ITS KEEPS RAINING IN COUNTRY
COUNTRY ITS FUCKING AUTUMN AND THIS
hard non-toxic
buying pussy or smoking weed could get you more jail time than being traitor your countrymen apparently
hard non-toxic
Fellas gotta invade Iran for feminism and gay rights.
hard non-toxic
What's Jesus's 's favorite gun? A nail gun.
hard non-toxic
Who is the only superhero that could beat Captain America? Captain vietnam I'm not weird, you're just a faggot
hard non-toxic
easy
Why is the monkey sad? Just learned that it will be nigger million years. A red bull a day keeps the niggers away
easy
easy
honestly hate Indians, Muslims, really anything with brown, black or yellow coloured skin I hate.
easy
What was Adolf Hitler's biggest mistake? Not killing the 9,000,001 Jew.
easy
Disclaimer: None of these things I' I'm about to tell you 100% accurate. Just *mostly* accurate.
easy
There are always exceptions.
countdown more companies following suit after seeing the way that was handled.
easy
Thanks for the #RT! @ marshaawiihtt @ PrettyFru @cgm807 @ poeeraavenadyy @ @axelmojave @wal- labemu? 62 Have a great week!
easy
that moment when you silence a mmge and they start hackpodalling and you just KNOW it's ahout go down
easy
still have two hunting knives from when used fish and shont of self defence home.
easy
Table 3: 30 posts used in the online user study.
13


Reviews:
[{"review_id": "Kv5vNngu3o", "reviewer": "Reviewer_7QKF", "summary": "", "questions": "- l. 055: What is a „hard … instance“ of bias? Please add more examples in the text in the beginning for all kinds of bias as this is a relevant point to understand not only on page 3 but from page 1. \n- Could you add a more detailed explanation of what is understood as a „mental shortcuts“ right in the explanation section? In my opinion, it is not fully clear what exactly is meant by this term, how exactly this is grounded in scientific theory, etc.\n- Could you please add a definition of subtle (e.g., 33, 67, 162, …) vs. overt bias (lexical overlap, etc.)? It is not really clear what the difference would be concretely in your work and how this difference affects data selection, the study, and the interpretation.\n- It would be helpful to provide a little bit more details on what „dual process theory“ is, what the core notions are, and how these are leveraged / implemented.\n- L. 171: I am not completely convinced of the process of using misclassified instances as „harder“ instances than correctly classified ones based on model performance only. How do you justify this? The model could be biased / bad at this for reasons other than toxicity is not easy to identify? How do you control for other phenomena which might be the real source of misclassifications?\n- l. 177: Did you at least check results from one potential other LLMs (e.g. an open-source model)? Why did you use exactly this model only?\n- Discussion section: Please elaborate a little bit more about human disagreement w.r.t accuracy — maybe there is no ground truth to agree on?\n- Could please disentangle limitations, ethical considerations and the broader impact?\n\n// edit: Thanks to the authors for their responses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and we are glad you find our work interesting and the study well-designed.\n\n\n**Generated explanations could be biased towards a certain view**\n\nWe are glad that you pointed out the issue with biases in language models. This is, in part, what motivates our work on a human-AI collaborative moderation setup, rather than full delegation to a model. We acknowledge that there isn’t a universal set of guidelines that applies to different communities. That said, in practical moderation scenarios (e.g., Facebook, Twitter/X, or a specific subreddit), there are often prescribed or imposed viewpoints (e.g., by terms of service or moderation guidelines of the platform such as [Facebook's rules](https://web.archive.org/web/20230322180903/https://www.theguardian.com/technology/2018/apr/24/facebook-releases-content-moderation-guidelines-secret-rules)). As outlined by [Rottger et al., 2022](https://aclanthology.org/2022.naacl-main.13/), there are two contrasting study paradigms in toxicity annotations: a descriptive study, where variation in annotators’ views of biases is studied, and prescriptive study, with set guidelines that annotators follow; since most real world moderation settings are prescriptive, we opt for a prescriptive set of guidelines in our study as well. We will extend on this discussion in future versions of this work, and please see L335-350 (Limitations) for a relevant discussion.\n\n**How are “mental shortcuts by annotators” defined?**\n\nWe follow definitions of *mental shortcuts* by [Tversky and Kahneman, 1974](https://pubmed.ncbi.nlm.nih.gov/17835457/): “mental shortcuts are heuristics that are employed in making judgements under uncertainty”. In particular, Tversky & Kahneman discuss the representativeness heuristic, which is very relevant to our work: when annotators are asked to label whether a post is toxic, they may use a heuristic of similarity to other toxic posts (e.g., swearwords causing annotators to flag toxicity, [(Sap et al., 2019)](https://aclanthology.org/2022.naacl-main.431/)). [Malaviya et al., 2022](https://aclanthology.org/2022.emnlp-main.438/) have further discussions connecting different heuristics to annotators for NLP tasks. We will add relevant context and definitions in the final paper.\n\n**How is subtle/hard vs. overt/easy toxicity defined?**\n\nWe consider a post subtle/hard if it will likely be missed by an annotator using simple heuristics like the representativeness heuristic described above. For example, a *hard toxic* post may be sexist without derogatory slurs, and a *hard non-toxic* post may be expressing positive sentiment while containing profanity. We acknowledge that this classification could be fuzzy in real moderation settings, but the authors did our best to select unambiguous examples for our study. In addition, we would like to point out that the easy vs. hard classification is only intended for testing the effectiveness of our framework on different types of examples, and it has no bearing on the deployment of our system. We will clarify the classification of toxicity and incorporate this discussion into our paper.\n\n**What is dual process theory?**\n\nDual process theory describes people’s thought and decision making arises from two processes: one “fast” and automatic, the other “slow” and deliberate. In the context of this work, “fast” refers to moderators using associative mental shortcuts for labeling. In contrast, “slow” refers to thoughtful moderation. That said, there isn’t a direct operationalization of the theory in our paper, and we don’t claim that hard toxic or hard non-toxic instances can only be correctly labeled using slow thinking. We will expand on relevant discussions in the paper.\n\n**Misclassified instances are not necessarily harder**\n\nIndeed, instances could be misclassified due to a variety of reasons other than instance difficulty (e.g., mislabeling, model biases). Due to exactly this, two authors went over all misclassified examples by the classifier and selected examples we agreed were hard, and can be unambiguously labeled (see L169-175). Thus, misclassifications caused by model biases do not influence the examples selected for our user study.\n\n**No ground truth to agree on**\n\nAs mentioned previously, our work follows a prescriptive paradigm, under which we assume one correct label. During data selection, two authors verified and agreed that every instance can be unambiguously labeled under the moderation guidelines we provide to annotators (Figure 6 in the appendix contains the guidelines).\n\n**Results from one potential other LLMs**\n\nIn preliminary experiments, we tried generating explanations with a fine-tuned T5-Large model, and found it substantially worse at spelling out the stereotypes in posts. Therefore, we chose GPT-3.5 (text-davinci-003), which was one of the best available LLMs at the time.\n\n**Code release and reproducibility**\n\nWe will release the code for generating explanations with GPT-3.5 and toxicity classifier training in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like the approach and the idea and think that this work makes a focused and interesting contribution.\nBased on the given information, the study  seems well-designed and carefully conducted. Another strength is the nuanced interpretation including side findings regarding the hard non-toxic examples.", "reasons_to_reject": "I am a little concerned regarding bias in the generated explanations: Every human, and thus, also every model trained on human-generated text, contains bias. On a meta-level, specific perspectives of the world will still be amplified in explanations for content moderation, namely based on whatever bias a human expert or a trained model introduces into their explanation, e.g. values based on a paradigm / view of life where individualism is valued higher than the collective (which is not the case in every culture) or where democracy is the preferred political system (not true for all countries, maybe also not preferred by some cultures). \nI would have liked to see at least a comment about the fact that these explanations still introduce a certain kind of bias (which could even be positive) which either needs to be accepted or addressed in further steps/future work.\n\nAnother point that I would to see addressed in a more nuanced way is defining what a subtle vs. toxicity is and how one can differentiate between both on which level (lexical, frame, ... levels). Here, more detailed definitions + references to related work should be added. In this regard, I am also missing a clearer interpretation of what a hard vs. an easy example of toxicity is and how the authors can be so sure that this holds for every use case. Here, I would highly recommend adding comments on limitations regarding whether a given post is toxic or not based on culture, subjective opinions, etc. \n\nFor scientific reproducibility, it would be very helpful if you shared. the code you used.", "questions_to_authors": "- l. 055: What is a „hard … instance“ of bias? Please add more examples in the text in the beginning for all kinds of bias as this is a relevant point to understand not only on page 3 but from page 1. \n- Could you add a more detailed explanation of what is understood as a „mental shortcuts“ right in the explanation section? In my opinion, it is not fully clear what exactly is meant by this term, how exactly this is grounded in scientific theory, etc.\n- Could you please add a definition of subtle (e.g., 33, 67, 162, …) vs. overt bias (lexical overlap, etc.)? It is not really clear what the difference would be concretely in your work and how this difference affects data selection, the study, and the interpretation.\n- It would be helpful to provide a little bit more details on what „dual process theory“ is, what the core notions are, and how these are leveraged / implemented.\n- L. 171: I am not completely convinced of the process of using misclassified instances as „harder“ instances than correctly classified ones based on model performance only. How do you justify this? The model could be biased / bad at this for reasons other than toxicity is not easy to identify? How do you control for other phenomena which might be the real source of misclassifications?\n- l. 177: Did you at least check results from one potential other LLMs (e.g. an open-source model)? Why did you use exactly this model only?\n- Discussion section: Please elaborate a little bit more about human disagreement w.r.t accuracy — maybe there is no ground truth to agree on?\n- Could please disentangle limitations, ethical considerations and the broader impact?\n\n// edit: Thanks to the authors for their responses.", "paper_topic_and_main_contributions": "The authors propose a framework for AI-generated explanations to assist users in content moderation. The idea is to support moderators with their decision-making by providing information about he entities and events of a given post. They authors conduct a user study and show that expert-written explanations support moderators while AI-generated explanations sometimes hinder correct moderation, i.e. explanation quality matters and AI models should be improved w.r.t. this.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "moCDQ2Ljn6", "reviewer": "Reviewer_5KD6", "summary": "", "questions": "Question A: How do you plan to address the challenge of identifying high-quality examples and generating human explanations, as mentioned in the limitation section?\n\nQuestion B: What are the suggestions to improve the accuracy of the generated explanations in future iterations of the BIASX framework?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and we are delighted that you consider our framework easy to implement and empirically effective. We are really grateful that you consider this work a significant contribution to content moderation and online safety.\n\n**Low accuracy of the generated explanations and directions for improving accuracy**\n\nIndeed, explanations generated with GPT-3.5 have somewhat low accuracy on the hard-toxic instances (60%), which points to the need for combining our framework with models that are more capable of understanding toxicity. For example, GPT-4 reports 80% accuracy on the hard toxic instances, and thus we expect our framework to be even more helpful to moderators with GPT-4 explanations. Future work could also combine effective prompt strategies (e.g., chain-of-thought) with explanation generation to improve explanation quality.\n\n**Challenge of identifying high-quality examples**\n\nBased on [Han and Tsvetkov, 2020](https://aclanthology.org/2020.emnlp-main.622/), we use misclassification as a signal for example difficulty. While this approach is somewhat effective in filtering out the easy instances, the remaining instances contain a mix of mislabeled and ambiguous examples, which requires additional human verification. One practical way of identifying high-quality instances could be through dataset cartography [(Swayamdipta et al., 2020)](https://arxiv.org/abs/2009.10795), which characterizes individual instances using training dynamics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper demonstrates through a large-scale crowdsourced user study that participants benefit from these explanations in correctly identifying subtle toxicity. This empirical evidence strengthens the validity of the proposed framework.\n2. The proposed framework - BIASX is very easy to implement enhances content moderation by providing free-text explanations of statements' implied social biases. This is a significant contribution to the field of content moderation and online safety.", "reasons_to_reject": "A potential reason to reject this paper could be the relatively low accuracy of the generated explanations. The study demonstrates that only 60% of model explanations for hard toxic content are accurate, which could lead to downstream errors when applied in real-world content moderation scenarios. This imperfection in the model's performance might limit its practical utility and effectiveness.", "questions_to_authors": "Question A: How do you plan to address the challenge of identifying high-quality examples and generating human explanations, as mentioned in the limitation section?\n\nQuestion B: What are the suggestions to improve the accuracy of the generated explanations in future iterations of the BIASX framework?", "paper_topic_and_main_contributions": "The paper introduces BIASX, a novel framework aimed at assisting content moderators by providing free-text explanations of implied social biases in statements. Through a large-scale crowdsourced user study (N=454), it demonstrates that these explanations assist participants in correctly identifying subtly toxic content. However, the introduction of explanations leads to an increase in labeling time and mental demand, which could be potential drawbacks in real-world applications. The study also emphasize the superiority of human-written explanations over machine-generated ones. Finally, the authors highlight the challenges in identifying high-quality examples and generating human explanations, suggesting the need for further research.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xmsYfbKweO", "reviewer": "Reviewer_rNFj", "summary": "", "questions": "A. What's the distribution (mean, variance for example) of labeling time in the qualification stage for each condition of workers?\n\nB. Is there a pre-defined standard for crowd workers that they should follow when playing the role of moderator?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you like the framing of this work, and consider our study well-designed.\n\n**Outliers/Individual labeling speed differences could lead to inaccurate labeling time estimates**\n\nLike the reviewer pointed out, there are factors that could bias labeling time (e.g., individual speed differences, multi-tasking workers switching between tabs). This is precisely why we use the median labeling time statistic for each individual annotator, so that the measurement is more robust to outliers (e.g., if the worker took a break during one of the questions). We would like to further point out that we use a large sample size for our study (>100 participants in each group), and it is very unlikely that one group ends up with a substantially larger percentage of fast/slow annotators than another group. In addition, the 95% confidence intervals of the median labeling time measurements (Figure 2) are sufficiently tight for our results to be statistically valid even when individual speed differences are taken into account.\n\n**Careful thinking does not necessarily correlate with longer labeling time**\n\nThank you for this acute observation. We are indeed using labeling time as one of the possible indicators for careful thinking, and we acknowledge that increased labeling time alone does not definitively suggest more careful reasoning. That said, in a controlled environment it’s reasonable to attribute this observed increase in time to more careful reasoning; the large sample size and computed statistics (e.g., tight 95% confidence interval) make it unlikely that our observations are due to noise. We want to further point out that the reported careful reasoning is further evident from the improvements in worker performance on hard instances when explanations are provided. We will elaborate on this discussion in our paper.\n\n**Standard for annotation**\n\nWe do provide guidelines of annotation (e.g, label definitions and training examples) to all annotators, which can be found in Fig. 6, 7, 8 in the appendix. This standard for annotation roughly follows Reddit’s moderation guidelines. As we discussed in the limitations section (L335-350), we opt for a prescriptivist perspective [(Rottger et al., 2022)](https://aclanthology.org/2022.naacl-main.13/), which is often adopted by actual platforms’ terms of service (TOS) and community guidelines. Please see the first point in our response to Reviewer 7QKF for a related discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The framing of this problem is good and this paper is well-structured.\n\n2. The user study is large with most parts well-designed. \n\n3. It addresses the importance of adding explanations to difficult content for moderators.", "reasons_to_reject": "1. My main reason to reject is the design to answer the 2nd research question: \"Do biasx enable moderators to think carefully about moderation decisions\".\n\nFirst, the MTurk workers are randomly assigned to one of the four conditions. There may be chances that some workers performing tasks (even other tasks) very fast may fall in the same condition. Do the authors considering calculate labeling time for each worker during qualification stage, and re-calculate that in each condition to reduce individual labeling speed biases?\n\nSecond, while the intuitive makes sense, \"think carefully\" does not necessarily correlate with \"labeling time\". For example, some crowd workers may perform multiple tasks, switch between tabs, which may result in an increase in labeling time too. It would be better to base this on a theory, data or literature. Or mention this in the limitations.\n\n2. \"crowdworker are asked to play the role of content moderators\", it should be better justified. The role, standard, or platform for moderators may vary a lot. ", "questions_to_authors": "A. What's the distribution (mean, variance for example) of labeling time in the qualification stage for each condition of workers?\n\nB. Is there a pre-defined standard for crowd workers that they should follow when playing the role of moderator?", "paper_topic_and_main_contributions": "This paper focuses on the problem of social biases, which indeed requires much attention in content moderation.\nIt introduces a framework that can help to improve content moderation quality.\nThe authors conduct a user study by crowdsourcing answers from more than 450 Mturk workers.\nThe main contributions are made by answering three questions: (1) this framework helps, (2) the explanations enable moderators to think more carefully, and (3) the quality of explanation matters.\n\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper overall is well written. \n\nFor figure 2: numbers in percentage are hard to see and can be a little larger.\n\nFor figure 4: text can be larger."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a framework for AI-generated explanations to help improve content moderation quality, and they evaluate that framework with a large-scale crowdsourced user study to demonstrate that the framework helps moderators think more carefully during content moderation decisions. 

Reviewers all agreed that this was a well-designed study and a well-written paper that makes a focused and interesting contribution to the NLP community and to work on content moderation and online safety, through the framework and the empirical results.

Reviewers wanted to see clearer explanations of the choice of the moderation framework (i.e., using a prescriptive framework, rather than a more descriptive approach to identifying bias) and the background of the participants in the study (i.e., that they were not themselves moderators). In addition, more discussion of the Tversky and Kahneman framework and more contextualization of what makes for difficult items to annotate would be useful for readers.
Decision: Accept-Main