Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive
Authors: Tharindu Cyril Weerasooriya
Sujan Dutta
Tharindu Ranasinghe
Marcos Zampieri
Christopher M Homan
Ashiqur R. KhudaBukhsh
Venue: EMNLP 2023 Main (2023)
Keywords: human annotation
fairness
noise audit

Abstract: Offensive speech detection is a key component of content moderation. However, what is offensive can be highly subjective. This paper investigates how machine and human moderators disagree on what is offensive when it comes to real-world social web political discourse. We show that (1) there is extensive disagreement among the moderators (humans and machines); and (2) human and large-language-model classifiers are unable to predict how other human raters will respond, based on their political leanings. For (1), we conduct a ***noise audit*** at an unprecedented scale that combines both machine and human responses. For (2), we introduce a first-of-its-kind dataset of ***vicarious offense***. Our noise audit reveals that moderation outcomes vary wildly across different machine moderators. Our experiments with human moderators suggest that political leanings combined with sensitive issues affect both first-person and vicarious offense. The dataset is available through https://github.com/Homan-Lab/voiced.

Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying
Human and Machine Disagreement on What is Offensive
Tharindu Cyril Weerasooriya** Sujan Dutta
Tharindu Ranasinghe
Marcos Zampieri * Rochester Institute of Technology
Christopher M. Homan
Ashiqur R. KhudaBukhsh
o Aston University
George Mason University
cyril@mail. rit. edu, sd2516érit edu, t. transsingheeastoon ac. uk mazgla@rit edu, cmhvcs@rit. edu, axkvse@rit.edu
Abstract
1.1 Definitions
Noise audit While limited literature exists on investigating the generalizability of offensive speech detection systems across datasets (Arango et al., 2019), political discourse (Grimminger and Klinger, 2021; Maronikolakis et al., 2022), vulnera- bility to adversarial attacks (Gröndahl et al., 2018) 8), unseen use cases (Sarkar and KhudaBukhsh, 2021). and geographic biases (Ghosh et al., 2021), to the best of our knowledge, no work exists on a compre- hensive, in-the-will evaluation of offensive speech filtering outcomes on large-scale, real- real-world politi- cal discussions. One key impediment to performing in-the-wild analysis of content moderation systems is lack of ground truth. It is resource resource-intensive to annotate representative dataset of social media posts test content moderation at scale. Much to the spirit of the celebrated book, "Noise: A Flaw in Human Judgment" by Kahneman al. (2021). we bypass this requirement through noise audit of several well-kun well-known offensive speech classifiers on a massive social web dataset with documented political dissonance (KhudaBukhsh et al., 2021, 2022).
This paper discusses and contains content that is offensive or disturbing. Offensive speech detection is key component of content mod- eration. However, what is offensive can be highly subjective. This paper investigates how machine and human moderators disagree what is offensive when it comes to real-world social web political discourse. We show that (I) there is extensive disagreement among the moderators (humans and machines); and (2) hu- man and large-language-modde classifiers unable to predict how other human raters will respond, based on their political leanings. For (I), conduct noise audit at unprece- dented scale that combines both machine and human responses. For (2), we introduce first- of-its-kind dataset' of vicarious offense. Our noise audit reveals that moderation outcomes vary wildly across different machine modera- tors. Our experiments with human moderators suggest that political leanings combined with sensitive issues affect both first-person and vi- carious offense.
1 Introduction
As defined in Kahneman et al. (2021), noise audit measures outcome variability across multi- ple (competent) decision systems. Kahneman et al. (2021) show, in real-world scenarios like judicial sentencing or insurance settlements that contain diverse body of decisionmakers, that noise is widespread. Our paper seeks to understand how content moderation outcomes vary across different offensive speech classifiers (dubbed machine mod- erators) in political discourse the granularity of individual social media posts.
Offensive speech web platforms is a persistent problem with wide-ranging impacts (Benson, 1996; Chandrasckharan et al., 2017; Perez and Heater, 2021). Among many reasons why moderation fails to eliminate the problem, is the reality that peo- ple often disagree on what is offensive (Waseem, 2016; Ross et al., 2016). In this study, we break new ground by investigating disagreement through a large-scale noise audit and by introducing the notion of vicarious offense.
Existing literature reveals that annotator's gender, race, and political leanings may affect people's perception of what is offensive con- tent (Cowan et al., 2002; Norton and Sommers, 201 1; Carter and Murphy, 2015; Prabhakaran et al.,
Vicarious offense
* "Tharindu Cyril Weerasooriya and Sujan Dulta are equal- contribution first authors. Ashiqur R. KhudaBukhsh is the corresponding author. 'The dataset is available through https: http:://gitii wtwb..ittuu.oomm /Homan- Lab/voiced

---

CNN
Pootentialyy oriminal? It WAS Tucking criminal and Trump and all associated with the insurrection should be in prison."
mate selection (Huber and Malhotra, 2017), and employment decisions (Gift and Gift, 2015). Is it possible that such negative views also affect our ability to perceive what we dub vicarious offense?
Conmennt
Personal Offente Doo find it offendive?
Vicarious Offerse
"Do you others will find oftensive?"
We thus break down our study on how politi- cal leanings affect perception of offense into two parts: (1) annotators answer questions relevant to direct, first-person perception of offense; (2) an- notators answer about vicarious offense. For any given social media post d, we have two points of view. For each X, Y E (Republican, Democrat, Independent]- where * y/ we have: Personal (or 1st person): how offensive a(n) X finds d: and Vicarious (or 2nd person): how offensive a(n) X' thinks a(n) Y would find d. With three dis- tinct choices of X for the Personal, and six distinct choices for ()) for the Vicarious, we have nine vantage points (see Figure 1).
Moderators
X
X
IND
X
X
x
X
Figure 1: An illustrative example from YouTube comments on CNN news videos highlighting nuanced inconsistencies between machine moderators and human moderators with dif- ferent political leanings. We use the majority vote to aggregate individual machine and human moderator's verdicts. The per- sonal offense is listed in the left column outside the grid. The vicarious offense is presented in the right column with the grid. While Republicans feel that the comment Potentially criminal? It WAS fucking criminal and Trump and all asso- ciated with the insurrection should be in prison. will invite equal ire from the Independents, the Independents, actually, do not find it offensive. For the machine moderators (MM), we have nine MM M,, (Section 3.1) and ChatGPT API (C), a large language model (LLM). Both types of MMs are able to detect personal offense; only the GPT model (Section 6) is able to identify the personal offense. The appendix contains more illustrative examples (Figure 17).
Our study is the first of its kind to explore how well we can predict offense for others who do not share the same political beliefs. In the era of grow- ing political polarization in the US where sectar- ian us vs. them often dominates the political dis- course (Finkel et al., 2020), our study marks one of the early attempts quantify how well the political opposites understand each other when asked be on their opposites' shoes.
2021; Sap et al.. 2022). We explore this phe- nomenon by introducing the notion of vicarious offense in which we ask a timely and important question: how well do Democratic-leaninn users perceive what content would be deemed as offen- sive by their Republican-leaning counterparts or vice-versa?
1.2 Research Questions and Contributions
Research Questions
We address the following
research questions:
For instance, Figure shows the post, Poten- tially criminal? It WAS fucking criminal and Trump and all associated with the insurrection should be in prison., and we imagine asking three human annotators crat, and one independent-wweee post is personally offensive to them AND whether they think those with different political affiliations would find it offensive. Figure shows that the Republican rater thinks it is personally offensive them, and Independents would find offen- sive, but not Democrats. We also ask a machine model whether the post is offensive (it thinks it is), and whether it is offensive to Republicans (yes). Democrats (no) and independents (no). We do NOT ask humans to rate how they expect machines would respond.
RQ1: How aligned are offense predictions of machines moderators across different ma- chine moderators?
Republican, one Demo- they think the
RQ2 How aligned are offense predictions of human moderators across different political beliefs?
RQ3 How the alignment between offense predictions of human moderators and ma- chine moderators?
Contributions
Our contributions are as follows: (1) We conduct a noise audit at an unprecedented scale that reveals considerable variations in in-the- wild content moderation outcomes across nine dif- ferent well-known machine moderators: (2) We conduct a detailed annotation study to un- derstand the phenomenon of vicarious offense and release the fosttof-its-kind dataset where annota- and vicarious offense.
Documented evidence indicate that negative views towards the opposite political party have af- fected outcomes in settings as diverse as allocating scholarship funds (Iyengar and Westwood, 2015),
tors label both

---

Unlike most studies on political biases which prof- fer binarized world-view of US politics, we con- sider all three key players in the US democracy: the Democrats, the Republicans, and the Independents; (3) We release an annotator-level dataset, dubbed VOICED (Vicarious Offense Identification Corpus for Estimating Disagreementss, available at: http consist- ing of 2,310 social web posts, that sheds critical insights into how well political groups understand each other's perception of offense in general and also when sensitive issues such as reproductive rights or gun control/rights are in the mix; and (4) Finally, we close the loop by analyzing how hu- man moderators (dir)e with their machine mod- erator counterparts on what is offensive, including the use of ChatGPT vicarious predictor.
tion has been studied before (Sap et al., 2021). Our work contrasts with Sap et al. (2021), in the fol- lowing key ways: (1) our introduction of vicarious offense, a novel offense perspective not considered heretofore; (2) the inclusion of the Independents in our annotation study; and (3) unified analysis of alignment between machine and human modera- tors.
s://g thub. com//oomn-Lab/voll
3 Methods
3.1 Machine Moderators (a.k.a., MMs)
We investigate nine open sourced offensive lan- guage identification models as machine moderators, From these models, we trained eight models on the following well-known offensive speech datasets ob- tained from Twitter, Facebook, Gab, Reddit, and YouTube: (1) AHSD (Davidson et al.. 2017); (2) HASOC (Mandl et al., 2020); (3) HatEval (Basile et al., 2019); (4) HateXplain (Mathew et al..
2 Related Work
We outline some of the key inspirations of our work in this section. Throughout the paper, we cite sev- eral other relevant papers.
2021); (5) OLID (Zampieri et al., 2019); (6)
TRAC (Kumar et al., 2020): (7) OHS (Qian et al., 2019); (8) TCC. Following work by Ranasinghe and Zampieri (2020), we transform the labels of instances present in above datasets into two broad labels: offensive and not offensive, which corre- spond to the level A of the popular OLID taxonomy (Zampieri et al., 2019) widely used in offensive speech classification. We trained BERT-LARGE- CASED models each of the training sets in these datasets following text classification objective. As the (9) and final model we used publicly avail- able Detoxify (Hanu and Unitary team, 2020). which is a ROBERTA- BAAEEAAAA--EEEE model trained on data from Unintended Bias in Toxicity Classification by Jigsaw Team (cjadams et al., 3017), it is a dataset that is used in training pipelines of Perspective API (see Figure 15 in Appendix for analysis using Per- spective API).
The primary inspiration of our noise audit ap- proach is the celebrated book, "Noise: A Flaw in Human Judgment" by Kahneman et al. (2021). However, our knowledge, our paper the first to apply this method in auditing offensive speech clas- sifiers bypassing the requirement of a large number of annotated samples.
Our research benefits from the vast literature offensive speech classificatinn that has yielded several valuable datasets that we use in this pa- per (Davidson al., 2017: Mandl et al., 2020; Basile al., 2019; Mathew et al., 2021; Zampieri et al., 2019; Kumar et al., 2020; Qian et al., 2019). However, while limited literature exists on inves- tigating the generalizability of offensive speech detection systems across data-sets (Arango et al., 2019), vulnerability to adversarial attacks (Grön- dahl et al., 2018), unseen use cases (Sarkar and KhuhaBukhsh, 2021), and geographic bi- ases (Ghosh et al., 2021), to our knowledge, no in-the-wild, comprehensive performance evalua- tion of offensive speech classifiers on political dis- course has been conducted before.
3.2 Human Moderators
Annotation Study Design. Our survey is grounded in prior human annotation literature studying subjectivity in offense perception (Sap et al., 2020; Kumar et al., 2021).² 3 We host the sur- vey in Qualtries and make it only visible to users registered as living in the US We set restrictions on MTurk due to the nature of the study. We release
Annotator subjectivity has been widely studied in (Prelec, 2004; Pavlick and Kwiatkowski, 2019; Dumitrache et al., 2018; Poesio al., 2019; Liu al., 2019; Weerasooriya al., 2020; Basile, 2020; Wecrasooriya et al., 2023). Specifically, how politi- cal influence may factor in offensive speech annota-
? Avaiilable https: https::/www..dr kaggle. htteee.co/////////e//eee com/c/jigsaw- tox rn--coment-clan wwtssffiiciioonii challcheeenge
'A version of this survey can he found on this link https: //github. ceatcuub.co/Hooaaaae HawpHoooaaaaaaaaeeaaeeaaaa

---

our study in batches of 30 data items in total with 10 items from each news outlet but with varying levels of MM disagreements. Each batch consisted of 10 instances each from Doffensive. debbtee:: and D noroOBeesivve Each instance is designed to be anno- tated by 20 annotators. We not only asked if each item was offensive to the annotator, but how some- one with different political identity would find it offensive. Our study was reviewed by our Institu- tional Review Board and was deemed as exempt.
Annotator Compensation. We compensate the annotator 0.1 USD for each instance. Each batch with 30 instances would thus fetch USD. Com- pensation is grounded in prior literature and dis- cussed in detail the Appendix. We allow the annotators to leave a comment on our study at the end. No annotator complained about compensation while many praised our task's novelty.
Data
4.1 Dataset for Noise Audit
Pilot Study and Annotator Demographics. Since MTurk has documented liberal bias (Sap al., 2022), we first conduct a pilot study with 270 examples (nine batches) to estimate political repre- sentation. 117 unique annotators participate this pilot. We observe a strong Democratic bias in the annotator pool (66% Democrat, 23% Republican, and 1% Independent).
We evaluate our moderators on dataset of more than 92 million comments on 241,112 news videos hosted between 2014, January to 2022, Aug 27 by the official YouTube channels of three prominent US cable news networks: CNN, Fox News, and MSNBC. We consider this dataset because of the broad participation, topical diversity, and recent lit- erature indicating substantial partisan and idcologi- cal divergence in both content and audience in these news networks (Stanley, 2012; Bozell, 2004; Gil de Zúñiga et al., 2012; Hyun and Moon, 2016; Dutta et al., 2022; Ding et al., 2023), with prior work reporting considerable political dissonance (Khud- aBukhsh et al.. 2021, 2022). Overall, our dataset comprises three million randomly sampled com- ments, one million from each of the three YouTTube channels denoted by D enn- Dfurr and D msshic spectively. Temporal and linguistic analyses on the dataset are included in the Appendix B.
To ensure comparable political representation, we set restrictions for the subsequent annotation batches to have at Icast six annotators from each political identity (18 annotators in total). The re- maining two spots are given to the annotators who first accept the jobs regardless of their political iden- tity. We also re-run batches from our pilot study to ensure they all contain at least six annotators from each political identity.
Final Study. Adding the political identity based restrictions aided in building representative dataset for this work (see Appendix C). We con- ducted a total of 37 batches of our survey of 30 items each, following the same survey structure as the pilot study.
4.2 Dataset for Human Moderators
In order to compare and contrast machine moder- ation and human moderation, we first construct a representative set of easy and challenging exam- ples from the machine moderators' perspective. For each corpus D, conduct stratified sampling from three subsets: (1) subset where most MMs agree that the content is not offensive (denoted by norOffensive:: (2) a subset where most MMs agree the content is offensive (denoted by Doffensive); and (3) a subset in the twilight zone where nearly half of the models agree that the content is offensive with the other half agreeing that it is not (denoted by Ddeiaaed). Formally.
Demographics of the final study annotator pool; Political Leaning: 35% (267) registered as Democrats, 35% (266) as Republicans, and 30% (220) as an Independent.
Gender: 47% Female, 53% male, and one non- binary annotator.
Race: Similar to the pilot study, majority of the annotators White Caucasian, with limited representations from the Asian, Black or African American, and American Indians communities (in line with Sap et al. (2020)).
Age: The study had annotators from all age groups above 18 years, majority of the annotators were from the age group 25-34.
Dawtttonsne
i < offenseSSore (d) < 1, if < offenseSSore((d (cf) < if N & offenseScore(cc < N.
d
Dashard
Dogtrsnee
where denotes the total number of offensive speech classifiers considered (in our case, N = 9),
'Detailed annotator demographies of the pilot study and the final study are presented in the Appendix C.

---

and offenseScore(d) returns the number of MMs that deem d offensive ([O, ND.
ANSD
1.00
0.43
0:333
00.52
0.54
0.02
0.43
0:42
Deccody
We have three news outlets, three sub-corpora defined based on MM disagreement, and five time periods yielding 45 different combinations of news networks, temporal bins, and MM disagreement.
06
HASOC
0.13
0:30
1.00
0.02
0.12
0:24
0.11
0:30
0:13
0.06
0.02
0.02
1.00
0:36
0.01
0.26
0:39
Hotesplatr
0.32
0.12
0.02
0.27
0.54
0.54
0.24
0.20
0.47
1.00
0.04
0:53
0.
0:36
We weigh each of these combinations equally and sample 1,110 comments (D general) In addition, we sample 600 comments with the keyword gun gun) and 600 more with the keyword abortion (D (Pabortion) to shed light on human- human-machine dis- agreement on hot-button issues. Filtering relevant content by single, general keyword has been pre- viously used in computational social science litera- ture (Halterman et al., 2021; Dutta et al., 2022). It is high-recal approach to obtain discussions relc- vant to reproductive rights and gun control/rights without biasing the selection towards event-specific keywords (e.g., Row v. Wade or Uvalde).
GUI
0:43
0.1
0.10
0.32
0.33
0:53
0.01
1.00
19
0.49
TCC
0.42
0.13
0.27
0.02
0:57
0.01
410
Decod6
HAS0C
Maye
015
164
Figure Agreement between machine moderators. A cell (i,j) presents the Cohen's к agreement between machine moderators M: and M, Majority ttyiityiin is a machine moderator that takes the majority vote the nine individual machine moderators.
content has offenseScore E {4,5). These are the comments on which the MMs have maximal dis- agreement. To summarize, large fraction of the so- cial web represents disputed moderation zone and possibly requires human moderators' assistance. In what follows, investigate how human modera- tors fare when they are tasked with the difficult job of determining offense in political discourse.
5
Results
5.1 Machine Moderators in the Wild
RQ1: How aligned are offense predictions of ma- chines moderators across different machine moder- ators?
Figure 2 summarizes the pairwise agreement re- sults (Cohen's 's κ) on the entire dataset. Results restricted to a specific news network are qualita- tively similar. Our results indicate that no machine moderator pair exhibits substantial agreement (x score 2 0.81), only handful exhibit moderate agreement (0.41 < score 0.60), and several pairs exhibit fair, slight, or no agreement. When we quantify agreement across all machine moderators, the overall Fleiss' K across Dem, Djox. and D mesnbc are 0.27, 0.25, and 0.22, respectively.
5.2 How Well Do Human Moderators Agree?
RQ2: How aligned are offense predictions of hu- man moderators across different political beliefs? We break RQ2 down into two sub-parts the first (RQ2a) investigating first-person offense, the second one (RQ2b) examining vicarious offense. RQ2a: How aligned are different political iden- tities on the perception of offense in first-person? ? Figure 3 (in Appendix Table 7) summarizes the con- fusion matrices between human annotators of dif- ferent political identities. We first observe that for any pair of political identities, the human-human agreement higher than the best human- machine agreement achieved in our experiment. We next note that while human-human agreement is gen- erally higher than human-machine agreement, the highest human- Cohen's 's x achieved between the Independents and Democrats (0.43) is still at the lower end and is considered as moderate agree- ment (McHugh, 2012). Within the political identity pairs, the Democrats and the Independents are most aligned on their perception of offense. This result is not surprising. Historically, Independents lean more toward the Democrats than the Republicans
We next examine the distribution of machine moderators' aggregate verdicts individual com- ments. As already mentioned, offenseScore re- turns the number of MMs that deem d offensive (IO, N7).
large fraction (nearly half) of the content is not flagged by any of the MMs, whereas a minus- cule proportion (0.03% is flagged as offensive by all. The content with offenseScore = (~ 17.5%) is particularly interesting. It indicates only one of the nine MMs marks these comments as offensive. Therefore, the moderation fate of every comment in this bin is highly volatile. If any other MM than the one that flags it is deployed, the comment will not be censored. We also observe that 10.1 % of the

---

evidenced by the Gallup survey where 47.7 + 2.98% of the Independents report that they lean Democrat as opposed to 42.3 + 3.08% Indepen- dents reporting leaning Republican (Gallup, 2022). RQ2b: Which political identity best predicts vicar- ions offense? In our vicarious offense study, we request the annotators to predict out-group offense labels. Hence, we have information about say, what Democrats believe Republicans find as offensive. Since we also have first-person perspectives from the Republicans, we can tally this information with the first-person perspective to find out how well annotators understand the political others.
Machine moderators: no:Offensive Human Moderators: offensive
So a woman wants an abortion, and SHE happens to have an airplane ticket that SHE got. The airline company can be sued?
Maybe read U.S. Constitution. Abortion not in unlike the 2nd Amendment. Please stop showing planned parenthood ads. don't I agree with abortion. republicans are so far right that facts don't matter any- more. it is political theater. 0 dollars of Federal money has been spent on abortion. the war is on ahortions. other people trying to dictate how other people should live their lives.
Table I:
Illustrative examples highlighting disagreement be- tween machine moderators and human moderators on Pabomton- The blue, yellow, and the red cells consider Democrats, Inde- pendents, and Republicans human moderators, respectively
We indicate the vicarious offense predictor in superscripts for clarity. Republicans means Democrats are predicting what Republicans would find offensive. Figure 4 and Table 8 (in Appendix) indicates that Republicans of vicarious offense. On both cases of predicting vicarious offense for the Democrats and the In- dependents, they do worse than the Independents and the Democrats, respectively. We further note that while Independents and Democrats can predict vicarious offense for each other reasonably well, they fare poorly in predicting what Republicans would find offensive. Hence, in terms of vicarious offense. Republicans
more. On one hand, certain camps believe that web platforms censor a particular political believ- ers unjustly more (Barrett, 2022). On the other hand, different groups often believe that poor plat- form censorship led to some of the recent political crises (Derysh, 2021). Figure 3 (in Appendix Ta- ble 6) examines to which political party machine moderators are most aligned with. We observe that while all three political identities have comparable agreement with machines on what is offensive, Re- publicans align slightly more with the machines on what is not offensive. Existing literature hy- pothesized that conservatives may focus more on linguistic purity than the liberals while determining toxic content (Sap et al., 2022). We note that of all the instances in D general that contains the keyword fuck, 94% of them were marked as offensive by the Republicans whereas Democrats marked 88% of them as offensive.
the worst predictors
the least understood po- litical group while they also struggle the most to understand others.
Finally, we present compelling result that shows why inter-rater agreement could be mislead- ing. Figure and Table 9 (Appendix) suggest that the Democrats and Independents have the high- est agreement on what Republicans would find as offensive. However, Table 8 already shows that neither the Democrats nor the Independents under- stand well what Republicans actually find offensive. Hence, if we have pool of web moderators com- prising only Democrats and Independents, their evaluations of what Republicans find as offensive will be reasonably consistent; however, it may not reflect what Republicans truly find as offensive.
Consider the following examples without any profanity; yet the Democrats marked them as offen- sive but the Republicans did not:
More fear-mongering. only .006% of people die of covid. Russia has reason to lie about the death total.
Diversiy====less White people, White sham- ing. I see everyone as Americans not by their Skin color; The real Racist say black, white, brown pride. It is possible that the Democrats found these exam- ples offensive because they did not align with their liberal views.
5.3 Machine and Human Moderators
RQ3: How is the alignment between offense pre- dictions of human moderators and machine moder- ators? Recent reports indicate increasing scrutiny on big-tech platforms' content moderation poli- cies (Douek, 2021). The discussions center around two diametrically opposite positions: these plat- forms are not doing enough, or they need to do
We notice that some obfuscated profanity es- caped machine moderators while human moder- ator groups caught them (c.g., cocksuxxxer, or 3-letter company). We also observe that hu-

---

1.0
gests that content moderation is highly nuanced topic where different political groups can exhibit different levels of tolerance to offensive content depending on the issue.
mi
0.14
0.14
0.082
0.098
REP DEM
0.14
1
0.28
0.32
0.5
What is offensive and should this offensive post be removed from social media can be subjective. as our study indicates. However, when we ask the annotators about fairness of the 2016 and 2020 elections, we notice a more worrisome issue: erod- ing trust in democracy. 5% and 10% of the an- notators believe that the 2016 and 2020 elections were not conducted in a fair and democratic man- ner, respectively. Democrats doubt the faimesss of 2016 election more while the Republicans doubt the fairness of 2020 election. This result sums up the deep, divergent political divide between the left and the right in the US and asks all the stakcholders social media platforms, social web users, media, academic and industry researchers, and of course the politicians to think about how to improve political discourse and restore trust in democracy.
model a
0.6
0.14
0.28
1
0.39
0.62
HumanMa) IND
0.082
0.32
0.39
0.7
-0.4
0.098
0.5
0.62
0.7
-0.2
ml
DEM
REP model_b b
IND HumanMaj
Figure 3: Agreement between machine moderators and hu- man moderators. A cell (i.5) presents the Cohen's 's и agree- ment between moderators. ML is a machine moderator that takes the majority vote of the nine individual machine moder- ators. The majority responses from the human moderators are included in the figure Democrat (DEM), Republican (REP), Independent (IND), and Human Majority (HumanMaj).
mans having deeper contexts allows them to re- spond differently. A dismissive comment about Caitlyn Jenner, a transgender Olympic Gold Medal- ist, is unanimously marked by all groups of human moderators as offensive which the machine moder- ators marked notOffensive (see Table Table and see Figure 17).
5.5 Ablation Studies
When we consider sensitive issues, we find the over- all disagreement observed our general dataset worsens further. Table 5 contrasts the pairwise disagreement between human-human moderators and human-machine moderators across D general- D abont and 'D gum- We first observe that machine- human agreement is substantially lower on the -ssue-specific corpora across all political identi- ties. We next note that some the moderator pairs achieved negative Cohen's 's к values on Dgun even on first-person offense perspective indicating con- siderable disagreement (McHugh, 2012).
5.4 On Censorship and Trust in Democracy
Beyond first-person and vicarious offense, ask the annotators range of questions censorship and election fairness. We explicitly ask every anno- tator if she believes the comment should be allowed on social media. We find that the comments in- dividual political groups marked as offensive on D Pgenenal. the Democrats, Republicans, and Indepen- dents believe 23%, 23%, and 17% respectively, should not be allowed on social media. This im- plies that in general political discussions, Indepen- dents are more tolerant than the two political ex- tremes on what should be allowed on social media. On D gmm. the Republicans exhibit slightly more in- tolerance than Democrats and Independents and want to remove 26% of the offensive content as op- posed to 23% and 14% by the Democrats, and the Independents, respectively. However, on Dabortion the Democrats exhibit more intolerance seeking to remove 23% of the offensive content opposed 21% from both Independents and Republicans. We note that Independents are much more sensitive to reproductive rights than gun control/rights or general political discussions. Our study thus sug-
The pairwise group dynamics behave differ- ently on different issues. While Independents ex- hibit considerable agreement with Republicans on D Pabortion they show little or no agreement on Dewn- Interestingly, while neither the Republicans nor the Independents agree lot with Democrats D ubortiinn these two groups (Independents and Re- publicans) are well-aligned on what Democrats would find offensive in Dabortion- However, once again, when we tally that with what the Democrats actually find as offensive, we see the agreement on the pairs (Democrats, Democrats and { Democrats, Democrats are substantially lower. Table lists few instances that machine mod- crators marked as notOffensive however, human moderators belonging to specific political identities

---

marked them offensive.
Machine moderators: offensive notOffensive
Human Moderators:
We conduct noise audits on a broad range of datasets (Twitter, GAB, Reddit, YouTube) and ob- serve qualitatively similar results (presented in the Appendix B.3).
Republicunts Evangelicunts are scourge to JesusBa- bies snatched from the parents worse then abortion u shameless bastards
Star. We can allow him to murder woman having abortion 5th Avenue and Dickbag trump is horrible for our country. He is a lying who have Americas dumbest supporters
President Pussygrabber
6 Discussion and Conclusion
In this paper, we present two novel perspectives on moderating social media political discourse: dis- agreement between machine moderators, and dis- agreement between human moderators. Our key contributions are (1) a comprehensive noise audit of machine moderators; (2) VOICED, an offensive speech dataset with transparent annotator details; (3) a novel framework of vicarious offense; and (4) a focused analysis of moderation challenges present in dealing with sensitive social issues such reproductive rights and gun control/rights.
its ALL Bidens fault with his open borders!!! TRUMP
WAS WAY BETTER THAN THIS IDIOT
YOU STUPID DEMS NEED TO LOOK AT WHATS HAPPENING TO EUROPE
This really stinks. The democraps really schitf their pants now!!!
Table 2: tween machine moderators and human moderators on D Pameatl The blue and red cells consider Democrats and Republicans human moderators, respectively
Illustrative examples highlighting disagreement be-
Traditional supervised learning paradigm as- sumes existence of gold standard labels. While recent lines of work have investigated disagreement among annotators that stems from the inherent subjectivity of the task (Pavlick and Kwiatkowski, 2019; Ghosh et al., 2021; Prabhakaran et al., 2021; Davani et al., 2022), our analyses of political dis- cussions highly sensitive issues reveal that there could be practically no agreement among anno- tator groups and depending on who we ask, we can end up with wildly different gold standard la- bels reminiscent of alternative facts (Jaffe, 2017). Our current work is primarily descriptive. We ad- dress the elephant in the room and quantify the challenges of offensive content moderation in po- litical discourse both from the machine and human moderators' perspectives. We believe our dataset will open the gates for modeling ideas considering multiple vantage points and yielding more robust systems.
particularly when it comes to polarizing topics and political preference. As evidenced in Table 2, our analysis reveals that the topic and targets of po- tentially offensive post (e.g. politician, political party, etc.) seem to be more important human moderators than to machine moderators as auto- matic methods often rely on the presence of profan- ity to assign a post offensive. This observation in line with datasets and annotation taxonomies that consider the target of offensive posts as the cen- tral aspect of offensive speech identification such as OLID (Zampieri et al., 2019), HASOC (Mandl et al., 2020) and others. The new vicarious of- fense dataset presented in this paper is a valuable resource that can be used for further analysis.
Evaluating Vicarious Offense as a Perspective: The vicarious offense is a novel perspective that is not captured by existing classification models used in our noise audit. In this section, we use ChatGPT API (C)$ (gpt3. 5turbo,y 3.5), a large language model (LLM) to predict the vicarious offense that humans are able annotate as a comparison. For given comment (comment), we use the following prompt: "You are an emotionally intelligent assis- tant who is registered partyA. Classify the intent of the user's text to be offensiive/disrespeectu to someone registered as partyB with ONLY ONE OF THE FOLLOWING CHOICES: 1. Very offen- sive, 2. Extremely offensive, 3. Slightly offensive, 4. Moderately offensive, 5. Not all offensive.
Our study raises the following points to ponder upon.
Issue focused analysis: In Section 5.5, our study barely scratches the surface of -ocused analy- sis. Studies show that there are political disagree- ment between the left and the right on several other policy issues that include immigration (Card et al., 2022), climate change (Fisher et al., 2013), racism in policing (Dutta et al., 2022), to name a few. We believe our study will open the gates for follow research expanding to more issues.
Style vs content: We observed an important in- terplay between the style and the content of posts
Phttps: hhttps:/pplatform http://laaafrrm..p.aaa... setnaai..oom/ooss/odeess///////
t-3-5

---

-1.0
1.0
0.3
0.3
0.3
0.3
0.3
0.3
DEM IND
0.1
1.0
0.1
0.6
0.2
0.3
0.2
0.2
0.2
0.3
0.1
0.3
0.2
0.4
Jim
0.9
-..8
DEM REP
0.1
0.6
0.1
1.0
0.1
0.2
0.1
0.3
0.1
0.2
0.1
0.2
0.1
0.4
DEM REP
IND DEM
0.1
0.3
0.1
0.2
0.1
1.0
0.1
0.4
0.1
0.5
0.1
0.3
0.1
0.5
model
IND REP
0.1
0.2
0.1
0.3
0.1
0.4
0.1
1.0
0.1
0.2
0.1
0.3
0.1
0.5
REP
0.3
0.8
0.8
0.8
0.2
0.1
0.4
REP DEM
0.1
0.3
0.1
0.2
0.2
0.5
0.2
0.2
0.1
1.0
0.1
0.6
0.1
0.5
REP DEP_DEM_lmm lim
REP IND
0.1
0.3
0.2
0.2
0.2
0.3
0.2
0.3
0.2
0.6
0.2
1.0
0.2
0.5
0.2
REP IND_]im Jimm
0.3
0.9
0.8
0.8
0.8
0.8
0.1
HumanMaj
0.1
0.4
0.1
0.1
0.5
0.1
0.5
0.1
0.5
0.1
0.5
0.1
1.0
E
DEM IND
DEN IND jim
DEM REP
DEM REP Jim
DEM
IND DEM Ilm
IND REP
IND REP Jim
DEM
REP DEM Jim
REP IND
REP JND, Jim
HumanMaj
model_b
Figure 4: Agreement between human moderators and GPT model on vicarious offense. A cell (i.j) presents the Cohen's 's agrocment hetween LJ.M model L. and My, Here ml is machine moderator that takes the majority vote of the nine individual machine moderators. The vicarious offense is denoted with par tyA_par ty8. The scores without suffix after the vicarious offense denote the majority of the lahels from the human moderators. The suffix 11m denotes the vicarious offense predicted hy the LLM. HumanMaj indicates the majority response considering all human moderators.
(comment)" In this prompt, partyA and partyB refer Democrat, Republican, indepen- dent. Note that, asked similar question the human moderators "How offensive do you think partyA will find this comment?". Results of this study are included Figure 4.
ation can be potentially gruesome and affect the mental health of the moderators (Solon, 2017). We maintain small batch size (30 YouTube com- ments), one-third of which is marked notOffen- sive by almost machine moderators to minimize the stress on annotators. fact, many of the an- notators left a comment at the end of the study indicating that they enjoyed this task. While our goal is to broaden our understanding of first-person and vicarious offense perception and has the poten- tial to robustify machine moderators, any content filtering system can be tweaked for malicious pur- poses. For instance, an inverse filter can be made that filters out notOffensive posts while filtering the offensive ones.
In this case, the agreement of the L predictions of the vicarious offense was similar to what was observed in the human and ML moderators' agree- ment scores for the first first-person offense identifica- tion. The LLM agreed less with the human modera- and more with the machine moderator majority. This adds evidence how challenging offensive language identification for machine moderators LLMs and non-LLMs alike for this dataset.
Ethics Statement
Limitations
Our study was reviewed by our Institutional Review Board and was deemed exempt. Our YouTube data is collected using the publicly available YouTube API. We do not collect or reveal any identifiable information about the annotators. Content moder-
Our paper has the following limitations. Fine-grained political identities: Unlike recent papers studying political polarization that only con- sider conservative and liberal idcologies (Demszky et al., 2019), our paper marks one of the earliest at-

---

tempts to include the Independents in the mix. That said, political identities can be far more nuanced than the three options a voter can register for. For example, a human moderator can be fiscally con- servative but socially liberal. Understanding both inst-person and vicarious perspectives of offense considering more fine-grained political identities merits deeper exploration.
tainty, we do not model uncertainty in this study.
Acknowledgments
This research has been partially supported by ESL Global Cybersecurity Institution Seed Fund and a gift from Google LLC. We would like to ex- press our gratitude to the human annotators who took part in the study, as well as the anonymous re- viewers for their valuable feedback and suggestions. Additionally, we thank Zohair Hassan and Sheeraja Rajakrishnan for their insightful discussions.
Our dataset was limited to English posts, pri- marily focusing on one platform (YouTube). Ide- ally, study would consider other platforms and languages to provide a greater degree of external validity.
References
Beyond US politics and political polarization: Finally, the framework of vicarious offense has a broader appeal. While we apply this US polities, there is rising political polarization in many other countries (Gruzd and Roy, 2014; Silva, 2018). It does not also have to be always political differences. The vicarious offense framework can also be used to understand religious polarization (Palakodety* et al., 2020; Chandra et al., 2021; Saha et al., 2021). Beyond US annotator populations: We primar- ily conduct our human annotation studics through Amazon Mechanical Turk. Even though a broad annotator demographic was targeted in this study, an extension this could be looking at the broader human annotation platforms while expanding it to other regions.
Aymé Arango, Jorge Pérez, and Barbara Poblete. 2019. Hate speech detection is not easy you may think: A closer look at model validation. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 45-54.
Fan Bai, Alan Ritter, and Wei Xu. 2021. Pre-train or an- notate? domain adaptation with constrained budget. pages 5002-5015.
Paul M. Barrett. 2022. Musk says Twitter is biased against conservatives facts say otherwise. The Hill.
Valerio Basile. 2020. It's the end the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In CEUR WORKSHIP PROCEEDINGS, volume 2776, pages 31-40. CEUR-WS.
Learning to Predict Vicarious Offense: The vicarious offense perspectives collected from the human annotators are challenging to teach an ML model. Our initial round of experiments included a closed LLM, ChatGPT as a baseline. ChatGPT is a black box in this setting, and hope to ex- plore this phenomenon further other publicly available systems built on work by Ouyang al. (2022) (InstructGPT) and Touvron al. (2023) (Llama 2). Perspective API is another service that is used for predicting personal offense, we use Detoxify (Hanu and Unitary team, 2020) as model that is trained on a dataset part of the Per- spective API for consistent reproducibility. We include responses from ChatGPT and Perspective API for the human annotated VOICED dataset in Appendix Figure 15.
Valerio Basile, Cristina Bosco. Elisabetta Fersini. Nozza Debora. Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti, et al. 2019. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. 13th International Workshop on Semantic Evalua- tion, pages 54-63. Association for Computational Linguistics.
Thomas W Benson. 1996. Rhetoric, civility, and com- munity: Political debate computer bulletin boards. Communication Quarterly, 44())355-378.
Brent Bozell. 2004. Weapons of mass distortion:
The coming meltdown of the liberal media. National Review.
Michael Bugert, Nils Reimers, Shany Barhom, Ido Dagan, and Iryna Gurevych. 2020. Breaking the subtopic barrier in cross- docssed event coreference resolution. In Tex:2Story @ ECIR.
Modeling uncertainty Vicarious offensive is re- lated concepts such as soft labeling (Vyas et al.. 2020) and Bayesian truth serum (Prelec, 2004). We generally work with hard labels and where Bayesian truth serum is about predicting uncer-
Dallas Card, Serina Chang, Chris Becker, Julia Mendel- sohn, Rob Voigt. Leah Boustan, Ran Abramitzky, and Dan Jurafsky. 2022. Computational analysis 140 years of us political speeches reveals more positive

---

but increasingly polarized framing of immigration. Proceedings of the National Academy of Sciences, 19.
the International AAAI Conference on Web and So- cial Media. 17(1):161-17..
Evelyn Douek, 2021. More Content Moderation Is Not Always Better. WIRED.
Evelyn R Carter and Mary C Murphy. 2015. Group- based differences perceptions of racism: What counts, to whom, and why? Social and personality psychology compass, 96):269-280.
Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, and Chris Welty. 2018. Crowdtruth 2.0: Quality metrics for crowdsoureing with disagree- ment. arXiv preprint 188.v:888.00000
Mohit Chandra, Manvith Reddy, Shradha Sehgal, Saurabh Gupta, Arun Balaji Buduru, and Ponnu- rangam Kumaraguru. 2021. " a virus has no reli- gion": Analyzing islamophobia on twitter during the covid-19 outbreak. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media, pages 67-77.
Sujan Dutta, Beibei Li. Daniel S. Nagin, and Ashiqur R. KhudaBukhsh. 2022. A murder and protests, the capitol riot, and the chauvin trial: Estimating dis- parate news media stance. In Proceedings of the Thirty-First International Joins Conference on Arti- ficial Intelligence, IJCAI-22, pages 5059-5065. Al for Good.
Eshwar Chandrasckharan, Umashanthi Pavalanathan, Anirudh Srinivasan, Adam Glynn, Jacob Eisenstein, and Eric Gilbert. 2017. You can't stay here: The efficacy of reddit's 2015 5 ban examined through hate speech. Proceedings of the ACM on Human- Computer Interaction, I(CSCW):1-22.
Eli Finkel, Christopher A Bail, Mina Cikara, Peter H Ditto, Shanto lyengar, Samara Klar, Lilliana Mason, Mary C McGrath, Brendan Nyhan, David G Rand. et al. 2020. Political sectarianism in america. Sci- ence, 370(6516):533-588)
Emily Chen, Ashok Deb, and Emilio Ferrara. 2021. # election2020: the first public twitter dataset on the 2020 us presidential election. Journal of Computa- tional Social Science, pages 1-18.
Dana R Fisher, Joseph Waggle, and Philip Leifeld. 2013, Where does political polarization come from? locat- ing polarization within the us climate change dehate. American Behavioral Scientist, 37(1):70-92.
cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, nithum, and Will Cukierski. 2017. Toxic comment classification challenge.
Gallup. 2022. Party Affiliation. Gallup.
Sayan Ghosh, Dylan K. Baker, David Jurgens, and Vinodkumar Prabhakaran, 2021. Detecting cross- geographic biases in toxicity modeling on social me- dia. In Proceedings of the Seventh Workshop on Noisy User Aser-generated Text, W-NUT 2021, Online, November 11, 2021, pages 313-328. Association for Computational Linguistics.
Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002. Hate speech and constitu- tional protection: Priming values of equality and freedom. Journal of Social Issues, 58(2):247-263
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective An- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
Karen Gift and Thomas Gift. 2015. Does politics influ- ence hiring? evidence from randomized experiment. Political Behavior, 3753)655.-675.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. In Proceedings of the 11th International AAAI Confer- ence on Web and Social Media, ICWSM '17, pages 515.
Homero Gil de Zúñiga. Teresa Correa, and Sebastian Valenzuela. 2012. Selective exposure cable news and immigration in the us: The relationship between fox news, cnn, and attitudes toward mexican immi- grants. Journal of Broadcasting & Electronic Media, 56(4):597-615.
Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Juraf- sky. 2019. Analyzing polarization in social media: Method and application to tweets on 21 mass shoot- ings. In Proceedings of NAACL-HLT 2019, pages 2970-3005. ACL.
Lara Grimminger and Roman Klinger. 2021. Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Of fensive Speech and Stance Detection. In Proceed- ings of the Eleventh Workshop Computational Approaches to Subjectivity, Sentiment and Social Me- dia Analysis, pages 171-180, Online. Association for Computational Linguistics.
Igor Derysh. 2021. Despite Parler backlash, Facebook played huge role in fueling Capitol riot, watchdogs say. Salon.
Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N Asokan, 2018. All you need is" love" evading hate speech detection. In Proceedings of the I 11th ACM workshop on artificial intelligence and security. pages 2-12.
Xiaohan Ding, Michael Horning, and Eugenia H. Rho. 2023. Same words, different meanings: Semantic polarization in broadcast media language forecasts polarity in online public discourse. Proceedings of

---

Anatoliy Gruzd and Jeffrey Roy. 2014. Investigating po- litical polarization on twitter: A canadian perspective. Policy & internet, 6(1 (1)):88445,
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive lan- guage datasets with annotators' disagreement. ArXiv, abs/2109.13566 135663
Andrew Halterman, Katherine A. Keith, Sheikh Muham- mad Sarwar, and Brendan O' O'Connor. 2021. Corpus- level evaluation for event QA: the indiapoliceevents corpus covering the 2002 gujarat violence. In ACL/IJCNLP 2021, Findings of ACL, pages 4240- 4253.
Tong Liu, Akash Venkatachalam, Pratik Sanjay Bongale, and Christopher Homan. 2019. Learning predict population label distributions. In Companion of The 2019 World Wide Web Conference, WWW 2019, pages 1111-1120. ACM.
Laura Hanu and Unitary team. 2020. Detoxify. Github. https://gtthu..co// htt..//mmanaaaa//dttiiiiii
Thomas Mandl, Sandip Modha, Anand Kumar M, and Bharathi Raja Chakravarthi. 2020. Overview of the hasoc track at fire 2020: Hate speech and offensive language identification tamil, malayalam, hindi, english and german. In Forum for Information Re- trieval Evaluation, pages 29 32.
Gregory A Huber and Neil Malhotra. 2017. Political homophily in social relationships: Evidence from online dating behavior. The Journal of Politics, 79(1):269-28)
Ki Deuk Hyun and Soo Jung Moon. 2016. Agenda setting in the partisan tv news context: Attribute agenda setting and polarized evaluation of presiden- tial candidates among viewers of nbc, cnn, and fox news. Journalism & Mass Communication Quar- terly, 93(3):509-529
Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar. Sahana Udupa, and Hinrich Schuetze. 2022. Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments. In Find- ings of the Association for Computational Linguis- ficx: ACI. 2022, pages 1089-1 104, Dublin, Ireland. Association for Computational Linguistics.
Shanto Iyengar and Sean J Westwood. 2015. Fear and loathing across party lines: New evidence on group polarization. American Journal of Political Science, 59(3):690-707.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann. Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: benchmark dataset for ex- plainable hate speech detection. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- lelligence, EAAI 2021, Virtual Event, February 2-9, 202 I, pages 14867-14875. AAAI Press.
Alexandra Jaffe. 2017.
Kellyanne Conway: WH Spokesman Gave AAtermative Facts' on Inauguration Crowd. NBC News.
Daniel Kahneman, Olivier Sibony, and Cass R Sunstein. 2021. Noise: A flaw in human judgment. Little. Brown.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-288
Ashiqur R. KhudaBukhsh, Rupak Sarkar, Mark S. Kam- let, and Tom M. Mitchell. 2021. We Don't 't Speak the Same Language: Interpreting Polarization through Machine Translation. In Thirty-Fifth AAAI Confer- ence on Artificial Intelligence, AAAI 2021, pages 14893-1444 14901. AAAI Press.
Michael I Norton and Samuel R Sommers. 2011. Whites see racism as a zero-sum game that they are now losing. Perspectives on Psychological science, 6(3):215-218
Ashiqur R. KhudaBukhsh, Rupak Sarkar, Mark S. Kam- let, and Tom M. Mitchell. 2022. Fringe news net- works: Dynamics of US news viewership following the 2020 presidential election. In WebSei '22: 14th A ACM Web Science Conference 2022, pages 269-278. ACM.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller. Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumerie, Kurt Thomas, and Michael Bailey. 2021. Designing Toxic Content Classification for Diversity Perspectives. arXiv:2106.04511 [es]. ArXiv: 2106.04511.
Shriphani Palakodety', . Ashiqur R. KhudaBukhsh", and Jaime G. Carbonell. 2020. Mining insights from large-scale corpora using fine-tuned language mod- els. In ECAI 2020 24th European Conference Artificial Intelligence, pages 1890-1897.
Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2020. Evaluating aggression iden- tification in social media. In Proceedings of the Sec- ond Workshop on Trolling, Aggression and Cyber- bullying. pages 1-5, Marseille, France. European Language Resources Association (ELRA).
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 7:677

---

Sarah Perez and Brian Heater. 2021. Apple suspends Parler from App Store. TechCrunch.
Maarten Sap. Swabha Swayamdipta, Laura Vianna. Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5986. Association for Computational Linguis- tics.
Massimo Poesio, Jon Chamberlain, Silviu Paun, Jun- tao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers). pages 1778-1789, Min- neapolis, Minnesota. Association for Computational Linguistics.
Rupak Sarkar and Ashiqur R. KhudaBukhsh. 2021. Are chess discussions racist? an adversarial hate speech data set (student abstract). In Thirty-Fifthh AAAI Con- ference on Artificial Intelligence, AAAI 2021, pages 15881- AAAT Press.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator- labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138. Association for Computational Linguistics.
Bruno Castanho Silva. 2018. Populist radical right par- ties and mass polarization in the netherlands. Euro- pean Political Science Review, 10(2):219-244.
Olivia Solon. 2017. Facebook is hiring moderators. But is the job too gruesome to handle? The Guardian.
Drazen Prelec. 2004. A bayesian truth serum for sub- jective data. science, 306(5695):462 466.
Alessandra Stanley, 2012. How MSNBC Became Fox's Liberal Evil Twin. Online: accessed -e-September- 2020.
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. A benchmark dataset for learning to intervene in online hate speech. In EMNLP-IJCNLL pages
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- hert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswani, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ama Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- steim, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang. Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Tharindu Ranasinghe and Marcos Zampieri. 2020. Mul- tilingual offensive language identification with cross- lingual embeddings. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP). pages 5838-5844 Online. As- sociation for Computational Linguistics.
Björn Ross, Michael Rist, Guillermo Carbonell, Ben- jamin Cabrera, Nils Kurowsky. and Michael Wojatzki. 2016. Measuring the Reliability of Hate Speech An- notations: The Case of the European Refugee Crisis In Proceedings of NLP4CMC.
Punyajoy Saha, Binny Mathew, Kiran Garimella, and Animesh Mukherjee. 2021. "short is the road that leads from fear to hate": Fear speech in indian what- sapp groups. In Proceedings of the Web Conference 2021, pages 1110-1121.
Maarten Sap, Saadia Gabriel, Lianhui Qin. Dan Jurafsky. Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implica- tions of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 5477-5490. Association for Computational Linguis- tics.
Nidhi Vyas, Shreyas Saxena, and Thomas Voice. 2020. Learning soft labels via meta learning.
Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 1 138- 142.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2021. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. CoRR, abs/2111 11:2111.0770 1000997.
Tharindu Cyril Weerasooriya, Tong Liu, and Christo- pher M. Homan. 2020. Neighborhood-basee pooling for population-level label distribution learning. In ECAI 2020 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago

---

de Compostela, Spain, August 29 September 2020 Including 10th Conference on Prestigious Applica- tions of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Appli- cations, pages 490-497. IOS Press.
Tharindu Cyril Weerasooriya, Sarah Luger, Saloni Pod- dar, Ashiqur KhudaBukhsh, and Christopher M. Homan. 2023. Subjective crowd disagreements for subjective data: Uncovering meaningful CrowdOpin- ion with population-level learning. In Proceedings of the ō1st Anmual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 950-966, Toronto, Canada. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-43 45, Online. Association for Computational Linguistics.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In NAACL-HLT, pages 1415- 1420.
Savvas Zannettou, Barry Bradlyn, Emiliano De Cristo- faro, Haewoon Kwak. Michael Sirivianos, Gianluca Stringini, and Jeremy Blackburn. 2018. What is gab: bastion of free speech or an alt-right echo chamber. In Companion Proceedings of the The Web Confer- ence 2018, pages 1007-1014.

---

A Supplemental Material
times, and the mean value is taken the final reported result.
A.1 Experimental Setup and Reproducibility
The paper presents novel concepts that are more involved in understanding human annotations, but a significant portion of the experiments were con- ducted through ML models. The code to repro- duce the first stage of the paper (noise audit) in- volves runninng the nine MM moderators through the YouTube dataset. The adaptation of the MMs for rumning the experiments is included in https: / //gi thub. cww Laaoonaann-aaaaiiiaaoa time for the predictions (1 million comments) on an Ubuntu 18.04, Intel i6-7600k (4 cores) at 4.20GHz, 32GB RAM, and n Vidia GeForce RTX 2070 Super 8GB VRAM machine averaged hours. The exper- imental code for analyzing the human responses is included in the repo.
B.1 Language Analysis on the Dataset of Human Moderators
here stypid amer ica aedloo choice ther teee
wiblack sen by chil0 Life wan t Sake
milldrn now
d
go.abortio
right
Women baby
needsspoott
Country catholic womaneven
note oe mur der ahe Hout
god
pn
democ
cra t Ropy shac
The run
traing sorth control O
apo
don
Wi
support t trump
bables eel iill preghent
#hinkway
give
really
gun banback
kil
State Pi
vote
el tbera a tine aW thrrthineppp chppeepple
usp
crrpubbl wwwblicanaiaee can biden
end hypocr goung
fuck
pa ke ameri Wit
can
car eiibellee
abortion clinic illegal lying getting
Figure Word cloud on the Abortion dataset
B
Machine Moderators
fdaki-
santstuppo
country
kidthinkamerra waa.iiian...iin
arth Literaai
know
stop
Parameter
Value
adam epsilon
le-8
batch size
64
COP
NOW
guy
epochs
3
amer ca
fuck
ma ke SERRET
Even shooting
learning rate
le-5
peop
e
one
warmup ratio
warmup steps
0
0.1
eun
trump
right
keep
u
max grad norm max seq. length
1.0
crimina time
need mamoocrateie
256
gradient accumulation steps
POLACe democr
aagggnn control
Figure 6: Word cloud on the Gun dataset
Table 3: BERT Parameter Specifications.
Tieetine
amer
iinn
göing
thinksed
Count
Black clackortee
They trained BERT on the datasets Table 4, which has achieved state-of-the-ee on a variety of offensive language identification tasks. From an input sentence, BERT computes a feature vector h I&d, upon which we build classifier for the task. For this task, we implemented softmax layer, i.e., the predicted probabilities are ( coftmaz(Wi) where W kkxd is the softmax weight matrix and & the number of labels. For the experiments, they used the BERI-LARGI-CASLD model available HuggingFace (Wolf et al., 2020).
nan lylne
tell
Ruy
say
am
Gaa
rake
need
Retieest cro
democrat
peop
e
Juln
to
look
thing
right
stupic
back
biden
nothing
rool
good
now
id1ot
sany
see
resl
W
vote e
repub
presid
dresideeti
U
Know
mant
Figure 7: Word cloud on the General datasct
To train the models they used a GeForce RTX 3090 GPU to train the models. They divided the dataset into training set and validation set using 0.8:0.2 split. For BERT they also used the same set of configurations mentioned in Table in all the experiments. They performed early stopping if the validation loss did not improve over ten evaluation steps. All the experiments were conducted three
B.2 Temporal Evolution of the Dataset
Figure 8 demonstrates that our dataset has sparse user engagement via comments between 2014 and 2018. Figure demonstrates how machine moder- ators agree on the level of toxicity.

---

Training
Testing
Dataset
Inst.
OFF %
Inst.
OFF
Data Sources
Reference
AHSDD
4,956
Twitter
Davidson al. (2017)
HASOC
5,604
0.36
1,401
0.35
Twitter, Facebook
Mandl al. (2020)
HatEval
9,000
1,434
Twitter
et (2019)
HateXplain
11,535
3,844
Twitter, Gab
Mathew (2021)
OHS
8,285
0.21
2,090
0.20
Reddit
Qian (2019)
OLID
13,240
0.33
860
0.27
Twitter
Zampieri (2019)
TCC
12,000
2,500
Wikipedia
URL
TRAC 4,263
1,200
Facebook, Twitter, You'
Kurar (2020)
Table 4: The eight datasets with the number of instances (Inst.) in the training and testing sets, the OFF % in each set, the data source, and reference.
fluencer Ben Shapiro (0.22+0.000); (3) YouTube
MBNEC
CNN
comments on political influencer Bryan Tyler Co- hen (0.26=0.0002;; (4) YouTube comments on BBC news (0.23=0.001);; (5) YouTube comments The Young Turks (0.26::0.001);; (6) Newsmax TV (0.20=00.00);; (7) a Twitter dataset on US poli- tics (Chen et al., 2021) (0.25=00002): (8) A Red- dit r/politics dataset (0.25 and (9) a Gab dataset (Zannettou et al., 2018) (0.29=10.001), Hence, our qualitative noise audit claims hold over several datasets across different social networks.
Fox Nows
201 2018
40
Figure 8: Temporal trend showing number of comments made about news videos on three news networks' offi- cial YouTube channels over time.
0.34
0.32
Feead bapps
0.30
0.28
0.78
0.22
2014
2015
2018
2017
2018
2019
2000
7021
2022
war
Figure 9: Temporal trend in agreement among ma- chine moderators on toxicity. bands represent 95% confi- dence interval.
B.3 Analyses on Other Datasets MM Moderators
We conduct large-scale analysis of YouTube chan- nels of prominent US cable news networks. While these networks attract a diverse audience, legiti- mate curiosity is whether the qualitative findings in noise audit hold across other YouTube chan- nels or platforms. With similar experimental set- tings, we conduct noise audits on the following nine datasets (Fleiss' k presented within parenthe- ses): (1) YouTube comments on The Hill (0.18 0.002); (2) YouTube comments on political in-
"nttps://Figshare.ccom http:///iiisateeecomaaaaccc////a
tics_Dattaett2222223

---

C Human Moderators
In Figures 12, and race of the annolator population. The majority of the annotators belong to the white or Caucasian race and the age group of 25-34. The overall annotator demographics are described in Figure 13.
the distributions for ages
This section includes in-depth analysis of the popu- lation of human annotators who participated in study.
We also asked the question if the presidential election in the years 2016 and 2020 was conducted in fair and democratic manner. The interesting in- sight from these two figures show how the members of losing party thinks the election wasn't conducted in fair manner. In 2016, the Republican president Donald J. Trump won the election and out the an- notators who thought wasn't fair are Democrats, similarly in 2020 the Democratic nominee Joseph R. Biden Jr. won the election and majority of the annotators who believe election wasn't 't fair are Re- publicans.
C.1 Human Annotation Study
We conducted our human annotation study using Qualtrics for the surveys and Amazon Mechanical Turk for annotator recruitment. We have included a copy of the survey (in human readable format) as well as in format that can be easily imported to Qualtrics in our code repo. The repo also includes resources running similar human annotation study.
C.2 Distribution of Independents in Election Voting
Gender
Political Leaning Independent Republican
100%
Requblicans Independents
80%
of Total Count People
Democral
30%
Percontage
60%
35%
20%
12%
0%
Female
Male
Year
Figure 11: Distribution of the annotators based on their political leaning and gender. The % denotes total per- centage from the whole population who belong to cach subgroup.
Figure 10: Distribution of political identities reported historical Gallup surveys (Gallup, 2022) over 19 years.
C.3 Annotator Compensation
Initial pilot by the authors estimated 12S/hour com- pensation (completion time: 15 minutes; compen- sation: 3S/task) more than US minimum wage (7.25$) and falls within the range reported in the literature (c.g., 6$/hour in Leonardelli et al. (2021); 7.25S/hour in Bugert et al. (2020); and 13S/hour in Bai et al. (2021)). The final study yielded 7.8S/hour median compensation (completion time:
23 minutes).
C.4 Demographic Analysis
The distributions for gender did not show any im- balances the population had equal representation from each gender (50% from each). Interestingly, the majority of the Democrat population was fe- male and there was a higher population of male annotators who were Independents.

---

Figure 12: Distributions based on Race (left) and Age (right). The counts denotes number of annotators from the whole population who belong to each subgroup. The colors denote the political leaning.
Netiva
American ine American or Alaska Native
Hawelian Other Pacific
Asian
Blacko Africcn American
Other
Proferr not
White Caucasian
40%
115
119
98
104
20%
1.0%
107
104
34
1
1
Femtlle
Male
Femele
Male
Female
Male
Prefeer say
M/le
Male
Female
Female
Male
Non-bimary /third
40%
85
10%
40%
17%
of People
97
93
Sf
1556
53
1.0%
20%
10%
115
33
36
38%
71
0%
0%
Female
Maie
Non-binary/ thind gen...
18.
25. 84
36-44
45.54
55--44
65
older
Figure 13: Overall demographic distribution of the annotator pool. The colors denote the political identities.

---

D Annotator Agreement
annotator was exposed. We use this metric sup- plement the analysis from the prior sections. Figure contains histograms for each perspective consid- ered based the political leaning of the annotators. Based on the analysis from the Figure 14, there are some interesting observations for this study, Based on the highest agreement bins from the fig- ure and the highest bin (0.9 to 1.0). The number of Democrat leaning annotators that showed the strongest agreement was surpassed by other politi- cal leanings across all the three perspectives con- sidered. Majority the Independents showed the strongest agreement across the three perspectives, bypassing their Democrat and Republican leaning annotators. And majority of the Republican lean- ing annotators showed the strongest agreement with the overall human majority label.
D.1 Disagreement Across Datasets Raw
Results
Moderatoons
Moderators
D Dawena
Davens
Pas
Machines
Repuhlicans
023
0.04
0.07
Machines
Democrats
0.19
0.06
0.11
Machines
Independents
0.17
4002
Republicans
Democrats
0.34
0.05
-0.01
Independenis
Independents
Republicans
0.39
0.36
-0.03
Democrals
Democrats
0.38
-0.05
-0.02
Democrats
Democrats fu
Republicans
Repuhlicans"
0.37
0.01
0.00
Republicans
Republicans
0.35
0.15
0.03
Independents
Independents"
0.37
0.18
0.04
Independents
Independems" Pu
0.43
Repuhlicans
Repuhlicans
0.49
0.01
00.4
Democrats
Democrats
0.44
0.57
0.10
Independents
Independents
0.37
0.06
0.05
Table Disagreement across different annotation
datascts.
D.2 Understanding Human Agreement
Gr Anectator
Group Arnatater
DEM
DEM
DEM
Bettlll
REP
REP
REP
Agramattn Magarty
Figure 14: Histograms of the agreement scores for hu- man annotators. (top to bottom) The rows denote polit- ical leaning, also represented with colors, Democrat, Independent, and Republican. (left right) The first set of graphs capture the agreement for annotators against other annotators from the same political leaning. Sec- ond set captures the agreement of the annotator against the majority the opposing groups. Third set repre- sents the agreement against the majority opinion of the entire population of annotators.
Moving beyond Cohen's k. in this section we study the agreement of a single annotator against three perspectives.
(1) other annotators who are from the same political leaning (in-group), (2) the majority of the other annotators from the opposing groups (out-group), and (3) the overall human an- notator pool majority (humans). We study these agreements by calculating the match score, i.e., the ratio matched labels for the perspective consid- ered divided by the number of total data items the

---

Additional results to the discussion on Figure 3. The tables below demonstrate the confusion matrix for each moderator pair.
(a) Cohen's x 0.17
(b) Cohen's N is 0.19
(c) Cohen's r: is 0.23
Rep
nottifiansiee 8.725
oliasiv 2500%
alossoy 2525%
Mofooie afrechee
norOffennin offenctee
10009
mowfffonei ofenchee
11 45%
ML
0.38%
58.72%
ML
8.12%
$7.
ML
7,30%
S7,619
Table 6: Confusion matrices between machines and humans
(a) Cohen's & is 0.34
(b) Cohen's 0.39
(c) Cohen'ss 's &
Dem
nonOffonshr
nonOfforatoe 3.81%
affentiee
To 08%
71.10
Rep
6.78%
73915
noOfffeniive offoruteee
5.91%
75.28%
Table 7: Confusion matrices hetween humans with different political identities
(a) Cohen's is 0.37
(h) Cohen's is 0.38
(c) Cohen's 's & is 0.37
Dem notOifforbie 8.12
Ind noneaffomntee 7.27%
affomnter 1031%
affencier 7,09%
Kep
Proe
neddfferrziiee ayformee
Pry
3,30%
(d) Cohen's 's is 0.35
(e) Cohen's is 0.46
(l) Cohen's 0.43
Rep
Dem
Etd nonOffimsiee
mofofffrenee 1.27%
affersio
nonOffenmte
offonmiee 3.90%
offorstor
Reg
monifeennse offerunee
Deu
8,34%
Ind
7,069%
Table 8: Confusion matrices between vicarious offense and ground truth
(a) Coben's 's is 0.49
(b) Cohen's is 0.44
(c) Cohen's 0.37
Dea
Lad
Roffefereti 9,5%
affersioe
nonOifee 6.54%
offensioe 4,70%
nonOffinsie
offoriiv
Rep
offersice
3.5%%
Den
7.26%
X1235
39%
Table 9: Contrasting vicarious offense predictions
The confusion matrices below includes results from comparing the human annotation input against other closed systems such as ChatGPT (gpt3.5turbo, v3.5) and Perspective API. In the figures, gpt_labe input from ChatGPT and papi labi_label input from Perspective API. For papi_label as per the API documentation' 8 we used 0.7 as the threshold for toxicity classification. For obtaining the gpt_label, we use the prompt "You are an with
Figure 15:
emotionally intelligent assistant. Classify the intent of the user's ONLY ONE OF THE FOLLOWING CHOICES: 1. Very offensive, 2. Extremely offensive, 3. Slightly offensive, 4. Moderately offensive, or Not at all offensive."
be offeensiveddisrsppeetull
Soous Maurta)
stesharunD 1.0
11
1:00
0.93
0.38
2.51
2:14
0.%
5.08
MPc
722
02
0:20
1.00
0.42
0.00
0:12
0:15
E.07
-18
P13
0.25
0.42
1.00
5.71
6.2
0.9
0.0k
11
3.11
0:23
1.00
24
26
03
04
3.14
0:15
0.29
14
03
04
69
61
PMN mmi af
61
81
-12
mocel

---

Figure 16: Additional Examples the Examples in Figurel
"Potentially fucicing Trump and sevocisent ed with the insureation should be prison."
CN
uniher desth and agony farilly shoulcers
any regular person they yours, makes
worship guns more than poople, including young Innocent children"
INO
lane you jokingt ill do you radly want any advice on any subject from Caitlyn Jenner??3??*
Eg5
happen. was people had onough. Thank you president Trump. Blddan did how the votes were staten.."
my president.
"Friendly advartinn
company: Good old days are back."
IND D33
Figure 17: Illustrative examples highlighting nuanced inconsistencies between machine moderators and human moderators with different political leanings. For every comment, majority vote is used aggregate individual machine and human moderator's 's verdicts. An angry emoji and green checkbox indicate offensive and norOffensive labels, respectively. These real-world examples are drawn from comments on YouTube news videos of three major US cable news networks: Fox News, CNN, and MSNBC. Each example is annotated by 20 human moderators with at least six Republicans, Democrats, and Independents. Nine well- nell-known offensive speech data sets are used to create nine machine moderators. The grid summarizes vicarious offense where annotators belonging to row political identity asked predict vicarious offense perspectives of the other political identities mentioned in the columns.


Reviews:
[{"review_id": "LEA1pFGI0Q", "reviewer": "Reviewer_Gr8n", "summary": "", "questions": "A. What is the motivation to use the generic datasets for the special case of identifying offenses in US-based political conversations? It is possible models may not learn well because the datasets do not provide enough information related to democrats, republicans, and independent groups based offense.\n\nB. Do the authors try to use few-shot ChatGPT/Perspective API or other open-source APIs for vicarious and first-person offense identification? It is possible the results might be changed greatly. However, it is still a possibility that can be understood after running the experiments.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that all three reviewers found our work exciting and methodologically sound. We thank them for their constructive criticism, improvement suggestions, and pointers to relevant literature. We have already addressed most of them in the rebuttal. We will address all of them in the final version. \n\nWe thank the reviewer for their feedback on our own. We will address the improvements suggested to the paper along with sections to clarify the issues raised in the reviews. \n\n> A. What is the motivation to use the generic datasets for the special case of identifying offenses in US-based political conversations? It is possible models may not learn well because the datasets do not provide enough information related to democrats, republicans, and independent groups based offense.\n\nQuestion A: As the reviewer suggests, we have included ChatGPT and Perspective API results. A key takeaway from our new results on first-person offense with ChatGPT and Perspective API is that they are *not* significantly better aligned with humans as compared to other machine moderators. \n\nWe agree with the reviewer that generic datasets may not fully capture the nuances of the current US political schism. However, several of these datasets are sourced from Twitter, Reddit, and GAB and are likely to contain political content. For example, in OLID [Zampieri et al. https://aclanthology.org/N19-1144/], authors sampled replies from both sides of the political spectrum, using politically charged keywords such as MAGA, Antifa, Conservatives, Liberals, etc. Also, our research question here is how often models trained on these datasets agree when evaluated in the wild. The nine datasets that we selected are well-known within the community and the audit reveals that their in-the-wild agreement is low. Our ablation analysis indicates qualitatively similar results in other datasets unrelated to US politics as well (Line 1082 reports results on BBC News comments). \nWe are happy to include any additional methods the reviewer recommends (e.g., PaLM 2). Also, we need to keep in mind that running experiments on millions of data points is computationally prohibitive (PaLM2 allows 30 requests per minute, and ChatGPT costs will be prohibitively large). The analysis of the human annotated dataset of 2,310 items with Perspective API and ChatGPT took ~1 hr working around with the API restrictions in place. \n\nFinally, as the API versions can be modified externally (e.g., a recent study reports ChatGPT’s declining math skills [Chen, Zaharia, Zhou et al., 2023, https://arxiv.org/pdf/2307.09009.pdf], in can have unforeseen reproducibility challenges. In contrast, the offline models (will be made publicly available upon acceptance) we use enable other researchers to reproduce our work.  \n\n>B. Do the authors try to use few-shot ChatGPT/Perspective API or other open-source APIs for vicarious and first-person offense identification? It is possible the results might be changed greatly. However, it is still a possibility that can be understood after running the experiments.\n\nIn response to Question B, about utilizing the few-shot approaches ChatGPT/Perspective API: We did analyze with the ChatGPT and the Perspective API on the overall alignment of the offensiveness against the current set of models and human annotators (see the table). Looking at the results (Cohen’s Kappa, Figures 1A and 1B), the humans agree more with the ML and ChatGPT label and less with the Perspective API. This is an observation that is common looking at F1 and recall (Figures 2 and Figure 3). We will include these findings in the final version of the paper.\n\n### Additional Results\n\nThese models are from the GeneralOffense models and TB is toxic BERT.    \n-   **ml** - this is the majority label from all the models (GO and TB)\n-   **gpt_label** - this is the label when ChatGPT 3.5 is used for classification (prompt is included in details). \"*You are an emotionally intelligent assistant. Classify the intent of the user's text to be offensive/disrespectful to someone with ONLY ONE OF THE FOLLOWING CHOICES: 1. Very offensive, 2. Extremely offensive, 3. Slightly offensive, 4. Moderately offensive, or 5. Not at all offensive.*\". This prompt is similar to the one we've used for analysis of the vicarious offense.  \n- **papi_label** - this label is from Perspective API Scores. Following the guidelines from the [documents](https://developers.perspectiveapi.com/s/about-the-api-score?language=en_US), we used the cut off as 0.7.\n-   **HumanMaj** - this is the majority opinion of the humans.\n-   **DEM**,**REP**, and **IND** - majority opinion of all the Democrat, Republican, and Independent leaning human annotators respectively.\n\n**Cohen's Kappa**\n|   |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.30 |  0.33 |       0.51 | 0.14 |        0.15 |         0.08 |\n| REP        |  0.30 |  1.00 |  0.42 |       0.63 | 0.15 |        0.15 |         0.07 |\n| IND        |  0.33 |  0.42 |  1.00 |       0.71 | 0.09 |        0.10 |         0.04 |\n| HumanMaj   |  0.51 |  0.63 |  0.71 |       1.00 | 0.11 |        0.11 |         0.05 |\n| ml         |  0.14 |  0.15 |  0.09 |       0.11 | 1.00 |        0.60 |         0.39 |\n| gpt_label  |  0.15 |  0.15 |  0.10 |       0.11 | 0.60 |        1.00 |         0.27 |\n| papi_label |  0.08 |  0.07 |  0.04 |       0.05 | 0.39 |        0.27 |         1.00 |\n\n**Recall**\n|  |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.63 |  0.64 |       0.70 | 0.61 |        0.60 |         0.60 |\n| REP        |  0.67 |  1.00 |  0.69 |       0.77 | 0.64 |        0.62 |         0.61 |\n| IND        |  0.72 |  0.74 |  1.00 |       0.83 | 0.61 |        0.60 |         0.58 |\n| HumanMaj   |  0.87 |  0.90 |  0.88 |       1.00 | 0.65 |        0.62 |         0.61 |\n| ml         |  0.56 |  0.56 |  0.54 |       0.54 | 1.00 |        0.78 |         0.74 |\n| gpt_label  |  0.57 |  0.56 |  0.54 |       0.54 | 0.83 |        1.00 |         0.70 |\n| papi_label |  0.56 |  0.55 |  0.53 |       0.54 | 0.75 |        0.68 |         1.00 |\n\n**F1-Scores**\n\n|  |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.65 |  0.66 |       0.75 | 0.55 |        0.57 |         0.40 |\n| REP        |  0.65 |  1.00 |  0.71 |       0.82 | 0.54 |        0.56 |         0.37 |\n| IND        |  0.66 |  0.71 |  1.00 |       0.86 | 0.50 |        0.52 |         0.34 |\n| HumanMaj   |  0.75 |  0.82 |  0.86 |       1.00 | 0.50 |        0.52 |         0.33 |\n| ml         |  0.55 |  0.54 |  0.50 |       0.50 | 1.00 |        0.80 |         0.66 |\n| gpt_label  |  0.57 |  0.56 |  0.52 |       0.52 | 0.80 |        1.00 |         0.57 |\n| papi_label |  0.40 |  0.37 |  0.34 |       0.33 | 0.66 |        0.57 |         1.00 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is interesting and exciting. \nThe need to focus on vicarious offense is indeed important from the societal point of view.\nResults and discussions are properly backed by political theories.\nThe comparison of human-human moderators is well discussed and presented.\nOverall, the writing of the paper is impressive as it is well written and understood easily.", "reasons_to_reject": "I am not convinced how authors have claimed about machine moderators. I feel that not enough importance is given to the preparation of machine moderators. Similar architectures of BERT /RoBERTa models are trained on different datasets and used as machine moderators to evaluate their performance on unseen political data. Due to the very simple architecture and the fact that I think these datasets are general datasets (with high imbalance) that are not entirely based on political data, it is possible that these models do not perform well on first-person offense detection and therefore have not been well aligned. I like that the authors in the discussion section used ChatGPT for identifying vicarious offense, but they have not shown how ChatGPT performs in identifying first-person offense compared to the models used in the study. Nevertheless, major claims are made about the inability of machine moderators to identify offense. In my opinion, these claims need to be backed up by better models, APIs such as Perspective API, or recent open-source/APIs-based large language models.", "questions_to_authors": "A. What is the motivation to use the generic datasets for the special case of identifying offenses in US-based political conversations? It is possible models may not learn well because the datasets do not provide enough information related to democrats, republicans, and independent groups based offense.\n\nB. Do the authors try to use few-shot ChatGPT/Perspective API or other open-source APIs for vicarious and first-person offense identification? It is possible the results might be changed greatly. However, it is still a possibility that can be understood after running the experiments.", "paper_topic_and_main_contributions": "The paper focuses on first-person offense and vicarious offense in US-based political conversations. The paper then investigates 3 research questions based on the alignment between machine-machine, human-human, and machine-human moderators to identify first-person and vicarious offense present in the conversations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None in my opinion."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think that the illustrations and tables should be better placed. The figures that are mentioned on one page are located after 2 pages, which sometimes makes it difficult to follow them. It is always better to place the tables and figures before you mention them.\n\nTypos: Figure ?? in section 5.3\nA.4, B.1 only headings are mentioned"}}, {"review_id": "2nnMLbf8jr", "reviewer": "Reviewer_Vd66", "summary": "", "questions": "1. I was wondering whether it would be useful to add an example case of vicarious offense in the introduction to better showcase this concept. Even though the definition is adequate, this is a new concept and would therefore benefit from a readily available example. Maybe you could repurpose the Fig. 1 caption to specifically point out what vicarious offense is.\n\n2. Can this study have a temporal element as well? It would be interesting to see how the findings evolve throughout the years.\n\n3. Is there a reason you did not use the Perspective API? You did use TCC (model 8 in your list), but I am wondering why the Perspective API was avoided. While I am not advocating for its use, a lot of contemporary work uses it so I found this decision interesting. If there is an explicit reason behind this, including it in the paper would be informative.\n\n4. For RQ2, I would have liked to have seen self-alignment as a baseline: what is the alignment within Democrats (Dem^Dem)? How well are Democrats at predicting what other Democrats find offensive? This would be interesting as a comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that all three reviewers found our work exciting and methodologically sound. We thank them for their constructive criticism, improvement suggestions, and pointers to relevant literature. We have already addressed most of them in the rebuttal. We will address all of them in the final version. \n\nWe thank the reviewer for their feedback on our work. We will incorporate the suggestions and references to the final version of the paper. As for the questions raised; \n\n>Q1: I was wondering whether it would be useful to add an example case of vicarious offense in the introduction to better showcase this concept. Even though the definition is adequate, this is a new concept and would therefore benefit from a readily available example. Maybe you could repurpose the Fig. 1 caption to specifically point out what vicarious offense is.\n\nQuestion 1, on adding the vicarious offense to the image caption. We will rephrase the caption to include details about the vicarious offense for the readers. \n\n>Q2: Can this study have a temporal element as well? It would be interesting to see how the findings evolve throughout the years.\n\nQuestion 2, we analyzed the overall toxicity of the dataset in a temporal dimension (appendix Figure 11). But due to space constraints, we had to move the analysis to the Appendix. \n\n>Q3: Is there a reason you did not use the Perspective API? You did use TCC (model 8 in your list), but I am wondering why the Perspective API was avoided. While I am not advocating for its use, a lot of contemporary work uses it so I found this decision interesting. If there is an explicit reason behind this, including it in the paper would be informative.\n\nWe did analyze with the ChatGPT and the Perspective API on the overall alignment of the offensiveness against the current set of models and human annotators (see the table). Looking at the results (Cohen’s Kappa, Figures 1A and 1B), the humans agree more with the ML and ChatGPT label and less with the Perspective API. This is an observation that is common looking at F1 and recall (Figures 2 and Figure 3). We will include these findings in the final version of the paper.\n\nThese models are from the GeneralOffense models and TB is toxic BERT.    \n-   **ml** - this is the majority label from all the models (GO and TB)\n-   **gpt_label** - this is the label when ChatGPT 3.5 is used for classification (prompt is included in details). \"*You are an emotionally intelligent assistant. Classify the intent of the user's text to be offensive/disrespectful to someone with ONLY ONE OF THE FOLLOWING CHOICES: 1. Very offensive, 2. Extremely offensive, 3. Slightly offensive, 4. Moderately offensive, or 5. Not at all offensive.*\". This prompt is similar to the one we've used for analysis of the vicarious offense.  \n- **papi_label** - this label is from Perspective API Scores. Following the guidelines from the [documents](https://developers.perspectiveapi.com/s/about-the-api-score?language=en_US), we used the cut off as 0.7.\n-   **HumanMaj** - this is the majority opinion of the humans.\n-   **DEM**,**REP**, and **IND** - majority opinion of all the Democrat, Republican, and Independent leaning human annotators respectively.\n\n**Cohen's Kappa**\n|   |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.30 |  0.33 |       0.51 | 0.14 |        0.15 |         0.08 |\n| REP        |  0.30 |  1.00 |  0.42 |       0.63 | 0.15 |        0.15 |         0.07 |\n| IND        |  0.33 |  0.42 |  1.00 |       0.71 | 0.09 |        0.10 |         0.04 |\n| HumanMaj   |  0.51 |  0.63 |  0.71 |       1.00 | 0.11 |        0.11 |         0.05 |\n| ml         |  0.14 |  0.15 |  0.09 |       0.11 | 1.00 |        0.60 |         0.39 |\n| gpt_label  |  0.15 |  0.15 |  0.10 |       0.11 | 0.60 |        1.00 |         0.27 |\n| papi_label |  0.08 |  0.07 |  0.04 |       0.05 | 0.39 |        0.27 |         1.00 |\n\n**Recall**\n|  |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.63 |  0.64 |       0.70 | 0.61 |        0.60 |         0.60 |\n| REP        |  0.67 |  1.00 |  0.69 |       0.77 | 0.64 |        0.62 |         0.61 |\n| IND        |  0.72 |  0.74 |  1.00 |       0.83 | 0.61 |        0.60 |         0.58 |\n| HumanMaj   |  0.87 |  0.90 |  0.88 |       1.00 | 0.65 |        0.62 |         0.61 |\n| ml         |  0.56 |  0.56 |  0.54 |       0.54 | 1.00 |        0.78 |         0.74 |\n| gpt_label  |  0.57 |  0.56 |  0.54 |       0.54 | 0.83 |        1.00 |         0.70 |\n| papi_label |  0.56 |  0.55 |  0.53 |       0.54 | 0.75 |        0.68 |         1.00 |\n\n**F1-Scores**\n\n|  |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.65 |  0.66 |       0.75 | 0.55 |        0.57 |         0.40 |\n| REP        |  0.65 |  1.00 |  0.71 |       0.82 | 0.54 |        0.56 |         0.37 |\n| IND        |  0.66 |  0.71 |  1.00 |       0.86 | 0.50 |        0.52 |         0.34 |\n| HumanMaj   |  0.75 |  0.82 |  0.86 |       1.00 | 0.50 |        0.52 |         0.33 |\n| ml         |  0.55 |  0.54 |  0.50 |       0.50 | 1.00 |        0.80 |         0.66 |\n| gpt_label  |  0.57 |  0.56 |  0.52 |       0.52 | 0.80 |        1.00 |         0.57 |\n| papi_label |  0.40 |  0.37 |  0.34 |       0.33 | 0.66 |        0.57 |         1.00 |\n\n>Q4: For RQ2, I would have liked to have seen self-alignment as a baseline: what is the alignment within Democrats (Dem^Dem)? How well are Democrats at predicting what other Democrats find offensive? This would be interesting as a comparison.\n\nQuestion 4, it is an interesting question. We did not ask this in our study. But we will include it as a future consideration in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel study analyzing perceptions of offense from different political spectra, as well as alignment with machine learning models. This study is vital in current political discourse and discussion on hate speech.\n\n2. Description of crowdsourced annotators is thorough and gives a solid picture of the annotator pool.\n\n3. The list of machine moderators is extensive, leading to a thorough analysis.\n\n4. Findings are interesting and the experiments conducted robustly.", "reasons_to_reject": "Nothing significant. See questions for comments/suggestions for improvement.", "questions_to_authors": "1. I was wondering whether it would be useful to add an example case of vicarious offense in the introduction to better showcase this concept. Even though the definition is adequate, this is a new concept and would therefore benefit from a readily available example. Maybe you could repurpose the Fig. 1 caption to specifically point out what vicarious offense is.\n\n2. Can this study have a temporal element as well? It would be interesting to see how the findings evolve throughout the years.\n\n3. Is there a reason you did not use the Perspective API? You did use TCC (model 8 in your list), but I am wondering why the Perspective API was avoided. While I am not advocating for its use, a lot of contemporary work uses it so I found this decision interesting. If there is an explicit reason behind this, including it in the paper would be informative.\n\n4. For RQ2, I would have liked to have seen self-alignment as a baseline: what is the alignment within Democrats (Dem^Dem)? How well are Democrats at predicting what other Democrats find offensive? This would be interesting as a comparison.", "paper_topic_and_main_contributions": "This work operates in the domain of political discourse, performing a large-scale study on the perception of offense from across the political spectra as well as from machine moderators. Further, a dataset produced by their study is released, labeled additionally for \"vicarious offense\".", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "lines 193 and 210 and 232: Usually footnotes are added after punctuation for more compact text. Such as this: text.1\n\n"}, "extra_text": {"Missing_References": "There are two works that perform studies on offense in political discourse:\n\n1. Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection, Lara Grimminger, Roman Klinger, 2021. (work collects offensive tweets from democrats and republicans targeting the other community)\n\n2. Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments, Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, Hinrich Schuetze, 2022. (one of the domains examined is hate speech in political discourse, collecting data targeting politicians, state, civil rights advocates, etc.)\n\nWhile these two differ from this work since they only collect data for hate speech in political discourse and do not perform a noise audit, I find they are still pertinent (slightly contradicting lines 167-169)."}}, {"review_id": "StwZBvrzNp", "reviewer": "Reviewer_G1Gg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that all three reviewers found our work exciting and methodologically sound. We thank them for their constructive criticism, improvement suggestions, and pointers to relevant literature. We have already addressed most of them in the rebuttal. We will address all of them in the final version. \n\nWe appreciate the comments about our work. \n\n>Some presentation problems make the last two sections difficult to follow. I suggested improvements in the corresponding section from review.\n\nWe will include all the changes in the final version of the paper and work on the formatting to avoid any misaligned sections.\n\n>Since each machine moderator is trained on a different dataset, they are expected to have a low agreement. Training on a concatenation of datasets or evaluating more public APIs, such as Perspective API, would generate more informative noise audit results.\n\nAdding a machine moderator trained on the union of the datasets is an excellent suggestion. We will include this result in the final version. However, the datasets we use are well-known within the community and there is no documented understanding of how aligned they are on in-the-wild evaluation sets. Our study addresses that gap.   \n\nWe have included additional results with the Perspective API on the human annotated dataset. In the final version, we will include a noise audit with Perspective API and the model trained on the union of the datasets.  \n\n### Additional Results\n\nThese models are from the GeneralOffense models and TB is toxic BERT.    \n-   **ml** - this is the majority label from all the models (GO and TB)\n-   **gpt_label** - this is the label when ChatGPT 3.5 is used for classification (prompt is included in details). \"*You are an emotionally intelligent assistant. Classify the intent of the user's text to be offensive/disrespectful to someone with ONLY ONE OF THE FOLLOWING CHOICES: 1. Very offensive, 2. Extremely offensive, 3. Slightly offensive, 4. Moderately offensive, or 5. Not at all offensive.*\". This prompt is similar to the one we've used for analysis of the vicarious offense.  \n- **papi_label** - this label is from Perspective API Scores. Following the guidelines from the [documents](https://developers.perspectiveapi.com/s/about-the-api-score?language=en_US), we used the cut off as 0.7.\n-   **HumanMaj** - this is the majority opinion of the humans.\n-   **DEM**,**REP**, and **IND** - majority opinion of all the Democrat, Republican, and Independent leaning human annotators respectively.\n\n**Cohen's Kappa**\n|   |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.30 |  0.33 |       0.51 | 0.14 |        0.15 |         0.08 |\n| REP        |  0.30 |  1.00 |  0.42 |       0.63 | 0.15 |        0.15 |         0.07 |\n| IND        |  0.33 |  0.42 |  1.00 |       0.71 | 0.09 |        0.10 |         0.04 |\n| HumanMaj   |  0.51 |  0.63 |  0.71 |       1.00 | 0.11 |        0.11 |         0.05 |\n| ml         |  0.14 |  0.15 |  0.09 |       0.11 | 1.00 |        0.60 |         0.39 |\n| gpt_label  |  0.15 |  0.15 |  0.10 |       0.11 | 0.60 |        1.00 |         0.27 |\n| papi_label |  0.08 |  0.07 |  0.04 |       0.05 | 0.39 |        0.27 |         1.00 |\n\n**Recall**\n|  |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.63 |  0.64 |       0.70 | 0.61 |        0.60 |         0.60 |\n| REP        |  0.67 |  1.00 |  0.69 |       0.77 | 0.64 |        0.62 |         0.61 |\n| IND        |  0.72 |  0.74 |  1.00 |       0.83 | 0.61 |        0.60 |         0.58 |\n| HumanMaj   |  0.87 |  0.90 |  0.88 |       1.00 | 0.65 |        0.62 |         0.61 |\n| ml         |  0.56 |  0.56 |  0.54 |       0.54 | 1.00 |        0.78 |         0.74 |\n| gpt_label  |  0.57 |  0.56 |  0.54 |       0.54 | 0.83 |        1.00 |         0.70 |\n| papi_label |  0.56 |  0.55 |  0.53 |       0.54 | 0.75 |        0.68 |         1.00 |\n\n**F1-Scores**\n\n|  |   DEM |   REP |   IND |   HumanMaj |   ml |   gpt_label |   papi_label |\n|------------|-------|-------|-------|------------|------|-------------|--------------|\n| DEM        |  1.00 |  0.65 |  0.66 |       0.75 | 0.55 |        0.57 |         0.40 |\n| REP        |  0.65 |  1.00 |  0.71 |       0.82 | 0.54 |        0.56 |         0.37 |\n| IND        |  0.66 |  0.71 |  1.00 |       0.86 | 0.50 |        0.52 |         0.34 |\n| HumanMaj   |  0.75 |  0.82 |  0.86 |       1.00 | 0.50 |        0.52 |         0.33 |\n| ml         |  0.55 |  0.54 |  0.50 |       0.50 | 1.00 |        0.80 |         0.66 |\n| gpt_label  |  0.57 |  0.56 |  0.52 |       0.52 | 0.80 |        1.00 |         0.57 |\n| papi_label |  0.40 |  0.37 |  0.34 |       0.33 | 0.66 |        0.57 |         1.00 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The release of a novel dataset that can be valuable for further research regarding vicarious offense.\n- The annotation process is well explained.\n- Experimental evaluation is complete: comparing machine and human moderators, generating informative results, and discussing the impact of what was achieved.", "reasons_to_reject": "- Since each machine moderator is trained on a different dataset, they are expected to have a low agreement. Training on a concatenation of datasets or evaluating more public APIs, such as Perspective API, would generate more informative noise audit results.\n- Some presentation problems make the last two sections difficult to follow. I suggested improvements in the corresponding section from review.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work conducted a noise audit of nine offensive speech classifiers on a dataset of more than 92 million YouTube comments, revealing considerable variations in the results. Also, it annotated a dataset with views from people from different political parties regarding their perception of offense and also about vicarious offense, that is, to predict offense for others who do not share the same political belief. Finally, it analyzed human and machine moderators' agreement on what is offensive.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Please cite the Appendix subsections in the paper so we can refer easily to them.\n- Table 1 is not cited in the paper.\n- Line 381: The cited result (0.43) is extracted from the Appendix. Consider reorganizing so the result appears in the main paper.\n- Figure 3 caption is incorrect. A cell [i, j] is not referring to machine moderators' agreement since the figure is also showing human moderators' agreement.\n- Line 478: Figure reference is missing.\n- Figure 4 uses a different vicarious offense notation from what was previously explained in the paper (in lines 401-404).\n- The results using ChatGPT to predict the vicarious offense are in the Discussion and Conclusion section, which should be in the Results section.\n- In the Appendix, Figures 5, 6, and 7 are not cited.\n- In the Appendix, Section B.1 is empty."}}]

Author rebuttal:

Meta-review: The reviewers all thought that this paper covered an important topic from an interesting perspective, in a technically sound way. They also raised various questions and concerns throughout their reviews that would be worth addressing in a revised version of the paper. 

The main suggestion I want to highlight, that was shared across all reviews, was the choice of machine moderators. Reviewers noted some obvious omissions (e.g., Perspective), and suggested some natural explanations for why they might disagree (divergence in training data), that the authors do not appear to have mentioned. The sense I got from reviews was that this could be addressed via writing edits:
* why were moderators like Perspective not highlighted? (the authors could choose to include additional results from their responses, or they could explicitly provide a reason, e.g., black-box nature of that particular algorithm)
* what are features of the machine moderators, that might contribute to noise? here the authors could go beyond simply listing moderators in a paragraph of text; a table could help to explicitly contrast algorithm/training data differences.
Decision: Accept-Main