Analyzing Norm Violations in Live-Stream Chat
Authors: Jihyung Moon
Dong-Ho Lee
Hyundong Justin Cho
Woojeong Jin
Chan Young Park
Minwoo Kim
Jonathan May
Jay Pujara
Sungjoon Park
Venue: EMNLP 2023 Main (2023)
Keywords: Norm Violation
Toxicity Detection
Live Streaming

Abstract: Toxic language, such as hate speech, can deter users from participating in online communities and enjoying popular platforms.
Previous approaches to detecting toxic language and norm violations have been primarily concerned with conversations from online forums and social media, such as Reddit and Twitter. 
These approaches are less effective when applied to conversations on live-streaming platforms, such as Twitch and YouTube Live, as each comment is only visible for a limited time and lacks a thread structure that establishes its relationship with other comments. 
In this work, we share the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms.
We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. 
We articulate several facets of live-stream data that differ from other forums, and demonstrate that existing models perform poorly in this setting. 
By conducting a user study, we identify the informational context humans use in live-stream moderation, and train models leveraging context to identify norm violations. Our results show that appropriate contextual information can boost moderation performance by 35\%.

Analyzing Norm Violations in Live-Stream Chat
Jihyung Moon'; Dong-Ho Lee Hyundong Cho², Woojeong Jin', Chan Young Park Minwoo Kim*, Jonathan May?, Jay Pujara', Sungjoon Park' 'SoftlyAI Research, ?University of Southern California,
CCrneeiee Mellon University, "Selectstar
(jihyung, moon, dangho. Lue,sunnooon wttppoonn..rrkksso....oo ai {jcho,jonmay, Jommay.jpujarajjsstt
{dongho. lee, woojeong. jinj@usc. {chanyoun]ecs, cmu. edu (mkkim)@selectttar..c
Abstract
Asynchronous
Synchronous
A:nn nn 02:23
A:D0a = 02:23:10
Toxic language, such as hate speech, can deter users from participating in online communities and enjoying popular platforms. Previous ap- proaches to detecting toxic language and norm violations have been primarily concerned with conversations from online forums and social media, such as Reddit and Twitter. These ap- proaches are less effective when applied to con- versations on live-streaming platforms, such as Twitch and YouTube Live, as each comment is only visible for limited time and lacks thread structure that establishes its relationship with other comments. In this work, we share the first NLP study dedicated to detecting norm violations conversations on streamin platforms. We define norm violation categories in live-stream chats and annotate 4,583 mod- erated comments from Twitch. We articulate several facets of live-stream data that differ from other forums, and demonstrate that ex- isting models perform poorly in this setting. By conducting user study, we identify the in- formational context humans use in live-stream moderation, and train models leveraging con- text to identify norm violations. Our results show that appropriate contextual information can boost moderation performance by 35%.
B: 11:07
Bo 02:23:22
C
18:23
C:oo 02:23:57
Bann 02:24:03
Train
?
Perspective
OpenAI
Delphi
Test
Figure 1: A Motivating Example. Chat in the syn- chronous domain has different characteristics than those in the asynchronous domain: (1) the temporal gap be- tween chats and message length are much smaller; and (2) relationships between chats are less clearly defined. Such differences make chats in the synchronous domain more difficult to be moderated by existing approaches.
et al., 7017; Founta et al., 20 Basile et al., 2019; ElSherief et al., 2021), Reddit (Datta and Adar, 2019; Kumar et al., 2018; Park et al., 2021), Stack- overflow (Cheriyan et al., 2017) and Github (Miller et al., 2022), efforts that extend them to live stream- ing platforms have been absent. In this paper, we study unique characteristies of comments in live- streaming services and develop new datasets and models for appropriately using contextual informa- tion to automatically moderate toxic content and norm violations.
1 Introduction
Conversations in online communities studied in previous work asynchronous: utterances are grouped into threads that structurally establish con- versational context, allowing users to respond prior utterances without time constraints. The lack of time constraints allows users to formulate longer and better thought-out responses and more easily reference prior context.
Interactive live strcaming services such as Twitch and YouTube Live 3 have emerged as one of the most popular and widely-used social platforms. Un- fortunately, streamers on these platforms struggle with increasing volume of toxic comments and norm-violan noom-violating + While there has been ex- tensive research on mitigating similar problems for online conversations across various platforms such as Twitter (Waseem and Hovy, 2016; Davidson
On the other hand, conversations live stream- ing platforms are synchronous, i.e. in real-time, utterances are presented in temporal order without thread-like structure. Context is mostly estab- lished by consecutive utterances (Li et al., 2021). The transient nature of live-stream utterances en-
Authors contributed equally.
'ntups:/gittub..oo/////
httpss//www.twitcht
httt::://www.youuube.
https///swwyyyiiiia.//////cocm/m Guideliis

---

courages fast responses, and encourages producing multiple short comments that may be more prone to typos (70% of comments are made up of words). Figure shows an illustration of the con- trasting temporal and length patterns between the asynchronous and synchronous platforms.
Twitch API IRC
Moderation (Ban, Timeout, Delete)
Top 200 Streamers
Annotators
Stage
Moderated Message
Gather chall rules
Stage 2
Moderattd Mensage
Cht Context
Owing to these different characteristics, we find that previous approaches for detecting norm viola- tions are ineffective for live-streaming platforms. To address this limitation, we present the first NLP study of detecting norm violations in live-stream chats. We first establish norms of interest by col- lecting 329 rules from Twitch streamers' channels and define 15 different fine-grained norm cate- gories through an iterative coding process. Next, we collect 4,583 moderated chats and their cor- responding context from Twitch live streams and annotate them with these norm categories ($2.1- $2.3). With our data, we explore the following research questions: (1) How are norm violations in live-stream chats, i.e. synchronous conversa- tions, different from those in previous social media datasets, i.e. asynchronous conversations?: (2) Are existing norm violation or toxicity detection mod- els robust to the distributional shift between the asynchronous and synchronous platforms? 1S $3.3); and (3) Which features (e.g., context and do- main knowledge) are important for detecting norm violation in synchronous conversations? ($3.2)
Stage
Chat/ Video Context
Moderated Messagee
Knowlodge Statement
Multi-clase Amnotation
CHAT RULES
Examples
Define Norms
Incivility
Map rules
Dicrimination
Raciem
Data Construction. Norms are manually defined based on the chat rules of the top 200 streamers. and annotators annotate the violated norm moderated event by three stages.
Figure 2:
data collection procedure is illustrated in Figure 2. We first select 200 top Twitch streamers and collect moderated comments from their streamed sessions ($2. I). To understand why these chats are moder- ated, we collect chat rules from these streamers and aggregate them define coarse and fine- fine-grained norm categories ($2.2). We design three-step annotation process to determine the impact of the chat history. video context, and external knowledge on labeling decisions ($2.3). Lastly, we present analysis of the collected data ($2.4).
2.1 Data Collection
From our explorations, we discover that (1) live- stream chats have unique characteristics and norm violating behavior that diverges from those in pre- vious toxicity and norm-vion norm-violation literature; (2) existing models for moderation perform poorly on detecting norm violations in live-stream chats; and (3) additional information, such chat and video context, are crucial features for identifying norm violations in live-stream chats. We show that incor- porating such information increases agreement for categorizing moderated content and that selecting temporally proximal chat context is crucial for enhancing the performance of norm vio- lation detection models in live-stream chats.
We collected data using the Twitch API and IRC from the streamers with videos that are available for download among the top 200 Twitch streamers as of June 20227. We specifically looked for com- ments that triggered a moderation event during a live stream (e.g. user ban, user timeout), and col- lected the moderated comment and the correspond- ing video and chat logs up to two minutes prior to the moderation event. Logs of moderated events from August 22, 2022 to September 3, 2022 were collected. We excluded comments that were mod- erated within less than second of being posted, as they are likely have been moderated by bots rather than humans.
2 Norm Vio-RT
2.2 Norm Categorization
To investigate norm-violations in live-stream chat, we first collect Norm Violations in Real-Time Con- versations (NormVine Vio-RT), which contains 4,583 norm-violating comments on Twitch that were mod- crated by channel moderators." An overview of our
Twitch streamers can set their own rules for their channels. and these channel-specific rules are es- sential for understanding why comments were mod- erated. We first collect 329 rules from the top
whtps://github.coom http://sshtubb.oomm//iichhttttto hanps///wwetttuteeeccocmmmmmmmee
Plcase contact the authors for the anonymized study data.

---

Coarse
Fine-grained
Target
Rule Examples
Discrimination
Diserimination
No racism, sexism or hamophobin.
Broadeister OIB
No HIB towards broadcaster
HIB
(Horassment, Intimidation, Bullying)
HIB
No HIB towards viewers, moderators, etc.
0OB
breadcasters, politicians, etc.
Please personal questions about me.
Privacy
Doxing
Do not share any personal information about yoursell or others.
NSFW
NSPW content (egg. Inappropriate
arts).
Sell-desituctive
No talk of suicide.
Inuppeopriate Contents
Illegal
No drug discussion of any kind.
Spoiler
Do not give game spoilers.
Controversial Topic
No drama, politics or religion.
Off Topii
Begging
No hepging for subscriptions money.
Excessive & Repeiitive Advertisements
No walls of lext.
Spam
No self promotion unless authorized.
Backscaing & Tall order
Don't tell me what to do. Don't ask for mod. down on other streamers. English only.
Meta-Rules
(Live streaming specific)
Mentioning broddnssters
Specific language only
Incivility (Miscellancous)
Incivility
Be nice, Re civil
Table I:
Live streaming norms. We map rules from top 200 Twitch streamers' channels to coarse and fine- fine-grained level norms. Some rules specify targets (OOB: Others Outside Broadcast, OIB: Others Broadcast).
200 Twitch streamers' channels. Next, following Fiesler et al. (2018), we take an iterative coding pro- cess such that the authors of this paper individually code for rule types with certain categories, come together to determine differences and then repeat the coding process individually. With this process, we aggregated similar rules into 15 different fine- grained level norm categories (e.g., controversial topics, begging) and cluster multiple fine-grained categories into 8 different coarse coarse-grained norm cat- egories (e.g., off-topic). To better understand the targets offensive comments in the HIB (Harass- ment, Intimidation, Bullying) class, we added an additional dimension to consider whether the tar- get is the broadcaster (streamer), participants in the channel (c.g., moderators and viewers), or someone not directly involved in the broadcast. We asked an- notators to assign "Incivility" to cases where anno- tators do not believe that a specific pre-defined rule type has been violated although moderated. Exam- ples of "Incivility" are provided in Appendix A.4. Table shows the resulting norm categories and corresponding fine-grained norms with examples.
Knowledge
Template
Platform Streamer
<span> [emoji, text] that <explanation>
<streamee>> <explanation>
Table knowledge statement template.
eration event (i.e. moderated comment) with the rule types it violates. To measure the importance of context in determining types of norm violations, annotators were asked to provide labels for three different scenarios with varying amounts of con- text: (I) Stage 1: annotate based on only the user's last message before the moderation event (single utterance); (2) Stage 2: annotate based on chat logs up to two minutes prior to the moderation (+chat context); (3) Stage 3: annotate based on chat logs and their corresponding video clip of the same duration (+video context). Since rules are not mutually exclusive (e.g., message can violate both discrimination & harassment), they are allowed to choose multiple rule types there are more than one violated rule at each stage. All the annotations are done with our internal annotation user interface (See Appendix A.1), To determine the final label for each moderated event, we aggregate the labels of annotators using a majority vote with heuristic rules (See Appendix A.2).
2.3 Violated Norm Type Annotation
We recruited three annotators who are fluent in En- glish and spend at least 10 hours a week on live streaming platforms to ensure that annotators un- derstood live streaming content and conventions. Their fluency verified through several rounds of pilot annotation work. Internal auditors continu- ously conducted intermittent audits ensure that annotators fully understood the guidelines.
Lastly, to examine how much external knowl- edge matters in understanding comments on live streaming platforms, we asked annotators to (1) in- dicate whether external knowledge is necessary to understand why a comment triggered moderation event and if so (2) describe what that knowledge is. We focus on two types of external knowledge: platform- and streamer-specifit knowledge includes the implicit meaning of partic-
Slatform-speciin
Annotators were asked to annotate each mod-

---

# Violates
Coarse
Fine-grained
Rules
stage
stuge
stage
Discrimination
Discrimination
13.98% (46)
2.34% (104)
2.25% (101)
2.34% (105)
HIB
HIB
22.49% (74)
21.33% (947)
26.55% (1,190)
27.80% (1,246)
Privacy
Doxing
0.60% (2)
0.34% (15)
0.36% (16)
0.36% (16)
Spoiler
0.60% (2)
0.02% (1)
0.02% (1)
0.02% (1)
NSFW
1.82% (6)
0.86% (38)
0.85% (38)
0.85% (38)
Inpppropriate Contents
Self-destructive
1.21% (4)
0.32% (14)
0.29% (13)
0.29% (13)
Illegal
0.30% (1)
0.16% (7)
0.07% (3)
0.07% (3)
Controversial Topic Begging
5,47% (18)
0.59% (26)
0.85% (38)
0.83% (37)
Topii
1.51% (5)
1.44% (64)
1,36% (61)
1.36% (61)
Excessive & Repetitive Advertisements
11.24% (37) 11.24% (37)
17.59% (781)
21.64% (970)
21 .2%% (960) 4.42% (198)
Spam
4.64% (206)
4.40% (197)
Mentioning other streamers
14.28% (47)
0.72%, (32)
10.62%; (476)
10.58% (474)
Meta-Rules (Live streaming specific)
Backseating Tall onder
5.16% (17)
3.45% (153)
(166)
3.77% (169)
Specific language only
10.03% (33)
0.97% (43)
6.94% (311)
6.94% (311)
Incivility Non- Soa-Identifialle
12.30% (546)
11.57%: (519)
11.51% (516)
Incivility (Miscellancous)
32.93% (1,462)
8,52% (382)
7.45% (334)
Total
329
4,439
4,482
4,482
Table Data Statisties. # of rules indicates number of streamers specifying the norm in their channels and # of violates indicates actual number of messages that violate corresponding norms.
ular emojis, emotes, and slang that are commonly used on Twitch. Streamer-specific knowledge in- volves the streamer's personal background and pre- vious streaming sessions. As shown in Table we provide templates for each type that annotators can casily fill out (More details in Appendix A.3).
% Agreement
Stage
Stage
Stage 3
Exact Match 39.71% (1.820)
41.10% (1.884)
40.67% (1.864)
Partial Match 11% (2,480)
75.03% (3,439)
75 15.211 (3,447)
Majority Vote 96.85% (4,439)
97.79% (4,482)
97.79% (4,482)
Table 4: Inter-annotator Agreement. Presents the per- centage of moderated events for 4.583 events while the number in parentheses indicates the number of events.
2.4 Data Statistics and Analysis
could not identify any reason for the moderation. We found that many cases are not identifiable, as shown in Table 3. It is natural that many cases are non-identifiable in stage 1, as annotators are only given the moderated comment and no context. However, the 7.45% non-identifiable cases that main even after stage 3 could be false positives, they could be cases where the moderation event oc- curred more than two minutes after problematic comment was made.
General Observations We identified three char- acteristics that distinguish real-time live-streaming chat from other domains. First, the majority of comments are very short; 70% of comments are made up of < 1 words. Additionally, they are often very noisy due to the real-time nature of communi- cation, which leads to high number of typos, ab- breviations, acronyms, and slang in the comments. Lastly, some comments use unusual visual devices such as ASCII art and "all caps". to make them more noticeable. This is because each comment is visible only for short time in popular streams (on average, there are around 316 chats per minute for the streamers in our data). The chat window in live streaming platforms can only display limited number of comments, so viewers are incentivized to use visual devices to draw the streamer's atten- tion in these fast-paced conditions.
Context improves inter-annotator agreement. Interestingly, providing context helps mitigate an- notator bias, as shown by the increase in inter- annotator agreement from stage to stages 2 and in Table 4. Here, the exact match determines whether all three annotators have exactly the same rules; partial match determines whether there is at least one intersection rule between three anno- tators; and majority vote chooses the rule types that were selected by at least two people. Also, non-identifiable and disagreement cases drop sig- nificantly when the contexts are given as shown in Table 3. Similarly for determining rule types, context also helps annotators identify targets for HIB and reduces inconsistencies between annota-
False positives in data. We find that the "Inci- vility" case contains many false positives, they include cases that seem to have been moderated for no particular reason. We asked annotators to put all miscellancous things into the "Incivility" category, and also to mark as "Incivility" if they

---

tors. Our observation emphasizes the importance of context in synchronous communication and dif- fers from previous findings that context-sensitive toxic content rare in asynchronous communica- tion (Pavlopoulos et al., 2020; Xenos et al., 2021). Analysis details are in Appendix A.2.
Model
Precision
Recall
F1
ToxiGen
0.31
0.91
0.46
Perspective API
0.39
0.95
0.56
OpenAl moderation
0.11
0.94
0.20
OpenAl content filter
0.55
0.86
0.67
Table 5: Performance (Binary F1) of toxicity detec- tion models on HIB and Discrimination data. Binary Fl refers the results for the 'toxic' class.
External knowledge helps annotations. To in- vestigate the impact of external knowledge on anno- tators' labeling decisions, we compare annotations made with and without external knowledge pro- vided. For examples with knowledge statements, we expect to see differences in annotation if ex- ternal knowledge is necessary to comprehend why they were moderated. Statistics show that 296 ex- amples (6.6%) require knowledge, with 183 exam- ples requiring streamer knowledge and 187 exam- ples requiring platform knowledge. Note that there are some examples require both. Details of statis- tics and examples are presented in Appendix A.3.
to the distributional shift from asynchronous con- versations to synchronous conversations and vice versa, and identify how important the context or domain knowledge are for detecting toxicity and norm violation in synchronous conversations.
3.1 Performance of Existing Frameworks.
To examine the difference in toxicity detection be- tween asynchronous and synchronous communi- cation, we investigate whether existing toxicity detection models are effective for synchronous communication. We evaluate the performance of four existing tools on Norm Vio-RT: Google's Per- spective API (Lees et al., 2022) OpenAI con- tent filter?, OpenAl moderation (Markov al., 2022) 10 and RoBERTa-large model fine-tuned on machine-generated toxicity dataset called Toxi- Gen (Hartvigsen et al., 2022). We only use exam- ples from the discrimination and HIB categories in NormVic Vio-RT, as they are most similar to the label space that the existing models are trained for (e.g.. hateful content, sexual content, violence, self-harm, and harassment). Categories are determined based on the stage consolidated labels, we do not provide any context the model. Additionally, we select an equal number of random chats from the collected stream to construct negative examples. To ensure the quality of negative examples, we only select chats that are not within two minutes prior to any moderation event as they less likely to con- tain norm violating chats. We also only select chats from users who have never been moderated in our data, To obtain the predictions from the models, we check whether toxicity score greater than or equal to 0.5 for Perspective API, and for OpenAI, check the value of the "flagged" field which indi- cates whether OpenAl's content policy is violated. We use binary classification outputs for ToxiGen. Table shows the results obtained from 2, 102 ex- amples with 1,051 examples each for toxic and non-
Norm Category Distribution Table 3 shows the norm category distribution of streamers' rules and the moderated comments. While the categories are not directly comparable to the ones defined in Nor- m Vio for Reddit (Park et al., 2021), we identified few similar patterns. First, in both domains, Ha- rassment and Incivility (i.e., Discrimination, HIB, Incivility) take up significant portion of the en- tire set of norm violations. Also, the two domains show a similar pattern where rules for Off-Topic, Inappropriate Contents, and Privacy exist but are relatively less enforced in practice. However, we also found that the two domains differ in various ways. For example, Spam and Meta-Rules cover significantly higher portions of both rules and mod- erated comments on Twitch than on Reddit. On the other hand, there are fewer rules about content on Twitch, which implies that streamers are less concerned about the content of the comments than Reddit community moderators. As our data shows that norm-vinn corm-violating comments on live chats exhibit distinctive rules and patterns, it suggests that the existing norm violation detection systems may not perform well without domain adaptation account for these distributional differences. We examine this hypothesis empirically in the following section and suggest appropriate modeling adjustments to better detect toxicity for real-time comments.
3 Toxicity Detection in Live-stream Chat this section, we first check whether norm vi- olation and toxicity detection models are robust
hnups://perspecciivap.ccm https///hwapaaccccooooopopoooooe

---

Context
All
Discrimination
HIB
Privacy
Inapt Contents
OIl Topic
Spam
Meta-Rules
Incivility
0.70 3300
0.11 1wm
0.52 aa
0.05 1we
0.12 jaa
1430
0.63 swo
0.68 1o00
0.28 poe
Single comtext
0.78 -200
0.07 twe
0.50 =ax:
0.03
0.18
0.05 =43
0.67
0.58
0.28
Multi-ue context (event)
0.25 _304
1um
0.44 1ase
1owm
0.14 1a12
143
0.60 1ooe
0.17 _00
Multi-user contest (utterance)
0.91 -000
0.04 100
0.61 1,35
0.00 1000
0.99 1o0s
0.10
1031
0.66 jem
0.68 201
0.24 2013
Multi user context (first)
0.95 -2:0
0.05 1uot
0.61 =3::
0.01
0.11
0.08
0.70
0.62
0.45
Broadeast category
0.77 _330
0,13 100m
0.48
1,38
0.02
1oon
0.13 1oss
0.12
1435
10m1
0.64 10od
0.30 _00
Rule text
0.75 _,:0
0.05 1o08
0.11 12:07
0.00
1000
0.12 10005
0.29
0.58
1om
0.38
2000
0.15 .00
Table 6: Performance on Norm Classification, macro Fl score for each coarse-level norm category. "All" refers to binary classification between moderated and unmoderated messages without considering norm category, Best models are bold and second best ones underlined. Scores average of 3 runs (3 random seeds).
toxic messages. The results illustrate that while existing models do not frequently produce false positives (high recall), they perform poorly in de- tecting toxic messages found in synchronous chats, with a detection rate of only around 55% at best (low precision).
Moderated User
Single-user context
Single Utterance
Moderation
Event
(first)
(stteenance)
Multi-user context (2 mins)
Figure 3: Multi-user context is chat logs that occurred up to two minutes before the moderation event while single-user context chat logs of moderated user in multi-user context and single utterance is the moder- ated user's last message before the moderation event.
3.2 Norm Classification in Norm Vio-RT.
To understand the model's ability to detect norm violations and how additional information can af- fect detection, we train binary classification models for each category with different types of context including conversation history, broadcast category, and rulc description following Park et al. (2021).
the moderation event. In all the Multi-user con- texts, we use N = 5; (5) Broadcast category is the category that streamers have chosen for their broadcast. It usually the title of game or set to "just chatting"; and (6) Rule text is a representative rule example shown in Table 1. The rule text is only used for training examples because is not possible to know which rule was violated for un- seen examples and we use randomly selected rule text for unmoderated negative examples in training examples. All contexts are appended to the input text (single ufterance) with special token ([SEP)) added between the input text and the context. Chat logs for multi-user context and single-user context are placed sequentially with spaces between chats. Training details and data statistics are presented in Appendix B.
Experimental Setup. For cach coarse-level cat- egory, we train RoBERTa-baad model with bi- nary cross entropy loss to determine whether the message violating the certain norm or not. Fol- lowing Park et al. (2021), we perform an 80-10-10 train/dev/test random split of moderated messages and add the same number of unmoderated messages in the same split. Next, for each binary classifica- tion, we consider the target category label as 1 and others as 0 and construct a balanced training data set. Appendix B (See Table 12). Here, the labels are based on stage 3.
To examine how context affects model perfor- mance, we experiment with four model variants with different input context: (1) Single user con- text is only the chat logs of the moderated user that took place up to two minutes before the mod- eration event; (2) Multi-user context (event) is N messages that directly precede the moderation event, regardless of whether it belongs to the mod- erated user; (3) Multi-user context (utterance) is N messages that directly precedes the single utter- ance, which is the moderated user's last message before the moderation event (i.e., chat 3 in Fig- ure 3).; (4) Multi-user context (first) is the first N messages of the collected two-minute chat logs. The intuition for this selection is that the modera- tion event may have taken place much earlier than
Experimental Results. Table 6 presents perfor- mance of norm classification for coarse-level norm categories. "All" refers to binary moderation de- tection, whether the message is moderated or not, and not the specific norm type. First, we can see that additional context improves the performance of "Al1," but context does not consistently improve the performance of category-specife norm classi- fiers. For example, context reduces performance for categories where the issues are usually lim- ited the utterance itself (e.g., discrimination and privacy). In contrast, categories that rely on the relationships between utterances, such as HIB and incivility, show improved performance with con-

---

1.00
0.955
0.95
stage
0.90
stage
0.85
Score
0.8
0.80
0.781
0.75
0.756
0.748
0.743
E2 0.7
0.703
0
0.65
(event)
multi (utterance) multi user (first)
0.60
context
single-user
multi-user (first)
Figure 4: Performance (F1 Score) of moderation de- tection by different ground truth label for each context.
3
5 10 15
20
25
Number context messages
Figure 5: Performance (F1 Score) frend of moderation detection with varying context length.
text. Secondly, multi-user context performs quite well compared to the other contexts, indicating that more global context that includes utterances from other users helps determine the toxicity of target utterances. Lastly, the strong performance of Multi- user context (first) suggests that earlier messages in the two-minute window are more important, mean- ing that the temporal distance between the mod- eration event and the actual offending utterance may be substantial in many cases. Thus, our re- sults encourage future efforts on developing more sophisticated approach for context selection.
to the moderated user's message helps with mod- eration performance the most (See ullerance and first). However, increasing the number of messages that directly precede the moderation event actually lowers moderation performance (See event). may be that most of this context serves as noise.
3.3 Distribution Shift in Norm Classification.
Existing tools often focus on identifying harmful speech, but Norm (Park al., 2021) also con- siders wider range of norm- corm-violating comments on Reddit, similar to Norm Vio-RT but in differ- ent domain. We compare Norm Vio and Norm Vio- RT by evaluating the performance of a model fine- tuned on Norm Vio with Norm Vio-RT, and vice versa, to examine the impact of distribution shift between these domains. We choose six coarse-level categories that overlap between the two, as shown in Table 7. To measure with-context performance, the previous comment history for Reddit and multi-user context (utterance) for Twitch simulate the most similar setup in both domains. Overall, experimental results show a pronounced distribution shift between Reddit (asynchronous) and Twitch (synchronous). Interestingly, models trained on Twitch are able to generalize better than models trained Reddit despite having 6x less training data. Specifically, models trained using the out-of-domain Twitch+context data perform comparably on the Reddit test those trained using in-domain Reddit+context data.
Availability of Context. To compare human de- cisions with those of our models, we conduct ex- periments varying the context available to anno- tators and models. For example, we expect mod- els trained with only single utterances to perform best when using stage (utterance only) labels ground-truth labels since humans also not given any context stage Indeed, in Figure using the stage labels as the ground truth labels yields the best performance for model trained without any context, while using the stage 2 (con- text) labels as the ground truth labels shows the best performance for model trained with previous chat history. Since our experiments only handle text inputs, it is not surprising that using stage (video) labels ground-truth labels yields worse performance than using stage 2 labels. However, interestingly, the gap is not large, which indicates that gains from a multi-modal model that incor- porates information from the video may be small and that single modality (text-only) models can be sufficient for the majority of moderation instances.
Related Work
Toxicity Detection Most toxic language data con- sists of explicit hate speech consisting of hate lexi- cons (Waseem and Hovy, 2016: Davidson et al., 2017: Founta et al., 2018; Basile et al., 2019), group identifiers (Warner and Hirschberg, 2012; Kennedy et al., 2020), and hateful phrase (Silva
Context Size. To understand how the amount of available context affects moderation performance, we compare the multi-user context configurations with various number of messages from one to 25. Figure 5 demonstrates that 15 to 20 messages prior

---

Category
Without Context
With Context
Reddit (Nomnvio)
Twitch (Normvio-RI)
R
T-R
T
R-T
R
T-R
T
R-T
ALL
ALL
0.99
0.84 tuss
0.70 2010
0.67 sute
0.99 1::00
0.98 sam
0.91
0.67 sew
Incivility
Incivility
0.67 1:z
0.16 taw
0.28 1014
0.09 2016
0.74 tow0
0.56 +0.14
0.24
10.12
0.09 201e
Harassmeut
HIB, Privacy
0.34 1am
0.19 Jam
0.51 1am
0.27 10m
0.41 10t0
0.20 10m
0.62 1012
0.26 :010
Spam
Spam
0.47 10m
0.22 1010
0.63 1am
0.28 jam
0.53 1am
0.27 1am
0.66 1001
0.28 1am
Of' Topic
Off Topic
0.25
0.12 1am
0.07 1030
0.00
0.28
0.12 1010
0.10
10M
0.00
Hate Speech
Discrimination
0.17 201e
1000
1010
0.02 2are
0.19 1010
0.06 1000
0.04
10te
0.02 1010
Content
Inapt. Contents
0.30
0.088 amm
0.12
0.00
0.37
0.05
2010
0.09
10m
0.00
Table Performance on distribution shift between norm violations in Reddit and Twitch. Macro Fl scores for each overlapped norm category. Scores are average of 3 runs (3 random seeds).
et al., 2016) in asynchronous communication (e.g.. Twitter, Reddit). However, models trained on such data may have spurious correlations that result in many false positives (e.g., group identifiers) (Sap et al., 2019; Kennedy et al., 2020; Hartvigsen et al., 2022: Lee et al., 2022). To reduce such bias, im- plicit hate speech, toxic language use without any explicit hateful words or phrases, has been ex- plored (Kennedy et al., 2018; ElSherief et al., 2021; Hartvigsen et al., 2022).
microaggressions that subtly expresses a prejudice attitude toward certain groups (Breitfeller et al., 2019; Han and Tsvetkov, 2020); and (5) Subjec- tivity measures annotation bias (Sap et al., 2022) and manage annotator subjectivity involved in la- beling various types of toxicity, which arises from differences in social and cultural backgrounds (Da- vani al., 2022). In this paper, we analyze the toxicity of synchronous conversations terms of the aforementioned dimensions by identifying ex- planation of toxicity as a form of norm categories (explainability), finding targets of HIB words (tar- get identificationn, leveraging context for both an- notation and modeling (context sensitivity), asking annotators for implied knowledge statement (impli- cation), and examining how human decisions align with machine decisions under different amounts of information (subjectivity).
Beyond Binary Toxicity Detection Treating tox- icity detection as a binary task may not be enough understand nuanced intents and people's ' reac- tions to toxic language use (Jurgens et al., 2019; Rossini, 2022). To holistically analyze toxicity, recent works take a more fine-grained and multidi- mensional upproach: (1) Explainability explains why a particular chat is toxic with highlighted ratio- nales (Mathew et al., 2021), free-text annotations of implied stereotype (Sap al., 2020; ElSherief et al.. 2021; Sridhar and Yang. 2022), or pre-defined vio- lation norms (Chandrasekharan et al., 2018; 8; Park et al., 2021). These explanations can be used not only to improve the performance of the toxicity detection model, but also to train models that gen- erate explanations; (2) Target identification finds the targets of toxic speech, such whether the target is individual or group, or the name of the group (c.g., race, religion, gender) (Ousidhoum al., 2019; Mathew al., 2021): (3) Context sensitivity determines toxicity by leveraging con- text, such as previous tweets (Menini et al., 2021), comments (Pavlopoulos et al., 2020; Xenos et al., 2021) or previous sentences and phrases within the comments (Gong et al., 2021). They show that context can alter labeling decisions by annotators, but that it does not largely impact model perfor- mance (Pavlopoulos et al., 2020; Xenos et al., 2021; Menini et al., 2021); )); (4) implication understands veiled toxicity that are implied in codewords and emojis (Taylor et al., 2017; Lees et al., 2021), and
Conclusion In this paper, we analyzed messages flagged by human moderators on Twitch to understand the na- ture of norm violations live-stream chats, pre- viously overlooked domain. We annotated 4,583 moderated chats from live streams with their norm violation category and contrasted them with those from asynchronous platforms. We shed light on the unique characteristics of live-stream chats and showed that models trained with existing data sets perform poorly in detecting toxic messages in our data, which motivates the development of special- ized approaches for the synchronous setting. Our experiments established that selecting relevant con- text is an important feature for detecting norm violations in the synchronous domain. we hope our work will help develop tools that enable hu- man moderators to efficiently moderate problem- atic comments in real-time synchronous settings and make the user-experience in these communities more pleasant.

---

6 Limitations
our workers, we maintained open communication channels, allowing them to voice any question, con- cerns, or feedback about the data annotation. This also helped to improve the quality of the collected data as we promptly addressed issues reported by workers throughout the process. We also give each annotation instance enough time so that we do not pressure annotators (40 days for 4,583 instances). We did not collect any personal information from annotators and we did not conduct any experiments with human subjects.
Our data, analysis, and findings have certain limita- tions. Our research is restricted to the English lan- guage and the Twitch platform, although the meth- ods used to detect rule violations in live-stream chat and collect data can be adapted to other lan- guages. Additionally, we recognize that our anno- tators were recruited from one country, which may result in lack of diversity in perspectives and po- tential societal biases. Furthermore, we established 2-minute context window for each moderated comment within the moderation event, but this may not capture all relevant context.
We confirm that we collected and used chats, also referred to as user content, in accordance with Twitch's 's Terms of Service and do not publicly re- lease the data as it may be in violation of laws against unauthorized distribution of user content. However, we intend to make the platform-specife knowledge statements we compiled available support future research on real-time chat in the live- streaming domain. During the collection process, we used the official Twitch API to monitor and retrieve chats.
Additionally, the small size of our human- annotated data may limit the generalizability of our findings to other situations, We recognize that our data set may not represent all instances of rule violations in real-world scenarios. This may be due to the biases of the moderators in choosing which users or comments to moderate or prioritiz- ing certain types of violations over others. Also, the randomly sampled data we annotated may not be representative of the entire population and the imbalance of rule violation classes in our data set may not contain enough samples of rare categorics to make definitive conclusions.
Lastly, we want to emphasize that careful consid- eration must be given to user privacy when using moderation events study norm violations. While users may be aware that their comments can be viewed by others in the chat room, researchers must also understand that users have the right to request not to be included in the data and establish a mech- anism for users to contact researchers to have their data removed, and refrain from publicly releasing the data and instead share it on need-to-knon basis to control who has access to the data.
Our experimental results indicate that models trained detect norm violation using our data are far from perfect and may produce errors. When such models are used in real world applications, this can result in overlooking potentially prob- lematic comments or incorrectly flagging non- problematic comments. Therefore, we recommend using Al-based tools to assist human moderators rather than trying to fully replace them. Practition- ers should also be aware that there may be users with malicious intent who try to bypass moderation by making their comments appear innocent. By em- ploying moderation models, malicious users may be better able to craft toxic messages undetectable by existing models. As mentioned above, having a final step of human review or verification of the model output will be beneficial. Additionally, it may be necessary to continuously update the model and limit public access it.
8 Acknowledgement
We would like to thank to Yeeun Shin (SoftlyAl) for managing the data annotation process and Kyu- min Park (SoftlyAl) for the development of the web annotation framework. Datumo, known SELECTSTAR in South Korea, provided crowd- soureing platform for the annotation of the data. This work was funded in part by the Defense Advanced Research Projects Agency (DARPA) under Contract No. N66001 1924033, Con- tract No. HR001 12290106, and Contract No. HR001 12290025, and with support from the Ke- ston Exploratory Research Award. The views and conclusions contained herein are those of the au- thors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ARO or the U.S. Govern- ment.
7 Ethical Considerations
We took several steps ensure that our data col- lection was ethical and legal. We set the hourly rate of compensation for workers at 116.15, which was well above the country's 's minimum wage at the time ($7.4). To ensure the safety and well-being of

---

References
Antigoni Maria Founta, Constantinos Djouvas, De- spoina Chatzakou, Ilaas Leontiadis, Jeremy Black- burn, Gianluca Stringhini, Athena Vakali, Michael Siriviamos, and Nicolas Kourtellis. 2018. Large scale crowdsoureing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Hongyu Gong, Alberto Valido, Katherine M Ingram. Giulia Fanti, Suma Bhat, and Dorothy I. Espelage. 2021. Abusive language detection in heterogeneous contexts: Dataset collection and the role of super- vised attention. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages
Luke Breitfeller, Emily Ahn, David Jurgens, and Yu- lia Tsvetkov. 2019. Finding microaggressions in the wild: A case for locating elusive phenomena in so- cial media posts. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1664-1674, Hong Kong, China. Association for Computational I Linguisties.
Xiaochuang Han and Yulia Tsvetkov. 2020. Fortify- ing toxic speech detectors against veiled toxicity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7732-7739, Online. Association for Computa- tional Linguistics.
Eshwar Chandrasekharan. Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman. Cliff Lampe, Jacob Eisenstein, and Eric Gilbert. 2018. The internet's 's hidden rules: An empirical study of reddit norm violations at micro, meso, and macro scales. Proceedings of the ACM on Human- Compan-e Inter- action, 21CS
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-genent generatedee dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 3309 -00993336..22 Dublin, Ireland. Association for Computational Linguistics.
Jithin Cheriyan, Bastin Tony Roy Savarimuthu, and Stephen Cranefield. 2017. Norm violation in online communities-a study of stack overflow comments. In Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XIII, pages 20-34. Springer.
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2021. Analyzing the forgetting problem in pretrain-finetuning of open- domain dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1121-1133, Online, Association for Computational Linguistics.
Srayan Datta and Eytan Adar. 2019. Extracting inter- community conflicts in reddit. In Proceedings of the international AAAI conference on Web and Social Media, volume 13, pages 146-157.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective An- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
David Jurgens, Libby Hemphill, and Eshwar Chan- drasekharan. 2019. just and comprehensive strat- egy for using NLP to address online abuse. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3658- 3666, Florence, Italy. Association for Computational Linguistics.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512-515.
Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods Natural Language Processing, pages 345-363, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Elaine Gonzalez, et al. 2018. The gab hate corpus: A collection of 27k posts annotated for hate speech.
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Da- vani, Morteza Dehghani, and Xiang Ren. 2020. Con- textualizing hate speech classifiers with post-hoe ex- planation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5435 5542,, Online, Association for Computa- tional Linguistics.
Casey Fiesler, Joshua McCann, Kyle Frye, Jed R Brubaker, et al. 2018. Reddit rules! characterizing an ecosystem of governance. In Twelfth International AAAI Conference on Web and Social Media.

---

Srijan Kumar, William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2018. Community interaction and con- flict the web. In Proceedings of the 2018 world wide web conference, pages 933-943.
Chan Young Park, Julia Mendelsohn, Karthik Radhakr- ishnan, Kinjal Jain, Tushar Kanakagiri, David Jur- gens, and Yulia Tsvetkov. 2021. Detecting commu- nity sensitive norm violations in online conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3386-3397, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, et al. 2022. Xmd: An end-to-end framework for interactive explanation based debugging of nlp models. arXiv preprint 2rXiv:2200. 16978.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vetioo, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high perf-rrrrrraceee deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-8035. Curran Associates, Inc.
Alyssa Lees, Daniel Borkan, Ian Kivlichan, Jorge Nario, and Tesh Goyal. 2021. Capturing covertly toxic speech via crowdsoureing. In Proceedings of the In-
First Workshop on Bridging Jluman-Compuen teraction and Natural Language Processing, pages 14-20, Online. Association for Computational Lin- guistics.
Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective api: Efficient multi- lingual character-level transformers. arXiv preprint arXiv:2202.1117
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and lon Androutsopoulos. 2020. Tox- icity detection: Does context really matter? In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4296- 4305, Online. Association for Computational Lin- guistics.
Han Li, Robert E Kraut, and Haiyi Zhu. 2021. Technical features of asynchronous and synchronous commu- nity platforms and their effects on community cohe- sion: comparative study of forum-based based and chat- based online mental health communities. Journal of Commuter-Mediatee Communication, 26(6):403- 421.
Patricia Rossini. 2022.
Beyond incivility: Under- standing patterns of uncivil and intolerant discourse in online political talk, Communication Research, 49(3):399-425.
Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2022. A holistic approach to un- desired content detection in the real world. arXiv preprint 2tt.:222080334444
Chinnadhurai Sankar, Sandeep Subramanian, Chris Pal, Sarath Chandar, and Yoshua Bengio. 2019. Do neu- ral dialog systems use the conversation history ef- fectively? an empirical study. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 32-33 32-37, Florence, Italy. Association for Computational Linguistics.
Binny Mathew, Punyajoy Saha, Seid Muhic Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 14867-14875.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1675 1678, Florence, Italy, Asso- ciation for Computational Linguistics.
Stefano Mcnini, Alessio Palmero Aprosio, and Sara Tonelli. 2021. Abuse is contextual, what about nip? the role of context in abusive language annotation and detection. arXiv preprint arXiv: 103. 14916.
Maarten Sap, Saadia Gabriel, Lianhui Qin. Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490, Online. Association for Computational Linguistics.
Courtney Miller, Sophie Cohen, Daniel Klug. Bogdan Vasilescu, and Christian Kästner. 2022. "did you miss my comment or what?" understanding toxicity in open source discussions. In In 44th International Conference on Software Engineering (ICSE'22) 22))
Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit- Yan Yeung. 2019. Multi- lingual and multi-aspect hate speech amalysis. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNI) pages 4675- 4684, Hong Kong, China. Association for Computa- tional Linguistics.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3884-5906, Seattle, United States. Association for Computational Linguistics.

---

Leandro Silva, Mainack Mondal, Denzil Correa, Fabrí- cio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In Tenth international AAAI conference web and social media.
Rohit Sridhar and Diyi Yang. 2022. Explaining toxic text via knowledge enhanced text generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 811-826. Seattle, United States, Association for Computational Linguistics.
Jherez. Taylor, Melvyn Peignon, and Yi-Shin Chen. 2017. Surfacing contextual hate speech words within social media. arXiv preprint arXiv:1711.1000
William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the Second Workshop dia, pages 19 26, Montréal, Canada. Association for Computational Linguistics.
Language in Social Me-
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88-93, San Diego, California. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Huggingface's transformers: State-of- the-ar art natural language processing, arXiv preprint arXiv:1910.33771.
Alexandros Xenos, John Pavlopoulos, and Ion Androut- sopoulos. 2021. Context sensitivity estimation in toxicity detection. In Proceedings of the 5th Work- shop on Online Abuse and Harms (WOATI 2021), pages 140-145, Online, Association for Computa- tional Linguistics.

---

A Annotation Details
Majority Vote
Stage
Stage
Stage
We engage in active discussions with annotators and provide detailed feedback after multiple rounds pilot study to ensure the data quality.
Non -on-iidetifiable
57.55% (545)
1.35% (16)
0.24% (3)
Brondcaster
6,53% (62)
59 83% (712)
59. 59.71% (744)
OIB
7.07% (67)
86.9%% (320)
32.34% (403)
0OB
0.32% (3)
1.09% (13)
1.61% (20)
Disagreement
28.51% (270)
10.84% (129)
6.10% (76)
A.1
Annotation U1
Total
947
1.190
1.246
To make it easy for annotators annotate with various types of contexts, we create an annotation tool. The annotation tool has three options and the user can select each option for each step annota- tion. Figure 6 shows the UT for step 1 which shows only the user's 's last chat (bad utterance) before the moderation event. Figure 7 shows chat logs up to two minutes ago based on the moderation events multi user context panel. To make it easier for annotators find previous chats from moderated users, we create single user context panel only display chat logs of the moderated user in multi user context. Figure 8 shows both chat logs and video context. The video context shows 1-minute clipped video around the moderation event.
Table 8: Inter-annotator Percent Agreement for Tar- gets of HIB. Presents the agreement percentage of HIB after majority vote. The numbers in parentheses indicate the absolute number of events.
Category
Platform
Streamer
Total
Discrimination
2
HIB
73
60
133
Privacy
0
Inappropriate Contents
Off Topic
Spam
25
25
50
Meta-Rules
25
42
Incivility
69
71
140
Total
187
183
370
Table 9; Data statisties of knowledge statements.
Target Agreement for HIB For cases consoli- dated as HIB with the majority vote, we further analyze the inter-annotator agreement of targer la- bels among annotators who have marked them as HIB. In cases where the annotator was unable to identify the target, we asked them to mark the tar- get "non-identifiable.. Table 8 shows that most HIB words (92.05%) are directed at someone the broadcast such as the broadcaster or viewers.
A.2 Annotation Consolidation
To determine the final label for each moderated event, we aggregate the labels of annotators us- ing a majority vote with heuristic rules. Each an- notator 0; identifies a list of violated rules L = (1,42, ... , for moderated event at each stage k- {1, 3. Here, we don't consider the target for HIB. We first evaluate the percentage agreement measure inter-annotator agreement in each stage by exact match and partial match. The exact match determines whether three annota- tors have exactly the same rules (Ca1 = La2 = Las) and partial match determines whether there is at least one intersection rule between three anno- tators ((Cu1 O La2 n La3) > 0). Table 4 shows the inter-annotator agreement percentage. We find that 98% of agreements from exact match are single label cases (i.e., 98% of exact matches have only one label) and many disagreements are resolved us- ing the partial match method. 92% disagreements that persist even with the partial match method are the case where one or two annotators marking a comment as violating the "Incivility" rule while the others do not. Finally, to determine the gold label using the annotations from the three annotators, apply a majority vole approach, choosing the rule types that were selected by at least two people. We discard approximately 3% of events that cannot be consolidated because all three annotators provided different labels.
A.3 Knowledge Statement.
Template. Table shows templates for the knowledge statement. For platform knowledge, annotators should fill out the span and what span means, and choose whether the span is emoji or text. For streamer knowledge, annotators should fill out the name of streamer and his/her personal background that may need to decide the label.
Data Statistics. Table shows data statistics of knowledge statement for each coarse-level norm category. Statistics show that 296 examples (6.6%) require knowledge, with 183 examples requiring streamer knowledge and 187 examples requiring platform knowledge. Note that there are some ex- amples require both. Statistics demonstrate that HIB and incivility require domain knowledge the most to understand the meaning behind them.
Knowledge statement examples. For each coarse-level norm category, we present its exam- ples (See Table 10).

---

Categury
Knowledge
Knowledge Statements
Discriminatonn
Streamer
Platfomm
[strcamer] playing Chinese
[ensde] is ennjii that macas Turkish.
Platfonn
Slaeee emaji that boredom, originating froes
askeep streum.
Streamer
[streomer] was LoLiee Legends) gume streamer, but seems to quit and play gamble.
Platfomm
Gambaa lest that mons gambling and the [streamer] ordered mode han whoever type
Spam
Streamer
bonned
"PogU".
Mor Rules
Platfumm
Shoutout
meam highlight roball memters in chat, prompting others to feelow chammol. [steamer]] is game streumer on Twitch.
Streamer
Incivility
Platfomm
laz lacome is text that meus sucrifiee People the chat suciifice themeevveseee timed oud typing them cum some kind of points.
(sreamet) declined pperson)'s fight
Off-topic
Platfonn
Tunkies is emoji that means implying supporis loward Russia (or Soviet Union).
Table 10: Knowledge statement examples.
A.4 Examples of Incivility.
C Ablation Study
C.1 Context Arrangement
Table 11 presents examples of chat moderation by streamers where the underlying reason for modera- tion is not apparent. The cases highlight potentially uncomfortable situations that streamers may en- counter.
To understand how the context arrangement in the input affects the performance, we conduct exper- iments with multiple variants of context arrange- ment moderation detection (See Table 15). First, the results show that randomly shuffled context consistently harm the performance. indicates that context order matters, contrast to the find- ings in dialog system study results (Sankar et al., 2019; He et al., 2021). Moreover, input as the sequential order of chats presented in the context- aware model (Pavlopoulos et al., 2020), or adding more contexts (e.g. broadcast category, rule text) degrade the performance. This indicates that the target text should always be placed first, and some contexts may not be helpful.
Chat
Action Ban
I'm so I'm really sad My mum said she wants to marry you
Table 11: Example cases of Incivility.
B Experimental Setup Details
Each fine-tuned experiment uses NVIDIA RTX A5000 GPU and uses FP16. We implement models using PyTorrc (Paszke al., 2019) and Hugging- face Transformers (Wolf et al., 2019). We use the Adam optimizer with maximum sequence length of 256 and batch size of 4. We set 100 epochs and validate the performance every 100 steps. The stopping criteria is set to 10. For each data, we searched for the best learning rate for our model out of [le-5, 2e-5, 5e-5, le-4, 3e-4J. Then, we report the average score of 3 runs by different ran- dom seeds (42, 2023, 5555). Each run takes 10 to 30 minutes. To determine the data distribution ra- tio between positives and negatives in the training data, we searched for the best distribution out of [1:1, 1:2, 1:5, Original] by random negative sam- pling. As shown in Table 12, we found that the evenly distribution (1:1) shows the most stable per- formance with the lowest standard deviation under with and without context. Data statistics for both Twitch and Reddit (Park et al., 2021) are presented in Table 13-14. Note that we report the number of data statistics after sampling the same number of negative samples as positive samples.

---

Calegory
No Context
Multi-user Context (utferance)
Multi-weer Context (firs)
1:1
1:2
1:5
Original
1:1
1:2
1:5
Original
1:1
1:2
1:5
Ohigital
1001
zowo
=301
0.00 cc.0
=uu
12:11
s003
=00
=t.>
162)
1841
1000
-301
0.34 1125
0.61 1905
s3d
13:30
1035
-00
2833
1835
1a35
Privac
0.05 1845
0.05 1009
0.11
-327
0.00 sc80
0.00 100
0.02 2301
0.08 12:2
0.00 1029
0.01 =06
0.07
0.07 1662
0.00 sox
Inape. Contenes
0.12
0.58
0.33
0.12 2136
0.09
0.36
0.62
0.36
0:11
0.28
0.54
0.38
Od Topic
0.007
0.19
0.28
0.00
0.10
0.13
0.38
01.00
0,8
0.15
0.57
0.00
0.63
0.68
0.67
0.61
0.86
0.72
0.71
0.89
0.75
Meto-Rules
1011
216
201
g00
2330
2000
200
2001
Insivlliyy
1084
2003
0.0
2016
(.00-100)
0.24 2002
0.31
2320
2234
s009
200 20:4
2600
Table Experimental Results on different Training data distribution. macro Fl score each coarse-level category, and scores average of 3 runs (3 random seeds). Excluding models with an F1 score of O, the model with the lowest standard deviation is bold for cach category and its context sctting.
Stage
Stage
Stage Development
Category
Train
Development
Test
Train
Development
Test
Train
Test
0
0
0
Discrimination
83
83
10
836
9
863
81
81
9
857
9
863
84
9
857
10
862
HIB
752
752
106
760
88
794
949
949
121
745
118
754
996
996
124
742
124
7748
Privacy
11
865
865
865
Inape. Conients
47
863
42
863
4
3
863
868
Oft Topic
70
70
858
8
8644
78
78
858
9
863
77
8
858
9
863
Spam
789
789
92
774
103
769
935
935
114
752
114
758
926
926
114
752
114
758
Meta-Rule Roll-Rules
23
92
774
778
Incivility
1,607
1,607
191
675
198
718
88
778
91
678
84
782
84
788
ALL
3,542
3,499
432
434
434
438
3,577
3,464
435
431
440
432
3,577
3,664
453
431
440
432
Table 13: Train/Dev/Test Statisties of Normvio-RT.
Train
Development
Test
Category
0
Incivility
1,787
1.787
252
4,962
230
4,901
Harassment
5,048
5.048
605
4.509
546
4,585
Spam
3,649
3,649
418
4.796
417
4,714
Off Topic
3,009
3009
326
4888
331
4800
Speech
4,930
4,464
Content
20.614
20,614
2,773
2,441
2,618
2,513
Table 14: Train/Dev/Test Statisties of Normvio (Park
et al., 2021).
Aregoeni
Context
Store
MM
0,703
single 0.747
0.701
mulii multi
single, user
0.708
+ [SEP]+ RAV Dicontel)))
multi user (event)
0.671
multi, user (untorance)
0.551
multi, user (finst)
0.952
[SBP]
(Paslopouios 2020)
(urierance)
mulii, (lios))
single user
0.777
15x [SEP] contxx [SEP] broadcast cat.
mult, user (unierance)
multi. user tevent)
0.504
0,571
mulli, (lind)
0.541
single uSer (event)
[ comext [SEP] sext
mulii msrr (stlerance)
multi, user (fins)
0.953
Table 15: Performance on moderation detection by different context arrangement. macro Fl score for "AI" in Table 6. Best models for each context are bold.

---

Single Only
Video Context
Bad Utterance
[2022-06-14 20)55:20), check this out chatr, playing
without r3deyojedi__a
Channel Setting
Single user
context
Figure 6: Step 1. single ufferance shows only the user's last chat before the moderation event.
Single
Context
Video Context
Bad Utterance
[2022-06-14 10:56:20): check this out chats, playing
without hand r3deyejedi__s
Channel Setting
AND SEE, SOLOQ, CLIMBING BACK TO CHALLENGER !YT Idiscord
League of Legands
English
Single user context
12022-06-14 10:54:42]: check this out chatr, playing
[2022-06-14
10:54:36]:
Isague without hand (3deyojed__aa bb448830.2B5 (--44883---665--44 4866-4471 4-1--4411---00
12022-06-14 10:54:36)
318318 122a41)
[2022
10:5437):
chatr, playing
[2022-08-14 10:54:37) me too
league without hand - radeyejedi_za (D: c37d80di #Bac- 4198 9683:
[2022-08-14 10:54:37: KEKW
022-08-14 10:54:37):
(2022-06-14 10:55:37]: [ban]
12022-08-14 10:54:38]:
[2022-08-14 10:54:40)
|2022 05-14 10:54:40]: D:
10----00--44 10:54:00) @Jankos
[2022-00-14 10:54:22: check this out chatt, playing
league r3deyejed__zs
[2022-06-14 10:54:42): D:
(2022-06-14 10:54:45):
12022-06-14 10:54:45) we dont see
Figure 7: Step context shows chat logs up to two minutes ago based on the moderation events (multi user context). single user context only shows the moderated user's messages within two minutes.

---

Show Selections
Step Single Ulterance Only
Step 2: 1 Text Contect
Step 2 Video Conteal
Video Context
Bad Utterance
12022-08-14 10:86:20]: chatr, playing
league without hand r3deyejedi__e _x0
Setting
CAN THIS MAN BE SO GOOD WHILE BEING YEARS OLD? COME AND SEE, SOLOO, CLIMBING BACK TO CHALLENGER IYT Idiscord
League Legends
Soplisn
00 1TD0
JANKOS
Single context
Multi
[2022-06-14 10:54:42) check this out chair, playing
nummgunri pimmazpiay LAVEATMU ie (urute-oo))
(2022-06-14 10:54:36): D:
Hb448330-5085- 4177--------
20022-06-14
10:54:36):
s166r8122a41)
[2022: 06 10:54:37): true
[2022-06/ 06-14 10:55/20): check
playing
league without hand r3deycjedl_za (ID) c37d600tt-89cc (att--------------088
4198-998---
10:54:37]:
[2022-05-14 10:54:37]: KEKW
838966F04e88)
12022-06-14 10:54:37]:::
(2022-08-14 10:55:37): [ban]
[2022-08-14 10:54:36)
(2022-06-44
10:54:40j
(2022-06-14 10:54:40]: D:
[2022-08-14
Jankks
[2022-05-14 10:54:4]]: check this out chatr, playing
without hand radeyejed__a
[2022-06-14 10:54:42): Dr
[2022-08-14 10:64:43) lel dont Shead
(202-06-14 10:54:45) we dont seell
Figure 8: Step 3. video context shows both chat logs and video context.


Reviews:
[{"review_id": "m8P03CZQLz", "reviewer": "Reviewer_HcKC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work. Rest assured, we will add the references you have recommended into our content. Thank you for taking the time to provide us with valuable insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Very well written paper and I would say the first to analyze synchronous and asyc chat message discrepancy and the important role of context in sync chats\nGood problem framing\nComparison against async chats e.g. on Reddit\nAnnotated dataset\n", "reasons_to_reject": "No reason", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors analyze synchronous and asyc chat message discrepancy and the important role of context in sync chats. The authors conducted analysis on Twitch stream chats and use Twitch channel moderated events and rules to build a benchmark dataset. The key highlight is that utilizing context helps improve the inter-annotator agreement and also the model performance. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://dl.acm.org/doi/abs/10.1145/3345645.3351104\nhttps://aclanthology.org/2022.wnut-1.7/ "}, "extra_text": {}}, {"review_id": "YW0s9P6Gtk", "reviewer": "Reviewer_YGch", "summary": "", "questions": "A few small questions:\n- In 3.2, how to apply the (6) Rule text-based model to testing data, considering the violated rule is unknown at prediction time?\n- I don't understand this sentence:\"The intuition for this selection is that the modera-391 tion event may have taken place much earlier than 392 the moderation event.\" What does it mean by moderation event is much earlier than moderation event?\n- An specific example showing why (4) Multi-user context is useful would be great for readers to understand the paper.\n- I am very curious about the performance of the proposed model in other domains, such as YouTube live comments.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your questions.\n\n#### **Q1. `Rule Text` experiments**\n\nAs mentioned in line 400, we do not use rule text for the inference since we do not know which rule is violated for unseen examples. We make an inference based on chat as it is without any context. This experimental procedure aims to assess whether the model possesses an intrinsic understanding of the underlying rationale behind norm violations.\n\n#### **Q2. Meaning of `The intuition for this selection is that the moderation event may have taken place much earlier than the moderation event`**\n\nApologies for any misunderstanding. The broadcaster or moderator can take appropriate moderation action within a few minutes of encountering problematic behavior. It is important to note that this action may not be instantaneous, resulting in a temporal gap between the user’s problematic behavior and the moderation action taken by the broadcaster or moderator.\n\n#### **Q3. Examples of scenarios in which the use of a `multi-user context` can contribute to improvement**\n\nThank you for your suggestion. We acknowledge the value of providing real-world examples as they can be helpful in illustrating our points. However, incorporating actual instances into the paper proved challenging. Nevertheless, we would like to provide the following example cases:\n```\nProblematic chat: 'GO M4CAC0S'\n```\nThe problematic chat can be challenging to identify as a violation of social norms for individuals who are unfamiliar with internet terminology. However, within a multi-user context, it was noticed that all the conversations taking place were constantly focused on making discriminatory statements towards Asian gamers. This surrounding context provides a clearer understanding of why this specific chat can be considered an example of discrimination.\n\n#### **Q4. Performance of model in other domains (YouTube live comments)**\n\nWe appreciate your suggestion and are eager to expand our efforts in developing a comprehensive framework for detecting toxic content in all types of synchronous chat platforms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed research problem is very interesting and has great practical impact.\n- It shows several interesting insights on this problem.", "reasons_to_reject": "none.", "questions_to_authors": "A few small questions:\n- In 3.2, how to apply the (6) Rule text-based model to testing data, considering the violated rule is unknown at prediction time?\n- I don't understand this sentence:\"The intuition for this selection is that the modera-391 tion event may have taken place much earlier than 392 the moderation event.\" What does it mean by moderation event is much earlier than moderation event?\n- An specific example showing why (4) Multi-user context is useful would be great for readers to understand the paper.\n- I am very curious about the performance of the proposed model in other domains, such as YouTube live comments.", "paper_topic_and_main_contributions": "This paper analyzes norm violations and toxic language in live stream chats, which have different characteristics compared to asynchronous platforms like Reddit or Twitter that have been the focus of most prior work. The authors collected a new dataset called NormVio-RT containing 4,583 moderated Twitch comments annotated for 15 fine-grained norm violation types. Experiments find existing toxicity detection models perform poorly on live stream data, motivating new approaches. Adding relevant context improves inter-annotator agreement on labels. Models using context outperform those without, especially for norms relying on relationships between messages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q9S7aovxv4", "reviewer": "Reviewer_9tw6", "summary": "", "questions": "1. Isn't the claim that the additional context helps in determining norm-violations fairly evident? why is this interesting?\n2. The same question is true for how additional context helps with inter-annotator agreement.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your questions.\n\n#### **R1. Rationale behind bad precision of existing toxicity detection models**\n\nOur findings suggest that the language used in online communities with a bulletin board format can differ from typical chat conversations, resulting in a shift in data distribution and synchronous/asynchronous communication styles. Additionally, the Twitch-trained model demonstrates a high level of generalizability (Table 7), indicating that typical chats may encompass a broader range of toxic behavior.\nAs a result, we believe that existing toxicity detection models trained on internet data from platforms like Twitter and Reddit may be adversely affected by this disparity.\n\n#### **R2. Details of experimental setup**\n\nApologies for any misunderstanding. We will strengthen our experimental setup in more detail.\n\n#### **Q1. The assertion that additional context is helpful for detection norm-violation and inter-annotator agreement appears to be self-evident.**\n\nI agree that it is evident that additional context can be helpful to detecting norm violation. Moreover, it can be helpful to any other NLP tasks as well (e.g., open-book QA).\nHowever, we demonstrate the importance of determining which specific context for detecting norm violation in a live-streaming environment which is beyond simple incorporation of context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The topics of the paper is relevant for the NLP community: detecting norm violations in live-steam media conversations. The previous work in this space has only been on non-live services such as Reddit\n2. The methodology is thorough with all relevant details that support the claims \n4. The paper is well presented with clear details on data collection, annotation strategy, experimentation, and analysis\n5. The collected and norm-categorized data might be useful for further research in this direction", "reasons_to_reject": "1. The authors mention that the existing models have bad precision for toxicity detection but don’t explain why. Further analysis on this end would be helpful\n2. More information is needed for the experimental setup ", "questions_to_authors": "1. Isn't the claim that the additional context helps in determining norm-violations fairly evident? why is this interesting?\n2. The same question is true for how additional context helps with inter-annotator agreement.", "paper_topic_and_main_contributions": "1. The paper proposes a new task: detecting norm-violations in interactive live chat streaming platforms\n2. The authors show that additional context such as audio and video helps in identifying norm-violations in live-stream chats\n3. The paper shows  that norm-violating behaviour is different for both non-live and streaming platforms and therefore  existing models are not as effective in detecting norm violations on streaming platforms\n4. The authors collect norm violating data and aggregate them into 15 fine-grained categories\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study on norm violations in live-streaming platforms. It introduces a new data set of moderated comments from Twitch and a taxonomy of 15 norm violation types. Models are trained that use context to identify norm violations that can improve overall moderation performance.

The reviewers are in agreement in their positive judgement of the paper. Overall, they consider the paper is well writen and appreciate the problem framing and importance of the work. The reviewers also appreciated the novelty of the work in dealing with live / synchroneuous chat and the insights gained into this problem, especially comparing it with async platform.

The data and annotation process are clear and it should form a solid basis for future work in this space.

The reviewers did not identify any substantial reason for rejection or area of improvement.
Decision: Accept-Main