Dimensions of Online Conflict: Towards Modeling Agonism
Authors: Matt Canute
Mali Jin
hannah holtzclaw
Alberto Lusoli
Philippa R Adams
Mugdha Pandya
Maite Taboada
Diana Maynard
WENDY Hui Kyong CHUN
Venue: EMNLP 2023 Findings (2023)
Keywords: online conversations
conflict
agonism
content moderation
TL;DR: A dataset and model for predicting conflict in online conversations, based on the context of the conversation

Abstract: Agonism plays a vital role in democratic dialogue by fostering diverse perspectives and robust discussions. Within the realm of online conflict there is another type: hateful antagonism, which undermines constructive dialogue. Detecting conflict online is central to platform moderation and monetization. It is also vital for democratic dialogue, but only when it takes the form of agonism. To model these two types of conflict, we collected Twitter conversations related to trending controversial topics. We introduce a comprehensive annotation schema for labelling different dimensions of conflict in the conversations, such as the source of conflict, the target, and the rhetorical strategies deployed. Using this schema, we annotated approximately 4,000 conversations with multiple labels. We then train both logistic regression and transformer-based models on the dataset, incorporating context from the conversation, including the number of participants and the structure of the interactions. Results show that contextual labels are helpful in identifying conflict and make the models robust to variations in topic. Our research contributes a conceptualization of different dimensions of conflict, a richly annotated dataset, and promising results that can contribute to content moderation.

Dimensions of Online Conflict: Towards Modeling Agonism
Matt Canute', Mali Jin", hannah holtzclaw', Alberto Lusoli', Philippa R Adams', Mugdha Pandya', Maite Taboada', Diana Maynard and Wendy Hui Kyong Chun'
Digital Democracies Institute, Simon Fraser University, Canada, *Department of Computer Science, University of Sheffield, UK, m. M..jjne@heffiie field. ac. ac. uk
ca
Abstract
Not all conflicts are equal, and the confusion over the political and social value of conflict partly stems from its diverse nature. Conflict exists on a sliding scale ranging from antagonistic conflict be- tween enemies (which is often silencing, undemo- cratic, and hateful because it is focused on delegit- imizing opponents' rights and status) to agonistic conflict between adversaries (which has productive potential for the emergence of democratic dialogue, dissent, and trust in the public and political sphere, since the struggle is over interpretation and not le- gitimacy to speak) (Mouffe, 2002; Wenman, 2013). As many across the political spectrum have noted, democracy needs agonism to widen voices and pre- vent public spaces from becoming totalitarian or meaningless consensus hubs. But how do we know which conflicts agonistic and which hateful? How do we know what kinds of counterspeech ef- fectively facilitate dialogue and which destroy it? Which follow the spirit of platforms' terms of use and which do not?
Agonism plays a vital role in democratic di- alogue by fostering diverse perspectives and robust discussions. Within the realm of online conflict there another type: hateful antago- nism, which undermines constructive dialogue. Detecting conflict online is central to platform moderation and monetization. It is also vital for democratic dialogue, but only when it takes the form agonism. model these two types of conflict, we collected Twitter conversations related to trending controversial topics. We introduce a comprehensive annotation schema for labelling different dimensions of conflict in the conversations, such as the source of con- flict, the target, and the rhetorical strategies deployed. Using this schema, we annotated ap- proximately 4,000 conversations with multiple labels. We then trained both logistic regression and transformer-hasee models on the dataset, incorporating context from the conversation. including the number of participants and the structure of the interactions. Results show that contextual labels are helpful in identifying con- flict and make the models robust to variations topic. Our research contributes a conceptu- alization of different dimensions of conflict, richly annotated dataset, and promising results can contribute to content moderation.
To begin answering these questions, we present an in-depth exploration of conflict online, using the platform once known Twitter (now "X") as a case study. Our work builds upon related Natural Language Processing (NLP) research fields such as abusive language, persuasion, and constructive comments. We use Twitter because, unlike the controlled environment of priori conflictual dis- cussions like those in Reddit's ' 'Change My View* (Monti et al., 2022; Srinivasan et al., 2019), it offers more organic setting.
1 Introduction
Conflict is everywhere online. From political protests to spirited debates over the latest TikTok trend, these conflicts are simultaneously celebrated as promoting democracy and condemned for fos- tering polarization and undermining public insti- tutions. Conflicts-inn -eeiicttt-ieess,,.eeeee arguments, or attitudes that oppose each other-are are also central to platform moderation and monetization: from the amplifica- tion of certain controversies provoke user en- gagement, to content takedowns to comply with platform's terms of use or national laws (Gillespie, 2022, 2018; Morrow et al., 2022; Douck, 2022; Zeng and Kaye, 2022).
To determine the nature of a conflict, context is central (Zosa et al., 2021: Hu et al., 2022; Ghosh et al., 2018). Currently, though, automatic content moderation mainly focuses single utterance rather than the ongoing conversation. Whether message fosters agonism depends on various di- mensions, including which groups are participating, how many participants, their level of interaction, and their relative power differences, all subject to

---

change over time.
ual conversations on Twitter; c) a detailed schema to annotate conversations for various dimensions of conflict and the resulting annotations, with good Inter-annotaie agreement; and d) a set of experi- ments that show the usefulness contextual infor- mation in predicting online conflict.
We introduce methodology for collecting and curating dataset of English Twitter conversations embodying various aspects of dimensions of con- flict. We collected conversations about trending events, as these often serve as catalysts, prompting individuals to interrogate their stance on current is- sues and deliberate their self-conception in relation to their views. We then annotated the conversations following our own coding schema, created with antagonistic and agonistic conflict in mind. These annotations took context into consideration rather than focusing on an isolated tweet.
Related Work
2.1 Conflict and agonism
Within this project, we understand conflict as the generative ground that spans hate to agonism. Ac- cording to political theorist Chantal Mouffe (2013), democratic speech or dialogue always bears traces of the conflicts from which it emerges. Democracy, therefore, necessarily entails conflict and negotia- tion; expand who counts citizen, to negotiate differing claims freedom or rights, and to vali- date collective decisions. This is agonistic conflict. Conflict can also be antagonistic: unproductive and undemocratic, when it silences individuals and groups, by shutting them out or harassing them with hateful speech. Democratic institutions are responsible for creating the space to allow con- flicts to take agonistic form, in which opponents are not enemies but adversaries among whom con- flictual consensus may emerge (see also Mouffe, 2002; Rancière, 1999, 2010; Wenman, 2013). This generative aspect of conflict has been neglected in discussions around content moderation on social media platforms, which frame the problem as free- dom of speech versus censorship (Douek, 2022; Gillespic et al., 2020; Gillespie, 2018).
Next, we trained logistic regression and trans- former models (BERT and GPT-3) on this dataset, to predict dimensions of conflict. The models are trained on human annotations of the conversations, enhanced with a) previously proposed labels for online conversation: constructiveness and toxicity; and b) contextual aspects such as cardinality (partic- ipant counts) and topology (interaction structure). We show that all models are sensitive to the specific words in the conversation, making them less gener- alizable across topics and domains. Incorporating conversational context, however, makes the models more robust, showing that cardinality and topol- ogy are important dimensions in the prediction of online conflict.
The potential use cases of this work include the measurement of productive (agonistic) versus un- productive (antagonistic) conflicts, providing in- sights into where learning and constructive dis- course can be fostered. This work ties in with attempts at reflective content moderation, where the goal is not simply to delete harmful content make it less visible (Zeng and Kaye, 2022), but also to identify and promote content that can be con- structive and productive towards democratic goals (Mouffe, 2013; Gillespic et al., 2020; Morrow et al., 2022). This kind of analysis has implications for detecting early signs of scapegoating and unproduc- tive disputes-cases where patterns of discourse do not necessarily break the terms of service, but can nonetheless bring harm over time. Our findings will be instrumental in shaping online discourse, aiming to hamess conflict driver of democratic conversation (agonism) rather than as a destructive silencing element (antagonism).
2.2 Abusive language
Abusive language online is a broad term that covers various forms of harmful or offensive communica- tion on the internet, such as hate speech, cyber- bullying, trolling, or flaming (Fortuna et al., 2020; Pachinger et al., 2023). Detecting and preventing abusive language online an important challenge for natural language processing (NLP) and social computing, and an extensive literature on the topic exists. In addition to the challenges of detecting a social phenomenon that the perpetrators often try to disguise, current industry solutions suffer from lack of interpretability, undermining their credibil- ity (MacAvaney al., 2019). Equally dangerous is the over-policing of certain communities and topics online (Saleem al., 2016).
Our main contributions are: a) a conceptualiza- tion of conflict online on a scale between antago- nism and agonism; b) a process to retrieve conflict-
Further, supervised classifiers require high- quality annotated data that may harm the annotators

---

and that may contain their biases (Sap al., 2022; Vidgen and Derezynski, 2021). We know context is also crucial in obtaining high-quality annotations (Ljubešić et al., 2022), and that some disagreement among annotators is to be expected (Leonardelli et al., 2021). All this prior work informs our annota- tions and explorations of machine learning models.
sarily have a successful outcome in terms of per- suading interlocutors.
2.4 Constructive comments
Research into high-quality online content has shown that constructiveness is useful dimension. Constructive comments build on and contribute the conversation, providing points of view and jus- tification for a particular opinion. They are not necessarily conflictual in nature, since they may simply build on the ongoing conversation. In a study of online news comments, Kolhatkar et al. (2023) propose that constructive comments seek to create civil dialogue, with remarks that are relevant to the article and not merely emotional provocations. Comments identified as constructive can be presented to future posters as prompts or examples of desirable behaviour or as nudges to de- polarize conversations (Stray, 2022). Our work on identifying conflict can contribute to the growing body of research on how to present content in such way that it contributes to productive, civil, and also agonistic discussion.
We emphasize, however, that we do not necessar- ily correlate the absence of toxicity or abuse with the presence of productive conflict. Abusive lan- guage research tends to characterize healthy and/or civil conversations those that are absent of tox- icity (e.g., Smith al., 2021; Hede et al., 2021). While that may be the case, healthy conversations are not necessarily agonistic. Agonism requires a certain level of disagreement as a source of political discussion and engagement.
2.3 Persuasion, argumentation, derailed conversations
Persuasion styles, rhetorical strategies, and argu- mentation styles all play a role in how we perceive and interpret conflict. Research in this area has pro- duced manual annotations of rhetorical strategies such as framing, hedging, modality, repetition, and rhetorical questions (Peldszus, 2014; Green, 2014; Hirst et al., 2014). These approaches tend to fo- cus on understanding which rhetorical approaches will be most effective in changing someone's 's mind (Habernal and Gurevych, 2016; Hidey and McKe- own, 2018).
Dataset
3.1 Data collection
We are interested in online conversations on con- tentious topics, so we used the Twitter Academic API v2 elevated access to gather replies containing certain keywords, starting with controversial topics. Then we consulted subject experts and representa- tives of equity-seeking groups as a way to increase the topic diversity within the dataset. This led a set of keywords as search terms (in Appendix A), After selecting tweet replies in English contain- ing the relevant keywords from each topic, we then extracted their surrounding conversation trees us- ing two traversal methods: depth and breadth. The former involved recursively collecting reply's ref- erenced tweet until reached the root message, the 7-message limit (since length is the last most frequent distribution before the start of the long tail of conversation thread). Breadth traversal involved capturing adjacent messages of a conversation by recursively creating new queries based on each re- ply's tagged author and the conversation ID of the reply.
Modeling work includes attempts to find argu- ments in text, area known argumentation min- ing (Mochales and Moens, 2011; 1; Lawrence and Reed, 2019; Harris and Di Marco, 2017). The goals include: a) identifying controversial topics in debates, news, Wikipedia articles, or online dis- cussions (Boltžžíć and Snajder, 2014; Kittur et al., 2007; Choi et al., 2010; Swanson et al., 2015; Stab and Gurevych, 3017); b) forecasting conversational derailment (Zhang et al., 2018); c) identifying con- versational strategies that will change the interlocu- tor's mind as in r/ChangeMy View (Monti al., 2022; Srinivasan et al., 2019); d) detecting conflict outside the conversation, as in r/ AsssoTeeAsshole posts (Welch et al., 2022).
While this previous research on persuasiveness informs ours, its goal is identify successful and unsuccessful argumentation styles. We are, first, looking for conflict, to then try and pinpoint exam- ples of agonistic discussions, which do not neces-
The annotated dataset contains equal mix of depth and breadth traversals. While the former en- ables more efficient data collection, the latter is useful for capturing the chaotic nature of conversa-

---

tions on most platforms, such as the one depicted in Figure 2.
Internalexttmaa??
Only conversations of length 3-7 messages in English were stored. This iterative process contin- ued over a period of three years (January 2020 December 2022), yielding a total of 220,626 con- versations."
Croep
Targer?
Policy natrudora
Systenic
Ponorra exporence
Common
Based on random sample of 1,000 conversa- tions, roughly 30% of these conversations likely involved first-time interactions", suggesting that these topics were contentious enough to spark de- bates among strangers in the comments of large accounts, creating virtual public forums.
Authority evoked?
Authorly rejeaed
US ab-obbeee then
there conflict?
Groups
compactiva
Unittert ciquactive
Fiptica cincaive
strategy?
3.2 Coding schema
Sexcurrt
To label the dataset, we developed an original cod- ing protocol based on an interdisciplinary review of literature on conffict, including media studies, political science, conflict resolution studies, and critical race theory (c.g.. Oetzel and moommyy 2006; Lamberti and Richards, 2019; D' Errico et al.. 2015; Itten, 2019; Yardi and boyd, 2010; Han et al., 2023).
Was context needed??
Concrnatinal
Media
ky
Figure 1: Coding schema for conflict
session during which annotators were introduced the project and its goals and were taught how to use the labelling platform (LabelStudio; see Appendix for screenshots). During the first week, the an- notators and two members of the research team worked together and labelled a subset of about 400 tweets. The annotation conducted in person, and each session had planned moments for dis- cussing disagreements and clarifying the gray areas of the annotation protocol. At the end of the train- ing session, we assigned a subset of the dataset to be codde. The research co-leaders held weekly in- dividual check-in meetings with annotators to trou- bleshoot issues and gather their feedback. These meetings also served to assess the emotional im- pact of annotation and to externalize the thoughts and emotions annotators encountered during their work. The entire research team also met every two weeks, to compare different annotation styles and discuss edge cases as a way to test the protocol's 's reliability. Each message was annotated by two annotators.
The initial protocol was first tested on subset conversations (see Section 3.3). We revisited the coding protocol twice throughout the course of the project based on coders' feedback and discussions over disagreements. The final coding protocol fol- lowed decision tree structure, where answering in the positive to question led to set of follow- up questions, shown in Figure 1. Appendix E contains an extensive discussion of each of the con- cepts in the figure, with examples.
3.3 Annotation and agreement
We recruited a team of four annotators. In selecting candidates, aimed to maximize demographic di- versity and cultural background. The team included two graduate and two undergraduate students from three departments at our university: Communica- tion, Political Science, and International Studies. The self-identified gender split was three women and one man, and the age ranges were: three 20-25 and one 26-30. At the beginning of the project, three members of the research team led training
Throughout the duration of the annotation cam- paign (May-December 2022), annotators labelled 4,022 conversations involving 9,472 individual au- thors with 22 labels. It two-tiered labelling system, where the annotators would first read the entire conversation and indicate whether there was conflict in the last message of the conversation given the context of the previous messages. They
'Raw data, annotations, and code used to extract the conversations are available our repository, which includes all code Section https://gil thub. con/Digital- Dattttlll--meeeessss...e Institute/De wittee//Dmeeiioooossoeeee of-On ine-Conflict
experiments
'First-time interactions were approximated by examining each account's ' prior 200 messages and checking if any of the accounts had interacted with each other previously.

---

would then annotate further aspects if the answer was 'yes', following Figure 1.
age'). Appendix C provides further information on the rate of conflict per topic. In summary, most topics contained some form of conflict, due to the way they were collected (trending topics). This makes the dataset possibly unbalanced, but also a rich source of how conflict develops online.
Since each binary label was annotated by two annotators, we can compute inter-annotate agree- ment using Cohen's 's kappa, & (Cohen, 1960). The initial question, 'did the last message in the conver- sation contain conflict? had x 0.65, moderate to substantial level of agreement (Landis and Koch, 1977). The kappa values for the follow-up ques- tions in Table show that some of the other labels foster less consensus. Annotators agreed, in gen- eral, whether the conflict internal or external to the tweet (with internal more difficult to adjudicate). A conflict is internal when all involved parties are also engaged in the conversation, i.e., it is conflict among the participants. An external conflict is dis- agreement about somebody else not in the conversa- tion, e.g.. a public figure (see Appendix É for more detail). Annotators also often agreed that conversa- tional context was needed. The level of agreement for rhetorical strategies (sarcasm, explicit directives and calls to action, association and metaphor analo- gies) was quite low, although consistent with the well-documented difficulty in annotating sarcasm (González-Ibiinee et al., 2011: Oprea and Magdy, 2020). In this paper, we mainly for conflict in the tweet, leaving other features for further study. We kept only conversations where the two annotators agreed on whether there was conflict. This yielded a total of 3,577 data points. One could argue that, by keeping only cases with clear agreement, we are in effect making the task "easier', Given the small size of the dataset, we follow this approach in order eliminate noise.
4 Conflict predictive models
For now, we focus only on finding conflict, leaving the issue of whether the conflict agonistic or not for future work (although we make some sugges- tions in Section 5). Our starting hypothesis is that we can find signals of conflict in the data. The tar- get variable of interest in this paper is the presence of conflict. Predictive features include: words in the text (bag of words model); constructiveness and toxicity labels; and context from the conversation, namely cardinality (number of participants) and topology (structure of participant interactions). By conversational topology we refer to the multi- threaded nature of online conversations, which have been described as polylogues (Marcoccia, 2004). Let examine it with an example, rep- resented in Figure 2. Amal sends out message about a new mural in their city. Boróka angrily replies that this mural is waste of lax-payer money, Deniz replies Boróka with a meme making fun of Amal, and Boróka sends the laugh emoji back to Deniz in response. Carlu tells Amal that a sec- tion of the mural is controversial and divisive and shouldn't have been publicly-funder funded. Eryl replies to Deniz clarifying that the section of the mural they are referring to not actually funded by the city. Eryl then says the same thing Carlu.
the binary label
Table I: Cohen's kappa x values for different features
Amal
Feature
Conflict (overall)
0.63
Target- Individual
0.90
External
0.58
Context Conversational
0.58
Carlu
Internal
0.48
Boróka
Rhetorical strategy Sarcasm
0.36
Rhetorical strategy Explicit
0.35
Rhetorical strategy Association
0.19
Context- Media
0.18
Eryl
Context Cultural
0.02
Deniz
Appendix B provides trend line for level of activity per topic over time. We saw that some top- ies, like 'Social Distancing" were discussed over long periods of time, whereas other topics peaked and declined quickly, perhaps having to do with specific events ('Will Smith Slap', 'Rogers Out-
Figure 2: An example conversation represented as graph
Each directed edge embeds information about
Names from the LEDIR repository (Sanders et al., 2020).

---

who received a notification from whom. If there is a path from C to A, that means that A received no- tification from C. For the above example, Deniz's message to Boróka sends a notification to both Amal and Boróka, but not to Carlu. Intuitively, it should help to know the directed graph's 's structure of conversation. It seems important to know, for example, that an utterance is part of larger con- versation sending out notification to five different people versus a back-and- forth conversation be- tween just two people. For the experiments, we en- coded this feature as a binary 'has bidirectionality" feature (i.e., is there back-and- lock-and-ford and-fforth interaction), but the dataset we will release has a full representa- tion of this dynamic. For instance, the conversation in Figure 2 is represented as: [(B, A), (D, B), (B, D), (C, A), (E, D), (E, C)1, which is also converted matrix form in the annotated dataset.
We also postulate (Hypothesis 1c) that the bag- of-words approach can be enhanced with additional labels. These labels are derived from other models that classify messages according to their construc- tiveness (Kolhatkar et al., 2023) and toxicity (Hanu and Unitary Team, 2020).* We added logit scores from each of these existing models into the same feature matrix. Table 2 shows confusion matrix for the bag-of-words (BOW) model with construc- tiveness and toxicity on the 716 conversations in our test set (we used a standard 80-20 split for train- ing and testing). We can see that this model is quite good at identifying both non-conffict and conflict, but it overpredicts conflict, as it is the majority in the imbalanced dataset.
Non-Conflict Conflict Total
Non-Conflict Conflict
27
87
114
602
602
4.1 Bag of Words model
Total
27
689
716
The first basic hypothesis (Hypothesis 1a) is that the presence of some words (unigrams and bi- grams) is predictive of conflict. We used logis- tic regression classifier with L2 regularization to predict whether the entire conversation represented conflict or non-conflict. It used a TF-IDF vectorizer to extract unigram and bigram features from the last message in the conversation, the message that we first extracted using keywords. We also used not just the last message, but the entire conversational context, with the same model (Hypothesis 1b). When we ran this model, we saw that some terms had coefficients with very high absolute values in predicting the presence of conflict. Although some of these terms are intuitively indicative of general conflict, such as the unigram 'people', some of the terms are most likely hyper-specific for particular conflicts that will have only happened once, such the unigram 'smith', referring to the Will Smith slapping incident. To reduce the risk of overfitting this model to particular topics on a new dataset or domain, and to have the model learn topic-agnostic linguistic patterns of conflict, we removed topic- related unigrams and bigrams by selecting the top 10 c- c-FF-IDF words from each topic and then re- moving those that seemed highly topic-specific (see Appendix D for lists of words removed by topic). We use this topic-agnostic dataset (Dataset 2) for the fine-tuning experiments in the next sec- tion. Dataset 1 contains all the words, without filtering.
Table Confusion matrix for BOW LR model with constructiveness and toxicity (Hypothesis 1c)
4.2 Transformer model
Our second main hypothesis (Hypothesis 2a) is that we can detect conflict using a transformer model, the BERT (Devlin et al., 2019) implementa- tion from HuggingFace." Further, we propose that, beyond the specific words in the message, the con- text of the conversation contributes to its likelihood of becoming conflictual. We examine different types of contextual information: the previous mes- sages (Hypothesis 2b); the previous messages with constructiveness and toxicity labels, as we saw in Section 4.1 (Hypothesis 2c); the cardinality of the conversation (Hypothesis 2d); and the topology, or structure the conversation (Hypothesis 2e).
We use technique inspired by Jin and Aletras (2021) incorporate contextual information into the BERT model. This approach has proven effec- tive for complaint severity classification by inject- ing linguistic features. The word representations from the embedding layer are combined with the contextual information using an attention gate to control the influence of different features. The com- bined representations are passed through the BERT encoder, followed by output layer. We set the
'https://github www/coiii/iacoeeeeeecooe http:/hhww.coomunitttarr/t/tttot hnttps://huggingface
http:/aaaaccccccecaeeeaaeeeeeeee co/ to/bbertbaae baserutcabee

---

max length 256 and keep the parameters in the attention gate the same as Jin and Aletras (2021).
with no topic words. This improvement with all the messages likely results from BERT's ability account for contextual sequences, as compared to the LR model.
4.3 Comparison of model results
Table shows Fl score results for the two main approaches we took, following the hypotheses de- scribed above. We first show results for a BOW logistic regression model under the three different conditions (last message, all messages, all mes- sages with constructiveness and toxicity). We then test the performance of BERT with the same condi- tions, plus we incorporate cardinality and topology. We also tested simple GPT-3 fine-tuned model using OpenAl's Curie fine-tuning API. Our test data comes from the human annotations (Dataset I), and the same data but with topic-related words removed (Dataset 2). The latter is more likely to be generalizable to new data about different top- ics, which is why we are interested in performance changes relative to Dataset 1. Results in the table are F1 score averages (average of three runs with random seeds 42, 43, 44).
Even more interesting is the effect of additional labels. When we incorporate constructiveness and toxicity, there is an improvement over the baseline of the last message and a slight decline from just including all the messages. However, the model more robust to the removal of topic words. The models with cardinality and topology have similar (opic-independent robustness. Cardinality leads to the lowest drop in performance (3.47 points) for Dataset 2, and topology also seems to show some topic independence. The GPT-3 model performs worse compared to all BERT combinations, and also suffers from drop in Dataset 2. We should note that the GPT-3 model that we use was trained on data collected roughly up the end of 2019 (Brown et al., 2020), so lacks knowledge about most of the topics in our data, which was collected later. serves, thus, as a good test for topic inde- pendence.
Model
Dataset
Dataset
We conclude that information about the conver- sational context is useful in pinpointing conflict, ad- ditionally contributing the type of contextual infor- mation that the model needs to be robust changes in topics and individual words.
LR Last Msg (Hla)
38.30
33,82
LR All Msg (HIb)
26.87
23.73
LR All Msg constr, tox (HIc)
38.30
24.81
BERT Last Msg (H2a)
94.58
85.26
BERT All Msg (H12b)
96.06
85.42
BERT All Msg constr, (I12c)
95.43
87.26
BERT All Msg cardinality (H2d)
92.83
89.36
Approximating agonism
BERT All Msg topology (H2e)
94.40
88.92
GPT-3 Fine-tune All Msg
91.95
85.85
We can attempt to approximate agonism well as other categories of these conversations given the three dimensions shown in Figure 3 by defining P^ as the Possibly Agonism Score, Pt as the Possibly Unproductive Score, and Ps as the Possibly Small- Talk Score as follows:
Table 3:
scores across datasets. constr construc- tiveness labels; tox toxicity labels
We can see from Table 3 that the simple logistic regression model (LR) results are quite low, but have drop for Dataset 2 that is comparable to that of the BERT models. There are no gains whatso- context,
PA = VIT - 0.0) +(S 100 (C 1.0)²
ever for the LR model from including in the form of knowing the prior messages in the conversation, or more information, such as the con- structiveness and toxicity labels. This might result from the absence of word sequence modelling in LR, which may hinder its ability to capture contex- tual dependencies between words.
Ps -==1- 1- -
1.0) (2)
P= 10-1-VT--000 (S- (C 0.0)2 (3)
where T is the Toxicity Likelihood, S is the Constructiveness Likelihood, and C' is the Conflict Likelihood.
The results from BERT are more interesting. First of all, simple model with just the last mes- sage shows F1 94.58, with 0.32 drop if topic words are not present. Including all messages helps considcrably, raising the score to its highest level, 96.06, but with an ever larger drop for Dataset 2,
Using these proxy scores, we can compare the ra- tios of unproductive versus agonistic conversations across different trending topics over time, shown in Figure 3. Part of our future work involves a

---

qualitative analysis of all the conversations in that green zone, investigate whether they have traces
Conclusions and future work
The long-term goal of our program of research is to identify agonistic conflict and distinguish it from less productive and democratic forms of conflict. The work presented here contributes definition of agonism and its operationalization in a coding schema, an important step approaching content moderation as a task of fostering agonistic dialogue, productive form of conflict that is essential to democratic dissent.
of agonism.
Furthering our analysis, we sampled the top 100 bidirectional conversations from the highest Pu and PA scores. A different set of annotators was then tasked with categorizing each conversation as either agonistic, antagonistic, neither. This re- sulted in inter-annotator agreement quantified by K 0.44, indicating moderate level of agreement. The distribution of resulting labels agreement shown in Table 4.
We introduce richly annotated dataset of online conversations containing conflict. Using this data, test methods that can identify conflict from versations, crucially incorporating contextual in- formation. We experiment with dimensions of the context that we believe can be proxies for agonistic conflict, including the presence of constructiveness and toxicity, the number of participants, and the topology of the conversation, which includes the level and direction of interaction. We show that the contextual information is key to identifying con- flict, especially because it helps the model remain topic-agnostic. This contextual approach can be helpful not just in identifying conflict and agonism, but also in detecting abusive language, as it pro- vides a wider view of the conversation, rather than whether an utterance is abusive or not in isolation.
Table Distribution of conversational labels
Label
Percentage
Agonistic 34%
Antagonistic 32%
Neither 35%
76.5% of the conversations coded as Agonism were sampled from the top 100 PA set, and 80% of the conversations coded Antagonism were sampled from the top 100 Py set. These findings suggest that while our proxy scores could mod- erately well at discerning between agonistic and antagonistic conversations, there is still room for improvement. Implementing secondary machine learning model on a larger dataset with labelled data from this secondary annotation exercise ap- pears to be promising next step in our goal to- wards modeling agonism.
We have made data and code available in repos- itory. 6 This includes: the dataset of 4,022 conversa- tions with annotations, the code collect conver- sations, the LabelStudio annotation scheme, and the code for all the experiments described. The appendices this paper include detailed informa- tion about the data collection process, the coding schema, and multiple examples of annotated con- versations from the dataset. The repository also contains links to raw data, larger dataset of con- versations collected but have not annotated (the entire dataset contains 220,626 conversations). We will, additionally, make available demo web appli- cation linked in the repository to experiment with the model results.
3D Scatterplot texicity, Constructiveness, and Conflict Likelihoods
088
Linethod
0.6
0.4
02
0.0
The next steps in our research program involve deploying the other dimensions in the data (us vs. them conflict; aadividuul/groupp conflict, etc.). We also plan to perform qualitative analysis of con- versations with high conflict, high constructiveness. and low toxicity, which we have defined as poten- tial zone of agonism. Further experiments will ex-
04
0.6 Execity Likethood
0.8
1.0
Figure 3: Conversations plotted by conflict, construc- tiveness, and toxicity likelihoods. We are postulating the green volume as the 'zone agonism'.
htt thub. com /Dig tal--eeocraciiss Institutee/Dimensionnss -fw.iioooooo-o-oonnnoonniii wnweo---OOnnnnnnooeee Conflict

---

tend this model to other topies in our larger dataset, test whether can be used for semi-automatic an- notation. We also plan to gather additional topics that generated discussion by querying keywords from monthly snapshots of the Wikipedia Por- tal:Current_ events and the Top_24 25_______ "epport through the Wayback Machine.
that we are the early stages of long and complex rescarch process.
Beyond identifying individual comments as con- structive, productive, or leading to agonism, it is also important to acknowledge the role of user and interface design in how comments are produced and presented (Masullo et al., 2022).
Defining and identifying agonism in conversa- tions is difficult task. We also acknowledge the difficulty of fostering the kinds of spaces that are conducive to agonistic debate, both online and of- fline, which is why an interdisciplinary approach with both quantitative and qualitative approaches such the one presented here is needed.
We also note more common limitations, includ- ing the source of the data (Twitter), the language of study (English), and the lack of demographic information about the participants in the discus- sions. We do not include information about lan- guage varicties and rhetorical strategies that may be characteristic of some online groups and linguis- tic communities and not in the mainstream. We do not know whether the conversations repre- sentative of some mythical mainstream culture of demographic groups with their own norms of debate and argumentation.
Limitations
Research on conflict consistently draws attention to its complex nature. Peace studies scholar Giorgio Gallo (2013) states that conflict can be character- ized by multiple, diverse, at times hidden, unde- fined, and evolving objectives. Thus, translating such a complex phenomenon into a set of labels output by a machine learning system is reductionist at best. Gallo notes that most research confict tends isolate it from its context, thus oversim- plifying the inquiry. We note that ours is one such simplification. The nature of online conflict, with long conversations unbounded by time and space limitations, unlike face-to-face communication and debate, renders more fine-graines and contextual- ized approaches impossible. We nevertheless at- tempt to incorporate context beyond the words in the individual messages message threads, by ex- amining features of the conversation, the number of participants, and the topology of the conversation.
Ethics Statement
We adhere the ACL Ethics Policy. In particular, we strive to contribute to socictal well- well-being and the public good by studying how conflict evolves in online conversations. We take the directive to avoid harms quite seriously. To avoid individual harm and respect privacy, we anonymize the tweets be- fore releasing them publicly (although the Twitter ids will provide link to the original). We also take into account duty of care for our researchers and annotators, and have provided them with opportuni- ties to debrief and protect their mental health when they read conflictual and hateful material.
We are concerned about the damage to the envi- ronment caused by training and fine-tuning large language models. We mitigated, in part, by using only pre-trained models. The BERT model was fine-tuned on a data center powered by hydroelec- tric energy, thus producing fewer CO2 emissions. Although our current system is far from perfect in detecting conflict, and we have not yet produced method to detect agonistic conflict, one risk of such systems lies in their misuse by employers, governments, or other social agents quell agonis- tic and productive conversations. For example, an employer may wish to suppress agonistic discourse that could lead to employee unionization.
Precisely because there is so much conflict on- line, dataset with about 4,000 conversations is not representative sample. The method of collection, where we started with topics likely to generate con- fliet (both hateful and agonistic), may also result in biased data. One alternative we could contemplate is to draw from datasets that have been labelled for toxicity, as those are more likely to contain conflict. Such data, however, does not necessarily contain agonistic conflict. Furthermore, as we mention in the conclusion section, we have not yet reached the stage of identifying agonism automatically. We hope that, by detecting conflict overall, we can ex- tract many instances automatically, leading us to a method for distinguishing antagonistic conflict from agonism. The main limitation of our study is
A Acknowledgements
We express our deepest gratitude to the coding team members who contributed to the success of this

---

Pranjali Jatinderjit Mann, Ashley Currie, Umer Hussain, and Judy Yac Young Kim. Addi- tionally, we are thankful for Zeerak Talat's assis- tance in developing and launching the project.
project:
speech datasets. In Proceedings the Twelfth Lan- guage Resources and Evaluation Conference, pages 6786-6794, Marseille, France. European Language Resources Association.
This research was supported by the joint contri- bution of the UK Economic and Social Research Council and the Canadian Social Science and Hu- manities Research Council. The study was con- ducted part of the "Responsible Al for Inclu- sive, Democratic Societies: A cross-disciplinary approach detecting and countering abusive lan- guage online" project Igrant number R/163157-11- IJ.
Giorgio Gallo. 2013. Conflict theory, complexity and systems approach. Systems Research and Behavioral Science, 30(2):156-11)
Debanjan Ghosh, Alexander R. Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755-792.
Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden deci- sions that shape social media. Yale University Press.
Tarleton Gillespie, 2022. Do not recommend? Reduc- tion as form of content moderation. Social Media & Society, 863):20533052221115555
References
Filip Boltužić and Jan Snajder. 2014. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop Argumen- tation Mining, pages 49 58, Baltimore, Maryland. Association for Computational Linguistics.
Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerrard, Robert Gorwa, Ariadna Matamoros- Fernández, Sarah Roberts, Aram Sinnreich, and Sarah Myers West. 2020. Expanding the debate about content moderation. Internet Policy Review. thttps://doi.org//// http:///00...0000000000..000
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learn- In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), volume 33, pages 1877 1901.
Roberto González- Ibáñiez Smaranda Muresan, and Nina Wacholder. 2011. Identifying sareasm in Twit- ter: A closer look. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581-586, Portland, Oregon, USA. Association for Computational Linguistics.
Yoonjung Choi. Yuchul Jung, and Sung Sung-Hyon Myaeng. 2010. Identifying controversial issues and their sub- topics in news articles. In Intelligence and Security Informatics: Pacific Asia Workshop, pages 140-153, Hyderabad. Springer.
Nancy Green. 2014. Towards creation of corpus for argumentation mining the biomedical genetics re- search literature. In Proceedings of the First Work- shop Argumentation Mining, pages 11-18, Bal- timore, Maryland. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient agreement for nominal scales. Educational and Psychological Mea- surement, 20( 203
Francesca D' Errico, Isabella Poggi, Alessandro Vincia- relli, and Laura Vineze. 2015. Conflict and multi- modal communication: Social research and machine intelligence. Springer.
Ivan Habernal and Iryna Gurevych. 2016. Which argu- ment is more convincing? analyzing and predicting convincingness of web arguments using bidirectional LSTM. In Proceedings of the 54th Annual Meet- ing of the Association for Compulational Linguistics (Volume I: Long Papers), pages 1589-1599, Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Catherine Han, Joseph Seering, Deepak Kumar, Jef- frey T Hancock, and Zakir Durumeric. 2023. Hate raids Twitch: Echoes of the past, new modali- ties, and implications for platform governance. arXiv preprint 2301
Evelyn Douek. 2022. Content moderation as systems thinking. Harvard Law Review, 136(2):526-60) 607.
Laura Hanu and Unitary Team. 2020. Detoxify, https: //gi tgitthu..com
Paula Fortuna, Juan Soler, and Leo Wanner. 2020. Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate
Randy Allen Harris and Chrysanne Di Marco. 2017. Rhetorical figures, arguments, computation. Argu- ment & Computation, 8(3):111-31.

---

Anushree Hede, Oshin Agarwal, Linda Lu, Diana Mutz, and Ani Nenkova. 2021. From toxicity in online comments incivility in American news: Pro- ceed with caution. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 2620-2630, Online. Association for Computational Linguistics.
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli, 2021. Agrecing to disagree: Annotating offensive language datasets with annotators' dis- agreement. In Proceedings of the 2021 Conference on Empiriaal Methods in Natural Language Process- ing, pages 10528-10539, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Christopher Hidey and Kathleen McKeown, 2018. Per- suasive influence detection: The role of argument sequencing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 5173 5180, New Orleans, LA.
Nikola Ljubešić, Igor Mozetič, and Petra Kralj Novak. 2022. Quantifying the impact of context the qual- ity of manual hate speech annotation. Natural Lan- guage Engineering, First View:1-14.
Graeme Hirst, Vanessa Wei Feng. Christopher Cochrane, and Nona Naderi. 2014. Argumentation, ideology, and issue framing in parliamentary discourse. In Proceedings of the Workshop on Frontiers and Con- nections between Argumentation Theory and Natural Language Processing, Forli-Cesena, Italy.
Sean Mac Hao-Ren Yao, Eugene Yang. Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PLoS ONE, 1428)22022
Michel Marcoccia. 2004. On-line polylogues: Conver- sation structure and participation framework in inter- net newsgroups. Journal of Pragmatics, 36(1):115- 145.
Yibo Hu, MohammadSaleh Hosseini, Erick Sko- rupa Parolin, Javier Osorio. Latifur Khan, Patrick Brandt, and Vito D' D'Orazio. 2022. ConfiBBRRT: A pre-trained language model for political conflict and violence. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx; Human Language Tech- nologies, pages 5469-5482, Seattle, United States. Association for Computational Linguistics.
Gina M. Masullo, Tamar Wilner, and Natalie Jomini Stroud. 2022. What social media could be: Norma- tive frameworks for evaluating digital public spaces, Social Media + Society, 8K4):2056305222113044
Raquel Mochales and Maric-Francine Moens. 2011, Ar- gumentation mining. Artificial Intelligence and Law, 19(1))1-22.
Anatol Valerian Itten. 2019, Overcoming social divi- sion: Conflict resolution times of polarization and democratic disconnection. Routledge.
Corrado Monti, Luca Maria Aiello, Gianmarco De Fran- cisci Morales, and Francesco Bonchi. 2022. The language of opinion change on social media under the lens of communicative action. Scientific Reports, 17(1):17920.
Mali Jin and Nikolaos Aletras. 2021. Modeling the severity of complaints in social media. In Proceed- ings of the 202/ Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2264-2274, 2274, Online. Association for Computational Linguistics.
Garrett Morrow, Briony Swire-Thompsone Jessica Mont- gomery Polny, Matthew Kopec, and John P. Wihbey. 2022. The emerging science of content labeling: Contextualizing social media content moderation. Journal of the Association for Information Science and Technology, 13(10):1365-1333
Aniket Kittur, Bongwon Suh, Bryan A Pendleton, and Ed H Chi. 2007. He says, she says: Conflict and coordination in Wikipedia. SIGCHI Conference on Human Factors in Computing Systems, pages 453 462, San Jose, CA.
Proceedings of the
Chantal Mouffe. 2002. For an agonistic public sphere. In Okwui Enwezor, editor, Democracy unrealized: Documenta 11, Platform 1, pages 87-97. Hatje Cantz.
Varada Kolhatkar. Nithum Thain. Jeffrey Sorensen, Lu- Dixon, and Maite Taboada. 2023. Classifying constructive comments. First Monday, 28(4):https: //doi. wrg.//gg////0 10. 5210/5111mm ffm. v28i4.111111 13163.
Chantal Mouffe. 2013. Agonistics: Thinking the world politicully. Verso.
Adrienne P Lamberti and Anne R Richards. 2019. Com- munication and conflict studies: Disciplinary con- nections, research directions. Springer.
John G Oetzel and Stella Ting- -oomeyy 2006. The Sage handbook of conftict communication: Integrating the- ory, research, and practice. Sage.
J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):159-177 174.
Vilviu Vlad Oprea and Walid Magdy. 2020. The ef- feet of sociocultural variables on sarcasm communi- cation online. Proceedings of the ACM on Human- Computer Interaction, CSCW, Arrtile 29.
John Lawrence and Chris Reed. 2019. Argument min- ing: A survey. Computational Linguisties, 45(4):765- 818.

---

Pia Pachinger, Allan Hanbury, Julia Neidhardt, and Anna Planitzer. 2023. Toward disanbiguating the definitions of abusive, offensive, toxie, and uncivil comments. In Proceedings of the First Workshop on Troxs-Cultural Considerations in NLP (C3NLP), pages 107-113, Dubrovnik, Croatia.
Dialogue, pages 217-226, Prague, Czech Republic. Association for Computational Linguistics.
Bertie Vidgen and Leon Derezynski. 2021. Direc- tions in abusive language training data, a system- atic review: Garbage in, garbage out. PLoS ONE, 15(12):c0243300.
Andreas Peldszus. 2014. Towards segment-based based recog- nition of argumentation structure in short texts. In Proceedings of the First Workshop on Argumentation Mining, pages 88-97, Baltimore, Maryland. Associa- tion for Computational Linguisties.
Charles Welch, Chenxi Gu, Jonathan K. Kummerfeld. Veronica Perez-Rosas, and Rada Mihalcea. 2022. Leveraging similar users for personalized language modeling with limited data, In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1742-1752, Dublin, Ireland. Association for Compu- tational Linguistics.
Jacques Rancière. 1999. Disagreement: Polities and philosophy. University of Minnesota Press.
Jacques Rancière. 2010. Dissensus: On polities and aesthetics. Bloomsbury.
Mark Wenman. 2013. Agonistic democracy: Con- stituent power in the era of globalisation. Cambridge University Press.
Haji Mohammad Salcem, Kelly P Dillon, Susan Be- nesch, and Derek Ruths. 2016. web of hate: Tack- ling hateful speech in online social spaces. In Pro- ceedings of Text Analytics for Cybersecurity and On- line Safety (TA-COS), LREC, pages 1-9, Portoroz.
Sarita Yardi and danah boyd. 2010. Dynamic de- bates: An analysis of group polarization over time on Twitter. Bulletin of Science, Technology & &Society, 30(5):316-32) 332-377..
Nathan Sanders, Pocholo Umbal, and Lex Konnelly. 2020. Methods for increasing equity, diversity, and inclusion in linguistics pedagogy. In Proceedings of the 2020 Anmual Conference of the Canadian Lin- guistic Association, pages 1-13, Online.
Jing Zeng and D. Bondy Valdovinos Kaye. 2022. From content moderation to visibility moderation: A case study of platform governance on TikTok. Policy & Internet, 14(1):79-95.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-590 Seattle, United States, Association for Computational Linguistics.
Justine Zhang, Jonathan Chang, Cristian Danescu- Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain. 2018. Conversations gone awry: Detecting early signs of conversational failure. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1350-1361 Melbourne, Australia. Association for Computational Linguistics.
Flaine Zosa, Ravi Shekhar, Mladen Karan, and Matthew Purver. 2021. Not all comments are equal: Insights into comment moderation from a topic-aware model. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). pages 1652 1662, Held Online. IN- COMA Ltd.
Josh Smith, Alex Krasodomski-Joma Maria Ola- nipekun, and Ellen Judson. 2021. A picture of health: Measuring the comparative health of online spaces. Technical report, Demos
Kumar Bhargav Srinivasan. Cristian Danescu- Niculescu-Mizil, Lillian Lee, and Chenhao Tan. 2019. Content removal as moderation strategy: Compliance and other outcomes the Change- My View community. In Proceedings of ACM Conference on Sluman-Computed Interaction, CSCW, page Article 163, Glasgow, UK.
A Appendix: Search keywords
The following are the list of keywords used for the search described in Section 3.1.
Christian Stab and Iryna Gurevych. 2017. Parsing argu- mentation structures in persuasive essays. Computa- tional Linguistics, 43(3):619-659,
(1) #hillel 1, #cancon, #CRTC, #onlinecemsorship. #online- harmsreduction, #StopAsianHatt #stopC11, 1, bean dad, bell books. biden loans, bill gates divorce, canadian truck- ers, capitol insurrection, china gamer ban. coastal gas link, convoy, defund the police, depp heard, el salvador bitcoin, elon twitter, evergrande, gamestop, groomers. hho test, india pakistan missile, ivermectin, libsoftiktok, metaverse. ocean fire. online harms, pandora papers, re- aldonaldtrupp. rittenhouse, rohb elementary, rec, rogers, sci_ Jub, social distancing, suez canal, taliban, tigray, tmx pipeline, west elm caleb, will smith slap, vaccine
Jonathan Stray. 2022. Designing recommender systems to depolarize. First Monday, 27s5::htps://doi. org/10. 5210/fmm v2715. 12684.
Reid Swanson, Brian Ecker, and Marilyn Walker. 2015. Argument mining: Extracting arguments from online dialogue. Proceedings of the 16th Annual Meet- ing of the Special Interest Group on Discourse and

---

B Appendix: Topic distribution over time
defund-the-polle police, defund
defund-the ttth--ooice-realoonnddtreee
police, realdon-
Figure 4 shows the number of conversations related to each topic over time. We can see that certain topics exhibit more enduring presence in the dis- course (Social Distancing), while others appear to be more transient, capturing attention for only day or two (Will Smith Slap, Rogers Outage). Each topic was sampled from its peak of discussion. We have observed that topics with high peaks (Will Smith Slap or India Pakistan Missile) tend to have a higher level of conflict than topics that last over longer periods of time.
aldtrump, defund
depp-heard: depp, heard, amber, johnny
elon-twitter: twitter, elon, musk, elonmusk
groomer: govrondesantis, groomers, travlingsnowman
india- madia-pakistnnmmisste pakistan, india, indian
libsoftiktok: libsoftiktok, taylorlorenz
metaverse: metaverse, crypto, bsc
protest-soccial-iite distaniiis protests, distancing
queen-elizabeth: us
roe-v-wade; roe, abortion, wade
russia-ukraine: ukraine, russia
salman-rushdic-atienn rushdic, salman
social-distancing: distancing
tigray ggray-cthiopia: tigray, ethiopia, tplf
vaccine: vaccine, covid
C Appendix: Summary statistics for top topics
E Appendix: Coding schema
Content warning: Examples in this section con- tain offensive language.
Table 5 presents summary statistics for the top top- ies (a total of 2,718 conversations, out of the 4,022 in the dataset). The first column shows the total number of conversations in that topic. The next few columns display the percentage of conversations in that topic with the label (e.g.. 62% of the Social Distancing conversations contained conflict, and 11% had sarcasm). We see that most topics con- tained some form of conflict (making the dataset unbalanced), but that presence of other dimensions of conffict varies widely. Using the percentage of test for statisti-
Annotators were presented with a LabelStudio interface that provided: the tweet annotate, the previous 3-7 messages in the conversation, and the labels choose from, as seen in Figure 5. Labels were always for the last tweet the conversation, but in the context of the entire thread. Here, we elaborate on the descriptions for each annotation decision, from the schema in Figure 1.
E.1 Is there confliet?
conflict per topic in Table 5, cally sigmificant differences in conflict rates across topics. Assuming non-normality and non-equal variances, we can use the Kruskal- Sruskal-Wallis test defined as follows:
A yes/no answer, based on the definition of conflict provided (see Section 2.1). Annotators were in- structed to label only the last message (e.g., the last tweet in Figure 5), but use the context (the previous tweets) if necessary. A 'yes" answer triggers all the decisions below.
12
R² Σ ny
H N(N 11)
3(N 11
(4)
E.2 Internal/external
If the annotators decided that there was conflict, they had to label the conflict internal the versation ('when it involves people/entities directly engaged in the conversation') ') or external. For ex- ample, if people are discussing Black Lives Matter as an organization, then the conflict is external. But if they discuss people in the conversation involved in BLM, then it's internal. A tweet can be both internal and external, so both labels are allowed. This label had relatively high level of agreement (see Table 1), so we feel this was valid distinction.
This calculation obtuined p-value < .005. We can confidently say that topics are clearly differ- ent in their rate of conflict, with discussions about Biden's 's loan forgiveness program being 100% con- flictual, and conversations about Facebook's Meta- verse being low in conflict (8% of the conversations in that topic had conflict).
D Appendix: Filtering out keywords
The following is the list of keywords removed from each topic as described in Section 3.1.
Examples (2) and (3) show instances of each.
Jan Jan-6th-ine 6th-insurrection: capitol, insurrection, trump
(2) [Internal] You are going to arrst me for standing 4' away from somcone? Go for it
TMX- ppppeinneccosttalgaattte
pipeline, tmx, trudeau
biden-loans: loans, student, biden
canada-day: canada
(3) [External] Honestly this point second wave is gonna have to happen for it get through their THICK SKULLS that they completely and utterly fucked this up
camadian-truckers truckers, canadian, convoy
canadian-truckers, protests: truckers, protests, canadian, canada

---

Distanding
Will Smith
Ward
Eion Twitter
Depp Heard
Libsoflkttkk
Biden Leans
Delund the Police
Wade
Tirry Ethiopla
Varcine
Rogers Outage
India Missle
7025-3001
z0200901
20210301
20210901
2022-2901
00210001
20234301
Figure 4: Volume of messages for some of the top topics over time
Topic
N
Conffict
Sarcasm
Metaphors
Directives
Target
Social Distancing
615
0.62
0.11
0.36
0.29
0.40
Will Smith Slap
320
0.97
0.20
0.76
0.05
0.96
Elon Twitter
268
0.90
0.23
0.67
0.14
0,91
Depp Hcard
233
0.97
0.11
0,70
0.08
0.95
Canadian Truckers
230
0.97
0.10
0.67
0.15
0,53
Libsoftiktok
191
0.97
0.18
0.55
0.09
0.80
Biden Loans
190
1.00
0.13
0.72
0.23
0.78
Defund the Police
157
0.94
0.06
0.41
0.42
0.33
Metaverse
131
0.08
0.18
0.50
0.05
0.59
Roe Wade
91
0.95
0.10
0.38
0.19
0.43
Tigray Fthiopia
90
1.00
0.06
0.68
0.40
0.31
Vaccine
76
0.89
0.20
0.24
0.30
0.45
Rogers Outage
69
0.48
0.15
0.20
0.30
0.41
India Pakistan Missile
57
1.00
0.21
0.61
0.12
0.46
Total/averaae
2.718
0.84
0.12
0.49
0.16
0.56
Table 5: Top topics and percentage of conversations in those topies that given select annotation labels
E.3
Who/what is the target?
(6) [Policy/insti tutions] That assumption is based on a busy car park. With miles of beach am sure it's easy to stay far enough away from others. Visiting Tesco's is 100x more dangerous
The target of the conflict can be more than one, so multiple labels are possible here. Some examples are presented in (6)-(6).
Individual
E,4 Authority evoked
Group
Policy/institutions
This question asked whether the position, stance, claim is recognized, consistent with, legitimated, supported by some form of authority. Annotators were instructed to consider how different forms of authority are invoked by the participants, again focusing on the last tweet the conversation.
Systemic issue or cause
Events
Other (specify)
[Individual]] Cummings should be sacked! And prose- cuted criminal law.
(5) [Group] Its unreasonable and unrationale to say that we can't let emotions take over. Imagine being a black person in America right now. We should be angry, we shouldn't be pushing down or hiding our feelings. We want justice, everything else is not important right now.
Personal experience. This could be an individ- ual encounter, an understanding, an insight, usually derived from proximity and epistemic positions.

---

This label had very low level of agreement with kappa value of 0.02. We believe the explanation of this distinction, as shown above, was unclear to annotators, perhaps also because these oppositional relations are not often explicitly stated. Some ex- amples are provided below. Note the dual label in (10).
ggrerondherred hernectinss the govermme in is only
money people get the you...
1eseone
whars
Iremectinn
toverdiive_nn (rremments geit120 (B ganenctthwead not nacoine thane 5125 a8llss vaconies
Is there conflict?
(10) [Us above them], [Us/them disjuntive] idk why y'all continue to defend people who prove themselves over and over again the rich will always find ways to hide their money and avoid paying their dues, if we continue to let them
Noa
Figure 5: Annotation interface
(11) [Us/ disjunctive] Please sign our petition to fire RCMP Commissioner Brenda Lucki here, and mayhe we can finally get some accountability for an organization that has gone completely off the rails, Keep fighting for what is right,
Common sense. Authority evoked by refer- ring to common sense knowledge, some sense of what is shared in the specific or cultural context (regardless whether common sense is actually true).
E.6 Rhetorical strategy
Factual/distant expertise institution. Par- ticipants support their statements by referring to experts, facts, or (statistics provided by) institutions.
A 'yes' answer here means that the annotator saw persuasion or appeals to sensibilities and mean- ings that were grounded in linguistic techniques, language moves, or other linguistic mechanisms. If the answer was 'yes', then they were asked to specify what type of strategy was deployed:
Authority is rejected. Participants reject or challenge authorities invoked by others.
Other (specify)
Explicit directives and calls to action Associations, metaphors, or analogies
(7) [Personal experience] Pretty sure for my son to go to boarding school he needed proof of vaccinations
Sarcasm
[Common sense] The mothers who raise these men play role. They, too, are responsible. Many them support the Taliban. It naive to believe otherwise
Other (specify)
(12) [Explicit directives], [Sarcasm] Yo ACLU and Amnesty International, calm down. Why sudden panic about "free speech" when Elon wanna buy Twitter?
[Factual/ distannt expertise indications are that the practice of with social dis- tancing have profound effect on population control. Perhaps even larger that what 'the pill' had in its day when it first came out.
institution! All
(13) [Associations, metaphors, or analogies] So seems like evergrande stroke deal which is good for now (kicking can down the road) and news just came out of China about reducing coal. Now just the Fed gotta worry about. Besides the fact that charts are looking ugly.
E.5 Groups being discussed
This is a binary question, asking whether the con- versation involves clearly identified groups or fac- tions, presented as an attempt to render conflict as isolated incompatibility between groups. If the an- swer is 'yes', this triggers another set of questions, about the relationships among those groups and how they are being discussed:
(14) [Sarcasm] So doxxing people who disagree with you OK, but not doxxing people who agree with you. Got it.
E.7 Meta questions
Finally, regardless whether the answer to the con- flict question was yes or no, annotators were asked three further questions, about context and about their own reaction.
Us above them
Us below them
Us/them conjunctive. The two or more groups being discussed involved in the conversa- tion are presented being allied, connected, or somehow related.
The first question was whether more context needed, beyond the thread. As we see in Figure 5, annotators were asked to label the last message and were allowed to read the previous messages on the screen. But they also had the possibility to click on the tweet and look at the context on
Us/them disjuntive. The two or more groups are presented as not allied, connected, or re-
lated.

---

the Twitter platform, including any media. If they clicked, they were asked answer "yes" to this question. Further, they could specify what kind of context was needed:
Cultural. The annotator needed to know more about the issue at stake. This could be current news topics, subcultural trends online, or other aspects of the topic.
Limer.. Soundhilin gevernmnt handost #hyaighh net
douluge go ARpecer- hmppeph
GregCanetson1 Gopendo Vast, Am hearnn someecty unemptoment senetiaa tolkss rerypleeed
mendare
Conversational. The conversation was miss- ing some elements, perhaps earlier than the messages included in the annotation platform, which made difficult to interpret.
again
yourre
upendo @GregGeneeent Nult wew larning CoP talking aroundl point. gandor.
spendo gGregfaneeent The sartaeli quote Wilsy.. Mussgl secceciasty fnam..ee
Mancciin regunting crean.
Missing content. Tweets in the thread had been deleted.
CregCametoon gispendo Us CoP was entical sh thutting down economy and ftlkks York. Rlog came mak-its new normel SCIENGE Now dems want people hed takingg barre sturtionns
Media. Videos images in the thread were not available, but seemingly necessary for in- terpretation.
Figure 6: Conversation, with labels shown in Figure
The second meta question asked about the emo- tional reaction of the annotator, simply asking 'How did you feel when reading the conversation?"
Some suggestions were provided:
Shock
Sadness
Disgust
Anger
Fear
Confusion
Indifference
Entertained
Hopeful
Grep
The third and final question was about level of confidence ('How confident are you about your analysis?'). Annotators were provided 1-5 scale, with being not confident all (more context was needed, tweets had been removed, tweet was inde- cipherable), and 5 being a high level of confidence or that the interpretation was straightforward.
Syukamo
Yes
Authority evoked?
Groups discuased?
Unltter cipunctiv
there conflic??
Rhetorical strategy?
Aecriatoo, meap
E.8
Full annotation
Was context needed??
No
Figure displays message (last message) and its context, to show the richness and complexity of the annotation scheme. Labels for the annotation of this example given in Figure 7.
How did you feel?
Livel confidence
Figure 7: Full annotation for conversation in Figure


Reviews:
[{"review_id": "4Tniccy8Tp", "reviewer": "Reviewer_APKa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the concise summary of the paper’s contributions and the appreciation for how well the problem was defined. \n\nWe agree that agreement for some of the annotations is low. We’d like to point out that the last two lines in Table 1 are meta-questions, that is, not true annotation labels. Those meta-questions, \"Context - Media\" and \"Context - Cultural\" are not meant to be included as target labels for model training, but rather serve to gauge the annotator's opinion about the need for context to understand a given conflict. This could have been explained more clearly in the paper, so we will correct this in the revisions. \n\nNevertheless, some of the other annotation labels have a relatively low level of agreement. This highlights the difficulty of the task for humans, which will likely make it a very difficult task for any NLP system. We believe the reasons for the low agreement in some of the categories are twofold: \n\n1. Complexity of the task. The more abstract aspects, such as \"Groups discussed\", are challenging to discern with the limited context available in online conversations. Rhetorical strategies are notoriously difficult in annotation, especially sarcasm. This is well-documented in existing literature and it is a research problem in its own right (Sanguinetti et al., 2018; Castro et al., 2019). \n\n2. Instructions and training. Annotators were told to focus on the conflict and the main aspects of the labelling (internal/external; target, etc.), and the more complex aspects may have received less attention. Joshi et al. (2019) report a wide range of inter-annotator agreement in sarcasm detection and suggest that framing in the guidelines is quite important. Our guidelines focus on the annotation of conflict and the high-level aspects. The annotation of rhetorical strategies received less attention and training. \n\nIn any case, we only use the “conflict” label for the experiments, and “conflict” has a moderate agreement level. The main aspects of the labelling also have a moderate level of agreement. It's only when we reach the third-level classification (the terminal nodes in Figure 1) that the agreement decreases significantly. \n\nIdeally, we would have extended the annotation with clearer instructions and more data, as suggested by the reviewer. At the moment, we are in a difficult transition time, as Twitter/X changes its policies for research access and researchers are re-evaluating how much more data can be obtained from that platform. \n\nTo address this issue, in the last month, we have developed a data collection platform that allows an individual researcher to save their Twitter/X reading data, as they scroll. We also plan to make the data and annotations not only available to all, but also editable. Users of the dataset can add their own annotations and comments, thus enlarging the data and contributing to higher inter-annotator agreement. The new annotations will be clearly marked to differentiate what is presented in this paper as a stable dataset from additions by the community. \n\nWe will use the extra page in the final version to describe the new data collection platform and the process for adding annotations.\n\nWe nonetheless believe this dataset, in its current format as used in the submitted version of the paper, is both rich and includes reliable labels for the top categories.\n\nWe are planning a follow-up study to perform qualitative analyses of the data, which will include refining the labelling and further work on identifying agonism. If this paper is accepted into EMNLP, it will form the theoretical basis and main dataset from which to launch future work on identifying agonism vs. antagonism.\n\nThank you for pointing out the need for more clarity in the figures. We will re-draw them to improve readability. \n\n**References:**\n\nCastro, S., D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria. 2019. Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629, Florence, Italy. Association for Computational Linguistics.\n\nJoshi, A., P. Bhattacharyya, and M. Carman. 2017. Automatic Sarcasm Detection: A Survey. ACM Comput. Surv. 50, 5, Article 73 (September 2018), 22 pages. https://doi.org/10.1145/3124420\n\nSanguinetti, M., F. Poletto, C. Bosco, V. Patti, and M. Stranisci. 2018. An Italian Twitter Corpus of Hate Speech against Immigrants. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The introduction provides a clear overview of the paper's subject matter, introducing the concept of conflict online and its significance in democratic dialogue. The distinction between agonism and antagonism is well-defined, setting the foundation for the subsequent research.\n\n\nThe writing style is clear and concise, making the paper easy to follow.\n\n", "reasons_to_reject": "The level of inter-annotator agreement for crucial aspects of conflict annotation is relatively low. While a moderate to substantial agreement was achieved for the binary conflict label (i.e., presence or absence of conflict), the lower agreement for other features, such as target, context, and rhetorical strategies, raises concerns about the reliability and consistency of the annotations.\n\nThe paper does not provide sufficient analysis or discussion regarding the reasons for the lower agreement on specific features.\n\nTo address this issue, the paper could explore potential strategies for improving inter-annotator agreement. This might include refining the annotation guidelines, or conducting additional training sessions to enhance consistency.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on detecting and differentiating two types of conflict to aid in platform moderation and content monetization. Twitter conversations related to trending controversial topics were collected, and a annotation schema was introduced to label various conflict dimensions, including the source, target, and rhetorical strategies employed. Logistic regression and transformer-based models, incorporating conversation context, were trained on the annotated dataset. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figures should be modified to be transparent"}, "extra_text": {}}, {"review_id": "ofPBEm3Dno", "reviewer": "Reviewer_bZMt", "summary": "", "questions": "I would like the authors to elaborate more on my doubts regarding the gap between motivations and experiments.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments about the organization and clarity of the paper. \n\nAlthough it is true that there is a difference between predicting conflict and having a better understanding of the causes of the conflict, we've argued that establishing a baseline for being able to successfully detect conflict is a necessary first step in the direction of understanding the nature of conflict. The paper lays out the overall problem and contributes to this understanding with the dataset and an initial set of experiments. As reviewer KsvZ points out, the data annotation followed a robust process that we believe is an important contribution to the field.\n\nAs we have discussed in our response to Reviewer APKa04, we will make the motivation more explicit and explain why the paper does not quite get (yet) to fully modelling agonism, because this is the first in a series of steps. Further, we felt that we needed a lot of space to document the problem and the dataset. We hope that this dataset and the lessons learned from our methodology can be used to help others working on a similar goal, and believe this is the kind of work that fits well in EMNLP, because it can advance the field by providing theoretical grounding and gold data for future experiments. \n\nWe will re-draw the figures to improve readability. \n\nFinally, could you elaborate on what the technical or methodological problems are specifically? We are also not sure how to make the hypotheses easier to track. They are described in Sections 4.1 and 4.2, each in bold font, with corresponding results in Table 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-organized and easy to follow.\nThe paper contains a good amount of discussion on the role of conflicts played in online discussion. Studying the problem of differentiating whether conflicts are agonistic or hateful is very interesting.\nThe authors contribute a dataset for conflict study with detailed description of their methodology and annotation schema. \n ", "reasons_to_reject": "The paper is motivated for better understanding the natural of conflicts with discussion on the multiple roles of conflicts, e.g., fostering perspective or producing hate. However, the data and experiments are mostly about a binary prediction task on conflicts. I think there's a gap between the motivation and the experiments.", "questions_to_authors": "I would like the authors to elaborate more on my doubts regarding the gap between motivations and experiments.", "paper_topic_and_main_contributions": "This paper investigates modeling Agonism using Twitter data. The paper is motivated by the role of agonism in demoncratic dialogs and the need of better understanding on the difference of agonism vs hate. The authors contribute a dataset with a methodology for conflict annotation. Using the data, the authors conduct a set of experiments to verify their hypothesis. The results show contextual information is crucial for conflict identification. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The hypothesis is hard to track. It would be better if the authors can add a list of all the hypothesis.\nFigure 1 and 3 are blurry. \nThe text in Figure 1 is hard to read."}, "extra_text": {}}, {"review_id": "P5Vrx2z4z9", "reviewer": "Reviewer_KsvZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a clear and encouraging review. We agree that it may be misleading to spend time in sections 1-3 talking about agonism, when we do not fully address the concept with our current findings. However, we believe it is important and valuable to establish the theoretical foundations of the work in conflict theory and communication. To clarify that this is the first step towards a longer-term project, we will reorient the introduction, while keeping the important explanations about the distinction that are of value to the current findings. \n\nAs we mention in response to Reviewer APKa04, we are also in the process of enhancing the dataset, which includes our creation of a new data collection platform for Twitter/X, given the recent changes in access to their APIs. We will add an explanation of this new method/process to the paper if accepted. \n\nIn our other responses, we also clarify how we will improve readability and expand on the motivation for the research project.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The motivation to distinguish between productive and unproductive conflict will be of wide interest.\n2) Robust data annotation processes were used. \n3) Potential limitations are clearly highlighted. \n", "reasons_to_reject": " The first few sections of the paper seem to indicate that distinguishing between agonism and antagonism is the goal of the present paper. So it is a little disappointing to get to section 4 and realize that this paper is an early, somewhat speculative step toward that end. This could be cushioned a bit by making this clearer in sections 1-3. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents work on identifying the presence of conflict within Twitter data as an early step toward automatically distinguishing between productive, good faith conflict and conflict that is unproductive, hateful, etc. The primary contribution of this paper appears to be the creation of a dataset with useful human annotations, some early results regarding prediction of conflict, and some ideas for distinguishing between agonism and antagonism in future work. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two  (APKa, KsvZ) reviewers feel strong excitement about this paper and find the soundness of the study to be quite sufficient in supporting its major claims. Reviewer bZMt urged authors to better clarify the paper’s seeming disconnect between the motivation and the operationalized tasks. Overall, there are some few issues that can be better addressed in the final version of the manuscript as suggested by the authors 1. Clarifying theoretical motivation in intro; tying back findings to theoretical motivation / terminology/ definitions (e.g., agonism), 2. Addressing low annotator agreement, 3. Making the motivation more explicit in connection to the tasks. 

The authors address the reviewer questions/ concerns quite sufficiently in their rebuttal.
Decision: Accept-Findings