Unveiling the Power of Argument Arrangement in Online Persuasive Discussions
Authors: Nailia Mirzakhmedova
Johannes Kiesel
Khalid Al Khatib
Benno Stein
Venue: EMNLP 2023 Findings (2023)
Keywords: Argument Mining
Persuasive Dialogues
Persuasion Strategies
ChangeMyView

Abstract: Previous research on argumentation in online discussions has largely focused on examining individual comments and neglected the interactive nature of discussions. In line with previous work, we represent individual comments as sequences of semantic argumentative unit types. However, because it is intuitively necessary for dialogical argumentation to address the opposing viewpoints, we extend this model by clustering type sequences into different argument arrangement patterns and representing discussions as sequences of these patterns. These sequences of patterns are a symbolic representation of argumentation strategies that capture the overall structure of discussions. Using this novel approach, we conduct an in-depth analysis of the strategies in 34,393 discussions from the online discussion forum Change My View and show that our discussion model is effective for persuasiveness prediction, outperforming LLM-based classifiers on the same data. Our results provide valuable insights into argumentation dynamics in online discussions and, through the presented prediction procedure, are of practical importance for writing assistance and persuasive text generation systems.

Unveiling the Power of Argument Arrangement in Online Persuasive Discussions
Nailia Mirzakhmedova Sunhaus-Unnversitae Weimar
Johannes Kiesel
-niivrs-Uniivessiätt Weimar
Khalid Al-Khatib University of Groningen
Benno Stein Bauhaus- Bauhaus-Universittä Universität Weimar
Abstract
PP
CMV: Today is best time period human history be allve the vast majortty of people.
With all the negativlly, really noad Incus the data and the posalive[]]
Previous research on argumentation in online discussions has largely focused on examining individual comments and neglected the interac- tive nature of discussions. In line with previous work, we represent individual comments as se- quences of semantic argumentative unit types. However, because it is intuitively necessary for dialogical argumentation address the oppos- ing viewpoints, we extend this model by clus- tering type sequences into different argument arrangement patterns and representing discus- sions as sequences of these patterns. These sequences of patterns are a symbolic represen- tation of argumentation strategies that capture the overall structure of discussions. Using this novel approach, we conduct an in-depth analy- sis of the strategies in 34,393 discussions from the online discussion forum Change My View and show that our discussion model is effec- tive for persuasiveness prediction, outperform- ing LLM--ased classifiers on the same data. Our results provide valuable insights into argu- mentation dynamics in online discussions and, through the presented prediction procedure, of practical importance for writing assistance and persuasive text generation systems.
almost not single important data point you can
Bay is trending up over time[]] Wars, fammnn, human rights. healthesee) inclisionn Intant mortaity]R] The lsst goas on and on and on. The only big change had boon access negative intarmation(]] but the actual texture of the world gotten better
time[]] What missing?
I'm looking for someone to point out the flow in my legic. The only thing can humen nature is supposed this connected to constant intormation[]]. that's easy to
change my
view.
U1
Currently in the devaloped wordd we are living through an unprecedinns ed mental health epidemic with skyrocketing suisides[V] Cost of living is! outstripping eernings for the majortty workers and houses are becaming expansiva[]] that samaane national mean, can longer tive th city g up in My generation the first that is predicted not to live as long us the previous generation on average(F) and all of this is taking place to baackrrop of the exciteentaa thvsst that is climata changelVV] The bast time period! longer
passed recentlylV)
The mental health crises I think real for that
U2
The best time easily sarly 2010s up 2017[V], sinca than we've seen
civil rights have been roliod back) Plenty of societios have collapsed beforelF). there's no reason ours sant(y)) So no, today abselltely not the beet time to be alive[V] We're in backwards shift. hooentully we can recover but can haintyy assume we will without activaty addtrassing the
that have
banwwards dactiine
OP What are you talking about?![R]] Becauso atrochies..
[F] act
[V] alue
[P] olicy
[T] estimony
[R] hetorical statement
1 Introduction
Figure 1: Illustration of one Change My View thread with two discussion branches (with commenters U1 and U2, respectively) and detected ADU types. The original poster's (OP) comment with A in the first discussion branch marks that branch as persuasive.
Convincing others can be quite challenging, even when equipped with a comprehensive set of argu- ments. The questions then arise: what kind of argu- ments are the most convincing and which should be presented first? Should one begin with facts or per- sonal experiences? The different answers to these and related questions are referred to as argumen- tation strategies (Al-Khatib et al., 2017). Several studics have empirically examined the arrangement of argumentative discourse unit (ADU) types, such facts or testimonies. in monological texts (e.g.. Al-Khatib et al. (2017)), or in individual comments within dialogues (e.g.. Hidey et al. (2017); Morio et al. (2019)). These studies have shown that the
arrangement of ADU types in a text can serve as a model of the argumentation that underlies the text, for example for predicting its persuasiveness.
In dialogues, such as the two discussion branches shown in Figure 1, it seems intuitive that there is no single best argumentation strategy for either side. Instead, the strategy needs to be dynamically adapted in response to the ongoing conversation. For instance, if the opponent concludes their ar-

---

gument with fact, countering them with another fact before suggesting policy might be more con- vincing. Notably, this dialogic nature has been mostly overlooked in the computational analysis of argumentative discussions so far.
for ADU types using a new approach that out- performs the previous state-of-the-art by 0.18 in terms of F1 score (Section 4). Moreover, we use our model to identify clusters representing the ar- rangement strategies used in these discussions, and show the utility of cluster sequenees for argumenta- tion analysis through the example task of predicting persuasiveness (Section 5).
In this paper, examine the nuances of ongo- ing argumentation in dialogues with the goal of un- derstanding the particular elements that contribute the success of particular debaters. Through a detailed analysis of ADU sequences, we seek to re- veal the underlying patterns and strategies utilized by skilled debaters. These strategies are essen- tial for advancing both theoretical understanding and practical application. Theoretically, they par- ticularly improve our understanding of persuasive dialogues, enabling the refinement of skills among novice debaters and thereby enhancing the overall quality of discussions. strategies serve as valuable guidance for the auto- mated generation of persuasive texts that resonate effectively with various audiences. Moreover, a set of core strategies can potentially form the backbone of advanced writing tools, providing substantial support to writers as they structure their arguments.
2 Related Work
Argumentative discussions play major role in computational argumentation analysis, for exam- ple when detecting counterarguments (Wachsmuth et al., 2018b) or whether two arguments are on the same side (Körner et al., 2021). I). A special em- phasis is placed on the Change My View platform due to its extensive user base, strict moderation, and user-provided persuasiveness rating. Several prior works examined individual comments, where wide range of linguistic, stylistic, and argumen- tative features have been employed to predict their persuasiveness (e.g.. Tan et al. (2016); Hidey et al. (2017); Persing and Ng (2017); Morio et al. (2019)). For analyses beyond individual comments, Ji et al. (2018) 18) employ features of both comments and the corresponding opening post, but do not look at dis- cussions that are longer than one comment to the opening post. Guo et al. (2020) consider whole discussions and employ several linguistic features. but none tied to arrangement. Our paper makes a significant contribution to the task of predicting persuasiveness, going beyond exploring individ- ual comments and delving into the arrangement of arguments within discussions.
practical terms, these
To reveal these strategies, we introduce a new model for argument arrangement. This model in- cludes identifying particular types of ADUs within a given argumentative discourse, analyzing the se- quences of these ADUs, and clustering these se- quences to reveal the patterns and strategies. We test our model using a large-scale dataset of persua- sive discussions gathered from Change My View.! We sampled two types discussion branches, di- alogue and polylogue, and identified 16 clusters of similar ADU type patterns, each representing a different strategy used in the discussions. The se- quence of clusters for each discussion (one cluster assignment per comment) then serves model of the discussion, which we evaluate against the task of determining the persuasiveness of the com- menter.? For this task of persuasiveness detection, our model outperforms several strong baselines.
A Argument models in online discussions Hidey et al. (2017) model unit types for premises (ethos, logos, and pathos) and claims (interpretation, eval- uation, agreement, or disagreement), revealing that the relative position of types can indicate a com- ment's persuasiveness. Morio et al. (2019), on the other hand do not distinguish between premises and claims. They use the following types: testi- mony, fact, value, policy, rhetorical statements. We employ the types of Morio al. (2019) for their simplicity, having shown impact on persuasiveness, and the easy to understand semantics behind the types (cf. Section 4.3), making them well-suited for modeling argument arrangement.
Overall, this paper introduces new model for identifying argument arrangement strategies through clusters of similar ADU type patterns (Sec- tion 3). We develop a large-scale dataset compris- ing 34,393 discussion branches and completely tag
'https:/ / //ts:///www.. http://wwwwwddd..//oo/wtt.d..ttw com/ r/ changemyview We gauge persuasiveness according to the standards of the Change My View community guidelines, where the original poster signals that their view has heen changed by marking comments with a A symbol in response.
3 All the resources developed in the paper can be found on: https: / higghuu.bb........ com/ /wbbi i -ede /ENNLEENPPP- 2.3

---

Original post clusters
10
Comment clusters
HA
1
H
ADU type identification
ADU type
ADU type pattem abstracting
ADU type pattern clustering
Prediction
pattern mining
Figure 2: Pipeline for modeling the overall strategy as flows of ADU type arrangements per its opening post and comments. Illustrated the two discussion branches of the example from Figure 1. 'hee clusters determined based on the abstracted patterss of all opening posts and comments in the dataset, respectively.
Argument arrangement in online discussions The investigation of argument arrangement so- cial media platforms and influence on persua- siveness remains a relatively understudied area with only a few works so far. Hidey et al. (2017) ex- plored whether the types of their proposed model (see above) follow specific order within Change My View discussions. They identified certain quential patterns, such the tendency for pathos premises to follow after claims of emotional eval- uation. Morio et al. (2019) modeled persuasion strategies by the positional role of argumentative units, also by examining Change My View discus- sions. They find that facts and testimonies were commonly positioned at the beginning of posts, indicating the importance of presenting factual in- formation before making claims. Conversely, pol- icy suggestions tend appear towards the end of posts, implying that recommendations or courses of action were treated as conclusions the argu- mentation process. The closest work to ours that by Hidey and McKeown (2018), who investigate the impact of argument arrangement on persuasive- ness, albeit only within individual comments. They predict persuasiveness using word-level features, Penn Discourse TreeBank relations, and FrameNet semantic frames. In contrast, our work incorporates a clustering step that facilitates grouping similar sequences argument unit types, enabling a more thorough exploration of arrangement strategies.
New York Times and associated them with per- suasive strategies. Through their analysis, they established specific rules for constructing effective editorials, like that arguments containing units type testimony should precede those containing units of type statistics. Wachsmuth et al. (201 8a) emphasized the importance of argument arrange- ment primary means in the context of gener- ating arguments with rhetorical strategy. They proposed and conducted manual synthesis of ar- gumentative texts that involves specific selection, phrasing, and arrangement of arguments following an effective rhetorical strategy. In contrast, our research specifically focuses on persuasive discus- sions. While there is an overlap in the argument unit types employed between our study and the work of Al-Khatib et al. (2017), the central focus in the former is the concept of evidence. More- over, our findings regarding argument arrangement hold the potential to be incorporated into the ap- proach by Wachsmuth et al. (2018a) for generating persuasive arguments.
Modeling Argument AoatmeentArragggee Arrangement
Although various studies have explored the selec- tion and phrasing of arguments in online discus- sions, often by analyzing the distribution of argu- ment attributes and linguistic features (e.g., Wieg- mann et al. (2022)), the effect of argument arrange- ment has received relatively limited attention. This paper aims address this research gap by inves- tigating the impact of argument arrangement, not only in predicting persuasiveness but also in gain- ing deeper understanding of how individuals en-
Argument arrangement in other domains A pioneering work of Al-Khatib al. (2017) in the study of argument arrangement identified evidence patterns in 30,000 online news editorials from the

---

Once the patterns are identified, they are grouped into clusters. The assigned cluster's ID is then con- sidered the sole representation of the arrangement of the opening post or comment. These clusters are identified using a clustering algorithm we use hierarchical agglomerative clustering for its sim- all patterns of the gage in persuasive discussions.
importance for the clustering and thus for our ap- proach, we propose and evaluate two measures:
In our approach to modeling argument arrange- ment in opening posts and comments, we em- play a three-step pipeline, as illustrated in Fig- ure 2. First, we identify the argumentative dis- course units (ADUs) and their corresponding se- mantic types within the opening post and comments (Section 3.1). Next, we mine and abstract the se- quence of ADU types to discover overarching pat- terns (Section 3.2). Finally, we categorize these patterns into clusters based on their similarity to other patterns (Section 3.3).
Edits The Edits approach calculates the normal- ized distance between two sequences by quan- tifying the minimum number of edits (inser- tions, deletions, or substitutions) needed transform one sequence into the other. We employ the widely used Levenshtein distance to determine the edit count. Yet, to ensure fairness, the resulting distance is normalized by the length of the longer sequence. This normalization ensures that edit made to shorter sequence has more significant im- pact on the overall distance compared to an edit made to a longer sequence.
3.1 ADU Type Identification
For this step, we utilize the five ADU types in- troduced by Morio et al. (2019): fact, policy, tes- timony, value, and rhetorical question. One ad- vantage of using these specific ADU types is the availability of a manually annotated dataset that serves as a valuable resource for fine-tuning a Large Language Model to effectively classify the ADU type in a given text. By identifying and categoriz- ing these ADU types, we lay the foundation for understanding the structure and organization of ar- guments within the discussion.
SGT Embeddings In the Embeddings approach, the ADU type sequences are transformed into a fixed-dimensional space using the Sequence Graph Transform technique proposed by Ran- jan et al. (2022). This technique offers the capability to capture a range of dependencies without introducing additional computational complexity. The distance between two em- bedded sequences is then determined by em- ploying standard distance metric, such Euclidean distance.
3.2 ADU Type Pattern Mining
Once the ADU types are identified, we examine the sequences in which these types appear within the comments. To enhance the mining of more general and reliable sequences, we incorporate the "Change" transformation proposed by Wachsmuth al. (2015). This transformation focuses on the transitions or changes between ADU types within a sequence. Through this abstraction, we empha- size the shifts in ADU types rather than the specific instances of each type. For example, if a sequence initially consists of [policy, policy, fact], it is ab- stracted to Ipolicy, fact].
After the distance matrix calculation using ei- ther of the above-mentioned methods, hicrarchical clustering can be applied. Initially, each sequence is considered individual cluster. The algorithm then proceeds to merge the most similar clusters it- cratively based on the distance between them. This process continues until all sequences are merged into a single cluster or until a predefined stopping criterion is met. The resulting hierarchical cluster- ing tree, also known as dendrogram, provides visual representation of the clusters' hierarchical relationships. It allows us to observe the formation of subclusters and the overall clustering structure. An optimal number of clusters can be determined either by setting a threshold on the dendrogram or by employing other methods, such as the Elbow method (Thordiike, 1953).
3.3
ADU Type Pattern Clustering
After performing clustering, the arrangement is represented by the longest common sequence of ADU types within each cluster. This way can gain insights into the prevailing ADU types and their sequential order, which represents the charac- teristic arrangement pattern within that particular cluster.
plicity and adaptability dataset.
A critical element for pattern clustering is the measure of distance between patterns. Due to its

---

4 Preparing a Dataset of Argument Arrangements in Online Discussions
the opening post comment without any subse- quent replies as a branch. These branches represent distinct trajectories within the discussion, highlight- ing the progression and development of the conver- sation from the opening post to subsequent com- ments. When the original poster perceives that a commenter has successfully influenced their view- point, even if only to minor extent, they indicate this by posting response that includes the A char- acter. In line with prior research, we consider only those branches that contain Á in a comment from the original poster as persuasive."
To evaluate the effectiveness of our argument ar- rangement model, applied it to real-world on- line discussions data, specifically focusing on the task of predicting persuasiveness. We collected a dataset comprising 34,393 discussion branches, extracted from 1,246 threads sourced from the Change My View online discussion platform via the public Reddit API. These discussions adhere to structured format enforced by the community, with persuasive discussions being identified by the presence of A character, in accordance with com- munity guidelines, which serves as our ground truth (Section 4.1). For our analysis, we catego- rize discussions into two scenarios, dialogue and polylogue (Section 4.2), and automatically iden- tif Argumentative Discourse Units (ADUs) and their respective semantic types (Section 4.3). Sub- sequently, we offer initial insights into the use of ADU types through statistical analysis of their positional distribution within discussion branch (Section 4.4).
4.2 Grouping Discussions by Scenario
For in-depth analysis of persuasiveness, cat- egorize the branches into two subsets based on different discussion scenarios:
Dialogue In this scenario, discussions unfold with a sequential exchange of arguments between the original poster and single unique com- menter. Here, both parties take turns present- ing their viewpoints and counterargument in a back-and-fort manner.
4.1 Structure of Change My View Discussions
Polylogue
This scenario encompasses discussion branches where multiple commenters engage with the original poster. These interactions in- volve multiple participants attempting to per- suade the original poster, resulting in a dy- namic, multi-party conversation.
Change My View provides dynamic and inclu- sive platform that encourages users to engage in thoughtful discussions, with a strong emphasis on presenting well-reasoned arguments. The discus- sions carefully moderated according the com- munity rules* ensuring the discussions maintain a high standard of quality and remain focused on the topic at hand.
This differentiation allows us to examine the ef- fectiveness of persuasive discourse different dis- cussion scenarios and gain insights into the dynam- ics of persuasion. Table 1 provides key numbers for each scenario: dialogues more often contain A and are on average shorter than polylogues. Moreover, out of the 1,246 total threads, nearly all contain at least one dialogue (1,236) and the major- ity contain least one polylogue branch (1,058).
In the Change My View platform, discussions, known as threads, begin when a user, referred to as the original poster, submits text-based opening post expressing and justifying their viewpoint on any controversial topic. As implied by the name of the platform, each such submission is chal- lenge to other users, known as commenters, to change the original poster's 's view by submitting tex- tual comments as replies. Both the original poster and commenters have the ability to contribute addi- tional comments. The original poster may choose to add comments to further defend their justifica- tions, while other commenters can provide counter- arguments or rebuttals. This iterative commenting process can lead to long chains of discussion. For the purposes of this paper, we define each path from
4.3 Automatic Identification of ADU Types
The fundamental building blocks of our approach (cf. Section 3.1) are the semantic ADU types pre- sented by Morio al. (2019). 6 The authors provide the following definitions and examples of the five ADU types:
"To avoid disclosing the label, the comments awarding and confirming the itself were excluded from our dataset. 'Mostly the same as those of Park et al. (2015).
'Change My View community rules: https: : /www. reddit. // htt.:////////cacaataaa chtpe///////aaaagggieieeeee wwwki//wkii//// rrules

---

Scenario
Threads
Branches
13
Avg. texts
A
A A
Dialogue 1,236
2,618
14,860
2.3
Polylogue 1,058
940
16,140
5.4 8.6
01
92
65
97
00
16
Both 1,246
3,393 31,000
2.8 5.4
-1.6
Table 1: Number of unique threads, branches (i.e., paths from opening post a leaf comment), and average num- ber of texts per branch (opening post and comments) in the dataset for each scenario and split between branches were delta was awarded (A) or not (f)).
Policy
-18
Statement
Testimony (T) is an objective proposition related to the author's personal state experience such as the following: do not have children.
Fact (F) is proposition describing objective facts that can be verified using objective evidence and therefore captures the evidential facts in persuasions: Empire Theatres in Canada has "Reel Babies" showing for certain movies.
Testiory
Value (V) is a proposition that refers to subjec- tive value judgments without providing state- ment on what should be done: it is absolutely terrifying.
Value
Figure 3: Distribution density of relative start positions (as a fraction of overall text length in a branch) for ADUs of each type, separately for branches with A (green) and without (red).
Policy (P) offers a specific course action to be taken or what should be done: intelligent stu- dents should be able to see that.
Rhetorical Statement (R) implicitly states the subjective value judgment by expressing fig- urative phrases, emotions, or rhetorical ques- tions: does it physically hurt men to be raped by women (as in PIV sex)?
4.4 Positional Role of ADU Types
The analysis of the human-annotated dataset by Morio et al. (2019) revealed slight differences the distribution of ADU types between persuasive and non non-persuasive comments. They observed that testimonies and facts tend to occur more frequently at the beginning of texts, while policies are more prevalent towards the end. Figure 3 shows the result of reproducing their analysis on larger dataset, utilizing automatically identified ADU types. As shown in the figure, our dataset exhibits a more balanced distribution of facts, whereas there is a no- ticeable increase in the usage of values and rhetori- cal statements towards the end of the discussions. We attribute these differences to our dataset con- taining not only the first comment to the opening post but complete discussion branches. Morcover, the figure suggests that even the position of ADU types alone can be though rather weak pre- dictor of persuasiveness. We anticipate that our exploration of ADU type arrangement will further enhance this predictive capability.
To identify the ADU types within the col- lected discussions, we fine-tuned pre-trained ELECTRA-Large model (Clark et al., 2020) on the Guman-annotatedd dataset of Morio al. (2019). The model was specifically trained for sequence labeling at the post level, enabling it to detect ar- gumentative discourse units and assign correspond- ing ADU types. We evaluated the performance of our model on separate hold-out test set from the dataset used by Morio et al. (2019). The model achieved an F1 score of 0.62, surpassing the perfor- mance of the original authors' model by 0.18 and in view is sufficiently reliable for large-scale anal- yses. Given that both our dataset and the one by Morio al. (2019) were sourced from Change My View platform, it is reasonable to expect a similar level of performance when applied to our dataset.

---

5 Analysis of Experiments and Results
Distance / Texts Cluster
Patterns
#
Avg. length
Most frequent
In this section, we present the results of modeling argument arrangement of Argumentative Discourse Unit (ADU) types introduced in Section 3.3 and propose several approaches for predicting the per- suasiveness of discussion branches. Furthermore, we conduct an experimental validation of our find- ings by incorporating cluster features in the task of persuasiveness prediction.
Embeddings
OP1
173
139
9.01
VTVRV
OP2
159
96
7.29
VFV
OP3
194
75
5.52
TV
OP4
99
90
9.72
VEVRV
OP5
74
68
10.26
VPVFV
OP6
141
33
4,43
VRV
OP7
134
38
4.61
VPV
OP8
102
90
8.52
VRVPV
OPY
53
43
8.55
VTVPV
OP10
117
1.00
V
5.1 Identified Arrangement Clusters
CI
46,085
595
VTV
Following our approach as described Section 3.3, we cluster the ADU type patterns in our dataset. Notably, we conducted separate clustering for the patterns found in opening posts and comments, rec- ognizing their distinct pragmatic purposes (justi- fying view vs. arguing). To determine ap- propriate number of clusters for each, we applied the widely-used Elbow criterion (Thorndike, 1953). Remarkably, this analysis revealed that both dis- tance measures led to the same number of clus- ters for both opening posts (10 clusters) and com- ments (6 clusters).
C2
13.758
765
4.60
C3
11,740
851
C4
9,013
622
C5
3,120
331
4.30
C6
1,767
408
Edits
OP1
390
278
8.82
VTV
OP2
106
105
16.30
VEVEVFVPV
OP3
138
5.71
VRV
OP4
202
TV
OP5
VPV
OP6
129
7.30
VFV
OP7
TVTV
VRVF
OP9
10.42
VEVTVR
OP10
112
PVVPY
Table 2 provides overview of the identified clusters. Notably, when employing SGT embed- ding similarity, the clusters exhibit more even dis- tribution of texts. Directly related that, the most frequent patterns per cluster tend to shorter when employing SGT embeddings, especially when clus- tering comments. This indicates that in this ap- proach short and frequent patterns are rather as- signed to different clusters than concentrated in single This approach holds promise for distin- guishing various arrangement strategies effectively. Moreover, the table shows that the average length patterns is quite different between clusters when using the Edit distance approach, which also de- sirable for the same reason. Furthermore, we con- firmed that the clusters also exhibit variations in the distribution density of ADU types, as demonstrated in Figure 3 for the entire dataset. The density plots for the clusters based on SGT embeddings can be found in the appendix (Figure 5).
C1
72,702
1,957
2.20
RV
C2
3,521
801
8.22
VPVPVP
C3
1,008
316
7.46
VEVEVF
C4
151
12
4.10
FVRV
C5
1,742
342
6.31
VFVFV
C6
2,869
144
3.78
VTV
Table 2:
Overview of the identified clusters in our dataset, either via the embeddings or edits distance, and either the opening posts (OPX) or comments (CX): number of texts (opening posts comments) they contain as well as the number unique patterns, their average length, and the most frequent one.
Length- Classifier As observed by Tan al. (2016) and reflected in our dataset (cf. Table 1), lengthy discussions on Change My View tend to receive fewer awards. To establish prelim- inary baseline, we employ logistic regression to classify discussion branches as persuasive or not based solely on their length feature.
Interplay Features In the study of Tan et al. (2016), the most effective features for the persua- siveness prediction were extracted from the inter- action between the opening post and the replies. They derived 12 features from similarity mea- sures (common words, similar fraction in response, similar fraction in the opening post, and Jaccard score) and subsets (all words, stop words, and
5.2 Approaches for Predicting Persuasiveness
In order to evaluate the effectiveness of our ar- gument arrangement model, employ it in the context of predicting persuasiveness within discus- sions: Given discussion branch, predict whether the original poster ultimately awards a in the end. We compare the following approaches:

---

Test Model
Scenario Polylogue
Dialogue
Both scenarios
F1
FIA
Pa
R
F1
FIA
Pa
Ra
F1
F1A
Pa
Ra
Unbalanced test set
Length-Basse Bagth-BBsedd Classifier
0.56
0.37
0.25
0.75
0.49
0.19
0.11
0.71
0.52
0.30
0.18
0.81
Interplay features
0.54
0.47
0.37
0.65
0.48
0.15
0.09
0.52
0.53
0.33
0.22
0.68
BERT
0.55
0.21
0.52
0.13
0.48
0.24
0.50
0.14
0.59
0.25
0.47
0.17
Edits
0.78
0.66
0.59
0.74
0.74
0.52
0.48
0.53
0.61
0.40
0.28
0.74
SGT Embeddings
0.82 0.71
0.66
0.77
0.75
0.53
0.53
0.56
0.61
0.42
0.28
0.81
BERT+Edie
0.80 0.68 0.60
0.77
0.72
0.47
0.45
0.49
0.77
0.58
0.57 0.58
BERT+SGT Embeddings
0.83 0.72 0.67
0.81
0.75
0.51
0.53
0.56
0.78
0.60
0.62 0.58
[Jama-7B Zero-Shot
0.24 0.18
0.12
0.44
0.23
0.17
0.11
0.41
0.23
0.16
0.10 0.34
Llama-7B Branches
0.32 0.27
0.18
0.71
0.30
0.24
0.15
0.65
0.29
0.25
0.16 0.65
[lama-78 Branches+Strategies
0.36 0.33
0.21
0.74
0.36
0.32
0.20
0.73
0.37
0.34
0.22 0.76
Balanced test
Length- Based Classifier
Length-Based
0.64 0.68
0.63
0.75
0.62
0.65
0.61
0.70
0.56
0.69
0.57
0.86
Interplay features
0.54
0.45
0.47
0.43
0.59
0.51
0.64
0.43
0.53
0.50
0.47
0.53
BERT
0.73
0.76
0.70
0.84
0.60
0.60
0.54
0.67
0.64
0.70
0.62
0.80
Edits
0.81 0.80
0.86
0.74
0.70
0.65
0.78
0.56
0.71
0.71
0.70
0.72
SGT Embeddings
0.84 0.84 0.90
0.78
0.75
0.71
0.87
0.60
0.72
0.73
0.70
0.77
BERT+Edin
0.80 0.68 0.61
0.77
0.68
0.61
0.79 0.50
0.70
0.69
0.70 0.56
BERT+SGT Embeddings
0.85 0.83
0.90
0.78
0.72
0.68
0.87 0.49
0.75
0.70
0.90 0.70
Llama-7B Zero-Shot
0.31 0.46
0.55
0.40
0.27
0.38
0,46 0.32
0.29
0.40
0.47 0.35
Llama-7B Branches
0.40
0.62
0.58 0.66
0.39
0.60
0.60 0.61
0.37
0.60
0.62 0.59
1Jama-7B Branches+Strategies
0.45 0.64 0.67 0.62
0.43
0.66
0.64 0.69
0.45
0.67
0.66 0.68
Table 3: Effectiveness of the tested models in predicting whether a discussion branch received A in the different scenarios and for both the unbalanced (full) and balanced test set. Employed measures are Macro-Fı score (Fi) as well as the F; score. precision, and recall for detecting delta discussions (Fi (F PA. RA).
content words). As lexical baseline approach, we employ the same features and use logistic regres- sion to classify the discussion branches.
essential role in predicting which round of debate (Pro vs. Con) makes more convincing argument on the debate.org website. In a similar manner, to leverage the strengths of both the ADU type rangements and BERT embeddings, the outputs from the previous two models. Two linear layers with softmax are used to predict the output probabilities over both these LSTM mod- els separately.
BERT-Based Classifier As a baseline that cap- tures both the semantic and syntactic information of the text, we employ classifier based on BERT Jurge embeddings (Devlin et al., 2019). For this ap- proach, concatenate the BERT embeddings for each turn in the discussion and pass them through bidirectional LSTM (Hochreiter and Schmidhuber, 1997) to capture contextual dependencies between different turns. The LSTM's outputs are then pro- cessed through a linear layer and softmax activation to predict persuasiveness.
combine
lurge
Llama-2-77 --7B Zero-Shot We utilize the out-of-the- box Llama-2 model with seven billion parameters (Touvron et al., 2023). Given the discussion branch, the model prompted to generate binary answers (yes/no) when asked whether the original poster's opinion was altered by the end of the discussion.
Argument Arrangement: Clusters In this ap- proach, we utilize bidirectional LSTM model in conjunction with a linear layer and softmax acti- vation for predicting persuasiveness labels based the identified cluster features. The input to the LSTM consists of sequence of cluster features identified by either the SGT embeddings or Edit distance approach.
Llama-2-7B Fine-Tuned Branches Using the same prompt as in the zero-shot approach, we fine-tune the Llama-2-7B model on the discussion branches, with the true yes/no answers provided in the prompt. We implement low-rank adapta- tion (Hu et al., 2022) with parameters scaling factor a 16 applied all linear layers. The model was fine-tuned for one epoch, as further training did not yield significant reduction the loss, with a batch size of 8 and a learning rate x 10 on a single NVIDIA A 100 GPU (40GB).
64 and
Combination of Contextualized Features and Argument Arrangement The study by Li al. (2020) demonstrated that incorporating argument structure features into an LSTM model plays an

---

Llama-2-7B Fine-Tuned Branches+SGT Clus- ters This approach closely mirrors the one above, with a slight variation in the prompt. Here, we extend the prompt with cluster features identified from the dataset using the SGT Embeddings ap- proach. Figure 4 illustrates the prompt template used in all three configurations.
t+ iastruction:
Below conversat ion between OP and usera along waen the peratasion stretegies employed by each of then. sead the discussion and decide by the c2 xas changed.
of di d'acussior opinion of
t+ C2 guaed acrategy
sepening post
itt Reply 1 [used stratagy XI:
Keommens 13
Reply N Iosed strategy Al:
Snommont
44 The opinion of the OP waa changed [yea/no]: <nodel response>
Figure 4: Example of prompt employ for zero-shot and fine-tuning experiments. The text colored in blue is adapted to the conversation at hand. The text colored in red is used only when identified clusters are employed.
5.3 Results of Predicting Persuasiveness
We present the results of our persuasiveness pre- diction analysis for all branches under the two sce- narios described in Section 4.2. Considering the class imbalance, we evaluate the approaches on the complete dataset as well balanced sample to address the imbalance issue. For the evaluation, we divide the branches into training and test sets, main- taining a ratio of 8 2. The predictions are then assessed on the held-out test set using the macro-Fi measure. Given a heavily skewed label distribu- tion, we want to emphasize the importance of A prediction, thus we provide more detailed evalua- tion results for this label. The effectiveness of the classifiers is reported in Table
Conclusion
This paper expanded our understanding of online persuasive dialogues by uncovering that debaters follow certain arrangement strategies. We intro- duced a new model for arrangement strategies that is based on patterns of argumentative discourse unit types. Clusters of such patterns correspond to dif- ferent strategies, with sequences of these clusters (one element per comment of a discussion branch) representing a whole discussion. This model was operationalized using a large-scale dataset compris- ing 34,393 discussion branches. In a comparative evaluation of ten approaches, we demonstrate the remarkable utility of these arrangement strategies in predicting persuasiveness - both if used as sole feature and in addition to others -, emphasizing their essential role in unraveling the dynamics of persuasive online discussions.
The classification results reveal several findings: First, the length baseline provides a starting point for comparison of the discussion branches in the balanced setting, achieving an overall Fj score of 0.56 in both scenarios and 0.69 F1 score for the A discussions. While the length baseline provides a basic understanding of the relationship between discussion length and persuasiveness, we recog- nize the need for more sophisticated methods to capture the nuances of persuasive discourse. Sec- ond, our experiments with encoding the discussions using BERT embeddings have yielded promising results, however, we have observed that incorpo- rating argument arrangement features further en- hances the prediction performance. In fact, solely using cluster features to represent the discussion dynamics is effective for persuasion prediction. We similar trend with the Llama-2-7B-bunn -ased approaches, where the performance of the model significantly improves after fine-tuning on the dis- cussion branches, but even more so when incorpo- rating the identified strategy features in the prompt.
Still, there's ample room for refining and ex- panding this research further. One aspect worth exploring is the development of more fine-grained categories for argumentative discourse units, such as incorporating the human value categories pro- posed by Kiesel et al. (2022). Besides, the iden- tified arrangement strategies have applications be- yond persuasiveness prediction. Exploring their potential in tasks such as writing assistance and text generation could broaden the scope of argu- mentative discourse analysis into new horizons.
7 Limitations
While our research contributes valuable insights into the prediction of persuasiveness and the role of argument arrangement, it is important to acknowl- edge certain limitations that may impact the gener- alizability and interpretation of our findings.
The enhanced prediction performance achieved by incorporating argument arrangement features highlights their significance capturing the intri- cate dynamics of persuasive discussions.

---

ADU Type Classification We rely on automated methods for the identification and classification of argumentative discourse units (ADUs), which may introduce errors or inaccuracies. Despite efforts to ensure accuracy, misclassifications or inconsisten- cies in ADU labeling could potentially impact the analysis and predictions. Further refinement and validation of the ADU classification process are necessary for more robust results.
tion strategies actoss topics. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 1351- 1357. Association for Computational Linguisties.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre- training text encoders as discriminators rather than generators. In 8th International Conference Learning Representations (ICLR 2020). OpenRe- view.net.
Limited Scope of Features Our study primarily focuses on ADU types and argument arrangement features in predicting persuasiveness. While these features have shown promising results, there are certainly other important linguistic, contextual, or stylistic features that were not considered in our analysis. Future research should explore combina- tions of such features with arrangement features, like we did with BERT embeddings, better un- derstand their individual and combined impact on persuasiveness prediction.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL-HLT) 2019). pages 4171-4186. Association for Computational Linguisties.
Zhen Guo, Zhe Zhang, and Munindar P. Singh. 2020. In opinion holders' shoes: Modeling cumulative in- fluence for view change in online argumentation. In The Web Conference 2020 (WebConf 2020), pages 2388 2399. ACM/ / IW3C2.
Only One Platform Our study focuses primar- ily on the Change My View platform, which may limit the generalizability of our findings to other social media platforms or contexts. The character- istics and dynamics of persuasive discourse may vary across different platforms, user demographics, and topics. Future research should explore the gen- eralizability of our findings to a broader range of platforms and contexts.
Christopher Hidey and Kathleen R. McKeown, 2018. Persuasive influence detection: The role of argument sequencing. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence (AAAI 2018), pages 5173-5180. AAAI Press.
Christopher Hidey, Elena Musi, Alyssa Hwang. Smaranda Muresan, and Kathy McKeown. 2017. An- alyzing the semantic types of claims and premises in an online persuasive forum. In Proceedings of the 4th Workshop on Argument Mining (ArgMin- ing @ EMNLP 2017), pages 11-21. Association for Computational Linguistics.
8 Ethics Statement
In terms of data collection, this study utilizes pub- licly available datasets that strictly adhere to ethical considerations, ensuring compliance with Change My View policies and maintaining the anonymity of the users who participated in the discussion.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 988) 1780.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In 10th International Con- ference Learning Representations (ICLR 2022). OpenReeiew..et.
As for our research findings, they have implica- tions for the identification and potential generation of persuasive text, especially within the limitations mentioned above. Hence, it is important to rec- ognize the potential misuse of this capability, as malicious actors may in the future be able to em- ploy our approach as one one many building blocks to rapidly create misleading or deceptive text for target audience with a profound persuasive impact, for example in chatbots.
Lu Ji, Zhongyu Wei, Xiangkun Hu, Yang Liu, Qi Zhang, and Xuanjing Huang. 2018. Incorporating argument- level interactions for persuasion comments evaluation using co-attention model. In Proceedings of the 27th International Conference on Computational Linguis- tics (COLING 2018), pages 3703-3714, Association for Computational Linguistics.
Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the Human Values behind Argu- ments. In 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022). pages 4459- 4471. Association for Computational Linguistics.
References
Khalid Al-Khatib, Henning Wachsmuth, Matthias Ha- gen, and Benno Stein. 2017. Patterns of argumenta-

---

Erik Körner, Gregor Wiedemann, Ahmad Dawar Hakimi, Gerhard Heyer, and Martin Potthast. 2021. the Same Side of an Argument. In 26th Conference on Em- pirical Methods in Natural Language Processing (EMNLP 2021), pages 10130-10138. Association for Computational Linguistics.
Henning Wachsmuth, Manfred Stede, Roxanne EI Baff, Al Khatib, Maria Skeppstedt, and Benno Stein. 2018a. Argumentation synthesis following rhetorical strate- gies. In Proceedings the 27th International Confer- ence on Computational Linguisties (COLING 2018), pages 3753-3765. Association for Computational Linguistics.
On Classifying whether Two Texts
Jialu Li, Esin Durmus, and Claire Cardie. 2020. Explor- ing the role of argument structure in online debate persuasion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP 2020), pages 8905-8912, Online. Asso- ciation for Computational Linguistics.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018b. Retrieval of the Best Counterargument with- out Prior Topic Knowledge. In 56th Annual Meet- ing of the Association for Computational Linguistics (ACI. 2018), pages 241-251 Association for Compu- tational Linguistics.
Matti Wiegmann, Khalid Al Khatib, Vishal Khanna, and Benno Stein. 2022. Analyzing persuasion strate- gies of debaters on social media. In Proceedings of the 29th International Conference Computational Linguistics (COLING 2022). pages 6897-6905. Inter- national Committee on Computational Linguistics.
Gaku Morio, Ryo Egawa, and Katsuhide Fujita. 2019. Revealing and predicting online persuasion strat- egy with elementary units. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCCNLP), pages 6274-6279.
A Appendix
Joonsuk Park, Cheryl Blake, and Claire Cardie. 2015. Toward machine-assisted participation in erulemak- ing: an argumentation model evaluability. Pro- ceedings of the 15th International Conference on Artificial Intelligence and Law (ICAIL 2015), pages 206-210. ACM.
For reference, Figure 5 provides the density plots for the clusters based on embeddings. Figure pro- vides the density plots for the different ADU types in replies based on OP cluster and whether a was awarded or not.
Isaac Persing and Vincent Ng. 2017. Why can't you convince me? modeling weaknesses in unpersuasive arguments. In Proceedings of the 26th International Joint Conference Artificial Intelligence (ICAA 2017), pages 4082-4088. ijcai.org.
Chitta Ranjan, Samaneh Ebrahimi, and Kamran Payn- abar. 2022. Sequence graph transform (SGT): fea- ture embedding function for sequence data mining. Data Min. Knowl. Discov, 36(2):668-708.
Chenhao Tan, Vlad Niculae, Cristian Danescu- Niculescu-Mize and Lillian Lee. 2016. Winning ar- guments: Interaction dynamics and persuasion strate- gies in good-faite online discussions. In Proceedings of the 25th International Conference on World Wide Web (WWW 2016), pages 613-624. ACM.
Robert Thorndike. 1953. Who belongs in the family? Psychometrika, 18(4):267-276.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-ture tuned chat models. arXiv preprint arXiv:2307.0928.
Henning Wachsmuth. Johannes Kiesel, and Benno Stein. 2015. Sentiment flow A general model of web re- view argumentation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP 2015), pages 601-611. The Association for Computational Linguistics.

---

Opening Post clusters
Comment clusters
3.5
Testimony
Testimony
2.0
3.0
2.5
Density
2.0
Density
1.5
1.0
0.5
0.5
0.0
0.1
0.2
0.4
0.5
0.6
0.7
0.8
0.9
100
4.0
Fact
2.00
Fact
3.0
1.50
1:25
Densiyy
Denaity
2.0
1.00
1.0
0.50
0.
0.25
0.0
0.2
0.9
Value
2.5
Value
1.0
Donsty
Density
0.8
0.6
1.0
0.0
0.0
0.1
0.2
03
0.4
0.5
0.6
0.7
09
Policy
Policy
3.0
1.75
2.5
1.50
Densiy
2.0
Denety
1.25
1.00
1.5
0.75
1.0
0.0
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Rhetorical Statement
Rhetorical Statement
1.25
Donstt
11.00
Densty
0.75
0.
0.1
02
033
0.5
05
077
08
0.9
1.0
0.0
0.1
0.2
04
0.5
06
0.7
08
09
10
Relative position
Relative position
Figure 5: Distribution density of relative positions (as fraction of overall text length) for ADUs of each ADU type, separately for the different embedding-based clusters (color-coded) of opening posts (left) and comments (right). The density graphs illustrate how patterns some cluster use certain ADU types rather the start and others types rather at the end of the texts.

---

PAO
SRRCC 020IE0 RECOCCS SEGCOS CENDICS REREECE
NO
Termos
100
Figure 6: Distribution density of relative start positions, expressed as a fraction of the overall text length, for each ADU type. Each row represents the distribution within one of the ten identified original poster (OP) embedding- based clusters (color-coded), and the corresponding reply branches, distinguished by whether they were awarded A (green) or not (red). The density graph reveals similarities and differences in the usage certain ADU types across commenters in response different OP strategies.


Reviews:
[{"review_id": "9k3fvnsX0B", "reviewer": "Reviewer_w6PD", "summary": "", "questions": "Question A: Did you happen to have some metric for cluster quality?\n\n\nQuestion B: Or, if not, do you have a rationale for why the clusters for embeddings and edits have very uneven sized clusters, but the clusters seem to be very different? e.g., the most frequent in Edit C1 is RV, but the most frequent in Embedding C1 is VTV. \n\nQuestion C: For Fig 3, all the green seems to taper off at the end. Do you happen to know what is happening? e.g., could they be artifacts of the last comments being the delta given?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s careful consideration of our paper and their insightful feedback. We address the raised concerns below and provide additional context for our work.\n\nIn regard to the strengths of our work, we appreciate the reviewer’s acknowledgment of the novelty in releasing a new CMV dataset. We are glad that the reviewer finds our approach novel and points out its interpretability. \n\n__Response to Reasons to Reject:__\n \nWe understand the reviewer’s concern about BERT being no longer considered state-of-the-art, but we chose it due to its reliability and simplicity. In this regard, it is crucial to note that the paper’s focus is not to achieve state-of-the-art results in predicting persuasiveness, but to analyze successful arrangement strategies in discussions. The persuasiveness classification is just a validation process that works well with BERT and can work (maybe even better) with more advanced models.\n\n__Response to Questions for the Authors:__\n\n- __Question A:__ Due to the unlabeled nature of our data, we faced limitations in assessing quality metrics for the resulting clusters beyond the internal evaluation measures we used to select the number of clusters via the Elbow criterion. Nevertheless, we conducted a thorough manual validation of the outcomes to guarantee the coherence, relevance, and absence of excessive overlap within the clusters. We will include this clarification in the updated version of the paper. \n- __Question B:__  Based on manual inspections, we hypothesize that the varying cluster sizes between the Embeddings and Edits method can be attributed to Edits method being less able to capture the inherent sequential similarities in ADU patterns. We'll make sure to highlight our hypothesis in the revised version.\n- __Question C:__ Indeed, one could wonder whether the text that awards the delta would have an influence on our results. However, this is not the case, because, to prevent the inference of delta prediction solely from the presence of the delta symbol within the discussion branch, we intentionally excluded the final comment awarding the delta to the user. We will make sure to clarify this point in the paper. However, since Figure 3 looks specifically at the start positions of ADUs, it might be that persuasive comments tend to have longer ADUs at the end. Although this hypothesis needs further verification and might be an interesting area for future research. \n\n__Response to Typos, Grammar, Style, and Presentation Improvements:__\n\nWe appreciate the reviewer's recommendation to emphasize the significance of the new CMV dataset. In the revised paper, we will explicitly highlight the substantial time gap between existing datasets and ours, underscoring the importance of incorporating the latest content for a comprehensive analysis. \n\n__Question to the Reviewer:__\n\nThe reviewer mentions the following:\n> The paper validates the edit and embeddings features with predictive performance, so this may be in the reader's mind as they read the paper.\n\nCould you kindly elaborate on what is meant by this? We would appreciate further clarification, as the identified strategies using the mentioned approaches can indeed be solely used for persuasiveness prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a newer dataset for CMV is interesting, especially given that Tan et al is from 2016\n  - Section 4.4 on the analysis of the dataset with ADUs is quite interesting, although not central to the paper\n- Idea for using patterns of ADU is novel and can be interpretable as a feature \n- Innovations (Edit and Embeddings features) do seem to help a lot in their quantitative experiments ", "reasons_to_reject": "- Models used are fairly weak (BERT) compared to what we have available.\n   - This could lead to the paper having less visibility\n   - The paper validates the edit and embeddings features with predictive performance, so this may be in the reader's mind as they read the paper\n", "questions_to_authors": "Question A: Did you happen to have some metric for cluster quality?\n\n\nQuestion B: Or, if not, do you have a rationale for why the clusters for embeddings and edits have very uneven sized clusters, but the clusters seem to be very different? e.g., the most frequent in Edit C1 is RV, but the most frequent in Embedding C1 is VTV. \n\nQuestion C: For Fig 3, all the green seems to taper off at the end. Do you happen to know what is happening? e.g., could they be artifacts of the last comments being the delta given?", "paper_topic_and_main_contributions": "\n\nThe contributions are explicity:\n- They release a dataset for reddit's Changemyview\n- They use the ADUs from Morio et al 2019 and argue that the order/structure of these ADUs matter\n- They identify two ways to operationalize this structure: edits and embeddings\n- They evaluate the utility of these structures by evaluating their predictive power on whether or not a user changes their mind given a branch.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I'd emphasize the usefulness of a new dataset for CMV. I think a very easy concern is \"why duplicate? However, there has been almost 10 years of content since then. "}, "extra_text": {}}, {"review_id": "rJpBkjrQUw", "reviewer": "Reviewer_Sf3y", "summary": "", "questions": " 1. Frequent pattern mining algorithms could be an alternative to identifying such patterns. Did you experiment with patterns with exact sequences of ADUs, instead of the current approach and what were the potential limitations of such an approach? \n", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful assessment provided by the reviewer regarding our paper, which delves into the analysis of argument arrangements in online persuasive discussions. We would like to address the points raised by the reviewer. \n\nFirst and foremost, we are grateful for the recognition of our paper's main contributions. The identification of argument arrangement strategies based on clusters of similar Argumentative Discourse Unit (ADU) type patterns is indeed a pivotal aspect of our work. We appreciate that the reviewer acknowledges the novelty, significance, and interpretability of our approach.\n\n__Response to Reasons to Reject:__\n\nFor employing more recent models for persuasiveness prediction, we acknowledge the importance of staying up-to-date with the latest advancements in the field. However, our paper's main focus is not on proposing a new persuasiveness prediction model but rather on analyzing the argument arrangement structures and identifying common patterns in online discussions. While future research could explore more recent models, it doesn't detract from the contributions we present in our current work. However, during this short rebuttal period, we were able to conduct some additional experiments involving the approach proposed by Tan et al (2016) and in particular using the best performing features: ‘Interplay’. In terms of macro F1 scores, adding cluster features to the set increases the performance from 0.53 to 0.68 for root paths and from 0.51 to 0.58 for the whole paths. As for the ROC-AUC score, we get an increase from 0.62 (similar to what Tan et al report) to 0.75 for whole paths and from 0.57 to 0.67 for the root paths, respectively. These results show that also other models benefit from including such argument structure patterns as features.\n\n__Response to Questions for the Authors:__\n\n- __Frequent Pattern Mining Algorithms:__\nWe appreciate the reviewer’s suggestion to use Frequent Pattern Mining as it highlights a relevant approach. Such algorithms would be more suited for scenarios where the goal would be to discover the exact frequent sequential patterns within the data. However, in our paper, the goal is to analyze how argument arrangements can be grouped based on their sequential similarity. Thus, we found clustering to be more appropriate, as it allows us to explicitly specify the similarity measure. \n\n- __Experiments with patterns with exact sequences of ADUs:__\nWe experimented with both exact sequences of ADUs and abstracted versions of sequences in our research. While considering exact sequences might seem intuitive, we encountered several limitations with this approach. First, the sheer diversity of discussions makes it challenging to capture precise patterns that account for all possible variations. Additionally, the abstraction through the removal of repetitions allows for greater generalizability across diverse discussions, enhancing the applicability of our findings. Our observations here align with those of related work, for example, the results of Wachsmuth et. al. (2015) in the domain of sentiment analysis.\n\nWe're grateful for your insightful review and are looking forward to any additional feedback or inquiries you would like to share.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " 1. The method of identifying structure of ADUs based on pattern mining and pattern clustering is well motivated and supported by experiments. \n2. Analysis of roles of ADU types is interesting and can be further emphasized. \n3. The identified patterns as features for persuasiveness prediction provides an interpretable approach.  \n", "reasons_to_reject": " 1. Paper can explore more recent models for persuasiveness prediction. It would further strengthen the paper if more recent models also benefit from including such argument structure patterns as features. ", "questions_to_authors": " 1. Frequent pattern mining algorithms could be an alternative to identifying such patterns. Did you experiment with patterns with exact sequences of ADUs, instead of the current approach and what were the potential limitations of such an approach? \n", "paper_topic_and_main_contributions": " The authors study the argumentation in online discussions by analyzing the argument arrangement structures and finding common patterns representative of argumentation strategies used in online discussions. In particular, \n\n1. authors identify arrangement strategies based on clusters of similar ADU type patterns using pattern clustering methods (edits and embeddings based)\n2. provide a new CMV based dataset to study persuasiveness \n3. show that these identified patterns are predictive of persuasiveness. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "48HbQg99jr", "reviewer": "Reviewer_HDqE", "summary": "", "questions": "What is the benefit of creating a new CMV dataset?\n\n", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough evaluation of our paper on the impact of argument arrangement patterns on persuasiveness in online discussions. We address the points raised in the review below: \n\nWe thank the reviewer for acknowledging the significance of the problem we addressed. The impact of argument structure on persuasiveness is indeed an interesting area of research. We appreciate the acknowledgment of the sensibility of our ADU-pattern-based approach. Capturing the structure of the discussions through ADUs indeed provides a holistic understanding of the persuasion dynamics. \n\n__Response to Reasons to Reject and Missing References:__\n\n- __For the existence of studies on interactive relationships between arguments:__\nWhile some research touches on this aspect, we emphasize that the majority either neglected the truly interactive nature of arguments or focused on a simplistic dialogical structure involving a single OP post and one reply, which was either awarded a delta or not, which in turn raises the question of how these replies have been selected: was the delta awarded directly to this reply or after several back-and-forth turns? We address these concerns by investigating more complex levels of interactions, collecting a new dataset with an enhanced structure, and more up-to-date discussions. Our paper is novel in its deep examination of the intricate argumentative patterns within online discussions.\n\n- __For the additional references of Ji et al. and Guo et al.:__\nWhile these studies do touch upon the dialogical nature of CMV discussions, the work of Ji et al. primarily focuses on simple dyadic interactions (OP post and a single reply), not capturing the complex dialogue dynamics we address in our study. Guo et al, in turn, do consider whole discussions, but they target a different task of predicting the points of opinion change on every turn in a discussion. On the other hand, our method delves into multi-turn discussions and employed argumentation strategies. We will add the two studies to our related work section and discuss the difference between their approach and ours.\n\n- __For the lack of direct comparisons with existing approaches:__\nWe want to clarify that our approach is fundamentally distinct, as it delves into strategies employed in multi-turn dialogues, an area that has remained largely unexplored. The application of including these strategies as features for persuasiveness prediction is just one manifestation of the broader potential our approach holds, extending beyond this specific task. However, our clusters can indeed be combined with existing persuasiveness models. As a concrete example, we now conducted some additional experiments, involving the approach proposed by Tan et al (2016) and in particular using the best performing features: ‘Interplay’. In terms of macro F1 scores, adding cluster features to the set increases the performance from 0.53 to 0.68 for root paths and from 0.51 to 0.58 for the whole paths. As for the ROC-AUC score, we get an increase from 0.62 (similar to what Tan et al report) to 0.75 for whole paths and from 0.57 to 0.67 for the root paths, respectively. \n\n- __Leveraging an LLM for ADU Identification:__\nIndeed, our paper employs a pre-trained ELECTRA-1.75M model, fine-tuned for ADU identification. While the reviewer characterizes the quality verification as \"ad-hoc,\" we would like to clarify that our model's performance evaluation was rigorously conducted using a separate hold-out test set from the dataset of Morio et al. (2019). The model achieved a macro F1 score of 0.62, surpassing the original authors' model performance by 0.18. This evaluation encompasses a standard practice to validate the effectiveness of the model.\n\n- __The value of creating a new CMV dataset:__\nWhile we acknowledge the diversity of the initial CMV dataset, we would argue that CMV datasets are still quite similar to each other in terms of language and discussion structure (the latter being enforced throught moderation). However, we advocate for the creation of an updated version due to the time gap since its release in 2016, as was pointed out by the reviewer [w6PD](https://openreview.net/forum?id=0eWQVWvPgu&noteId=9k3fvnsX0B). Moreover, another of the new dataset's advantages lies in its improved structure: threads in the new dataset are organized into dialogue branches, which enhances our ability to capture and analyze the interactive nature of argument patterns more effectively. This structure is crucial for our proposed approach, which aims to uncover the nuanced strategies underlying persuasive interactions within discussions. \n\n__Response to Questions for The Authors:__\n\nPlease see our answer to the same question in the Response to Reasons to Reject and Missing References.\n\nThank you for your thoughtful review, and we look forward to any further feedback or inquiries you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies an interesting problem. \n- The proposed method of capturing argument structure makes sense", "reasons_to_reject": "The interactive relationship between arguments have been studied before. The paper does not provide comparison with existing approaches. \n\nThe paper leverages LLM for ADU identification. However, the quality verification is rather ad-hoc. The authors verify their model using a different CMV data and claim the two datasets are similar. (1) CMV contains topics from a large variety of domains. (2) If the two datasets are similar, what is the benefit of creating a new dataset?\n\n", "questions_to_authors": "What is the benefit of creating a new CMV dataset?\n\n", "paper_topic_and_main_contributions": "The paper studies the impact of argument pattern on persuasiveness in online discussion. The authors claim that existing research mostly focuses on individual comments and ignore the interactive relationship between arguments. They propose to use ADU (argumentative discourse unit) patterns to capture the structure of discussions. They fine-tune a pre-trained LLM to classify ADU type and propose two approaches, Edits and embeddings based, to cluster ADUs. The authors analyze the data they collected to understand the ADU type distribution and cluster statistics. To verify the effectiveness of using argument patterns for predicting persuasiveness, the authors incorporate the argument structure features into an LSTM model and compare the model with length based and BERT based baselines.\n\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Ji et.al., 2018. Incorporating Argument-Level Interactions for Persuasion Comments Evaluation using Co-attention Model. COLING.\nGuo et.al., 2020. In Opinion Holders’ Shoes: Modeling Cumulative Influence for View Change in Online Argumentation. 2388-2399. WWW.\n\n"}}]

Author rebuttal:

Meta-review: The work presents a dataset of Change My View (CMV) subreddit text annotated with argumentative discourse unit (ADU) typology by Morio et al. (2019). It argues that the ordering of the ADUs affect the persuasiveness and supports its claim with experimental results on determining whether a user has changed their view given a brand of comments.

Reviewers appreciated the study of ADU type arrangement patterns and strategies, leading to insights allowing tangible gains in persuasiveness prediction experiments. They also thought a new CMV dataset is good update to the existing one published in 2016.

A primary reviewer concern was the lack of experiment results using the latest models. While the authors provide additional results using LLaMA-2 (7B) in the rebuttal, I agree with the reviewers, and in fact the authors, that reframing the paper to focus on the analysis of argument arrangement will lead to a bigger impact in the community. Also, this work adopts the ADU typology from Morio et al. (2019). However, the typology is a minimally modified version of Park et al. (2015)’s typology. Consequently, the authors cite it, as well, for proper attribution.
Decision: Accept-Findings