Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks
Authors: Ramon Ruiz-Dolz
Stella Heras
Ana Garcia
Venue: EMNLP 2023 Main (2023)
Keywords: Argument Evaluation
Debate Analysis
Argumentation Theory
Computational Argumentation

Abstract: The lack of annotated data on professional argumentation and complete argumentative debates has led to the oversimplification and the inability of approaching more complex natural language processing tasks. Such is the case of the automatic evaluation of complete professional argumentative debates. In this paper, we propose an original hybrid method to automatically predict the winning stance in this kind of debates. For that purpose, we combine concepts from argumentation theory such as argumentation frameworks and semantics, with Transformer-based architectures and neural graph networks. Furthermore, we obtain promising results that lay the basis on an unexplored new instance of the automatic analysis of natural language arguments.

Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks
Ramon Ruiz-Dolz', Stella H Ana Carcía-Fornes
'Centre for Argument Technology, University of Dundee, Dundee DD1 4HN, United Kingdom ²VRIIN, Universitat Politècnica de València, 46022 Valècia,, Spain Truizdolz001@dumm ac. uk, (stehebar, agarcia)@dsic.upy
Abstract
take argumentative analysis tasks (Lawrence and Reed, 2019; Goffredo et al., 2023). One of the most popular tasks that has gained a lot of popular- ity in the recent years is argument mining, a task aimed at finding argumentative elements in natu- ral language inputs (i.e., argumentative discourse segmentation) (Jo et al., 2019), defining their argu- mentative purpose (i.e., argumentative component classification) (Bao al., 2021), and detecting ar- gumentative structures between these elements (i.c., argumentative relation identification) (Ruiz-Dolz et al., 2021a). Even though most of the NLP re- search applied to computational argumentation has been focused in argument mining, other NLP-based tasks have also been researched such as the genera- tion of natural language arguments (Mitsuda et al., 2022), the assesment of the persuasiveness of natu- ral language arguments (El Baff al., 2020), and the automatic generation of argument summaries (Bar-Haim et al., 2020) among others. However, it is possible to observe an important lack of research aimed at the evaluation of complete argumentative debates approached with NLP-based algorithms. Furthermore, most of the existing research in this topic has been contextualised in online debate fo- rums, considering only short text arguments and messages, and without professional human evalu- ation (Hsiao et al., 2022).
The lack of annotated data on professional ar- gumentation and complete argumentative de- bates has led to the oversimplification and the inability of approaching more complex natural language processing tasks. Such is the case of the automatic evaluation of complete pro- fessional argumentative debates. In this paper, we propose an original hybrid method au- tomatically predict the winning stance in this kind of debates. For that purpose, we com- bine concepts from argumentation theory such as argumentation frameworks and semantics, with Transformer-based architectures and neu- obtain
ral graph networks. Furthermore, promising results that lay the basis on an unex- plored new instance of the automatic analysis of natural language arguments.
1 Introduction
The automatic evaluation of argumentative debates is a Natural Language Processing (NLP) task that can support judges in debate tournaments, analysts of political debates, and even help to understand the human reasoning used in social media (c.g., Twit- ter debates) where argumentation may be difficult to follow. This task belongs to the computational argumentation area of research, a broad, multidis- ciplinary area of research that has been evolving rapidly in the last years (Atkinson al., 2017). Classically, computational argumentation research focused on formal abstract logic and computational (i.e., graph) representations of arguments and their relations. In this approach, the evaluation of argu- ments relied exclusively in logical and topological properties of the argument representations (Alfano et al., 2021). Furthermore, these techniques have been thoroughly studied and analysed, but from theoretical and formal viewpoint considering spe- cific cases and configurations instead of large, in- formal debates (Verheij. 2005; Caminada, 2006).
In this paper, we propose hybrid method for evaluating complete argumentative debates consid- ering the lines of reasoning presented by profes- sional debaters and taking into account the human evaluations provided by an impartial jury. Our method combines concepts from the classical com- putational argumentation theory (i.e., argumenta- tion frameworks and semantics), with models and algorithms effectively used in other NLP tasks (i.e.. Transformer-based sentence vector representations and graph networks). This way, we take complete professional debate including all the argumentation and rebuttal phases as an input, and predict the win-
The significant advances in NLP have enabled the study of new less formal approaches to under-

---

ning stance (i.e., in favour or against) for given argumentative topic. For that purpose, we define an original computational modelling of complete professional natural language debates that allows us to improve the prediction of the winning stance of complete argumentative debates compared to more conventional approaches. We present com- plete comparison of approaches relying exclusively NLP algorithms and approaches relying exclu- sively on argumentation theory concepts. From our findings, we can observe that the hybrid ap- proach proposed in this paper is the more adequate to tackle a complex NLP challenge such as predict- ing the winning stance in complete debates.
in an argumentative dialogue as in the case of the debates.
The global (i.e., debate) approach on the evalu- ation of natural language arguments was initially researched in (Potash and Rumshisky, 2017) where Recurrent Neural Networks were used to evaluate non- con-professional debates in a corpus of limited size and structure. Following this trend, in (Shirafuji et al., 2019), the authors propose a method based on the persuasiveness to predict the outcome of online debates using a support vector machine. Re- cently, in (Hsiao et al., 2022), the authors present an algorithm for predicting the outcome of non- professional debates of limited length and depth in online forums. Furthermore, in the previous work the considered urgumentative structures are simple, and the proposed methods depend exclu- sively on natural language features. All these works have two main aspects in common: first, they are focused exclusively in online text-based debates. where information easy obtain, but very lim- ited from an argumentative viewpoint; and second, the debates brought into consideration present short interactions and simpler arguments than the ones that can be found in professional debate.
2 Related Work
Classically, the computational representation and assessment of arguments has been conducted through argumentation frameworks and argumen- tation semantics (Dung, 1995). However, this line of research has been focused on abstract argumen- tation and formal logic-based argumentative struc- tures, and has not been properly extended to the informal natural language representation of human argumentation.
The automatic assessment of natural language arguments is a relatively new topic of research that has been addressed from different NLP viewpoints. Most of this research has been focused on perform- ing an individual evaluation of arguments or ar- gumentative lines of reasoning (Wachsmuth et al., 2017) instead of a global, interactive viewpoint where complete debates consisting of multiple, con- flicting lines of reasoning are analysed. Typically, the automatic evaluation of natural language ar- guments has been carried out comparing the con- vincingness of pairs of arguments (Gleize et al., 2019); analysing user features such interests or personality to predict argument persuasiveness (Al Khatib et al., 2020); and analysing natural lan- guage features of argumentative text to estimate its persuasive effect (E) Baff et al., 2020). The use of graph-based approaches to evaluate individ- ual argument structures has been recently explored in (Saveleva et al., 2021). In the same direction, (Marro et al., 2022) proposes framework for eval- uating three dimensions of arguments (i.e., cogency, rhetoric, and reasonableness) by producing natu- ral language embeddings from individual argument structures (c.g.. claim- premise). However, none of them considers the problem of argument evaluation
Interestingly, recent trend in the proposed argu- ment assessment algorithms from the more theoret- ical side of the computational argumentation area of rescarch also consists of leveraging structural information of the graph to estimate the acceptabil- ity of arguments by using neural networks instead of classic solvers (Kuhlmann and Thimm, 2019; Malmqvist et al., 2020; Craandijk and Bex, 2021). However, in these cases arguments are treated abstract entities without specific natural language being attributed to them, making their results more difficult to contextualise a real situation with natural language arguments.
The previously reviewed work evidences that fundamental concepts from computational argu- mentation theory are typically overseen in the argument-related NLP literature, and the used cor- pora contain debates far removed from the concept of a professional debate. Thus, we propose a new method that combines the advantages of both areas of research: formal argumentation theory and NLP. This way, our proposal enables the analysis of com- plete professional argumentative debates in both, length and argumentative depth, a task that has not been addressed in the literature yet.

---

3 Data
contains its own text, its stance (i.e., in favour against the topic of the debate), the phase of the debate where it has been uttered (i.e., introduction, argumentation, and conclusion), and a set of argu- mentative relations (i.e., inference, conflict, and rephrase) that make possible to capture argumenta- tive structures, the sequentiality in the debate, and the existing major lines of reasoning. Additionally, each debate has the scores of the jury that indicate which team has proposed a more solid and stronger argumentative reasoning. An in-depth analysis of the corpus structure and statistics can be found in (Ruiz- Diizz et al., 2021b).
In this paper, we approach the automatic prediction of the winning stance in complete natural language professional debates. For that purpose, we use the VivesDebate' corpus (Ruiz-Dolz et al., 2021b) to conduct all the experiments and the validation of our proposed method. This corpus contains the annotations of the complete lines of reasoning pre- sented by the debaters in a debate tournament, in- spired by the AIF (Chesnevar et al., 2006) standard, and the professional jury evaluations of the qual- ity of argumentation presented in each debate. is important to emphasise this aspect, as the av- erage length of the debates we analysed in this paper is 4819 words (30-40 minutes of length), and large language models have problems when working with long sequences of natural language text (Beltagy et al., 2020) Previously published corpora for the analysis of natural language argu- mentation always tended to simplify the annotated argumentative reasoning, by only considering in- dividual arguments, pairs of arguments, or consid- cring a small set of arguments, instead of deeper and complete lines of argumentative reasoning. For example, in argument mining (e.g.. US2016 (Visser et al., 2020) and Q730 (Hautli-Janisz et al., 2022)), argument assessment (c.g.. IBM-EviCom (Gleize et al., 2019)), natural language argument genera- tion/summarisation (e.g.. GPR-KB (Orbach et al., 2019), DebateSum (Roush and Balaji. 2020)). Fur- thermore, online debates with their crowd-sourced evaluations were compiled in (Durmus and Cardie, 2019), but argumentation was produced in short written paragraphs, and evaluations were based on anonymous votes from the community that did not require any justification. Therefore, the VivesDe- bate corpus is the only identified publicly available corpus that enables the study of the automatic eval- uation of natural language professional debates in their complete form.
4 Method
The human evaluation of argumentative debates complex task that involves many different as- pects such as the thesis solidity, the argumentation quality, and other linguistic aspects of the debate (e.g., oral fluency, grammatical correctness, etc.). It is possible to observe that both, the logic of argu- mentation and the linguistic properties play ma- jor role in the evaluation of argumentative debates. Therefore, the method proposed in this paper is designed capture both aspects of argumentation by combining concepts from computational argu- mentation theory and NLP. Our method is divided into two different phases: first, (i) determining the acceptability of arguments (i.e., their logical valid- ity) in a debate based on their logical structures and relations; and second, (ii) scoring the resulting acceptable arguments by analysing aspects of their underlying natural language features determine the winner of a debate. Figure presents an scheme with the most important phases and elements of the proposed method.
Before describing both phases of our method, it is important to contextualise proposal within the area of computational argumentation research. We assume that the whole argument analysis of natural language text has already been carried out: the argumentative discourse has been segmented, the argument components have been classified, and argument relations have been identified among the segmented argumentative text spans (see (Lenz et al., 2020)). Thus, a graph structure fined from given natural language argumenta- tive input. As depicted in Figure 1, the Argument Analysis containing the text of the arguments (i.e., node content), their stance (i.c., node colour), infer- ence relations (i.e., green edges), conflict relations
The VivesDebate corpus contains 29 complete argumentative debates (139,756 words) from uni- versity debate tournament in Catalan. Each debate is annotated entirely without partitions, and cap- turing the complete lines of reasoning presented by the debaters. The natural language text is seg- mented into Argumentative Discourse Units (7,810 ADUs) (Peldszus and Stede, 2013). Each ADU
be de-
'Avallable online https://doi.org//u 5281/
zenodo. (edoddo.531455 483
"The Longfonner supports sequences of up to 4096 tokens.

---

Phase
Argument Analysis
Argumentation
Semantics
Framework
Debate
Learning Sample
Phase II
[..]
[..]]
Winner
Predictor
[...]
dBen6umg Mioeli
Looser
Graph Network
Sample Init.
[..]]
[..]]
Figure I: Structural scheme of the proposed automatic debate evaluation method.
(i.e., red edges), and rephrase relations (i.e., yellow edges) among arguments can be valid starting point to the proposed method.
Thus, consider that set of arguments is conflict free if there are not any attacks between arguments belonging the same set:
(Lenz et al., 2019)
Definition 2 (Conffict-freee) Let AF A, R be argumentation framework and Args A. The set of arguments Args conflict-free iff -3ag. aj E Args (a;; Qrz) E R.
4.1 Phase I: Argument Acceptability
The first phase of the proposed method relies on concepts from computational argumentation theory. This phase can be understood as a pre-processing step from the NL. research viewpoint. Thus, the main goal of Phase / is analyse the argumenta- tive information contained in the argument graph, and computationally encode this information focusing on the most relevant aspects for natural language argumentation (see Figure 1, Framework Encoding and Argumentation Semantics).
We also consider that a of arguments is admissible if, in addition of being conflict-free, it able to defend itself from external attacks:
Definition 3 (Admissible) Let AF A. R be argumentation framework and Args A. The set of arguments Args is admissible iff Args is conflict-rree, and Vas E Args, -3a E A (at, ax)) E and (as,as) $ R, or -dog E Args: (aj, aq) E R
For that purpose, is necessary to introduce the concept of an abstract argumentation framework and argumentation semantics. Originally proposed by Dung in (Dung, 1995), argumentation frame- work is a graph-based representation of abstract (i.e., non-structured) arguments and their attack relations:
In this paper, we compare the behaviour of these two properties through the use of Naïve (conflict- free) and Preferred (admissible) semantics to com- putc all the acceptable extensions of arguments from the AF representations of debates. Naïve mantics are defined maximal (w.rt. set inclusion) conffict-free sets of arguments in given AF. Simi- larly, Preferred semantics are defined as maximal (w.rtt set inclusion) admissible sets of arguments in a given AF.
Definition (Argumentation Framework) An Argumentation Framework (AF) is tuple AF A, R > where: A is a finite set of arguments, and R is the allack relation on A such A A R.
At this point, it is important to remark that the criteria of selecting both semantics for our method is oriented by the principle of maximality. Since acceptable extensions will be used as samples to train the natural language model in the subsequent phase of the proposed method, we selected these se- mantics that allow us obtain the highest number of extensions, but keeping the most of the natural language information and maximising differences
Furthermore, argumentation semantics were pro- posed along with the AFs as set of logical rules to determine the acceptability of abstract argu- ment set of arguments. In this paper, following one of the most popular notations in argumentation theory, we will refer to these sets as acceptable extensions. These semantics rely on two essential set properties: conflict-freeness and admissibility.

---

return AF among the extensions (i.e., not accepting the sub- sets of given maximal extension, which would result data redundancy and hamper the learning process of the model).
GN receives graph as an input containing ini- tialised node features (i.c., U.,.... ........ data (i.e., (e1, .1),............ ................................ (Ck, Tk: 8k) E, where e are the edge features, r is the receiver node, and 8 is the sender node), and global features (i.e., u); and updates them according to three learnt update and three static aggregation p functions:
E V), edge
Algorithm 1 Argumentation Framework Encoding.
1: function GRAPHTTOAKKArruueenGGruu
2: AG < ArgumentGraph
3: rt AG.edges( tendges("conflicee flinctt"
& = x (cxs Ure Uspi 1)
C = p (E))
4: AG.remove Edgess
5:
cc + AG..ooneected
v - (c w' - (e,
= p (E')
(I)
6: AF <- NewGraph()
o' - p "(V"
7: for subgraph E cc
8:
arg + 0
This way, of computes edge-wise update of edge features, o² updates the features of the nodes, and 6² computed at the end, updating global graph features. Finally, p functions must be com- mutative, and calculate aggregated features, which are used in the subsequent update functions.
9: for node & subgraph do
10:
arg.append(node.Daie Data
11:
end for
12:
AF.addN .F..adNode((arg) (arga)g)
end for AFgessEdges(r)
Thus, the first step in Phase II is to build the learning samples from the previously computed extensions of AFs (see Figure Learning Sam- ple). An extension set of logically acceptable arguments under the principles of conffict-freeness and/or admissibility, However, there are no explicit relations between the acceptable arguments, since AF representations only consider attacks between arguments, and the conflict-fre principle states that there must be no attacks between arguments belonging to the same extension. Thus, in order to structure the data and make it useful for learning linguistic features for the debate evaluation task, we generate a complete bipartite graph from each extension. The two disjoint sets of arguments are determined by their stance (i.e., one set consisting of all the acceptable arguments in favour, and the other against), since argumentation semantics al- low to define sets of logically acceptable arguments but do not guarantee that they will have the same claim similar stance.
15:
16: end function
Therefore, we encode the argument graphs, re- sulting from a natural language analysis of the de- bate. abstract AF's using the proposed Algorithm 1. ADUs that follow the same line of reasoning (i.e., related with inference rephrase) abstract arguments, and the existing conflicts be- tween ADUs are represented with the attack re- lation of the AF. Then, both Naïve and Preferred semantics are computed on the AF representation of the debate. This leads to a finite set of extensions, each one of them consisting of set of acceptable arguments under the logic rules of computational argumentation theory. These extensions will be used learning samples for training and evaluat- ing the natural language model in the subsequent phase of the proposed method.
grouped into
4.2 Phase II: Debate Outcome Estimation
The second step consists on initialising all the required features of the learning samples for the GN architecture (see Figure 1, Sample Init.). Thus, we define which features will encode edge, node, and global information of the previously processed bipartite graph samples. Edges do not contain any relevant natural language information, tialise edge features identically (similar to previ- ous research (Craandijk and Bex, 2021)), so that node influence can be stronger when learning edge update functions. Nodes, however, are a pivotal aspect of this second phase since they contain all
The second phase of the method focuses on analysing the natural language arguments con- tained in the acceptable extensions, and determin- ing the winner of given debate. For that pur- pose, we use the Graph Network (GN) architecture combined with Transformer-base beddings generated from the natural language ar- guments contained in the acceptable extensions. A GN machine learning algorithm aimed learn- ing computational representations for graph-basee based data structures (Battaglia et al., 2018). Therefore,
sentence em-
ini-

---

the natural language data. Node features are ini- tialised from sentence embedding representations of the natural language ADUs contained in each node. Thus, we propose the use of a pre-trained language model to generate dense vector represen- tations of these ADUs, and initialise the vector features for learning the task. Finally, the global features of our learning samples encode the proba- bility distribution of winning/losing given debate (represented as acceptable Axtension-based bipar- tite graphs), and are binary label that indicates the winning stance (i.c., O for the team in favour and for the team against).
et al., 2008) libraries. Argumentation semantics have been implemented considering the NetworkX- based AF graph structures. Regarding Phase II, the language model and the dense sentence vec- tor embeddings have been implemented through the Sentence Transformers library (Reimers and Gurevych, 2019). We used a pre-trained XLM- RoBERTa architecture (Reimers and Gurevych, 2020) able encode multilingual natural language inputs into a 768 dimensional dense vector space (i.e., word embedding size). Finally, the Jraph' library has been used for the implementation of the graph network architecture, for learning its update functions (i.e., Equation 1), and for the probabilis- tic modelling defined in Equation 2. We used an Intel Core i7-9700k computer with an NVIDIA RTX 3090 GPU and 32GB of RAM to run all our experiments. The code implementation of the proposed method and the subsequent exper- iments is publicly available in https: / //g thub. com/ http//rruddoo/// httprrruooo//rruennttveeeee /wrrrrrrunnaalaaatonnn
The final step in the Phase II of our proposed method is focused on learning the automatic evalua- tion of argumentative debates (see Figure 1, Graph Network). In classical debate, there are always two teams/stances; in favour and against some spe- cific claim. In this paper, we approach the debate evaluation as a binary classification task. There- fore, at the end of the proposed method, we model the classification problem as follows:
It is also important to completely define the no- tion behind learning sample in the experimental setup, and how the data pipeline manages all these samples and structures them for training/evaluation. In our proposal, we defined a learning sample as an acceptable extension of given debate. Thus, different debates may produce a different number of learning samples depending on the argumen- tation semantics and/or the argumentation frame- work topology. This way, learning samples be managed from debate-wise wise viewpoint. Even though we used the learning samples individually (i.e., extension wwise) for the training of the proposed models, we will always consider debate-wing wise partitions of our data in our experimental setup. This decision has been made because it would be unfair to consider learning samples belonging to the same debate in both our train and test data partitions. The reported results could be misleading, and would not properly re- flect the strengths and weaknesses of our method. Therefore, we used 29 complete debates in our ex- periments (18 in favour, 11 against). To create the train-test data splits, we assigned 23 debates (80%) to the train split and 6 debates (20%) to the test split.
è arg max P(c[G) rEC
(2)
where C - ["F", "A"), depending on the win- ner of each debate (i.e., in *F" avour or "A" gainst. And G is a complete bipartite graph generated from the acceptable extensions of the AF pre-processing described in the Phase 1 of our method. We ap- proach this probabilistic modelling with three Multi Layer Perceptrons (MLP) consisting of two layers of 128 hidden units for each of the e update func- tions. Since the debate evaluation is an instance of the graph prediction task, it is important to point out that the architecture of the two MLP approaching df and d² are equivalent, and their parameters are learnt from the backpropagation of the MLP archi- tecture for dt Finally, the proposed GN model has a 2-unit linear layer (for binary classification) and a softmax function (for modelling the probability distribution) on its top.
extension-
5 Experimental Analysis
5.1 Experimental Setup
All the experiments and results reported in this pa- per have been implemented using Python and run under the following setup. The initial cor- pus pre-processing and data structuring (i.e., Phase 1) has been carried out using Pandas (McKinney et al., 2010) together with NetworkX (Hagberg
To evaluate and validate our proposal, we have calculated the values of the performance scores averaged after 3 sequential runs with a different
Phttps://jraph. https////jaaphrrrettheeeeossss

---

(random) initialisation. As for the performance scores, we have calculated the precision, the recall, and the weighted average Fl score in all of our experiments. We decided to report the weighted av- erage of the F1 score due to the existing variability of the class distributions on the learning samples generated from different debates included in the test split from one run to another.
graphs resulting from the argumentative analysis.
5.3 Our method
Apart from the baselines, we have experimented with two variants of our proposed method: the Naîve-GN and the Preferred-GN. In the Naive-GN. we applied the Naive semantics during the Phase 1, and then trained the graph network using the learning samples generated based on the principle of conflict-freedom. Conversely, in the Preferred- GN approach, we calculated the Preferred exten- sions during Phase and trained the graph network with the learning samples resulting from applying the principle of admissibility to our argumentation graphs. In the end, we obtained a total number of 471 Naive and 32 Preferred extensions from the 29 debates. The 471 Naive extensions were distributed follows: 203 learning samples be- longing to class 0 (i.e., in favour team wins in the 43. 1% of the cases), and 268 samples belonging to class (i.c., against team wins in the 5. of the cases). On the other hand, the 32 Preferred extensions were distributed as follows: 19 learn- ing samples belonging to class 0, and 13 samples belonging to class 1.
5.2 Baselines
We defined five baselines to compare the perfor- mance of our proposed method and validate our contribution. The selected baselines have been defined to provide better understanding of the benefits of using transversal approaches a prob- lem such as the prediction of the winning stance in a complete and long argumentative debate, where complex natural language and argumentative rela- tions are present.
For that purpose, we have used a random base- line (RB) that assigned randomly class to each debate as our most basic baseline. Moreover, we have considered two baselines that rely exclusively on concepts from computational argumentation the- ory (i.e., semantics) to determine the winner of a debate. These are the Naive argumentation the- ory baseline (Naive-ATB) and the Preferred argu- mentation theory baseline (Preferred-ATB) which compute diferent set of acceptable extensions based on the different admissibility principles, and then calculate the majoritarian set of arguments grouped by stance. Therefore, the winning team is decided by having more acceptable arguments in the extensions produced by the argumentation semantics. To compare argumentation theory ex- clusive approaches with algorithms relying exclu- have defined two
From the variations in data distributions in both approaches, it is possible to observe how the admis- sibility principle is much more strict from logical perspective than the conflict-free principle, and has significant repercussion on the number of accept- able extensions (i.e., learning samples) produced. Therefore, more learning samples are produced by the Naive semantics which can be leveraged by the Phase Il of the proposed method.
5.4 Results
As we can observe in Table 1, the best results in all the evaluation metrics were obtained by the Naïve- GN approach. The Preferred-GN was the second best in terms of precision and recall, but performed similar the argumentation theory baselines in terms of Fl. Argumentation theory baselines, how- ever, performed similar to the random baseline in all the metrics. This observation is telling us that relying exclusively on formal logic aspects in infor- mal environments such as natural language debates, must not be enough to approach linguistically com- plex tasks such as the automatic evaluation of natu- ral language argumentation. Finally, we have also been able to observe that both baselines relying exclusively on NLP algorithms and techniques per- formed worse than the random baseline. The main
sively in NLP techniques, more baselines. The first one is the Longformer (Beltagy al., 2020) architecture, which we have fine-tuned to predict the winner of a debate directly from the text transcripts of the debates. We decided to use this model since it represents one of the best options to deal with longer texts in the bate-offthee art. We fine-tuned the model for 3 epochs our training data with a learning rate of 5e-5. As for the second natural language baseline, we have used the Graph Network Baseline (GNB), in which we ignored the Phase of our proposed method and we trained the graph network directly on the argument
'https: hhttpp://huggnggffae httpa//nnnaaaeaaaaalaa/aaeaeaaaa ct/al//aalla co/allenai/ longformer-base-4488

---

Experiment Model
Evaluation Metrics
Precision
Recall
Weighted-F1
0.5
0.32
RB
0.59
0.55
0.55
0.22
0.31
0.17
0.33
Naive-ATB
0.58
0.46
0.48
Longformer
Graph Network Baseline
Preferred-AII ATB
0.51
0.55
0.53
0.5
0:45
Longformer
0.50
0.25
0.33
0.5
0.33
0.2
Nalve-GN
PrefemediGGN
GNB
0.29
0.44
0.33
0.22
0.24
0.5
0.1
Naïve-GN 0.64
0.65
0.64
Prefemed-GN 0.62
0.61
0.52
Favour
Against
Against
Table I: Precision, recall, and weighted-F1 results of the automatic debate evaluation task. The reported re- sults have been averaged from 3 randomly initialised sequential runs.
Figure Aggregated confusion matrices.
represent the error distributions, have calculated the percentages after 3 runs of debates classified as in Favour (F) or Against (A) winning stances. We can observe how the ATBs present an almost uniform distribution the error, similar to what can be expected of a random baseline. This goes in line with the reported results in the previous section. Conversely, the Longformer assigned to every debate the F winning stance (i.c., the majority class) giving more emphasis the needs for larger corpora to make use of LLMs. Finally, we can ob- serve the impact of the differences between the use of Naive and Preferred semantics in our methods on the error distributions. With the Naïve-GN, the model managed to correctly identify almost all the debates where A was the winning stance, but had some problems when doing it in Favour. On the other hand, the Preferred- GN did the opposite, all the debates where was the winning stance were correctly classified, but in the case of A debates, the model miss-classified most of them.
cause of this problem can be probably attributed to the lack of data our domain. State-of-the- art NLP algorithms rely on large amounts of data, which we do not have when it comes to analysing and evaluating complete professional debates in natural language.
We have also detected some interesting findings when looking specifically at each group of exper- imental baselines. Regarding the ATBs, it is pos- sible to observe how, after runs, the Preferred- ATB is consistently providing better results than the Naīve-ATB. This finding makes lot of sense, since (despite their bad general performance) relying on the principle of admissibility is more informative from the argumentative viewpoint than doing on similar be-
the principle of conflict-freedmm haviour can be found within the NLP baselines. We can observe that the GNB has a significantly better recall than the Longformer, meaning that the GNB is doing better generalisation than the Longformer. This behaviour can be attributed to the fact that, apart from the limitations in the size of the corpus for relying exclusively on state-of- the-art NLP techniques, learning representations from graph-structured data is better idea than just using the whole text (i.e., debate transcripts) as the unique input for the models.
Discussion
In this paper, we have defined an original hybrid method approach the winning stance prediction of complete natural language argumentative de- bates. For that purpose, we present a new instance of the debate assessment task, where argumentative debates and their underlying lines of reasoning are considered in comprehensive, undivided manner.
5.5
Error Analysis
The proposed method combines aspects from for- mal logic and computational argumentation theory, with NLP and Deep Learning. From the observed results, several conclusions can drawn. First, has been possible to determine that our method performed better than approaching independently
Apart from looking at the performance scores, have also analysed the behaviour of the proposed method and the baselines from the error perspec- tive. Figure 2 depicts the aggregated confusion matrices of the three runs reported in Table To

---

the debate evaluation task from either the argu- mentation theory or the NLP viewpoints. Further- more, we have observed in our experiments that conflict-free semantics produce a higher number of acceptable extensions from each AF compared to the admissibility-basee semantics. This helped to improve the learning process of the task in similar way to that achieved by data augmentation techniques. Thus, better probabilistic models of natural language distributions that are not too con- strained formal logic and graph topology can be learnt by our model.
Grant TEED2021-1312955--22 funded by AFI/00.3035/50110001000/// European Union NextGeneraationEU//PRRR Spanish Government project PID2022-111341RBB IO0, and the 'AI for Citizen Intelligence Coaching against Disinforma- tion (TITAN) project, funded by the EU Horizon 2020 research and innovation programme under grant agreement 101070658, and by UK Research and innovation under the UK governments Horizon funding guarantee grant numbers 10040483 and 10055990.
This paper represents a solid starting point of research in the evaluation of complete natural lan- guage professional debates. As future work, we are interested in considering finer-grained features for the evaluation of argumentation such thesis solidity, argumentation quality, and adaptability. We also plan to extend our method with acoustic features, considering aspects such as the intonation or the fluency.
References
Khalid Al Khatib, Michael Völske, Shahbaz Syed, Niko- lay Kolyada, and Benno Stein. 2020. Exploiting personal characteristics of debaters for predicting persuasiveness. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7067-7072, Online. Association for Computational Linguistics.
Gianvincenzo Alfano, Sergio Greco, Francesco Parisi, and Irina Trubitsyna. 2021. Argumentation frame- works with strong and weak constraints: Semantics and complexity. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages
Limitations
The present paper describes a new method for the automatic evaluation of complete argumentative debates. However, several limitations are described along the paper. Even though the proposed method can be generalised to any argumentative domain, the reported results are constrained to the domain of the corpus used in the experiments to validate our proposal. As discussed in Section 3, it has not been possible to identify any other corpus suitable for upproaching the task as presented in this work. Thus, we are not able to evaluate our approach when considering different argumentation topics domains. Furthermore, used different base- lines validate the improvements achieved by our proposed method, but was not possible to use pre- vious research as reference. Finally, since we used real debates, the experimental configuration was also highly dependent on the specific debate struc- tures. Extending the corpus may help to provide
Katie Atkinson, Pietro Baroni, Massimiliano Gia- comin, Anthony Hunter, Henry Prakken, Chris Reed, Guillermo Simari, Matthias Thimm, and Serena Vil- lata. 2017. Towards artificial argumentation. Al magazine, 38(3)25-33.
Jianzhu Bao, Chuang Fan, Jipeng Wu, Yixue Dang, Ji- achen Du, and Ruifeng Xu. 2021. A neural transition- based model for argumentation mining. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6354 6364, Online. Association for Computational Linguistics.
Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kan- tor, Dan Lahay, and Noam Slonim. 2020. From ar- guments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, pages 4029-4039, Online. Association for Computational Linguistics.
more solid results and explore the performance of our proposal when generalising to multiple topics and domains.
Peter W Battaglia, Jessica Hamrick, Victor Bapst, Al- varo Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational in- ductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01 0/261.
Acknowledgements
This work has been developed thanks to the funding of the following projects:
Iz Beltagy, Mathew E Peters, and Arman Cohan. 2020. Longformer: The Long-document transformer. arXiv preprint 05X
Grant PID2021-1236330B-C1 funded by MCIN/AEI/ 13039/50 100011100,,

---

Martin Caminada. 2006. On the issue of reinstatement in argumentation. In European Workshop on Logics in Artificial Intelligence, pages 111-123. Springer.
Fa-Hsuan Hsiao, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2022. Modeling inter round attack of online debaters for winner prediction. In Proceedings of the ACM Web Conference 2022, pages 2860-2869.
Carlos Chesnevar, Sanjay Modgil, Iyad Rahwan, Chris Reed, Guillermo Simari, Matthew South, Gerard Vreeswijk, Steven Willmott, et al. 2006. Towards an argument interchange format. The knowledge en- gineering review, 21 21(4):293-316
Yohan Jo. Jacky Visser, Chris Reed, and Eduard Hovy, 2019. A cascade model for proposition extraction in argumentation. In Proceedings of the 6th Workshop Argument Mining, pages 1--44, Florence, Italy. Association for Computational Linguistics.
Dennis Craandijk and Floris Bex. 2021. Deep learning for abstract argumentation semanties. In Proceed- ings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intel- ligence, pages 1667-1673.
Isabelle Kuhlmann and Matthias Thimm. 2019. Us- ing graph convolutional networks for approximate reasoning with abstract argumentation frameworks: A feasibility study. International Conference on Scalable Uncertainty Management, pages 24-37. Springer.
Phan Minh Dung. 1995. On the acceptability of ar- guments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence. 77(2):321-377
John Lawrence and Chris Reed. 2019. Argument min- ing: A survey. Computational Linguistics, 45(4):765- 818.
Esin Durmus and Claire Cardie, 2019. A corpus for modeling user and language effects in argumentation on online debating. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguisties, pages 602-607, Florence, Italy. Associa- tion for Computational Linguisties.
Mirki Lenz, Premtim Sahitaj, Sean Kallenberg, Christo- pher Coors, Lorik Dumani, Ralf Schenkel, and Ralph Bergmann. 2020. Towards an argument mining pipeline transforming texts to argument graphs. Com- putational Models of Argument: Proceedings of COMMA 2020, 326:263.
Roxanne El Baff, Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. 2020. Analyzing the Per- suasive Effect of Style in News Editorial Argumenta- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3154-3160, Online. Association for Computational Linguisties.
Mirko Lenz, Stefan Ollinger, Premtim Sahitaj, and Ralph Bergmann. 2019. Semantic textual similar- ity measures for case-based retrieval of argument graphs. In Case- Case-Based Reasoning Research and De- velopment: 27th International Conference, ICCBR 2019, Orzenhausen, Germany, September 8-12, 2019, Proceedings 27, pages 219-234. Springer.
Martin Gleize, Eyal Shnarch, Leshem Choshen, Lena Dankin, Guy Moshkowich, Ranit Aharonov, and Noam Slonim. 2019. Are you convinced? choos- ing the more convincing evidence with a Siamese network. In Proceedings of the 57th Annual Meet- ing of the Association for Computationall Linguistics. pages 967-976, Florence, Italy. Association for Com- putational Linguistics.
Lars Malmqvist, Tommy Yuan. Peter Nightingale, and Suresh Manandhar. 2020. Determining the accept- ability of abstract arguments with graph convolu- tional networks. In SAFA@ @ COMMA, pages 47-56.
Santiago Marro, Elena Cabrio, and Serena Villata. 2022. Graph embeddings for argumentation quality assess- ment. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 4154-4164, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Pierpaolo Goffredo, Elena Cabrio, Serena Villata, Shohreh Haddadan, and Jhonatan Torres Sanchez. 2023. Disputool 2.0: A modular architecture for lulti-layer argumentative analysis of political de- bates. In Proceedings of the AAAI Conference on Ar- sificial Intelligence, volume 37, pages 16431-16433
Wes McKinney et al. 2010. Data structures for statisti- cal computing in python. In Proceedings of the 9th Python in Science Conference, volume 445, pages 51-56. Austin, TX.
Aric Hagherg, Pieter Swart, and Daniel S Chult. 2008 Exploring network structure, dynamics, and func- tion using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States).
Koh Mitsuda, Ryuichiro Higashinaka, and Kuniko Saito. 2022. Combining argumentation structure and lan- guage model for generating natural argumentative di- alogue. In Proceedings of the 2nd Conference of the Asia- Chapter of the Association for Compu- tational Linguistics and the 12th International Joint Conference on Natural Language Processing, pages 65 71.
Annette Hautli-Janisz, Zlata Kikteva, Wassiliki Siskou, Kamila Gorska, Ray Beeker, and Chris Reed. 2022. Qr30: A corpus of argument and conflict in broad- cast debate. In Proceedings of the 13th Language Resources and Evaluation Conference, pages 3291- 3300. European Language Resources Association (ELRA).
Matan Orbach, Yonatan Bilu, Ariel Gera, Yoav Kantor, Lena Dankin, Tamar Lavee, Lili Kotlerman, Shachar Mirkin, Michal Jacovi, Ranit Aharonov, and Noam

---

Slonim. 2019. dataset of general-purpose purpose rebuttal. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNLPP, pages 5591- 5601, Hong Kong, China. Association for Computa- tional Linguisties.
Bart Verheij. 2005. Evaluating arguments based on toulmin's 's scheme. Argumentation, 19(3):347-37)
Jacky Visser, Barbara Konat, Rory Duthie, Marcin Kos- zowy, Katarzyna Budzynska, and Chris Reed. 2020. Argumentation in the 2016 presidential elections: annotated corpora of television debates and social me- dia reaction. Language Resources and Evaluation, 54(1):123 154.
Andreas Peldszus and Manfred Stede. 2013. From ar-
gument diagrams to argumentation mining in texts: A survey. International Journal of Cognitive Infor- matics and Natural Intelligence (IJCINI), 7(1):1-31.
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberd- ingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguisties: Volume 1. Long Pa- pers, pages 176-187, Valencia, Spain. Association for Computational Linguistics.
Peter Potash and Anna Rumshisky. 2017. Towards de- bate automation: recurrent model for predicting debate winners. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Pro- Copenhagen, Denmark. Association for Computational Linguisties.
cessing. pages
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Allen Roush and Arvind Balaji. 2020. DebateSum: large-scale argument mining and summarization dataset. In Proceedings of the 7th Workshop on Ar- gument Mining, pages 1-7, Online. Association for Computational Linguistics.
Ramon Ruiz-Dolz, Jose Alemany, Stella M Heras Bar- berá, and Ana García-Fornes. 202 Transformer- based models for automatic identification of argu- ment relations: cross-domain evaluation. JEEE Intelligent Systems, 36(6):62-70.
Ramon Ruiz-Dolz, Montserrat Nofre, Mariona Taulé, Stella Heras, and Ana Gareia-Fornes. Fornes. 2021b. Vives- debate: A new annotated multilingual corpus of argu- mentation in debate tournament. Applied Sciences, 11(15):7160.
Ekaterina Saveleva. Volha Petukhova, Marius Mosbach, and Dietrich Klakow. 2021. Graph-based argument quality assessment. Proceedings of the Interna- tional Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1268 1280, Held Online. INCOMA Ltd.
Daiki Shirafuji, Rafal Rzepka, and Kenji Araki. 2019. Debate outcome prediction using automatic persua- siveness evaluation and counterargument relations. In LaCATODA/BiG@ IJCAI, pages 24-29.


Reviews:
[{"review_id": "2EKmTcKDe1", "reviewer": "Reviewer_cvvq", "summary": "", "questions": "A) Your proposed approach is evaluated only on a single dataset of Catalan debates. Can you discuss how you expect the hybrid technique to transfer to other debate domains and languages? Do you plan to test generalization capability on other debate datasets?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer cvvq,\n\nFirst of all we would like to thank you for your insightful feedback and constructive comments. In our response, we would like to address the two mentioned reasons to reject and the question with hopes of improving your impression in our work.\n\nWe would like to start my response by addressing your concern about the lack of some state-of-the-art related research by mentioning that we were aware of the mentioned works but we finally did not include them in our Related Work section because of the space limitation and their limited connection to our proposal. We wanted to make it clear since the very beginning of the paper the huge difference between the work done on the evaluation of individual arguments (even though they might contain some supporting/conflicting structures such as the ones included in Wachsmuth at al., 2017) from evaluating a complete argumentative debate where many different arguments are used by the teams representing two stances in a very interactive dialogue. Said that, we are willing to improve our work by including not only the mentioned references but other references such as (Lenz et al., 2019), (Kuhlmann and Thimm, 2019), (Malmqvist et al., 2020) and (Goffredo et al., 2023) where the concepts of argument mining, argumentation semantics, and argument graph networks are brought together. Given that in the camera ready version of the work we will have an additional page, the space limitation should not be a problem, and we will be able to improve the contextualisation of our work with other recent research in this direction.\n\nAnother important aspect criticised in your review is the lack of direct comparison between our proposal and other techniques for the task of debate evaluation. This is a more difficult challenge, mainly due to the nature of existing annotated corpora containing argumentative debate annotations. As discussed in the Section 3 of our paper, the VivesDebate is the first and only publicly available argumentation corpus that contains the annotations for the entire debate, i.e. it annotates all Argument Discourse Units of the complete debate, thus being able to establish relationships along the complete lines of argumentative reasoning presented by the debaters, including in addition the objective scores of a professional jury on the argumentation presented in each debate by each team. This is the main reason why a direct comparison of our method with previous research has not been possible. \n\nSo far, the existing debate datasets that contain relevant annotations to evaluate our method fall into two main categories: (i) contain short text-based debates/arguments (see Wachsmuth at al., 2017, Durmus and Cardie, 2019), or (ii) divide the argumentative debate transcripts into smaller snippets and note the argumentative structures and their relations in these snippets, but miss possible relations between arguments belonging to different text snippets, e.g. that have been proposed further apart in time (see Visser et al., 2020 or Hautli-Janisz et al., 2022).\n\nIn shorter (and simpler) argumentations, fitted LLMs might be sufficient to provide convincing results. However, in full argumentative debates such as those included in the VivesDebate corpus, the use of such approaches is limited by the size of the natural language inputs. For this reason, we have not been able to directly compare the performance of our proposed method to any previous work. However, in our evaluation we have not limited ourselves to naive baselines (e.g. random or majority), but have included algorithms used in similar recent research, such as the Longformer.\n\nFinally, we would like to respond your question. The language and the domain can undermine the generalisation of the trained models included in our system. However, as long as corpus of complete debates with their corresponding evaluations are shared in English, a more complete system where the complete argument mining pipeline combined with our proposal can be trained and evaluated. However, given the nature of our proposal, it might not be informative to evaluate our proposal in other kinds of shorter/simplified argumentative debates such as the one released in (Wachsmuth at al., 2017) or (Durmus and Cardie, 2019), since a LLM will be able to provide better results. The most interesting part of our proposal is approaching the challenge that represents the size of complete long argumentative debates, and the way in which we encode these debates so that they can be used to train Neural Network-based algorithms (a graph network in our case). We are also very interested of evaluating the generalisation capability of our system on other languages/domains and in including spoken features into the evaluation algorithm in future works.\n\nWe hope that with this response we will be able to improve your perception on the Soundness and Excitement of our work. \n\nThank you very much again for your time and dedication,\n\nThe authors.\n\nNEW REFERENCES\n\nLenz, M., Ollinger, S., Sahitaj, P., & Bergmann, R. (2019). Semantic textual similarity measures for case-based retrieval of argument graphs. In Case-Based Reasoning Research and Development: 27th International Conference, ICCBR 2019, Otzenhausen, Germany, September 8–12, 2019, Proceedings 27 (pp. 219-234). Springer International Publishing.\n\nKuhlmann, I., & Thimm, M. (2019, December). Using graph convolutional networks for approximate reasoning with abstract argumentation frameworks: A feasibility study. In International Conference on Scalable Uncertainty Management (pp. 24-37). Cham: Springer International Publishing.\n\nMalmqvist, L., Yuan, T., Nightingale, P., & Manandhar, S. (2020). Determining the Acceptability of Abstract Arguments with Graph Convolutional Networks. In SAFA@ COMMA (pp. 47-56).\n\nHautli-Janisz, A., Kikteva, Z., Siskou, W., Gorska, K., Becker, R., & Reed, C. (2022, June). Qt30: A corpus of argument and conflict in broadcast debate. In Proceedings of the 13th Language Resources and Evaluation Conference (pp. 3291-3300). European Language Resources Association (ELRA).\n\nGoffredo, P., Cabrio, E., Villata, S., Haddadan, S., & Sanchez, J. T. (2023, June). DISPUTool 2.0: A Modular Architecture for Multi-Layer Argumentative Analysis of Political Debates. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 13, pp. 16431-16433).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Tackles a complex and underexplored NLP problem - evaluation of professional debates with complete logical argumentation - using a hybrid approach.\n\n- The details of the hybrid technique combining concepts from argumentation theory seem sound and applicable to debates. \n\n- Well-written with a clear explanation of all concepts for audiences unfamiliar with argumentation theory.\n\n- Provides compelling evidence that bridging logical and neural techniques is beneficial for argument-mining problems.\n\n- Could inspire more creative hybrid modelling and use of structured knowledge in NLP systems.", "reasons_to_reject": "My biggest issue with this paper is the lack of comparison to other techniques for the debate evaluation task, which makes it difficult to situate the novelty of the proposed approach. The authors only compare against baselines they define, without discussing prior proposals from the literature. Given the lack of comparisons, it is unclear whether this approach substantially advances upon the state-of-the-art for this task.  \n\nAdditionally, the literature review overlooks highly relevant prior work. The authors claim in line 110 that \"Most of this research has been focused on performing an individual evaluation of arguments or argumentative lines of reasoning (Wachsmuth et al., 2017) instead of a global, interactive viewpoint where complete debates consisting of multiple, conflicting lines of reasoning are analysed\". However, Wachsmuth et al. (2017) proposed a Reasonableness dimension that evaluates the whole argument, considering counter-arguments and rebuttals. In a debate scenario, counter-arguments and rebuttals are what constitute the bare bones of it so one would imagine that a dimension like Reasonableness would at least be discussed. Furthermore, Marro et al. (2022) also introduced a technique to assess the reasonableness of full argument graphs, combining text and graph structure. Yet neither of these directly related works are acknowledged or compared to. The lack of citations and comparisons to existing techniques that also evaluate full argument graphs raises concerns. The paper does not sufficiently situate its contributions among prior methods or demonstrate advancement over closely related approaches. A more rigorous review and comparison to previous debate evaluation and reasonableness assessment research is needed.", "questions_to_authors": "A) Your proposed approach is evaluated only on a single dataset of Catalan debates. Can you discuss how you expect the hybrid technique to transfer to other debate domains and languages? Do you plan to test generalization capability on other debate datasets?", "paper_topic_and_main_contributions": "This paper proposes a new method for automatically evaluating and predicting the winning stance in professional argumentative debates. The key problem it addresses is performing debate analysis and evaluation on complex natural language arguments, which have been relatively unexplored. \n\nThe main contributions are:\n\n- Introduces a hybrid technique combining concepts from argumentation theory and NLP to model complete reasoning lines in debates and determine a winning stance.\n\n- Demonstrates superior performance of the hybrid technique over pure NLP or argumentation theory baselines on the debate evaluation task.\n\n- Provides detailed experimental analysis and error analysis lending insight into the benefits of combining logical and linguistic knowledge for this problem.\n\nOverall, this engineering-focused paper makes contributions in methods, and analysis towards advancing NLP for a complex argumentation problem.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- [Graph Embeddings for Argumentation Quality Assessment](https://aclanthology.org/2022.findings-emnlp.306) (Marro et al., Findings 2022)\n\n- [Yes, we can! Mining Arguments in 50 Years of US Presidential Campaign Debates](https://aclanthology.org/P19-1463) (Haddadan et al., ACL 2019)"}}, {"review_id": "Xyp1UCTOKv", "reviewer": "Reviewer_5zCQ", "summary": "", "questions": "- You wrote that \"both baselines relying exclusively on NLP algorithms and techniques performed worse than the random baseline. The main cause of this problem can be probably attributed to the lack of data in our domain\". But did you consider testing simpler probabilistic models for NLP, such as a Bayesian classifier, which performs well with small datasets?\n\n- Did you consider comparing the results of your method with those of a generative LLM, which have been widely used for complex NLP tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5zCQ,\n\nFirst of all, we would like to thank you for your review and your constructive comments.\n\nIn our response, we would like to address the two questions and the mentioned reasons to reject with hopes of improving your impression in our work.\n\nRegarding the first question, we considered the use of Support Vector Classifiers and Linear Regressors in our experiments. However, in our task setup, each debate represents a sample with an assigned label to predict the winning team (i.e., In favour or Against). Each debate has an average length of 4819 words (approximately 40 minutes of continued argumentative natural language being uttered by both teams). Such a complex input is a huge constraint for almost every NLP algorithm from classical Machine Learning to recent Deep Learning and generative LLMs. This is the main reason why it was not feasible to include simpler probabilistic models in our results section.\n\nThe answer to the second question and the provided reasons to reject is closely related to the answer provided above. Our answer is that yes, we tried to use GPT-3.5-turbo in the automatic evaluation of debates, but the debates were longer than what the OpenAI API allowed us to use as a prompt. Therefore we were not able to test these approaches in our work. However, I am sure that it is going to be an interesting line of research to be explored in the near future.\n\nThank you very much again for your time and dedication,\n\nThe authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work achieved results surpass those previously reported in the literature for this task.\n\n- Detailed analyses present original and interesting outcomes of argumentation processing. The realization that \"learning representations from graph-structured data is a better idea than just using the whole text\" holds significant importance within the current context of NLP research, wherein generative LLMs are garnering widespread attention.\n\n- The incorporation of error analysis enriched the discussions.\n\n- The text provides a concise and well-structured literature review on automatic debate processing.\n\n", "reasons_to_reject": "- Despite the strengths of the proposed work, the attained F1 score by the best model still appears relatively low. As a matter of fact, this is not a poor outcome when considering the conventional models employed for the task, but it's possible that the proposed model might be outperformed by newer Generative LLMs. It would be important to discuss the influence of these novel models on the task's execution.", "questions_to_authors": "- You wrote that \"both baselines relying exclusively on NLP algorithms and techniques performed worse than the random baseline. The main cause of this problem can be probably attributed to the lack of data in our domain\". But did you consider testing simpler probabilistic models for NLP, such as a Bayesian classifier, which performs well with small datasets?\n\n- Did you consider comparing the results of your method with those of a generative LLM, which have been widely used for complex NLP tasks?", "paper_topic_and_main_contributions": "The article presents an original method for evaluating the winning stance in complete debates, under real-world circumstances involving professional debaters. The proposed method is a hybrid approach that combines argumentation frameworks and semantics on one side, and Graph-Network and Transformer-based architectures on the other.\n\nThe argumentation frameworks module categorizes arguments as either Conflict-free or Admissible. Subsequently, the arguments are processed by a hybrid model that integrates a Graph-Network architecture with embeddings generated by a Transformer model. The winner is estimated as the participant with the highest count of acceptable arguments.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 452: has been carried OUR using Pandas\n\n- Line 624-5: learning representations from A graph-structured data"}, "extra_text": {}}, {"review_id": "WxqqTDM8lo", "reviewer": "Reviewer_6Vnd", "summary": "", "questions": "A. How is the removal of arguments having an attack relation on them justified?\nB. Question on extension of work - often debates are not neatly categorized into phases of speech and rebuttal - how does this architecture handle that?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 6Vnd,\n\nFirst of all we would like to thank you for your insightful feedback and constructive comments. In our response, we would like to address the two mentioned reasons to reject and the questions with hopes of improving your impression in our work.\n\nRegarding the first concern and reason to reject, we would like to provide further clarifications. First of all, we have evaluated two different sets of argumentation semantics: Naive and Preferred. The former are based on the concept of conflict-freeness, meaning that all the different sets of arguments that do not contain conflicting arguments are acceptable. The later are based on the concept of admissibility, meaning that in addition to being conflict free, the set of acceptable arguments must be able to defend themselves from the attacking arguments. Let us illustrate these two semantics with a very basic example, let us assume argument A attacking argument B. Considering the Naive semantics, we will have two sets of acceptable arguments {A} and {B}. Conversely, considering the Preferred semantics we will only have one set of acceptable arguments: {A}. Therefore, our proposed method does not remove arguments that have an attack relation on them, only the implementation with Preferred semantics might do that. Furthermore, the best performing implementation of our proposed method is the one based on the Naive semantics, and among other possible reasons this might be one of them to make it more successful (as discussed in the paper). \n\nOn the other hand, the second major concern is the specific impact of atomic arguments to the probability of winning/loosing a debate. In our method, we use the Graph Network architecture to automatically score the whole debate rather than scoring individual arguments and aggregating these scores to predict the winner/looser. Therefore, the GN takes into consideration the natural language included in the arguments and the conflicting relations to learn the parameters and estimate the outcome of a debate. However, we agree that the name of the section may be misleading for the reader and we will modify it for the camera ready version of the paper to: 4.2 Debate Outcome Estimation.\n\nFinally, we would like to answer the questions. Regarding the question A, we provided a clear justification of it in the first paragraph of our response. Regarding the question B, our proposed method is robust to variations in the structure of the debates, since our algorithms are not using explicit information of the debate structure as their input. We just need an argument graph (e.g., the outcome of an argument mining pipeline) representing the debate as an input to run our proposed method.\n\nWe would also like to mention that we will add further references to make more clear the argumentation theory baselines and improve the understandability of our work. Furthermore, regarding the reproducibility score, we will release the code upon acceptance making it easy to reproduce our experiments.\n\nWe hope that with this thorough response we will be able to improve your Excitement about our work and improve your perception about its Soundness. \n\nThank you very much again for your time and dedication,\n\nThe authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting two phase approach to argument quality detection.\n2. Innovative model proposed to handle debate outcome prediction,", "reasons_to_reject": "There are some decisions taken by the author/s that I would like some clarifications on, as to my understanding, they impact the outcome of the paper in a major way.\n1. The removal of arguments that have an attack relation on them prior to them being scored strikes me as being quite extreme. There may be a good argument that has a substandard attack on it. Removal of that argument from the set of admissible arguments is likely to make the model non representative, as human adjudicators may still credit the argument. It may also be the case that an argument is too bad to be attacked. Most debates have a defined time for speakers to rebut their opposition and often the arguments that are focused on are the ones most fundamental to the case that the opposition is running. It makes little sense at that point to spend valuable time on rebutting throwaway statements. In the case of the method proposed by the authors, a good argument that is rebutted badly would be inadmissible and a bad argument that is deemed not worth rebutting would be admissible. This may cause issues in the subsequent steps.\n2. I am unclear on how arguments impact the probability of winning the debate. While section 4.2 is titled argument scoring, I am not clear where that process is actually happening.", "questions_to_authors": "A. How is the removal of arguments having an attack relation on them justified?\nB. Question on extension of work - often debates are not neatly categorized into phases of speech and rebuttal - how does this architecture handle that?", "paper_topic_and_main_contributions": "This paper proposes a new method to evaluate natural language debates based on a hybrid approach of combining argumentation theory and NLP techniques. The approach taken seems to be an interesting one, with the potential of leveraging the improvements made in both fields to perform the challenging task of debate outcome prediction. The model used is, to my knowledge, an original one and comparisons are made to reasonable baselines although some details are missing. The two phase approach taken to argument scoring is another good contribution.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Spelling error on line 654 - miss-classified -> misclassified"}, "extra_text": {"Missing_References": "In section 5.2 the author/s discuss Argument Theory Baselines and subsequently utilize them in table 1. I was not able to find a citation for this and the author/s do not go into details of what these baselines entail. Either a citation or some details on the baselines used would be appreciated here."}}]

Author rebuttal:

Meta-review: This paper proposes a new method for automatically evaluating and predicting the winning stance in professional argumentative debates. The key problem it addresses is performing debate analysis and evaluation on complex natural language arguments, which have been relatively unexplored. 

The reviewers agree that the [hybrid] approach is very interesting and the results are strong. In addition, the paper was well-written and clear, with a generally strong literature review. The authors provide a clear error analysis which helps support the results and conclusions nicely. The hybrid approach is also technically sound and might provide inspiration for additional NLP tasks.
Decision: Accept-Main