Contextual Interaction for Argument Post Quality Assessment
Authors: Yiran Wang
Xuanang Chen
Ben He
Le Sun
Venue: EMNLP 2023 Main (2023)
Keywords: argument
argument quality
contrastive learning
large language models

Abstract: Recently, there has been an increased emphasis on assessing the quality of natural language arguments. Existing approaches primarily focus on evaluating the quality of individual argument posts. However, they often fall short when it comes to effectively distinguishing arguments that possess a narrow quality margin. To address this limitation, this paper delves into two alternative methods for modeling the relative quality of different arguments. These approaches include: 1) Supervised contrastive learning that captures the intricate interactions between arguments. By incorporating this approach, we aim to enhance the assessment of argument quality by effectively distinguishing between arguments with subtle differences in quality. 2) Large language models (LLMs) with in-context examples that harness the power of LLMs and enrich them with in-context examples. Through extensive evaluation and analysis on the publicly available IBM-Rank-30k dataset, we demonstrate the superiority of our contrastive argument quality assessment approach over state-of-the-art baselines. On the other hand, while LLMs with in-context examples showcase a commendable ability to identify high-quality argument posts, they exhibit relatively limited efficacy in discerning between argument posts with a narrow quality gap.

Contextual Interaction for Argument Post Quality Assessment
Yiran Wang Xuanang Chengl, Ben He Le Sun*
UUiieesitty of Chinese Academy of Sciences, Beijing, China 2 Institute of Software, Chinese Academy of Sciences, Beijing, China (wangyiran20, chenxuanang1 (etgg1lmmmmmmsueemmmm uc...ac..ccc.................... cn, benhe@ucas. ac. cn, sunle@iscas.ac. ac.
Abstract
are many debate websites on the Internet, such as Quora, Kialo, and Zhihu. Usually, those websites present multiple arguments in masonry layout. People tend to view those arguments in very short period of time, then give those arguments an overall estimation of their strength based on their impres- sion the relative quality of different arguments. In this regard, on the internet, when confronted with a large volume of text, peoplc's evaluations of the quality of arguments often stem from quick impressions rather than careful consideration after thorough reading and thinking.
Recently, there has been an increased empha- sis assessing the quality of natural language arguments. Existing approaches primarily fo- cus on evaluating
quality of individual ar- gument posts. However, they often fall short when it comes to effectively distinguishing ar- guments that possess narrow quality margin. To address this limitation, this paper delves into two alternative methods for modeling relative quality of different arguments: Su- pervised contrastive learning that captures the intricate interactions between arguments. By incorporating this approach, we aim to enhance the assessment of argument quality by effec- tively distinguishing between arguments with subtle differences. 2) Large language models (LLMs) with in-context examples that harness the power of L.LMs and enrich them with in- context demonstration. Through extensive eval- uation and analysis on the publicly available 1BM-Rank-300 dataset, we demonstrate the periority of our contrastive interaction approach over state-of the-ar art baselines. On the other hand, while I.LMs with in-context examples demonstrate commendable ability to identify high-quality argument posts, they exhibit rela- tively limited effectiveness in quantifying argu- ment quality and distinguishing between argu- ments with narrow quality gap. Code is avail- able at https://gitub..oom/u wwom//ccaYW//CCC.. Y///Coltextual- Interaction-fon for-A-Ao
Argumentation is frequently perceived as a tool for facilitating various forms of reasoning, includ- ing decision-makiee and persuasion. However, these approaches often assume that the individuals involved will exhibit purely rational behavior. In contrast, human behavior is known to blend ratio- nal and emotional elements in guiding their actions. has been suggested that a substantial link ex- ists between the process of argumentation and the emotions experienced by the participants in that process. According to the study of social science (Benlamine et al., 2015; D' E'rrico et al., 2018; 8; Li and Xiao, 2020; Hilton, 2008), the strength of an argument relies much more the argument's 's emo- tional appeal rather than the argument's logical coherence, which means it is hard capture the quality of arguments only with their own context. The first impression made by comparing different arguments plays an important role in assessing the argument's 's quality. However, how to model this comparing procedure remains uncharted. Indeed, most current AQ assessment approaches (Marro et al., 2022: Gurcke et al., 2021a; Wachsmuth et al., 2016; Persing and Ng, 2017) consider an argument's quality based on its own context under the restricted perspective logical, rhetorical, or dialectical. Only a few works tried to solve the
1 Introduction
Given the highly subjective nature of argumenta- tion, arriving at standard answer for a contentious topic is often challenging, as diverse opinions exist. Consequently, assessing the quality of arguments is a complex task, it necessitates assessing not only the relationship between arguments and the topic at hand, but also the quality of the argument itself (Wachsmuth et al., 2017b). Nowadays, there

---

AQ assessment problem through a monolithic view (Fromm et al., 2022). Some tried capture a fea- ture that may affect the quality of an argument, such as argument structure (Li et al., 2020) and discourse structure (Liu et al., 2021). However, revealing those features requires extra annotation costs under the guidance of linguistic specialists which makes those methods hard to be applied to large-scale datasets. Since argument quality is relative concept, it's hard to distinguish the slight difference between them, especially for arguments with subtle differences.
of arguments was discussed through dialectic and rhetoric (Aristotle and Kennedy, 2006). Based on classical theories on arguments (Johnson and Blair, 2006; Hamblin, 1970; Perelman and Olbrechts- Tyteca, 1969; Eemeren and Grootendorst, 2003), Wachsmuth et al. (2017b) identify logical, rhetori- cal, and dialectical as the three aspects of AQ. With recent progress in natural language processing, AQ has been studied and applied in various domains, including student essays (Wachsmuth et al., 2016), news editorials (E) Baff et al., 2020), and social media discussions (Wachsmuth et al., 2017c; Ski- talinskaya et al., 2021).
This study presents a novel investigation into measuring the nuanced differences between argu- ments for quality assessment. Specifically, we amine the contextual interaction of content between various argument posts to enhance the quality as- sessment process. We explore two different ap- proaches to simulate the comparison of different contexts as humans evaluate multiple arguments: 1) Supervised contrastive learning for cross-argune argument interaction that pulls together those arguments with similar quality, while pushing apart other argu- ments whose quality is at both ends. 2) Utilization of Large Language Models (LLMs) with in-context examples for the AQ assessment. Extensive evalua- tion on the standard HAM-RRnk-30K dataset (Gretz et al.. 2020) demonstrates that contrastive learn- ing surpasses state-of-the-art baselines in terms of overall assessment accuracy across arguments of varying quality ranges, well as in distinguishing arguments with similar quality. While LLMs with in-context examples exhibit effectiveness in rec- ognizing arguments at extreme quality ends, they fall short compared to contrastive learning when quantifying argument quality and differentiating between arguments with close quality gaps.
In the current research on AQ assessment, the main focus has been on studying as a sub-task in Argument Mining (AM). However, due to the extreme subjectivity of AQ, there is no clear defini- tion for it. As result, it is believed that there are various factors that can influence AQ. Wachsmuth et al. (2017b) summarized 15 factors that affect argument quality, categorizing them into logical, rhetorical, and dialectical aspects. Gurcke et al. (2021b) assessed argument quality specifically terms of sufficiency with human efforts, hypothe- sizing that the conclusion of a sufficient argument can be derived from its premises. Li et al. (2020) assessed the persuasiveness of arguments by ana- lyzing their argument structure using factor graph model. On the other hand, Singh al. (2021) fo- cused on explicating implicit reasoning (warrants) in arguments with the help of trained experts. Falk and Lapesa (2023) attempted to enhance the AQ as- sessment performance by incorporating knowledge from other dimensions into the prediction process through multi-task learning. These studies consider each perspective of argument quality separately, which limits their holistic view of the concept. Ad- ditionally, some of these approaches require addi- tional annotations (Marro et al., 2022), bringing difficulty in application to large-scale datasets.
Major contributions of this paper are: 1) An in- vestigation of modeling contextual interaction for the AQ assessment. 2) Proposal of a contrastive interaction approach that outperforms state-of-th the- art baselines on the IBM-Ram --n--ann-3000 dataset. 3) An exploration of LLMs with in-context examples for the AQ assessment underscores the limited perfor- mance of LLMs in quantitatively recognizing the subtle difference between arguments.
Cross-document Interaction. The idea of model- ing cross docsm-d interactions has been widely explored in machine learning and learning to rank tasks. Pang et al. (2019) proposed SetRank, which utilizes self-attention mechanism to capture local context information from cross-document interac- tions and learn permutation equivalent representa- tions for the input documents. In related area, van den Oord et al. (2018) introduced unsupervised contrastive learning, a method for extracting useful representations from high- nigh-dimensional data. This
2 Related Work
A Aggument Quality (AQ) Assessment. The prob- lem creating a convincing argument has its ori- gins in ancient Greece, where the persuasiveness

---

technique has been influential in various domains and has shown promising results in learning mean- ingful representations. Building on the concept of contrastive learning, Khosla et al. (2020) extended it to the supervised setting, enabling effective uti- lization of label information.
Contrasiive Iearning
Reps for ranked batch
Divide
M,
Reps for original batch
Rank
Large Language Models with In-context Exam- ples. In-context learning (ICL) (Honovich al., 2022) has achieved tremendous success on large language models (LLMs). The main concept be- hind in-context learning is to leverage analogies for learning. In-context learning uses a small num- ber of examples to create a demonstration con- text, often expressed through natural language tem- plates (Min et al., 2022). Specifically,, query ques- tion and demonstration context concatenated to form a prompt, which is then inputted into LLM for prediction. Besides, to enhance the efficiency and performance of existing LLMs, especially for non-opensource APIs like ChatGPT and GPT-4, OverPrompt (Li et al., 2023) enables parallel pro- cessing of multiple inputs within a single prompt using in-context learning. Liu et al. (2022) aim to explore more effective strategy for carefully selecting in-context examples, which can amplify GPT-3's in-context learning capabilities more ef- ficiently than random sampling. Recently, some studies have employed LLMs for ranking and qual- ity assessment tasks. Sun et al. (2023) examined the use of generative language models like ChatGPT and GPT-4 for relevance ranking in Information Retrieval (IR). Koemi and Federmann (2023) uti- lized the GPT series models assess the quality of machine translation and proposed the GEMBA metric.
Figure The supervised contrastive learning for cross- argument interaction. The representations (Reps) of arguments with median quality are pulled closer (-++), pulled far (-+) from other arguments whose qual- ity are at both ends.
We explore an alternative approach that lever- ages the capabilities of LLMs. Specifically, we utilize LLMs with in-context examples, which be viewed means of incorporating contextual interaction as prompted demonstrations. this up- proach, the models analyze and process text while considering the provided examples within a given prompt. To imbue the LLMs with knowledge of argument quality, we supply the models with ex- ample arguments of varying quality levels in the prompts. We then prompt the model to assess and rate the provided arguments based the standards by those examples.
3.2 Supervised Contrastive Learning for Cross-argument Interaction
The architecture of our proposed supervised con- trastive learning method is shown in Figure 1. Dur- ing the training phase, each original batch B, con- sists of 4n arguments from the training set, shuffled in random order. Each argument is concatenated with its corresponding topic using the delimiter [SEP] tokens, and a [CLS] token is added to the head of the sequence: /CLS) Topic [SEP] Argu- ment [SEP]. The argument representation, which serves as the input to the contrastive objective, re- mains consistent with the backbone architecture, such RoBERTa (Liu et al., 2019). After being en- coded, we extract the last layer's hidden units of the [CLS] token as the representations for arguments in the original batch B,, obtaining a representation list H. = [h1,0- 12,00 ... hl
3 Method
3.1 Overview
To imitate the reaction when humans read different arguments, we explore two alternative approaches for this comparing procedure, especially for the dif- ferentiation between arguments with a close quality gap. The first approach is based on supervised con- trastive Learning (CL) (Khosla et al., 2020), which aims to maximize the similarity between similar pairs of samples, while simultancously minimizing the similarity between dissimilar pairs. Addition- ally, a reasoning module based on graph neural networks introduced to leverage the discourse relation and logical structure arguments for their quality assessments.
To determine the anchor candidate for arguments with median quality within the batch, the original batch B, is ranked based on the descending order of their true labels (human-annotated scores) be an ordered one B,, with the ordered represen- tation list H. = [ h2,s 1: hang], simplified

---

[b,, h2.... han]. Then, divide this ranked batch into four equal-size mini-batches, denoted as BR and He which 8 E [1, 2,3, -I. For exam- ple, the representations in the second mini-batch is HG2- = [h2n h2m 12: /mn]- To increase the distinction between median arguments with two extreme ends, the second and third mini-batches are chosen as the anchor candidate sets, and the other two mini-batches only serve as negative sam- ple sets. That is, we hope that the arguments in the second (or third) mini-batches could be as close as possible, and as far away as possible from the arguments in the remaining three mini-batcher The contrastive loss is defined as follows:
domain of argument quality assessment. Some studies have investigated the impact of discourse relations on the quality of argumentation (Dur- mus et al., 2019; Li et al., 2020). In the field of reading comprehension, Huang et al. (2021) use Discourse- Aware Graph Network (DAGN) to cap- ture advanced discourse features that can effec- tively represent passages to solve logic QA tasks, Herein, we further adapt this method to the do- main of quality evaluation for arguments, leverag- ing graph neural networks to learn the discourse structure contexts.
Specifically, as shown by the example in Fig- ure 2, DAGN treats discourse units as nodes and constructs a graph structure with certain conjunc- tions serving edges to generate the graph rep- resentation. Differently, we supplement common conjunctions (in Appendix A.1) used in arguments as edges when constructing the graph structure. Fi- nally, the original [CLS] token of the argument and the generated graph representation contacted as the representation of the argument.
E hism(
Cig = -log EnncH./HH csim(haks)/F
La n
1
Σ
(2)
wherein s is 2 or 3. sim is the cosine similarity scores, and is the temperature parameter.
Apart from the contrastive alignment, the model is also trained under the ground-truth scores using a mean squared error (MSE) loss:
E5
E2
E4
Enhenced engument
mmse MSE(Upred: Utrue)
(3)
Without sandstory recirement blder
Et
individials is who would onherviise be able
wherein Upred the predictions the model, and Mtrue the ground-truth scores.
thenselves lves without 50b
bLock young people groduntes
The final loss is given by a combination of the contrastive loss and the MSE loss. The contrastive loss is to encourage the model to learn discrimi- native representations for anchor candidates and in-bateh positives, while also promoting separa- tion from the in-batch negatives. The MSE term measures the discrepancy between the model's pre- dictions and the ground truth labels.
Who in desperate need employment
andia path,
The example of the reasoning using discourse Figure 2:
relation within the context argument.
3.4 Large Language Models with In-context Examples
L = B|aCaa + -1-1 Cal + Bm (4)
In this work, employ two large language mod- els, ChatGPT (gpt-3. 5-turbo) and Davinci-003 (text-davineee -003). Both ChatGPT and Davinei- 003 are based on the GPT-3.5 architccture, while Davinci-003 is an earlier version. Both of them offer excellent performance and can generate co- herent and contextually relevant responses in many text generation tasks. Therefore, we aim to explore whether these two models exhibit equally outstand- ing performance in the AQ assessment task.
wherein a and B are the weighting factors. By adjusting the value of 3, we can control the relative importance of the contrastive loss and the MSE loss in the overall training objective.
3.3 Reasoning Using Discourse Relation
Similar previous studies (Toledo et al., 2019; Lauscher et al., 2020; Habernal and Gurevych, 2016; Wachsmuth et al., 2017a; El Baff et al., 2018), we recognize the vital role played by the context and logical structure of an argument in the
Our approach aims to emulate the comparative evaluation carried out by human readers when browsing debate forums or websites. To achieve

---

this, we adopt a method that incorporates knowl- edge of argument quality, designing the prompt with in-context examples. Examples within the prompt can be considered as posts that humans have already seen and their corresponding evalu- ations in their minds when they browse different arguments. Figure 3 shows the detailed prompt for AQ ranking and comparison task. For the prompt, we first provide example arguments along with their corresponding topic and score. Then, we pro- vide the task instructions. Finally, we present the argument(s) to be evaluated or the arguments pair to be distinguished, followed by "Score:" or "Better argument: ", indicating that the model is required to complete the score or judgment.
of 2,000 pairs. These pairs were categorized based on the difference in argument quality scores. The categories include pairs with score difference of less than 0.25, ranging from 0.25 to 0.5, and ex- ceeding 0.5. The objective of this task is to assess how the model performs in terms of differentiat- ing between argument posts in different levels of quality gaps. Accuracy, the percentage of correct predictions made by the model over all argument pairs, is used to evaluate this task.
Implementation details. We use RoBERTa-base (Liu et al., 2019) and BERT-base (Devlin et al., 2019) as our backbone models. The hyper- parameters a and 3 in Eq.4 are set to 0.5 and 0.8. and the temperature in Eq.1 is set as 0.1. We fine- tuned the RoBERTa-base model for 5 epochs with learning rate of 2e-5 and batch size of 32, which align with the settings of the BERT model (Gretz et al., 2020). The vanilla model and models with DAGN are trained by MSE loss, while models with CL are trained based on the loss in Eq.4
4 Experiments
4.1 Experimental Setup
Dataset. IBM-Rankk33k (Gretz et al., 2020) is a large-scale argument dataset containing 30k argu- ments over 71 topics. Each argument in the dataset is annotated with a continuous quality score be- tween 0 and 1, and we use weighted average quality scores ground-truth labels since they negate the influence of non-reliable annotators. The training subset contains 49 topics with 20,974 arguments. The Dev subset contains 7 topics with 3,208 ar- guments and is used for tuning dyper-parameter and determining early stopping. The test subset contains 15 topics with 6,315 arguments.
As for demonstrations to LLMs, we employed two settings for example numbers N. The first ap- proach includes three examples consisting of a high-quality argument (1 point), medium -uallity argument (0.5 points), and a low-quality argument (0.1 points). The second setting involves providing ten examples in the prompt, with scores ranging from 0.1 to 1. The smallest difference in scores between each pair of examples is approximately 0.1 points. We experiment with these two settings for both AQ ranking and comparison tasks.
Task setup. We evaluate the effectiveness of in- corporating contextual interaction for argument quality (AQ) assessment on two tasks, namely, the AQ ranking, and the AQ comparison.
In AQ ranking tasks, we instruct LLMs to give a quality score of the input argument, and we test two settings on candidate arguments numbers S: 1) S = 1, rating individual argument, and 2) S 10, rating groups of ten arguments together, to evaluate the impact of the candidate arguments number AQ ranking performance. In AQ comparison task, to recognize if the LLMs have the ability to distin- guish the quality of arguments, we ask the LLMs to select the argument with better quality between the two options, that is, we instruct them in the prompts to return the number (1 or 2) corresponding to the higher-quality argument according to the specified format. The prompts used can be found in Figure 3. Due the randomness of scoring arguments us- ing LLMs, we conduct five experiments separately and report the average score or majority vote of the five sets of experiments as the final results. the naming of model variants, AE and &S stands
The AQ ranking task follows the official setup of HBM-Rank-30K
Given a list of arguments A = [a1, 02, ..... am] and their corresponding topics T = [ti, t2., .... fn]. the task is to assign ranking
for those arguments based on the descending or- der of their quality score S(a;, fz). The result of the argument quality ranking task evaluated with Pearson correlation (Cohen et al., 2009), Spearman correlation (Wissler, 1905), and NDCG (Normal- ized Discounted Cumulative Gain) (Järvelin and Kekäläänen, 2002) on all test samples.
The AQ comparison task involves predicting the argument of higher quality in a given argument pair, denoted as (A1, A2). To evaluate the performance of models on this task, we conduct experiments using three sets of argument pairs, each consisting

---

Hore is Rt cxomle engareris with thii correspeniigg topic and sore you standard orgment quility
qullity score to show the stancord argarent quility wecloationinn
collaction:
Eaarlee: orgment icontestt tople ccontoats Score rodedt W Foor argaest ceoontaatt topleceranee appiccerantats
Earple
Cecmplet: anga tople
PowrpTel: arganti:conteers
Extolled: ongurenti cooncetts tepls CONTTNTT Scoret srrereeeeeaaa
Dowrplell www.sttt.cccoo...... coontttttooottt tople: crontests soarecdect sa Bace the folloutin iegurenes with comvesponding tapte
Sxmplea: angarert:
Intruction
the aracment.o In
Give
Scorw:
Inttumion
onganent accurding atardard by
wursies. tetum the namber of better one, in
Angueeti: cconseets Topis:<tspic>
Candissta
Anganent:: cconbckt)
Tosicistopier VI
Arganntt:: EcontoNcy topác: CCOpLES An
Arganent::
Toplii
Scoreity Snoredly
(a) Prompt for AQ assessment
(b) Prompt for AQ comparison
Figure 3: The prompt for AQ assessment and comparison, where N represents the number of examples, and S
represents the number of arguments to be evaluated.
for the example arguments presented to the LLM, and the arguments that the LLM is asked to score for once. For instance, ChatGPT-3E- 1OS stands for ChatGPT which is prompted score a batch of 10 arguments while being shown with 3 examples.
gument, or presenting a counterargument. Dual
BERT was used as the backbone model.
The outcomes obtained from SVM BOW and Arg Length are referenced from (Gretz al., 2020). The results of BERT and TFR-BERT are directly cited from (Favreau et al., 2022). Additionally, RoBERTa w/ own adaptar and Dual BERT with Spark (ZS) are cited from (Falk and Lapesa, 2023; Deshpande et al., 2023), respectively.
Comparison models. We employ the following baselines our evaluation:
SVM BOW is a support vector regression ranker (Gretz et al., 2020). The training set is composed of the most frequent 1000 tokens and utilizes an RBF kernel and bag-of-words features.
4.2 Experimental Results
Tables 1 & 2 present the results obtained for the AQ ranking task and the AQ comparison task, spectively.
Arg Length (Gretz et al., 2020) evaluates argument quality based on its length in characters, following the intuition that longer texts may provide more detailed explanations.
Our proposed contrastive argument interaction outperforms state-of-the-ard baselines. The re- sults Table show that RoBERTa's s performance is superior to that of BERT. RoBERTa achieves a Pearson correlation of 0.5283 and a Spearman correlation of 0.4858. It also can be seen that our proposed contrastive learning (CL) for the argu- ment interaction outperforms the baselines with a Pearson correlation of 0.5580 and Spearman cor- relation of 0.5186. According to NDCG@K K, our methods outperform baselines by significant mar- gin. Moreover, RoBERTa + DAGN and RoBERTa + CL achieve a prominent performance with a Pear- son correlation of 0.55 and Spearman correlation of 0.51. The prominent performances of these 2 mod- els reveal that reasoning and comparing both have large impact argument quality. The experimen- tal results indicate that CL can consistently improve the AQ ranking performance over vanilla RoBERTa and BERT models. For the AQ comparison task, we exclusively incorporate RoBERTa- based mod- els, excluding LLMs, as they have demonstrated su-
RoBERTa and BERT (Favreau et al., 2022) con- catenate the argument with its corresponding topic. and generate quality score.
TFR-BERT with ensemble losses achieves state- of-the-art effectiveness on IBM-Rank-30k as shown in (Favreau et al., 2022). This approach is built upon the work by (Han et al., 2020), which incorporates several ranking losses in TFR-BERT. Favreau et al. (2022) applies a similar technique to evaluate argument quality by combining the output of multiple TFR-BERT models, each trained with a distinct ranking loss.
RoBERTa w/ own adpt (Falk and Lapesa, 2023) is a recent approach that integrates knowledge from different dimensions into the prediction process using multi-task learning. RoBERTa-base is used the backbone model for all the dimensions. Dual BERT w/ spark (ZS) (Deshpande et al..
2023) involves four types of augmentations for the AQ prediction by offering feedback, deducing hidden assumptions, providing a similar-quality ar-

---

Model
Pearson
Spearman
NDCG@5
NDCG@10
NDCG@1
RoBERTa
0.5283
0.4858
0.9304
0,9304
0.9427
BERT
0,5200
0,4800
0,8500
0,8700
0.8600
TFR- BERRT Ensemble Losses
0.5200
0.4700
0.8900
0.8900
0.8800
Basclines
RoBERTa w/ own adpt
0.4850
Dual BERT w/ spark (ZS))
0.,742
0.2884
SVM BOWT
0.3200
0.3100
Arg-length?
0,2100
0.2200
BERT
0.5278
0.4879
0.9191
0.9163
0.8872
BBRT-bascd
BERT w/ DAGN
0,5292
0.4943
0,9222
0,9258
0.9264
BERT w/ DAGN and CL.
0.5375
0.4949
0,9330
0,9372
0,9388
RoBERTa w/
0.5580
0.5186
0.9740
0,9614
0.9634
RoBERTa-bused
RoBERTa w/ DAGN
0,5511
0.5112
0,9654
0,9655
0.9645
RoBERTa w/ DAGN CL
0.5604
0.5174
0.9799
0.9769
0.9648
Davinci-003-3E-IS
0.1013
0.1092
0.7758
0.7758
0.7758
ChatGPT-3E-1S
0.1277
0.1382
0.7672
0.7672
0.7671
Davinci--M33100E-11
0.0411
0.0401
0.5901
0,5899
0.5899
ChatGPT-10FS
0.1221
0.15509
0.6270
0.6270
0.6271
L.Mss
ChatGPT-200--1
0.1130
0.1153
0.6657
0.6657
0.6658
ChaGGPT-30F--S
0.0863
0.0713
0.6301
0.6303
0.6303
Duvinci-M03-33-110
0.1680
0.1633
0.7781
0.7781
0.7780
ChaGGPT-3E-10S
0.1896
0.1903
0.8044
0.8044
0.8044
Table 1: Results of AQ ranking task. Our proposed models are based on contrastive learning (CL), discourse-awarc graph network (DAGN) and large language models (LLMs). The best results are marked in bold, and the results of best baseline are underlined. f denotes that the results are directly cited from corresponding papers.
perior performance compared to other models. The result in Table revealed that CL method achieves the highest accuracy in discerning argument pairs with difference in scores within 0.25 and above 0.5, reaching 65.45% and 89.6%, respectively. This confirms that our proposed approach effectively enhances the model's 's ability to distinguish subtle differences and creates a clear distinction between arguments with intermediate scores and those at the extreme ends.
demonstrated strong ability in text generation tasks (Bang et al., 2023; Shen et al., 2023; Laskar et al., 2023), they still have significant limitations com- pared our models when it comes quantify- ing text quality. Indeed, both Davinei-003 and ChatGPT underperform in the task of scoring text, whilst ChatGPT outperforms Davinci-003 as ex- pected. After closer look at the results, when there are 3 examples in prompt, we find that Davinci-003 exhibits stronger randomness in pre- dicting scores for low medium scores and tends to give scores similar those in the examples through observation of the predicted score distribu- tions in the three experiments. On the other hand, ChatGPT provides more diverse predicted scores, indicating its stronger comprehension ability com- pared to Davinci-003. When we attempted to pre- diet their quality by grouping candidate arguments into sets of 10, there
Accuracy 0.25 <D< 0.5
Model
0.25
D>
CharGPT-3E
53.20%
65.95%
76.25%
ChaiGPT-10E
48.80%
59.00%
78.30%
ChatGPT-20F
49,35%
52.40%
68.20%
Cha:GPT-30E
48.40%
52.65%
69.90%
Davinci-003-3E
54.20%
66.85%
73.80%
RoBERTa
63.15%
75.85%
88.35%
w/ CL
65.45%
77.60%
89.60%
slight improvement in the results compared before. This indicates that the interaction between arguments is also ef- fective in LLMs. When the number of examples in the prompt is increased to 10, the performance decreases compared to when there are only 3 exam- ples, particularly evident in the NDCG metric. The performance deterioration becomes even more pro-
w/ DAGN
64.45%
78.35%
89.00%
w/ DAGN and CL
64.75%
78.25%
89.45%
Table 2: AQ comparison results. denotes differ-
ence value of quality score between argument pairs.
Performance of LLMs. As shown by the results in Tables and 2, despite the fact that LLMs have

---

Topic
Argument
Grund-Truth
RoBERTa
RoBERTa w/ CL.
factory farming allows for greater food production feed growing
0.95
0.97
0.93
Factory farming
population.
Factory farming is a great way to get more food for everyone
0.84
0.95
0.87
social media allows access to social service resource.. food pantrics. various support groups, bousing and many other viable resources. il helps maintain contact with family and friends.
0.95
0.95
Social media
social media great tool make business personal connections
0.86
0,94
0.88
Table 3: Examples arguments with small difference, on which RoBERTa w/ CL can give more discriminative
scores than vanilla RoBERTa.
nounced as the number of examples increases to 20 and 30. Upon observing the experimental results, one possible explanation is that due to the strong memorization capabilities of the GPT-3.5 model, tends to heavily rely on the scores provided in the examples (details are shown in Appendix A.2). As the number of examples in the prompt increases, large language models tend to be more inclined to repeat the content of the examples from the prompt in their output, including the scores and text of the examples. This results in failure of the final evaluation outcome.
arguments in Table express their opinions support- ing factory farming. While both arguments discuss the advantages of factory farming in increasing agricultural output, slight differences in expression lead to variations in their quality scores, One argu- ment receives a high-quality score of 0.95, while the other argument scores around 0.8, indicating slightly lower quality. When using RoBERTa-base without argument interaction, the model assigns score of 0.97 to the argument with score of 0.95. However, for the other argument that has similar content but slight flaws in expression, RoBERTa- base still assigns high score of around 0.95. Mean- while, after introducing contrastive lcarning as a means of argument interaction, the model main- tains a score of 0.95 for the argument with a score of 0.95. Additionally, it can provide a score of 0.87 which is closer to the true score for the other argument. Similar results observed for the topic of Social media. These two examples demon- strate how the argument interaction can enhance the model's ability to discern subtle differences expressions.
Although the results Table indicate that the LLMs can distinguish between good and poor argu- ment quality when there is a significant quality gap (with difference in quality scores above 0.5), the accuracy of the LLMs drops to around 50% when the difference in argument quality is small (with difference in quality scores below 0.25). This ac- curacy is close to that of random selection, which suggests that LLMs are more inclined to randomly select rather than make judgments based on under- standing when the difference in argument quality is small. When the difference in argument quality is between 0.25 and 0.5, the comparison capabil- ity of the LLMs shows a significant improvement, reaching around 67%. However, there is still 10% gap compared the performance of the fine-tuned models. This phenomenon is also reflected in the performance the LLMs on the correlation met- rics for the quality ranking task, where it shows relatively poor results. However, it achieves a de- cent performance in terms of the NDCG metrics, which indicates LLMs are able to distinguish be- tween good and poor arguments but struggle to differentiate small differences in quality.
Distribution of predicted scores with argument interaction is closer to the real distribution. The histogram presented in Figure 4 illustrates that the fine-tuned RoBERTa-bass generates qual- ity logits that are more polarized compared the actual data distribution. In the ground truth, top arguments (score 2 0.9) make up about 34% of the distribution, while in the prediction without ar- gument interaction, they account for nearly 50%.
This suggests that the model fails to distinguish between decent and good arguments due to the difference in expression skills and the resulting emotional responses from readers. The histogram of the model with contrastive learning exhibits a distribution closer to the real data. It identifies top arguments at approximately 36%, indicating better differentiation between decent and good ar-
5 Discussion
Argument interaction helps distinguish the slight difference. As case study, two example

---

RuBERTa Ground
RoBERTa w/ Ground
01
Poreintage 1N
Peroenties
30-
20-
10-
19
02
0.1
0.
10
0.
02
06
10
Predictioon Score
Predi ctiion Score
(a) The prediction results of RoBERTa
(b) The prediction results of RoBERTa w/ CL
Figure Histogram of the prediction from RoBERTa and RoBERTa w/ CL compared with ground truth.
guments compared to the RoBERTa-bass model. These findings demonstrate that incorporating con- trastive learning enables the model to accurately capture subtle differences in quality and provide more precise assessments.
gument pairs for each of the three score differences under 3 examples. Despite the relatively limited experimental sample size, which led to some fluctu- ations in the results, the experimental findings largely consistent with the conclusions obtained in Table 2.
The potential of LLMss in AQ assessment tasks. LLMs have shown immense potential in argument quality (AQ) assessment tasks, particularly in ar- gument comparison. As evident from the results in Table 2, when there is a significant difference (above 0.5) in the quality of arguments, large lan- guage models can achieve comparable resolution capability with only a few examples. However, de- spite this, there is significant gap between large language models and fine-tuned models when the difference in quality between arguments is reduced. When the quality gap is small (below 0.25), the evaluation of quality by LLMs becomes closer to random guessing, indicating their limited ability to capture nuanced contexts by the LLMs involved in this study. Meanwhile, we have also observed that when it reaches certain number of examples in the prompts (e.g., 20 examples), due to the excellent memorization ability of the large model, it tends to replicate text and scores that appear in the prompts in the answers. How make the LLMs overcome the hallucination during the evaluation process, and return appropriate content, which can also help im- prove the performance of the large model in AQ tasks.
This observation suggests that LLMs, despite their proficiency in open-ended text generation (Arora et al., 2022), may lack precision in assign- ing precise AQ scores, leading to relatively lower performance in AQ ranking tasks.
6 Conclusion
In this study, we explored two altermative methods for assessing the quality of natural language argu- ments: supervised contrastive learning and large language models (LLMs) with in-context exam- ples. Comprehensive evaluation highlights the im- portance of considering contextual interactions for evaluating argument quality. We found that pervised contrastive learning, which captures intri- cate interactions between arguments, outperformed state-of-the-at baselines evaluating argument quality. However, LLMs with in-context examples showed limitations in quantifying argument qual- ity and distinguishing between arguments with narrow quality gap.
Acknowledgments
This work is supported by the National Natu- ral Science Foundation of China under grant No. U1936207 & 62272439, and the Fundamental Re- search Funds for the Central Universities.
To evaluate the ability of LLMs to assess the quality of arguments under the same topic, con- ducted experiments on the topic "We should adopt libertarianism". Due the limited number of argu- ments available for single topic (approximately 300 to 400), we conducted experiments on the argu- ment quality comparison task by selecting 100 ar-

---

Limitations
quality assessment with relevant knowledge. ArXiv, abs/2305.12280.
While the proposed contrastive learning approach enhances the effectiveness of AQ predictions, it is important to acknowledge that it may introduce additional computational costs. The inference cost increased we introduced DAGN and CL. The phenomenon that L.LMs failed in AQ ranking task can be attributed several potential factors. Bias in training data, model interpretabilityy and the need for Noomain-specifie fine-tuning are among the fac- tors that require careful consideration and mitiga- tion. We only conducted experiments on ChatGPT and Davinci-003, without testing GPT-4. From our experiments, it can be observed that LLMs hold great promise in AQ assessment tasks. While sub- stantial prospects exist for LLMs in the domain of quality evaluation tasks, overcoming the aforemen- tioned challenges remains an imperative subject for ongoing research.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, 0bs//810004805.
Esin Durmus, Faisal Ladhak, and Claire Cardie. 2019.
The role of pragmatic and discourse context in de- termining argument impact. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP", pages 3668-5678, Hong Kong, China. Association for Computational Linguistics.
Frans H. Van Eemeren and Rob Grootendorst. 2003, A Systematic Theory of Argumentation: The Pragma- Dialectical Approach. Cambridge University Press.
Roxanne EI Baff, Henning Wachsmuth, Khalid Al- Khatib, and Benno Stein. 2018. Challenge em- power: Revisiting argumentation quality a news editorial corpus. In Proceedings of the 22nd Confer- ence Computational Natural Language Learning, pages 454 464, Brussels, Belgium. Association for Computational Linguistics.
References
Aristotle and George A. Kennedy. 2006. On Rhetoric: A Theory of Civic Discourse, Oup Usa.
Roxanne El Baff, Henning Wachsmuth. Khalid Al Khatib, and Benno Stein. 2020. Analyzing the Per- suasive Effect of Style in News Eddttoiil Argumenta- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3154-3160, Online. Association for Computational Linguisties.
Simran Arora, Avanika Narayan, Mayee Chen, Lau- rel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Fred- eric Sala, and Christopher Ré. 2022. Ask any- thing: A simple strategy for prompting language mod- els. arXiv preprint arXiv:2210.02441.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hal- lucination, and interactivity. ArXiv, abs/2302.04022
Necle Falk and Gabriella Lapesa. 2023. Bridging argu- ment quality and deliberative quality annotations with adapters. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 2469-2488, Dubrovnik, Croatia. Association for Computational Linguistics.
Sahbi Benlamine, Maher Chaouachi, Serena Villata, Elena Cabrio, Claude Frasson, and Fabien Gandon. 2015. Emotions argumentation: an empirical eval- uation. In Twenty-Foutd Fourth International Joint Confer- ence on Artificial Intelligence.
Charles-Olivier Favreau, Amal Zouaq. and Sameer Bhat- nagar. 2022. Learning to rank with bert for argument quality evaluation. The International FLAIRS Con- ference Proceedings.
Michael Fromm, Max Berrendorf, Johanna Reiml, Is- abelle Mayerhofer, Siddhartha Bhargava, Evgeniy Faerman, and Thomas Seidl. 2022. Towards a holis- tic view on argument quality prediction. ArXiv, abs/2205.09800
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Ben- esty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coeffi- cient. Noise reduction in speech processing, pages 1-4.
Shai Gretz, Roni Friedman, Edo Coren-Karlik, As- saf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. In AAAI Conference on Artificial Intelligence.
Francesca D' Errico, Marinella Paciello, and Matteo Amadei. 2018. Behind our words: Psychological paths underlying the un/supportive stance toward im- migrants in social media. In 2018 IEEE 5th Inter- national Conference on Data Science and Advanced Analytics (DSAA), pages 649-656. IEEE
Timon Gurcke, Milad Alshomary, and Henning Wachsmuth. 2021a. Assessing the sufficiency of arguments through conclusion generation. arXiv preprint
Darshan Deshpande, Zhivar Sourati, Filip Ilievski, and Fred Morstatter. 2023. Contextualizing argument

---

Timon Gurcke, Milad Alshomary, and Henning Wachsmuth, 2021b. Assessing the sufficiency of arguments through conclusion generation. In Pro- ceedings of the 8th Workshop on Argument Mining, pages 67-77, Punta Cana, Dominican Republic. As- sociation for Computational Linguistics.
Jialu Li, Esin Durmus, and Claire Cardie. 2020. Explor- ing the role of argument structure in online debate persuasion. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pages 8905-8911 Online. Association for Computational Linguistics.
Ivan Habernal and Iryna Gurevych. 2016. What makes convincing argument? empirical analysis and de- tecting attributes convincingness in web argumen- tation. In Proceedings of the 2016 conference on empirical methods natural language processing, pages 121 1223.
Jiazheng Li, Runcong Zhao, Yulan He, and Lin Gui. 2023. Overprompt: Enhancing chatgpt capabilities through an efficient in-context learning approach.
Jinfen Li and Lu Xiao. 2020. Emotions in online de- bates: Tales from 4forums and convinceme. Proceed- ings of the Association for Information Science and Technology, 57(1)=2555.
Charles Leonard Hamblin. 1970. Fallacies. London, England: Vale Press.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Shuguang Han, Xuanhui Wang. Mike Bendersky, and Mare Najork. 2020. Learning-to-rank with bert in tf-ranking. arviv preprint arXiv:2004.08476.
Denis Hilton. 2008. Emotional tone and argumenta- tion in risk communication. Judgment and Decision making, 301):100.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. ArXiv, abs/2205.
Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2021. Exploring discourse structures for argument impact classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 3958-3388 1958-9969, Online. Association for Computational Linguistics.
Yinya Huang, Meng Fang, Yu Cao, Liwei Wang, and Xiaodan Liang. 2021. Dagn: Discourse- aware graph network for logical reasoning. ArXiv, abs/2103. 14349.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, 0tbs//997.11699
Kalervo Järvelin and Jaana Kekäläänen. 2002. Cu- mulated based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 4024):422-446.
Ralph Henry Johnson and Anthony Blair. 2006. Logi- cal self-defense. Idea.
Santiago Marro, Elena Cabrio, and Serena Villata. 2022.
Graph embeddings for argumentation quality assess- ment. In EMNLP 2022- O02-Coofereecce on Empirical Methods in Natural Language Processing.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020, Super- vised contrastive learning. ArXiv, abs/2004.1 1362.
Sewon Min, Mike Lewis, Luke Zeutlemoyer, and Han- naneh Hajishirzi. 2022. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791-2809, Seattle, United States. Association for Computational Linguisties.
Tom Koemi and Christian Federmann. 2023. Large language models are state-of-the-ar evaluators of translation quality. ArXiv, abs/2302. 14520.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang, 2023. A systematic study and comprehensive evaluation of chatgpt on bench- mark datasets. arXiv preprint arXiv:2305. 18486.
Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. 2019. Setrank: Learning a permutation innvataaiooniivaiiantt ranking model for information retrieval. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Anne Lauscher, Lily Ng. Courtney Napoles, and Joel Tetreault. 2020. Rhetoric, logic, and dialectic: Ad- vancing theory-basee argument quality assessment natural language processing. International Con- ference on Computational Linguistics.
Chaīm Perelman and Lucie Olbrechts-Tyteca 1969. The New Rhetorie: Treatise on Argumentation. Notre Dame, IN, USA: Notre Dame University Press.

---

Isaac Persing and Vincent Ng. 2017. Lighty--superised modeling of argument persuasiveness. In Proceed- ings of the Fighth International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 594-604.
for Computational Linguisties: Volume 1, Long Pa- pers, pages 176-187, Valencia, Spain. Association for Computational Linguistics.
Henning Wachsmuth, Martin Potthast, Khalid Al- Khatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017c. Building argument search en- gine for the web. In Proceedings of the 4th Workshop on Argument Mining, pages 49-59, Copenhagen. Denmark. Association for Computational Linguis- tics.
Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li, Weiming Lu, and Yue Ting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. ArXiv, abs/2303.
Keshav Singh, Paul Reisert, Naoya Inoue, and Kentaro Inui. 2021. A comparative study collecting high- quality implicit reasonings at large-scale. ArXiv, abs/2104.07922
Clark Wissler. 1905, The spearman correlation formula. Science, 22(558):309-311.
Gabriella Skitalinskaya, Jonas Klaff, and Henning Wachsmuth. 2021. Learning from revisions: Quality assessment of claims in argumentation scale. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1718-1722 Online. Association for Computational Linguisties.
Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjic Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large lan- guage models as re-ranking agent. arXiv preprint arXiv:2304.09542.
Assaf Toledo, Shai Gretz, Edo Cohen-Karlie Roni Friedman. Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim, 2019. Automatic argument quality assessment new datasets and meth- ods. Conference on Empirical Methods Natural Language Processing.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. ArXiv, abs/1807.0333 807.03748.
Henning Wachsmuth, Khalid AI-Khatib, and Benno Stein. 2016. Using argument mining to assess the argumentation quality of essays. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1680-1691, Osaka, Japan. The COLING 2016 Orga- nizing Committee.
Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yu- fang Hou, Graeme Hirst, Tryna Gurevych, and Benno Stein. 20 Argumentation quality assessment: Theory practice. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics /Volume 2: Short Papers), pages 250-255, Vancouver, Canada. Association for Computational Linguistics.
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu. Vinodkumar Prabhakaran, Tim Alberd- ingk Thijm, Graeme Hirst, and Benno Stein. 201 7b. Computational argumentation quality assessment in natural language. In Proceedings of 15th Con- ference of the European Chapter of the Association

---

A Appendix
A.1 Supplementary edges for graph
construction
We have added five prepositions and conjunctions with logical meanings that occur in the corpus. They are "in case", "contrary to", "against", "so to", and "in order to".
A.2 Supplementary experiments conducted under different example scores
To demonstrate the susceptibility of large models to the influence of example scores and their tendency to concentrate predicted scores around the example scores in the context of argument text assessment, we conducted experiments on 1000 candidate argu- ments on ChatGPT and Davinci-003 with prompt of three examples and one argument be evalu- ated at each time. As shown the Figure 5, the davinci-003 model was significantly affected by the example scores. As expected, chatGPT outper- formed the davinci-003 model.
Perocitace
0/:
10
Frediction Soore
(a) Prediction distribution of Davinci-003
0.
10
10
Prediotion
Soore
(b) Prediction distribution of ChatGPT
Figure Prediction distribution of LLMs under two different example scores, namely, 0.8, 0.4, 0.2 and 1, 0.6, O.I.


Reviews:
[{"review_id": "4eZ6kj9kHw", "reviewer": "Reviewer_gESR", "summary": "", "questions": "A. Would be interested in a detailed description of the context provided to the LLMs to score arguments.\nB. Do the topics that the arguments relate to have an impact in terms of scoring and if so, how is that captured in the model proposed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank you for your careful consideration of our paper and for providing valuable feedback. Please see below for our responses.\n\nR1. Number of runs and variance of LLMs: We chose to do 3 runs to balance between efficiency and effectivness. We acknowledge that due to the inherent variance in LLMs, a larger number of runs might provide a more comprehensive understanding of the model's behavior. We will conduct additional experiments with a greater number of runs in a revision.\n\nR2. Context details and data sources: We apologize for any ambiguity in our presentation. The arguments used as <context> in our prompt are from training set of IBM30k. Since AQ is a universal concept, the AQA effectiveness is evaluated on a mixture of arguments from different topics in prior art [1,2]. Following this paradigm, when selecting arguments to serve as examples, we chose arguments from different topics and did not impose specific requirements on stance of the arguments. The chosen examples are of diverse scores to enable them to represent arguments of varying quality. We hope that by providing LLMs with examples of arguments from different topics and stances, we can enhance its robustness in scoring arguments. We attempted to use arguments from the same topic as examples in the prompts. As prelimenary evaluation showed that this had little impact on the results, we did not investigate further with this setting. \n\nR3. Number of arguments as examples: We appreciate this concern and understand the potential impact of example size on the model's performance. To address this, we will experiment with varying context sizes to determine whether there is an optimal threshold beyond which further improvement in scoring becomes marginal.\n\nQA: See reply to R2.\n\nQB: A good argument should stay on topic during the process of reasoning and maintain strong relevance to the subject. Therefore, in our input, we have concatenated the topic with the argument. In this way, the argument representation has taken the related topic into consideration, which is further updated through interaction with other arguments.\n\n[1] Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, Noam Slonim: A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis. AAAI 2020: 7805-7813\n\n[2] Favreau C O, Zouaq A, Bhatnagar S. Learning to Rank with BERT for Argument Quality Evaluation[C]//The International FLAIRS Conference Proceedings. 2022, 35.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The ideas presented in this paper are on a whole, innovative and logical.\n1. Analyzing and understanding the structural similarity between arguments is quite an interesting concept, and the results shown in table 2 do suggest that the interactions that the author/s are trying to capture impact the scoring of said arguments.\n2. While there are some issues with the approach taken by the author/s to utilize LLMs for argument quality detection (detailed below), I believe the approach as a whole is an interesting one.\n3. The conclusion that LLMs can identify high quality arguments but cannot distinguish between similarly scored arguments well aligns with the capabilities of LLMs as of now.\n4. Reasonable baselines are used and the SCL method shows improvement on SOTA.", "reasons_to_reject": "There are some specific issues with the task of argument quality detection using LLMs -\n1. The author/s take the average of three runs for this task. Considering the variance inherent in LLMs and the relatively small size of the context, it may be possible that three runs are not enough. It would be good to see the impact of more runs and whether that causes a change in the results reported.\n2. More details need to be provided with regards to the context provided for the LLMs to be able to assign a quality score to an argument - are they arguments from IBM30k? If so, are they from the same topic or from different ones and does that impact the score assigned?\n3. The author/s provide a maximum of 10 arguments in the context for LLMs to score arguments. This seems a bit random, especially considering that IBM-30K has nearly 30,000 scored arguments. It may be interesting to see if the scoring is improved on provision of more context and whether there is a drop off in terms of improvement after a certain threshold.", "questions_to_authors": "A. Would be interested in a detailed description of the context provided to the LLMs to score arguments.\nB. Do the topics that the arguments relate to have an impact in terms of scoring and if so, how is that captured in the model proposed?", "paper_topic_and_main_contributions": "The main contributions of this paper that stood out to me are as follows:\n1. The idea that interactions between arguments can be used to gauge quality is a new one to me. The similarity between argument composition leading to an agreement of scores is an interesting direction to explore.\n2. The approach of combining Supervised Contrastive Learning with DAGN is an innovative one.\n3. The application of LLMs to the task of scoring arguments having been given examples of argument-score pairings is interesting.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2uehh1MVo5", "reviewer": "Reviewer_eHo2", "summary": "", "questions": "1. Please refer to point 2 above, would love to hear thoughts on it! \n2. In line 419, why is average a good metric?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank you for your careful consideration of our paper and for providing valuable feedback. Please see below for our responses.\n\nR1. Exploring LLMs further: We acknowledge this concern and realize that a deeper exploration of LLMs and their tendencies is important. While the reason why LLMs producing random results when the argument score difference is small is not elaborated upon in the paper, our guess is that the LLMs used were trained on highly diverse corpus, and were perhaps not specialized for the AQA task, such that they lack the ability to capture subtle differences between arguments. Additionally, as observed in Appendix 3, the LLMs appear to have the tendency to memorize scores from examples rather than evaluating quality based on comprehension. This is also a contributing factor to the limitations of the large model's performance on AQ tasks. However, it can be possible to improve the LLMs' AQA ability by supervised fine-tuning on sufficient high quality training data. \n\nR2&Q1. Context details and data sources: We apologize for any ambiguity in our paper. The <context> arguments used as examples in our prompt are sourced from the IBM30k training set. As we described in section 4, Implementation details, we choose examples with diverse scores to enable them to represent arguments of varying quality.  As per reply to R2 of Reviewer gESR, we present arguments from different topics following prior works, as presenting arguments from the same topic does not appear to significantly change the results.\n\nQ2: We sincerely apologize for the confusion. To clarify, for the ranking task using LLMs, we utilize an approach where the average of three runs is used. As for pairwise quality comparison task, we employ a majority vote method. Due to the inherent randomness of large models, we employ a methodology of conducting 3 experimental runs and averaging the results. This improves the stability and robustness of the experimental outcomes.\n\nTypos & Presentation: Thanks very much for the suggestions. We will update our paper based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of comparing LLMs to CL in this specific domain is very innovative. \n2. Very thorough testing and results that prove differences between the two methods- conclusions like LLMs guessing randomly when argument nuance is limited is derived through careful testing, and is an important conclusion that highlights the limitations of LLM, especially now, when LLMs and ChatGPT are in the spotlight. \n3. Table 1 summarizes everything in a nice manner. \n4. Table 3 and its finding is extremely interesting.", "reasons_to_reject": "1. I think SCLs are explained fairly well, and achieve SOTA results. However, LLMs are not explored futher. Why LLMs produce random results when argument score difference is small is not explored (just an observation, I wouldn't reject it based on this).\n2. Some details about LLMs can be useful- what examples are provided? How do you ensure they are truly representative? What topics are covered in one set? If not, did you try providing it different arguments for the same topic and then seeing if its still random?", "questions_to_authors": "1. Please refer to point 2 above, would love to hear thoughts on it! \n2. In line 419, why is average a good metric?", "paper_topic_and_main_contributions": "The authors propose two methods for modelling argument quality: SCL and LLMs. Results provided show SOTA improvement by using CL for IBM 30K dataset, while LLMs do well in certain instances (when two arguments are different enough). Sufficient baselines are used for comparison. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Abstract and Intro, when read together are a bit repetitive. \n2. Appendix is filled with important info. The prompt for ChatGPT is an important piece of information that needs to be included in the main paper in my opinion. "}}, {"review_id": "3333rcq83Z", "reviewer": "Reviewer_MHeS", "summary": "", "questions": "- A: CL and DAGN improve performance by a few percentage points. Therefore, the authors could discuss the limitations of incorporating contextual interactions on performance. What were the expectations in this regard? Was a larger increase expected based on the social\n- B: Adding DAGN into the comparison in Section 5 could reveal further interesting insights.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank you for your careful consideration of our paper and for providing valuable feedback. Please see below for our responses.\n\nR1. Strengthening the Introduction: We appreciate the reviewer's feedback regarding the introduction of our paper. We acknowledge the importance of emphasizing the integration of contextualization into social science research. In the revised version, we will highlight the significance of contextual interaction in computational argumentation and its relevance to social science research. For instance, how emotional expression in an argument affects people's first impression and subsequently influences their evaluation of the argument's quality. We appreciate any specific suggestions or insights to further enhance our work.\n\nR2. Justification of LLMs Choice: Considering the remarkable performance of LLMs in text summarization, which stems from their ability to understand textual content, it is reasonable to assume that this ability could also contribute to their effectiveness in accomplishing the AQA task. Based on this assumption, we posit that LLMs would exhibit favorable performance in AQA tasks.\n\nR3. Discussion of Performance Improvement Limitations: Our aim is to enable text interaction by introducing representations for structural analysis of text, emulating the comparative process humans undertake after reading and contemplating. Although this approach yielded the best performance, the improvements achieved were somewhat limited. We attribute this limitation to the concatenation of the [CLS] embedding with the DAGN embedding used to obtain representations of structure augmented argument. While this concatenation preserves semantic information and enhances structural details, it can introduce interference during contrastive learning, hindering the expected outcomes. From a sociological perspective, this limitation might be attributed to the greater significance placed on first impressions. Just as in our everyday lives, where the initial impression holds considerable weight, the subsequent impact of our thoughts may not be as influential as the role of that initial impression.\n\nR4. Comparison with DAGN: In our previous experiments, we attempted to introduce more words, including certain prepositions, as edges. The results indicated that finer decomposition of the structure does not necessarily lead to better performance. In fact, an excessive number of edges can result in a decline in performance. \n\nTypos & Presentation: Thanks very much for your thorough review. We will make the revisions to the paper based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Evaluating argument quality, especially as a ranking task, and in cases where the quality between arguments does not differ much, is a hard task of computational argumentation. This work contributes an approach that outperforms previous SOTA results by incorporating contextual interaction.\n- The paper is written in a very clear and well structured way.\n- The approaches are well motivated and the experiments are comprehensible. The results are very clearly summarized and interpreted during the evaluation.\n- The authors provide a very interesting analysis of the ability of LLMs (ChatGPT and Davinci) to distinguish arguments in terms of quality. In doing so, they reveal weaknesses of these highly praised approaches. Especially that the performance in distinguishing similar arguments is equivalent to random guessing is an exciting finding.\n- The effect of contextual interaction is nicely illustrated in the paper using a case study and a comparison of the distribution of ground truth and predictions. This analysis supports the hypothesis of this paper.", "reasons_to_reject": "- The paper's introduction appears to be its least strong aspect. In this section, the idea of contextualization could be emphasized more, along with better integration into social science research.\n- The use of ChatGPT and Davinci is motivated by their good performance on text summarization. Here, it should be explained in more detail why the authors use this as an indicator for likewise good performance on AQ, since the relation to the task at hand is not straightforward.", "questions_to_authors": "- A: CL and DAGN improve performance by a few percentage points. Therefore, the authors could discuss the limitations of incorporating contextual interactions on performance. What were the expectations in this regard? Was a larger increase expected based on the social\n- B: Adding DAGN into the comparison in Section 5 could reveal further interesting insights.", "paper_topic_and_main_contributions": "In this paper, the authors investigate the influence of contextual interaction on human assessment of argument quality. To this end, they propose to combine language models with supervised contrastive learning, as well as with a discourse-sensitive embedding of the argument. In addition, they investigate the potential of prompt-based approaches, where the context is part of the prompt. On the IBM-Rank-30k dataset, they evaluate the models’ ability for (1) argument quality ranking and (2) argument quality comparison. They show that the use of contrastive learning as well as discourse-sensitive argument embeddings with RoBERTa significantly outperforms previous approaches for (1), as well as strongly performs for (2), where it particularly excels in distinguishing arguments with only small differences in quality. They also find that prompt-based approaches perform significantly worse.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "- 144: attempt\n- 175: uses\n- 186: aim\n- 187: strategy\n- Figure 1: augments → arguments?\n- 257: the other\n- 409: In the AQ"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed on the importance of analyzing and understanding the structural similarity between arguments to assess their quality. They found the approach novel and sound. The reviewers also appreciated the author rebuttal which clarified the issues highlighted in the reviews.
Decision: Accept-Main