Exploring the Potential of Large Language Models in Computational Argumentation (2024)

Abstract: Computational argumentation has become an essential tool in various domains, including law, public policy, and artificial intelligence. It is an emerging research field in natural language processing that attracts increasing attention. Research on computational argumentation mainly involves two types of tasks: argument mining and argument generation. As large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language, it is worthwhile to evaluate the performance of LLMs on diverse computational argumentation tasks. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings. We organize existing tasks into six main categories and standardize the format of fourteen openly available datasets. In addition, we present a new benchmark dataset on counter speech generation that aims to holistically evaluate the end-to-end performance of LLMs on argument mining and argument generation. Extensive experiments show that LLMs exhibit commendable performance across most of the datasets, demonstrating their capabilities in the field of argumentation. Our analysis offers valuable suggestions for evaluating computational argumentation and its integration with LLMs in future research endeavors. 1

heading: 1 Introduction
text: Argumentation is a powerful and indispensable tool in various domains such as legality (Mochales and Moens, 2011; Grabmair et al., 2015), debating (Slonim et al., 2021; Li et al., 2020), and education (Stab and Gurevych, 2016). It plays a vital role
*Equal contribution. Guizhen Chen is under the Joint PhD Program between Alibaba and Nanyang Technological University.
†Liying Cheng is the corresponding author. 1Our data and code implementation are released at
https://github.com/DAMO-NLP-SG/LLM-argumentation.
in facilitating understanding between individuals by providing insights into different perspectives and their underlying reasons. Additionally, argumentation serves as a means of communicating convincing opinions, enhancing the acceptability of positions among readers. As computational argumentation becomes a growing research field in natural language processing (NLP) (Dietz et al., 2021; Habernal and Gurevych, 2016a; Atkinson et al., 2017; Wachsmuth et al., 2017; Holtermann et al., 2022; Barrow et al., 2021), researchers have dedicated considerable efforts to two distinct directions (Chakrabarty et al., 2019; Cheng et al., 2021; Alshomary et al., 2021; Bilu et al., 2019). The first direction, argument mining, focuses on understanding unstructured texts and automatically extracting various argumentative elements (Cabrio and Villata, 2018; Levy et al., 2014a; Rinott et al., 2015; Cheng et al., 2022). The other direction is argument generation, which aims to generate argumentative texts based on external knowledge (Hua et al., 2019; Schiller et al., 2020) or summarize key argument points. (Syed et al., 2021; Roush and Balaji, 2020).
Unlike classical structure prediction NLP tasks like named entity recognition that typically take a single sentence as the input and extract token-level information, computational argumentation tasks require discourse-level comprehension. This requirement makes it challenging and laborious to gather a large volume of labeled data for training, hindering the progress of research in this field. Fortunately, recent studies have shown that large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Tay et al., 2022; Touvron et al., 2023a) have demonstrated impressive performance on a wide variety of NLP tasks (Zhong et al., 2023; Pan et al., 2023b; Wang et al., 2023b; Cheng et al., 2023; Shen et al., 2023) in both zero-shot and few-shot settings. Given their strong capability in understanding long contexts and generating natural language, it is ex-
2309
citing yet still questionable how well LLMs can perform computational argumentation tasks without any supervised training.
In light of this, our objective is to investigate the performance of LLMs on diverse computational argumentation tasks. There are two main issues we aim to address in our study. Firstly, although there are existing surveys about argument mining (Peldszus and Stede, 2013), the systematic study of the broader definition of computational argumentation including argument mining and argument generation is under-explored. To bridge this gap, we categorize current computational argumentation tasks into two primary classes, comprising six distinct categories. In addition, we establish a standardized format and evaluation metrics for fourteen openly available datasets. Secondly, existing tasks and datasets either focus on argument mining or argument generation. To take a holistic approach, we propose a new task that integrates both argument mining and generation. This task is designed to generate counter speeches in response to debate speeches, which typically advocate a particular stance. We name them counter speech and supporting speech respectively in the remainder of our paper. This task requires the model to understand the argumentative structures in the supporting speech, meanwhile to generate the counter speech against the proposition. To facilitate the study, we construct a new document-to-document counterargument generation benchmark based on a debate database (Lavee et al., 2019).
To evaluate the performance of LLMs on computational argumentation tasks, we choose from both open-source and proprietary LLMs to conduct our main experiments, in zero-shot and fewshot settings. Our results reveal that LLMs exhibit promising performance in both argument mining and argument generation tasks. While LLMs might fail to achieve exceptionally high scores on specific metrics such as ROUGE, we hypothesize that the strict nature of these metrics could potentially underestimate the true potential of LLMs, which are inherently generative in nature. Human evaluation shows that LLMs are able to comprehend the core meaning of arguments and convey them effectively, even if the exact wording might not match. Collectively, these findings highlight the strengths of LLMs in grasping and effectively conveying the essence of arguments, showcasing their potential beyond what traditional metrics may suggest.
To summarize, our contributions include:
• We organize the existing computational argumentation tasks including argument mining and argument generation, and standardize the format of related datasets.
• We introduce a new task targeted at evaluating both argument mining and argument generation capabilities as a whole.
• To the best of our knowledge, we for the first time systematically evaluate the performance of multiple computational argumentation tasks using LLMs in zero-shot and few-shot settings.
• Extensive experimental results and analysis demonstrate the potential of LLMs in the computational argumentation research field and also suggest limitations in existing evaluation.

heading: 2 Background
text: Computational Argumentation Argumentation research has a long history (Walton et al., 2008; Hinton, 2019), aiming to persuade through logical propositions and achieve agreement among parties (Van Eemeren et al., 2004). Recently, computational argumentation has emerged as a significant field in NLP. The two main research directions are argument mining and argument generation, along with other directions such as persuasiveness of arguments (Habernal and Gurevych, 2016b) and quality assessment of arguments (Wachsmuth et al., 2017). Our work specifically focuses on argument mining and argument generation, where the detailed background can be found in Appendix A.
Large Language Models Recently, LLMs such as ChatGPT (OpenAI, 2023) have demonstrated strong capabilities in various NLP tasks. A surge of research has emerged to analyze and evaluate their performance on different types of tasks (Leiter et al., 2024; Liu et al., 2023d; Yang et al., 2024; Guo et al., 2023a; Laskar et al., 2023), including translation (Jiao et al., 2023; Liu et al., 2024), reasoning (Shakarian et al., 2023; Frieder et al., 2023; Liu et al., 2023b; Pan et al., 2023a), question answering (Tan et al., 2023; Pham et al., 2024), sentiment analysis (Zhong et al., 2023; Deng et al., 2023; Wang et al., 2023c; Zhang et al., 2023b; Nguyen et al., 2023a), text-to-SQL (Li et al., 2023; Liu et al., 2023a), dialogue understanding (Pan et al., 2023b; Fan and Jiang, 2023; Hu et al., 2023), relation extraction (Yuan et al., 2023; Nguyen et al., 2023b; Zheng et al., 2023), hate speech detection (Das et al., 2023; Hoang et al., 2024), summariza-
tion (Nguyen and Luu, 2022; Yang et al., 2023; Wang et al., 2023a; Luo et al., 2023; Zhang et al., 2023a), and trustworthiness (Zhao et al., 2023, 2024) etc. However, it still lacks a systematic and thorough evaluation of computational argumentation using LLMs. Therefore, our work aims to explore the field of computational argumentation using LLMs by covering multiple tasks.

heading: 3 Tasks and Datasets
text: In this work, we systematically review existing tasks and datasets of computational argumentation and organize them in Figure 1. To maintain a balance for different tasks and datasets, we restrict our assessment by randomly sampling 500 examples from each dataset.

heading: 3.1 Argument Mining
text: We focus on the detection of argumentative components and their relations, which are the fundamental tasks in argument mining. We include a range of datasets with varying levels of difficulty, from simple binary tasks such as claim detection and evidence detection to harder ones like evidence type classification and stance detection. While we do not cover joint tasks such as end-to-end argument mining (Eger et al., 2017; Bao et al., 2022), these tasks could be transformed into a sequence of subtasks of identifying argumentative components and relations, which could be handled by our evaluated tasks.
Claim Detection A claim is a statement or proposition that asserts something to be true or false. In the context of argument mining, a claim is a key argument component that forms the basis of reasoning and debate. In claim detection tasks, the goal is to automatically extract claims from articles related
to a specific debating topic (Levy et al., 2014b). We evaluate on datasets including IAM Claims (Cheng et al., 2022), IBM Claims (Levy et al., 2018a), and IBM Argument (Shnarch et al., 2020).
Evidence Detection Evidence is any information or data that supports or undermines a claim. In argument mining, evidence extraction involves automatically identifying and extracting relevant evidence from texts to substantiate claims (Rinott et al., 2015). Automating this process aids in comprehending and assessing arguments. By pinpointing relevant evidence, researchers can gain valuable insights into the underlying beliefs and motivations behind an argument. We evaluate evidence detection on the IBM Evidence dataset (Shnarch et al., 2018) and the IAM Evidence dataset (Cheng et al., 2022).
Stance Detection Stance represents a position towards a controversial topic, usually in the form of support and attack. Stance detection aims to determine whether a text supports, opposes, or remains neutral toward the topic. This task holds significance in domains such as politics (Habernal et al., 2017), fact-checking (Thorne et al., 2018; Guo et al., 2021), and journalism (Hanselowski et al., 2019), as it helps gauge public opinion and attitudes. Automated stance detection enhances the understanding and analysis of arguments across various applications. We use multiple datasets for evaluation, including FEVER (Thorne et al., 2018), IAM Stance (Cheng et al., 2022), IBM Stance (Levy et al., 2018b), and Multi-Target Stance Detection (MTSD) (Sobhani et al., 2017).
Evidence Type Classification Evidence type refers to the different categories of evidence that can be used to support or undermine a claim (Addawood and Bashir, 2016; Rinott et al., 2015). Examples of evidence types from previous works include statistics, expert opinions, facts, anecdotes, examples, etc. Automatic evidence type classification aids in understanding the strengths and weaknesses of an argument, particularly in fields such as debate, law, and policy. We use two datasets for evaluation, including IBM Type (Aharoni et al., 2014) and AQE Type (Guo et al., 2023b).

heading: 3.2 Argument Generation
text: We cover two main tasks: argument generation and argument summarization.
Generation Argument generation involves automatically generating arguments for or against a particular topic, to create persuasive and coherent arguments that can support or challenge a given position. We adopt the CounterArguGen dataset (Alshomary et al., 2021) for evaluation. There are two settings: generating a counter-argument given a claim with premises or generating based on a claim with weak premises.
Summarization The goal of argument summarization is to extract the main ideas and evidence supporting or challenging a particular claim or position and present them concisely and coherently. We evaluate two datasets: ConcluGen (Syed et al., 2021) and DebateSum (Roush and Balaji, 2020), which aim to summarize or give a conclusion for arguments. In the ConcluGen dataset, the corpus is augmented with three types of argumentative knowledge: topic, targets, and aspects. We study the effect of each argumentative knowledge and compare their respective performance with the base setting. In the DebateSum dataset, there are two settings. The abstractive summary generates a concise summary of the main points and arguments, while the extractive summary aims to extract relevant evidence from the passage to support the arguments.

heading: 3.3 Counter Speech Generation
text: Existing tasks in the field primarily center around either argument mining or argument generation. The former emphasizes language understanding, whereas the latter focuses on language generation. However, there is a lack of research comprehensively studying the overall argumentative capabilities of models. We contend that argument understanding and argument generation are two indispensable components of the broader computational argumentation landscape. Hence, a holistic perspective is necessary for evaluating the argumentative capabilities of models. Focusing solely on argument mining or argument generation provides only a partial understanding of their true potential.
In light of this, we propose a new task, counter speech generation, that aims to provide a more thorough evaluation of LLMs’ argumentative capabilities. This task serves as a means to assess the model’s capability to comprehend argumentative structures and generate counter-arguments accordingly. In debates, a supporting speech serves as a form of discourse intended to construct a specific idea or stance. It aims to provide compelling
arguments and evidence in favor of a particular viewpoint. Counter speech generation, therefore, involves the task of generating a responsive or opposing speech in reaction to the supporting speech.
To the best of our knowledge, this is the first document-to-document counterargument generation task that simultaneously assesses a model from multiple perspectives including claim detection, stance detection, and argument generation. Earlier works focus on mining and retrieval of counterarguments (Wachsmuth et al., 2018; Bondarenko et al., 2020; Jo et al., 2021), which does not involve argument generation. Some focus on generating an opposing argument for a given statement which are typically short, informal texts from online forums (Alshomary et al., 2021; Hua and Wang, 2018; Hua et al., 2019). In contrast, ours consists of complete, formal speeches that are in the form of long argumentative texts, which potentially contain multiple arguments. Our task requires the model to first mine and analyze the main arguments in the original speech, then construct a complete and cohesive speech that addresses each key point. This expanded scope challenges the model to have a deeper understanding of argumentative structures from longer passages, while also requiring a heightened capacity to generate complete counter speeches.
To facilitate this study, we process a debate dataset (Lavee et al., 2019) by matching each supporting speech with the corresponding counter speech in a pool of debate scripts. We randomly sample 250 speech pairs for our zero-shot experiments. Given the constraint on limited annotated samples, we evaluate in a zero-shot setting only. Appendix B shows a data sample of this dataset.

heading: 4 Experiments
text: In this section, we discuss our choices of models, methods, and evaluation metrics.

heading: 4.1 Models
text: In our investigation, we examine the effectiveness of LLMs in directly performing inference on argument-related tasks without any finetuning. To accomplish this, we evaluate on open-source and proprietary models, including ChatGPT (GPT-3.5-Turbo) from OpenAI (OpenAI, 2023), Flan-T5-XL, Flan-T5-XXL (Chung et al., 2022) and Flan-UL2 (Tay et al., 2022) from the Flan model family, as well as Llama-2-7B,
Llama-2-13B from the LLaMA2 series (Touvron et al., 2023b).

heading: 4.2 Methods
text: For tasks of a similar nature, we employ a consistent prompt format. More specifically, for argument mining tasks, we adhere to a standardized prompt template which consists of task definition and required output format. The task definition serves as a clear guideline for the LLMs to understand the task objective, while the required output format provides clarity on the expected output structure and restricts the generated response to a set of predefined labels to facilitate easier evaluation.
In contrast to argument mining tasks, output for argument generation tasks is more free-style and not constrained by any predetermined label space. The focus is on generating contextually relevant arguments. In order to tap into LLMs’ linguistic knowledge and reasoning abilities, we adopt the prompts advised by ChatGPT. The prompt templates are available in Appendix C.
To tackle counter speech generation, we propose three different approaches2, as shown in Figure 2. The first approach follows a pipeline method. We first identify the main claims from the supporting speech by determining if each sentence is a claim towards the given topic. We use Flan-T5-XXL due to its fast computation and strong capability in claim detection. After identifying all claims, we generate counterarguments that attack each claim detected in the supporting speech. For this step, GPT-3.5-Turbo is employed due to its strong generative ability.
Another pipeline approach is by generating a summary of the supporting speech. Initially, the key arguments in the supporting speech are summarized into a condensed representation of the main
2Note that other combinations of models could be used for each approach. Here we only employ the strongest model for each task/subtask, guided by the results in Section 5.
points. Subsequently, a counter speech is crafted to challenge these key arguments. In both steps, we use GPT-3.5-Turbo, which is adept at handling long inputs and comprehending long contexts.
Unlike the two-step approaches, the third method is a one-step process where we directly prompt GPT-3.5-Turbo to respond to the supporting speech by challenging the main arguments. This approach serves as a means of gauging the model’s ability to internally identify key arguments and generate a respective counter speech.

heading: 4.3 Evaluation
text: To evaluate argument mining tasks, we use both accuracy and F1 score as the metrics.
To assess argument generation and counter speech generation, we employ a wide range of automatic evaluation metrics, including ROUGE1, ROUGE-2, ROUGE-L (Lin, 2004), METEOR (Denkowski and Lavie, 2011) and BERTScore (Zhang et al., 2019). The ROUGE scores assess the quality based on the overlap with the reference arguments, while METEOR also considers synonyms, paraphrases, and stemming. On the other hand, BERTScore takes into account the semantic context. We also conduct human evaluation to complement the results from automatic evaluation.

heading: 4.4 Previous SOTA
text: To compare our results against existing state-ofthe-arts (SOTA), we either finetune pre-trained language models (PLMs) or leverage available checkpoints to conduct inference on our sampled test set. Training details are reported in Appendix D.

heading: 5 Results and Discussion
text: In this section, we discuss the main results and provide insights into the performance of various LLMs on argument mining, argument generation and counter speech generation.

heading: 5.1 Results on Argument Mining
text: Table 1 shows the zero-shot performance of three representative models, GPT-3.5-Turbo, Flan-UL2 and Llama-2-13B, across 11 argument mining datasets. Statistical tests3 are conducted to show if the LLM’s predictions are significantly different from the random observations. Results of other models including Flan-T5-XL, Flan-T5-XXL and Llama-2-7B are available in Appendix E.
3We use McNemar’s test (Mcnemar, 1947) following the guidelines by Dror et al. (2018).
involving multi-class are underlined red. Highest accuracy and F1 score for each task are in bold. ∗ indicates statistically significant results that are different from random observations with a significant level of α = 0.05.
To analyze, we categorize tasks into simple and hard tasks based on the number of classes involved. Binary classification tasks, including claim detection, evidence detection, IAM stance detection and IBM stance detection, are classified as simple tasks. Tasks with more than two labels, including evidence classification, FEVER stance detection and MTSD stance detection, are classified as hard tasks.
Overall, both GPT-3.5-Turbo and Flan-UL2 perform decently in the zero-shot setting, surpassing the random baseline with most results being statistically different. However, Llama-2-13B falls short of the random baseline in the majority of the tasks, notably in more challenging tasks like FEVER stance detection and evidence classification. This highlights its limitations in capturing nuanced stances and comprehending evidence types within the zero-shot context, which relies on sufficient prior knowledge in the model.
Comparing GPT-3.5-Turbo and Flan-UL2, Flan-UL2 consistently demonstrates higher profi-
ciency in tasks like claim detection, evidence detection, and certain stance detection tasks that are mostly binary classification tasks. However, its performance diminishes when encountered with tasks that involve more than two classes, such as MTSD stance detection and AQE evidence classification. In contrast, GPT-3.5-Turbo generally demonstrates superior performance in these multiclass scenarios.
Figure 3 shows the effects of increasing shots on different models and task difficulties. In general, while there remain certain gaps between the fewshot performance of LLMs and finetuned PLMs using the full training set, it is worth noting a significant trend: by simply prompting LLMs with less than 10 demonstrations, they are able to close the gaps and match the performance of finetuned PLMs trained with 500 samples.
Comparing among models, we notice that the choice of model is crucial, as different models exhibit varying levels of proficiency across different
tasks. While Flan models excel in simple tasks, the performances of Flan-T5-XL and Flan-UL2 lag behind that of GPT-3.5-Turbo in hard tasks, even with an increased number of shots. Overall, Flan-T5-XXL appears to be most robust, consistently demonstrating strong performance across diverse tasks.
Secondly, larger models are not necessarily superior to smaller models. Upon comparing the two LLaMA models, Llama-2-13B generally outperforms Llama-2-7B. However, one interesting exception surfaces when the input is minimal. Surprisingly, Llama-2-7B proves to be more effective than Llama-2-13B in simple tasks. For Flan models, larger models consistently outperform their smaller counterparts in simple tasks. This trend, however, does not hold in the case of more challenging tasks. Notably, Flan-T5-XL (3B) model performs comparably to Flan-UL2 (20B) in hard tasks, despite its significantly smaller size. Furthermore, the 11B Flan-T5-XXL model showcases remarkable performance, even though it is smaller than both Flan-UL2 and Llama-2-13B. This sug-
gests that, for certain complex tasks, the performance of the model may not be solely determined by its size.
Furthermore, increasing demonstrations have varying effects on different models. GPT-3.5-Turbo generally benefits from more shots. For Flan models, the gain in performance is not obvious. Llama models, on the other hand, exhibit mixed performance in response to more demonstrations. The larger model demonstrates notable performance improvement, particularly in hard tasks, when provided with more shots. However, the smaller model does not exhibit performance gain from additional demonstrations. In fact, when it comes to simple tasks, providing more shots has a negative impact. It appears that longer contexts might introduce noise or unnecessary information that could potentially hinder the performance of smaller models.

heading: 5.2 Results on Argument Generation
text: Table 2 presents the performance of GPT-3.5-Turbo on argument generation tasks.
Compared to existing SOTA, GPT-3.5-Turbo already outperforms previous methods in several tasks including CounterArguGen, ConcluGen in the base setting, as well as abstractive summarization in DebateSum.
Although previous methods excel in other ConcluGen settings, we attribute their high performance to additional annotations encoding specific aspects, targets, or topics. Such manual annotations are task-specific and extremely costly. GPT-3.5-Turbo, on the other hand, achieves comparable results across different settings regardless of the presence of encoded information. Furthermore, the contrasting results from different evaluation metrics reveal an interesting pattern: the ROUGE scores are generally low but the BERTScores are high. The low ROUGE scores indicate that there are only a few overlaps between the generated text and the reference text. The high BERTScore indicates that the semantic meaning of the generated text is highly similar to the reference text. This suggests that although the generated text may not match the reference text in terms of exact wording or specific phrases, it successfully captures the underlying semantic meaning. To further support this, we provide several illustrative examples in Appendix F. Both automatic evaluation and quality analysis show that GPT-3.5-Turbo grasps the essence of the content and conveys it effectively, even if the choice of words or phrasing differs from the reference.
For extractive summarization, previous method (Roush and Balaji, 2020) relies on word-level classification, wherein each word is predicted as either “underlined” or “not-underlined”, which is inefficient and compromises the coherence of the generated sentences. In contrast, GPT-3.5-Turbo avoids the high training cost and generates coherent sentences. Additionally, our quality analysis shows that GPT-3.5-Turbo is able to identify important information accurately. Case studies can be found in Appendix G.
In addition, we notice that GPT-3.5-Turbo exhibits incremental performance improvements as the number of shots increases. The performance gains are relatively modest compared to those observed in argument mining tasks. This implies that GPT-3.5-Turbo is inherently proficient in argument generation without necessitating more demonstrations.
We also evaluate other models including Llama-2-7B, Llama-2-13B and Flan-UL2 which
could accommodate long context. Performance of other models are available in Appendix H. All models exhibit similar trends with the above except Flan-UL2 - its advantage in extractive summarization is less apparent compared to the other models.

heading: 5.3 Results on Counter Speech Generation
text: Automatic Evaluation Table 3 shows the results from automatic evaluation. The end-to-end approach surpasses the summarization pipeline approach across all metrics. This highlights the model’s strong capability of internalizing and synthesizing information from the supporting speech without the need for intermediate steps.
Comparing the end-to-end approach to the claim detection pipeline approach, the former lags behind in ROUGE-1 and METEOR, but surpasses in BERTScore, ROUGE-2, and ROUGE-L. To determine which approach is superior, we conduct human evaluation for a more complete understanding of the performance of these two approaches.
Human Evaluation We hire 2 human judges who are professional English speakers to manually evaluate the quality of counter speeches generated by the claim pipeline approach and the end-to-end approach on 50 random samples. For each test instance, we provide the judges with supporting speeches along with randomly ordered counter speeches from the two methods, and ask the judges to individually evaluate the generation quality based on the following criteria:
• Fluency (Flu.): Is the generation fluent, grammatical, and without unnecessary repetitions?
• Persuasiveness (Per.): Is the text able to convince you to adopt a certain belief or attitude?
• % of arguments addressed (% Arg.): Does the counter speech address all claims/arguments in the supporting speech?
Fluency and persuasiveness, graded on a scale of 1 to 5, are to assess model’s argument generation capability. To evaluate the model’s argument mining
ability, we use % of arguments addressed, calculated by the number of addressed arguments in the counter speech over the total number of arguments in the supporting speech. This metric reflects how effectively the model is able to identify arguments, either explicitly in the two-step approach or implicitly in the one-step approach.
In Table 4, it is evident that the end-to-end approach outperforms the pipeline approach on all 3 metrics. In specific, the pipeline approach is not able to address as many arguments as the end-toend approach, possibly due to the potential loss of information during the intermediate step. In the pipeline method, information from the supporting speech undergoes processing, such as summarization or claim detection, before the final counter speech is generated. This might result in information loss or distortion, which could negatively impact the overall coherence and effectiveness of the generated response, which in turn affects the fluency and persuasiveness scores. In contrast, the one-step approach bypasses the intermediate stage, allowing the model to directly engage with the supporting speech and generate a counter speech in a more holistic manner. We show qualitatively in Appendix I.

heading: 6 Conclusion and Broader Impacts
text: In this paper, we have made several significant contributions to the field of computational argumentation research.
Firstly, our efforts in organizing the diverse landscape of argumentation-related tasks and standardizing the format of related datasets are crucial for future research in designing domain-specific largescale models for argumentation.
Secondly, we for the first time systematically evaluate the performance of multiple computational argumentation tasks using LLMs in zero-shot and few-shot settings. Traditional approaches for computational argumentation rely heavily on supervised fine-tuning that requires a large amount of labeled data, hindering the progress of research in this field. Our exploration of low-resource settings addresses a gap in previous computational argumentation research, demonstrating the potential of
LLMs in scenarios with limited training data. Furthermore, we introduce a new counter speech
generation benchmark that evaluates models’ capability in both argument mining and argument generation. Our extensive experimental results and analysis demonstrate the potential of LLMs in computational argumentation, while also highlighting existing limitations in evaluating computational argumentation tasks.
Overall, our paper provides important insights and valuable resources for researchers interested in the field of computational argumentation, which will potentially inspire further advancement in this exciting area.
Limitation
In the field of computational argumentation, there are more tasks involved. In this work, we only cover argument mining and argument generation, as these two categories are the most fundamental in this field. By understanding and establishing the performance of these core tasks, we could progress to tackle other tasks in our future study. In addition, it is challenging and laborious to conduct human evaluation on the full argument generation datasets. To address this, we could use GPT-4 as an evaluator (Liu et al., 2023c).

heading: Acknowledgments
text: This work was supported by DAMO Academy through DAMO Academy Research Intern Program. We would like to thank Wenxuan Zhang for providing feedback on the paper.

heading: A More Background
text: A.1 Argument Mining
Argument mining is a rapidly emerging field of NLP that aims to automatically identify and extract arguments and their components from textual data. With the increasing volume of digital text available online, the need for automated methods to analyze and understand arguments has become more pressing. By identifying the arguments in natural language text, researchers can better understand the underlying beliefs, values, and motivations that drive human behavior. As such, argument mining is a core task of research within NLP that is poised to make significant contributions to a wide range of fields.
A.2 Argument Generation
With the understanding of the argumentative structures within the text through argument mining, the next step is to explore how to generate arguments. Argument generation and argument summarization are two related tasks within computational argumentation that have the potential to transform the way we create and consume arguments. Argument generation involves the automatic creation of persuasive text, such as generating a sentence attacking another standpoint, that can be used to influence a group of readers. Argument summarization, on the other hand, involves the automatic summarization of arguments, enabling users to quickly and easily understand complex arguments without having to read through lengthy documents. For example, in the law domain, large amounts of legal documents need to be analyzed and understood in a time-sensitive manner. As such, argument generation and summarization are two key areas of research within NLP that have the potential to significantly streamline the process of argumentation in various domains.

heading: B Data Sample on Counter Speech Generation
text: Table 5 shows a data sample from our benchmark dataset for the proposed counter speech generation task. The topic is “Nationalism does more harm than good”. The supporting speech is the input, and the counter speech written by humans is considered the output.

heading: C Prompt Templates
text: C.1 Prompt Templates for Argument Mining Tasks
Table 6 shows the prompt templates for selected argument mining tasks, including claim detection and stance detection.
C.2 Prompt Templates of Argument Generation Tasks
Table 7 shows the prompt templates for argument generation tasks, including counter argument generation and abstractive summarization.

heading: D Training Details of SOTA
text: For argument mining tasks, we train sentence-pair classifiers based on pre-trained models such as BERT (Devlin et al., 2019) following the settings reported by Cheng et al. (2022). Dataset statistics can be found in Table 8. For datasets where training sets are not available, we randomly sample 500 data points to reserve for the test set and make use of all the remaining for training.
For CounterArguGen, we directly evaluate based on the released predictions (Alshomary et al., 2021).
For ConcluGen (Syed et al., 2021), we use the available checkpoints and conduct inference on our sampled test set. The released checkpoints were trained on the reported training sets, while our 500 samples were sampled from the test sets for proper train-test split.
For DebateSum tasks, we randomly sample 90000 data for train set, 10000 for development set, and 500 for test set, since the original train test split is not specified. Specifically, for abstractive summarization, we finetune a T5-base (Raffel et al., 2020), a popular and performant generative model, using the AdamW optimizer with a learning rate of 1e-4, a fixed batch size of 4, and 3 training epochs. For DebateSum extractive summarization, we follow the settings reported by Roush and Balaji (2020).

heading: E Additional Results on Argument Mining
text: Table 9 shows the zero-shot performance of Flan-T5-XL, Flan-T5-XXL and Llama-2-7B on argument mining tasks.

heading: F Quality Analysis on Argument Generation Tasks
text: To further support our claims in Section 5.2, we show three examples of references and predictions from the ConcluGen dataset in Table 10. For instance, in the third pair, while the generated text uses “should” instead of “need to”, and “be held to the same standards” instead of “follow the same set of rules”, it effectively conveys the same meaning as the reference. These observations imply that the generated text might have used different wordings but the overall semantic meaning is similar to that of the reference text, which further supports our claims.

heading: G Case Study on Extractive Summarization
text: To further support our claims in Section 5.2, we show 2 examples in Table 11 and 12. It can be observed that Longformer tends to generate incoherent sentences, while GPT-3.5-Turbo can generate coherent sentences and can extract important information from the input.

heading: H Additional Results on Argument Generation
text: Table 13, 14 and 15 display the evaluation results of argument generation tasks using Llama-2-7B, Llama-2-13B and Flan-UL2 respectively.

heading: I Case Study on Counter Speech Generation
text: Table 16 shows a case study of the data sample shown in Table 5 for the proposed counter speech generation task. The pipeline approach by extracting claims first tends to generate repetitive phrases, and does not attack all the claims stated in the supporting speech. In contrast, the end-to-end approach is more concise and attacks the claims in the supporting speech.
Model Claim Detection Evidence Detection Stance Detection Evidence Classification
IBM Claims IBM Argument IAM Claims IBM Evidence IAM Evidence IAM Stance IBM Stance MTSD FEVER IBM Type AQE Type

heading: Future Work
text: Firstly, our efforts in organizing the diverse landscape of argumentation-related tasks and standardizing the format of related datasets are crucial for future research in designing domain-specific largescale models for argumentation. Secondly, we for the first time systematically evaluate the performance of multiple computational argumentation tasks using LLMs in zero-shot and few-shot settings. Traditional approaches for computational argumentation rely heavily on supervised fine-tuning that requires a large amount of labeled data, hindering the progress of research in this field. Our exploration of low-resource settings addresses a gap in previous computational argumentation research, demonstrating the potential of
LLMs in scenarios with limited training data. Furthermore, we introduce a new counter speech
generation benchmark that evaluates models’ capability in both argument mining and argument generation. Our extensive experimental results and analysis demonstrate the potential of LLMs in computational argumentation, while also highlighting existing limitations in evaluating computational argumentation tasks. Overall, our paper provides important insights and valuable resources for researchers interested in the field of computational argumentation, which will potentially inspire further advancement in this exciting area. Limitation
In the field of computational argumentation, there are more tasks involved. In this work, we only cover argument mining and argument generation, as these two categories are the most fundamental in this field. By understanding and establishing the performance of these core tasks, we could progress to tackle other tasks in our future study. In addition, it is challenging and laborious to conduct human evaluation on the full argument generation datasets. To address this, we could use GPT-4 as an evaluator (Liu et al. , 2023c)