Dynamic Knowledge Integration for Evidence-Driven Counter-Argument Generation with Large Language Models
Authors: Anar Yeginbergen, Maite Oronoz, Rodrigo Agerri
Accepted at: acl25

Dynamic Knowledge Integration for Evidence-Driven Counter-Argument Generation with Large Language Models

Anar Yeginbergen and Maite Oronoz and Rodrigo Agerri
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
{anar.yeginbergen,maite.oronoz,rodrigo.agerri}@ehu.eus

Abstract

This paper investigates the role of dynamic external knowledge integration in improving counter-argument generation using Large Language Models (LLMs). While LLMs have shown promise in argumentative tasks, their tendency to generate lengthy, potentially non-factual responses highlights the need for more controlled and evidence-based approaches. We introduce a reconstructed and manually curated dataset of argument and counter-argument pairs specifically designed to balance argumentative complexity with evaluative feasibility. We also propose a new LLM-as-a-Judge evaluation methodology that shows a stronger correlation with human judgments compared to traditional reference-based metrics. Our experimental results demonstrate that integrating dynamic external knowledge from the web significantly improves the quality of generated counter-arguments, particularly in terms of relatedness, persuasiveness, and factuality. The findings suggest that combining LLMs with real-time external knowledge retrieval offers a promising direction for developing more effective and reliable counter-argumentation systems. Data and code are publicly available.1

1 Introduction

Argumentation in Natural Language Processing (NLP) is becoming an increasingly active area of research, driven by the natural human tendency to express disagreement with claims or viewpoints expressed by individuals during information exchanges. In fact, it is becoming an indispensable tool in many application domains such as public policy, law, medicine, and education (Stab and Gurevych, 2017; Eger et al., 2018; García-Ferrero et al., 2024; Yeginbergen et al., 2024; Sviridova et al., 2024).

![Our approach to counter-argument generation integrating dynamic external knowledge.](page_1016_352_388_388.png)

Figure 1: Our approach to counter-argument generation integrating dynamic external knowledge.

It is possible to distinguish two main research lines in Argumentation in NLP: (i) argument mining, which involves analyzing unstructured texts to automatically identify and extract argumentative elements and their relations (Cabrio and Villata, 2018; Stab and Gurevych, 2017; Yeginbergen et al., 2024); (ii) argument generation, which focuses on generating argumentative texts using external knowledge sources (Hua et al., 2019; Schiller et al., 2021) or summarizing key argumentative points (Roush and Balaji, 2020; Syed et al., 2021; Chen et al., 2024).

This paper contributes to the second line of research by investigating whether dynamic integration of external knowledge helps LLMs to improve counter-argument generation. Counter-argument generation seeks to develop an effective framework for presenting alternative perspectives to an argument while ensuring the correctness of the message and incorporating factual evidence (Wachsmuth
et al., 2017, 2018). LLMs have shown promising potential to deal with debates or any disagreements by solely relying on the parametric knowledge encoded in the model (Chen et al., 2024; Alshomary and Wachsmuth, 2023). Moreover, most of the LLMs have security safeguards to avoid harmful interactions by any means (Bai et al., 2022; Ouyang et al., 2022; Touvron et al., 2023). While maintaining harmlessness is recommendable, it is also important to be reasonable, persuasive, and grounded with a factual basis when arguing (Khan et al., 2024).

Previous work on counter-argument generation has focused on specific aspects of the argument (Schiller et al., 2021; Saha and Srihari, 2023), integrating different external news sources (Hua et al., 2019), or by attacking different evidence of the initial argument (Jo et al., 2020; Chen et al., 2024). However, no previous work has considered integrating dynamic knowledge to inform counter-argument generation. Furthermore, we argue that existing datasets for counter-argument generation consist of excessively long (Hua and Wang, 2018; Hua et al., 2019; Chen et al., 2024) or extremely short (Schiller et al., 2021; Lin et al., 2023) arguments, which makes it difficult to accurately evaluate their quality. Thus, although modern LLMs tend to generate rather lengthy essay-style responses that may be highly persuasive, they lack coherence, logic, and factual evidence, restricting them to short single sentences is insufficient to study the complexity and desired pragmatic characteristics of a good counter-argument (Hinton and Wagemans, 2023; Ji et al., 2023; Carrasco-Farre, 2024; Verma et al., 2024). Finally, the manual evaluation of counter-argument quality is a challenging, time-consuming, and highly subjective task that requires knowledge of the subject matter and credible evidence to support or refute it (Wachsmuth et al., 2017; Wang et al., 2017; Hua et al., 2019). In this sense, we believe that traditionally used (Alshomary et al., 2021; Chen et al., 2024) reference-based automatic metrics such as BLEU, METEOR, or BERTScore fail to accurately capture the nuanced relationship between the generated counter-argument and the human judgment.

In order to address these issues, we propose using retrieved external knowledge from dynamic sources without any limits to particular outlets or databases, along with LLMs, to generate efficient, concise, evidence-based counter-arguments. Furthermore, we present a new curated dataset of concise and structured human-generated argument and counter-argument pairs in which the length of the counter-arguments is enough to study the main argumentative aspects while facilitating manual and automatic evaluation. While this paper employs both manual and automated evaluation methods, we introduce a novel automated evaluation approach using LLMs-as-a-Judge, designed to optimize correlation with human assessments.

Using the new dataset and dynamically retrieved external evidence, this work aims to answer the following research questions (RQ):

• RQ1: Does the integration of dynamic external knowledge into LLMs help to generate better counter-argumentation?

• RQ2: Which automatic evaluation method correlates better with human judgments?

• RQ3: To what extent do LLMs use retrieved external evidence in producing counter-arguments?

Figure 1 illustrates our proposed framework. First, we automatically generate queries that challenge the main points of the argument or claim for more fact-based information and feed those queries to the web search. Next, related evidence is retrieved, and, lastly, the claim and the retrieved evidence are provided as context to the LLM to generate a counter-argument. The main contributions of our paper are the following:

1. We publicly share a new dataset of manually curated arguments and counter-arguments.

2. A new method to dynamically integrate external knowledge retrieved from the web in LLM-based counter-argument generation.

3. Experimental results show that the generation of counter-arguments with LLMs is improved through the integration of dynamic external knowledge, with factual evidence demonstrating a particularly significant impact on pragmatic aspects, including relatedness, persuasiveness, and factuality.

4. Our automatic evaluation based on LLM-as-a-Judge reveals a higher correlation with human judgments compared with reference-based metrics such as BLEU, METEOR, or BERTscore.
2 Related Work

Prior research in efficient and persuasive argument generation has approached the problem from various perspectives, focusing on different aspects of arguments. For instance, Jo et al. (2020) investigated the ability of machine learning approaches to detect attackable sentences in the arguments, and they concluded that automatic approaches are more stable in this task than human annotators, depending on the subjectivity, topic, and tone of the argument. Alshomary et al. (2021) focused on generating counter-arguments by pinpointing and challenging weak premises. Schiller et al. (2021) produces arguments by specifying the desired aspect and stance in a sentence-level setting via Conditional Transformer Language Model (CTRL) (Keskar et al., 2019). Similarly, Saha and Srihari (2023) proposed a method to control both the topic and stance of an argument while enriching it with factual evidence using an encoder-decoder language model. Alshomary and Wachsmuth (2023) propose a counter-argument generation based on the stance of the conclusion of the argument. Lin et al. (2023) employed large language models for sentence-level counter-argument generation, implementing a Chain-of-Thought methodology. Hu et al. (2023) introduces an agent-based chain-of-thought (CoT) method for generating arguments by decomposing the claim and sequentially processing the argument for the final output. Finally, Chen et al. (2024) evaluated LLMs in several argument mining and generation tasks, showing the potential of LLMs for this particular task.

The use of external sources has demonstrated its effectiveness in the generation of alternative perspectives. Wachsmuth et al. (2018) analyzed the question of retrieving the best counter-arguments when no prior knowledge is available. Hua and Wang (2018) introduced a framework that incorporated information retrieval, but their approach was limited to using the Wikipedia database as the external source. However, Wikipedia primarily contains static factual information, which may not align with the dynamic nature of arguments. To address this limitation, Stab et al. (2018) expanded the scope by indexing all documents from the Common Crawl database for argument retrieval. Hua et al. (2019) proposed an enhanced framework that leverages databases from news outlets alongside Wikipedia to retrieve evidence and improve the quality of the generated counter-arguments.

All these efforts mainly refer to static databases as external sources, meaning that all the documents containing the evidence of the argument in question should be parsed in advance. Moreover, previous argument generation with external knowledge was proposed before the LLMs entered the race. In our work, we believe that we should test the capabilities of LLMs and that, in our ever-changing dynamic world, it is not efficient to rely on a pre-defined set of documents as an external source for factual and persuasive counter-argument generation. Instead, we propose to integrate knowledge from the whole internet as a source for finding factual evidence to generate better counter-argumentation.

3 Data

In order to perform our experiments and to guarantee a balance in the length of input argument and output counter-argument in the data, we constructed a new corpus for the evaluation of counter-argument generation. Previous work often focused on either (too short) sentence-level (Lin et al., 2023; Schiller et al., 2021) or (too long) paragraph-level (Hua et al., 2019; Hua and Wang, 2018) generation.

Given the capabilities of modern LLMs, it was essential to specify clear input and output data to ensure accurate, robust, and fair comparisons during evaluation. Thus, our objective is to create a dataset of argument/counter-argument pairs that would meet specific criteria. We argue that (i) generating single-sentence claim-based counter-arguments is insufficient to accurately assess the quality of counter-arguments produced by LLMs and (ii), counter-arguments that are too long make it extremely difficult to properly evaluate the argumentative quality of the generated text. Our analysis revealed that Large Language Models (LLMs), when not given explicit length constraints, tend to generate verbose, essay-like responses that frequently deviate from true argumentative form, lack substantiating evidence, and demonstrate poor coherence with the original argument while overemphasizing persuasive elements. To address these limitations, we propose generating counter-arguments with a maximum length of three sentences, focusing on conciseness, factual content, and direct alignment with the input argument.

With this aim in mind, we constructed a new dataset of argument and counter-argument pairs using the CANDELA corpus as a basis (Hua et al., 2019). The corpus consists of debates and dis-
cussions on various controversial topics from the r/ChangeMyView2 subreddit. The corpus is centred around real-world online debates where users post their opinions and evidence for any controversial topic and expect other users to provide reasoning for an alternative perspective.

<table>
  <tr>
    <th></th>
    <th># sentence</th>
    <th># words</th>
  </tr>
  <tr>
    <th>arguments</th>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Original</td>
    <td>16</td>
    <td>372</td>
  </tr>
  <tr>
    <td>Intermediate</td>
    <td>3</td>
    <td>83</td>
  </tr>
  <tr>
    <td>Final</td>
    <td>3</td>
    <td>61</td>
  </tr>
  <tr>
    <th>counter-arguments</th>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Original</td>
    <td>30</td>
    <td>921</td>
  </tr>
  <tr>
    <td>Intermediate</td>
    <td>5</td>
    <td>165</td>
  </tr>
  <tr>
    <td>Final</td>
    <td>3</td>
    <td>72</td>
  </tr>
</table>

Table 1: Average number of sentences and words in arguments and counter-arguments in the original, summarized (intermediate), and final versions of the data.

CANDELA is available in a format split by sentences, tokenized, and lowercased, making the reconstruction of the corpus necessary. To address this, we employed LLMs to convert the data back into a coherent, human-readable format.

Once the data was fully reconstructed in a human-readable format, we summarized all arguments and counter-arguments. To avoid any bias, we choose a language model different from those used in our experimentation, namely, Llama-3.1-70B-Instruct (Dubey et al., 2024). The generated summaries were then manually verified against the original data and re-summarized when required to ensure semantic and pragmatic correctness.

While the corpus includes data from real-world interactions and reflects arguments from natural exchanges of information, we found that not all topics were equally suitable to evaluate whether external knowledge benefits counter-argument generation. Specifically, topics lacking a scientific or factual foundation, namely, expressed on deeply subjective topics, often trigger LLMs to produce generic, safe responses due to their safety guardrails. Moreover, the external knowledge of the topics that lack ground-truth factual backing tends to be ambiguous and/or biased. An example of such arguments can be found in Appendix B.

To mitigate potential quality issues, we implemented a manual filtering process for the corpus, retaining only those arguments that demonstrated both high quality and direct relevance to the subject matter, specifically selecting instances where the incorporation of factual information would meaningfully contribute to argument generation.

Finally, we further manually refined the summaries to follow a structured argumentative format, emphasizing components such as the main claim, supporting evidence, and examples where applicable (see examples in Appendix C). This process resulted in a dataset of 150 high-quality 3-sentence paragraph-level argument and counter-argument pairs.

Examples of the final version of the corpus used in the experiments can be seen in Appendices A and C. The average distribution of sentences and tokens in the data is shown in Table 1.

4 Experimental setup

In this section, we describe the methodology for generating counter-arguments with dynamic knowledge integration for evidence-driven counter-argument generation. Preliminary experiments revealed that relying on static pre-defined document sets (such as Wikipedia) for factual evidence retrieval often yields incomplete information regarding specific argument topics, resulting in incoherent and unreliable evidence. We determined that external sources must contain topic-specific content, opinions, and observations directly relevant to the events described in the argument, particularly since claims may reference recent events that post-date the last update of pre-parsed sources. Consequently, our experimental design incorporates dynamic web-based data as the primary information source to address these limitations.

External knowledge is obtained through the web search tool provided by the Cohere API3. The process involved automatically generating five queries (averaging 67 words each) designed to challenge the validity and veracity of the original argument’s claims and premises by questioning key points that required factual substantiation. These queries were sequentially submitted to the web search engine, and the retrieved results, averaging 5,496 words in length, were incorporated as contextual information in the final prompt presented to the language model. The model then generated counter-arguments based on both the original argument and the retrieved contextual information.

To assess the role of external knowledge in
counter-argument generation with LLMs, we performed a comparative analysis using two system configurations: one incorporating external information and another relying solely on the model’s parametric knowledge. The model in the latter configuration received only the original argument as input and was tasked with generating counter-arguments using its internal knowledge base exclusively. This experimental design enabled us to evaluate whether LLMs display better performance when provided with external evidence for counter-argument generation, as measured by comparing the quality of outputs between the two configurations.

The experimentation is performed using two LLMs with strong results on text generation tasks in two different configurations: (i) the LLM using only the claim as a prompt to rely on its parametric knowledge and (ii) the LLM with dynamic external knowledge. This results in the following four models:

• **Command R+**: Command-R+, a 104B parameter model from Cohere For AI (2024).

• **Command R+ with external knowledge**: Command-R+ 104B with external evidence retrieved using Cohere’s API for web search.

• **Mistral-7B-Instruct**: Mistral-7B-Instruct-v0.3 (Jiang et al., 2023).

• **Mistral-7B-Instruct with external knowledge**: Mistral-7B-Instruct-v0.3 with external evidence retrieved via Cohere’s API for web search.

Importantly, all experiments are conducted in an inferential setting under default hyperparameters to assess the real capabilities of LLMs in counter-argument generation. This setup ensures a fair and robust evaluation of their performance in generating meaningful, well-reasoned, and factual counter-arguments. A list of all the prompts used at each generation step is illustrated in Appendix D.

4.1 Evaluation

Following Hua et al. (2019), we assess the quality of the generated counter-arguments using a pointwise 3-point Likert scale across five key dimensions: Opposition, Relatedness, Specificity, Factuality, and Persuasiveness.

It should be noted that the evaluation of generated counter-arguments is inherently subjective. While human evaluation is the gold standard, it is time-intensive, costly, and prone to individual biases (Wachsmuth et al., 2017; Hua et al., 2019; Chen et al., 2024). To address these limitations, along with human evaluation, we will also provide two types of automatic evaluation. First, using reference-based metrics such as BLEU, METEOR, or BERTscore previously used in counter-argumentation generation (Alshomary et al., 2021; Chen et al., 2024). Second, we propose to use the LLM-as-a-Judge approach. Leveraging the consistency, scalability, and efficiency of LLMs, this method enables rapid and reproducible scoring across the five dimensions. To the best of our knowledge, we are the first to propose an LLM-based evaluation for counter-argument generation.

Taking the gold standard counter-argument dataset built in Section 3, we generate a counter-argument for each claim using the four models listed above. The five counter-arguments (gold reference included) are then evaluated by human annotators and LLM-as-a-Judge annotators. More specifically, they are asked to assess each dimension by categorizing the example as *unsatisfactory (score: 1)*, *moderately satisfactory (score: 2)*, or *highly satisfactory (score: 3)* for each of the five dimensions listed above. Finally, reference-based metrics were computed by comparing the generated counter-arguments with the gold reference.

We will also compute the correlation between both automatic evaluation methods and human judgments, as establishing strong alignment is crucial for ensuring valid comparisons and reducing reliance on human evaluators, ultimately leading to a more robust and practical evaluation framework.

**Human evaluation.** We recruited human evaluators through the crowdsourcing platform Prolific\footnote{https://www.prolific.com/} to assess the quality of a sample of 75 generated counter-arguments across the five predefined dimensions. We performed two rounds of evaluations of three participants to ensure that the obtained manual judgments were of high quality. We set test questions in the questionnaire to determine whether the evaluations were performed fairly. Subsequently, we selected the four participants who correctly passed the test. All evaluators were compensated above the minimum rate recommended by the platform. An example of instructions provided for manual evaluation can be found in Appendix F.

**LLM-as-a-Judge.** We employ two state-of-the-art open LLM-as-a-Judge models, namely,
Prometheus (Kim et al., 2024) and JudgeLM (Zhu et al., 2023), and one proprietary model, Claude 3.5 Sonnet\footnote{https://www.anthropic.com/claude/sonnet}. The models are prompted with the same set of instructions for evaluation as those used for human annotators.

Reference-based evaluation. Following the previous evaluations on counter-argument generation (Hua et al., 2019; Lin et al., 2023; Chen et al., 2024) we evaluate using reference-based overlap and similarity metrics, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2019). BLEU and ROUGE compare the overlap between the counter-argument and the claim, whereas METEOR also considers synonyms, paraphrases, and word stems. Finally, BERTScore takes into account the semantic context and meaning of the text, going beyond surface-level word matching.

Ranking. Based on the evaluation scores per dimension, we obtain rankings of each of the 5 candidate counter-arguments (4 automatically generated and the gold reference) by summing the scores for each counter-argument, as provided by the human and LLM-as-a-Judge evaluators. Formally, let the evaluation process involve \( n \) counter-arguments, \( m \) dimensions, and \( e \) evaluators. Let \( Score_{i,j,d} \) represent the score assigned by evaluator \( j \) for counter-argument \( i \) on dimension \( d \).

The total score for counter-argument \( i \), summed across all dimensions for evaluator \( j \), is given by:

\[
T_{i,j,d} = \sum_d Score_{i,j,d}
\]

The final ranking for counter-argument \( i \), combining scores across all \( m \) dimensions for each evaluator \( j \), is calculated as:

\[
R_{i,j} = Rank(\sum_{d=1}^m T_{i,j,d})
\]

Where:

• \( T_{i,j,d} \) is the total aggregated score for counter-argument \( i \) for dimension \( d \) by evaluator \( j \).
• \( m \) is the number of dimensions, 5 in our case.
• \( R_{i,j} \) is the rank of the counter-argument \( i \) calculated by summing \( T_{i,j,d} \) from evaluator \( j \).

The counter-arguments are ranked in ascending order according to their calculated \( O_{i,j} \) values, where lower scores correspond to superior performance. This ordinal ranking methodology effectively normalizes individual scoring variations and minimizes evaluator bias, as it accounts for potential systematic differences in scoring tendencies among evaluators who might consistently assign either higher or lower scores, thereby ensuring a fair comparison.

The evaluation using reference-based metrics involves calculating scores for counter-arguments generated by each LLM in comparison to the gold reference, which enables the establishment of relative rankings among the LLMs’ outputs.

5 Results

We first calculate the correlation of the automatic metrics with human judgments. We then used the best automatic metrics to compute the overall rankings for the gold counter-arguments and those generated by the four models. Finally, we analyzed dimension-wise rankings obtained in the manual evaluation.

Correlation with Human Judgments. We would like to stress that manual counter-argument evaluation is a highly subjective and tedious process (Wachsmuth et al., 2017; Hua et al., 2019; Chen et al., 2024). Thus, ensuring that our automatic evaluation method correlates with human judgments is crucial. Therefore, we computed the correlation between every evaluation metric on the evaluator sample set of 75 examples, including human evaluation, using the method described in Section 4. The Spearman’s rank correlation coefficients (\( \rho \)) are reported in Figure 2. First, it can be observed that the three LLM-as-a-Judge methods obtain the highest correlation with human judgments (row marked in red). Thus, while JudgeLM and Prometheus obtain a strong correlation, Claude 3.5 Sonnet is the best method with a \( \rho \) of 0.82 (very strong correlation).

Reference-based metrics show a poor correlation with both human judgments and LLM-as-a-Judge evaluation methods. This suggests that reference-based metrics may be inadequate for evaluating the quality of counter-argument generation, as they fail to capture the essential dimensions established for human evaluation and disregard the context of the original argument or claim being countered.

Among the LLM-as-a-Judge methods, the lowest correlation was observed between JudgeLM and Prometheus. Upon a more detailed analysis,
Figure 2: Heatmap showing Spearman’s rank correlation coefficients between human evaluation and automatic metrics, including LLM-as-a-Judge metrics. The row marked in red represents the correlation of all the evaluation metrics to human preference.

we found that these two judges had the highest level of disagreement in evaluating Opposition and Persuasiveness. Prometheus tends to be stricter on Opposition, while JudgeLM does the same on Persuasiveness. Thus, each model assigned lower scores than the other for those particular dimensions.

Overall rank evaluation. Figure 3 reports the overall rankings obtained by summing all the scores from each dimension that each evaluator provided (as described in Section 4). The first observation is that 2 out of 3 LLM-as-a-Judge agree that Command R+ with external knowledge generates the best counter-arguments. Moreover, according to the manual evaluation, both Command R+ and Mistral-7B-Instruct with external knowledge are ranked at the top. Interestingly, results with respect to the ranking of other models vary, but all four evaluators agree that the human-generated counter-arguments (gold reference) are ranked worst.

Table 2 shows the scores obtained by the reference-based metrics. While Command R+ with external knowledge remains the best model, it is easy to see that the rankings are not aligned with respect to human preferences. Furthermore, although not directly comparable, the obtained scores are in the range of previous work evaluating counter-argument generation with these metrics (Chen et al., 2024).

Taking into account the high correlation of LLM-as-a-Judge methods with human preferences, we also evaluated the four models over the full corpus of 150 arguments and counter-arguments pairs. The results from each LLM-as-a-Judge evaluator can be found in the Appendix E (Figures 5, 6, 7, 8).

Point-wise evaluation. Figure 4 illustrates the average dimension-wise rankings from manual evaluation. We can see that Command R+ excels in Opposition compared to others, whereas both Command R+ and Mistral-7B-Instruct with external knowledge are valued the best on Persuasiveness and Relatedness. Moreover, Mistral-7B-Instruct with external knowledge and Command R+ obtain the highest Factuality scores. Overall, we can conclude that when external evidence is provided to the LLMs, the overall quality of the counter-argument improves. Nevertheless, the performance of generating counter-arguments based on parametric knowledge is quite high, especially for the larger models (Command R+).

6 Analysis of Results

To answer RQ3, we checked whether the generated counter-arguments indeed used the provided external knowledge or whether the retrieved evidence is already incorporated in the parametric knowledge of LLMs.

We used the sentence transformer model gte-base-en-v1.5’ (Li et al., 2023; Zhang et al., 2024) to calculate sentence-level cosine similarity between the generated outputs and provided external evidence. Our methodology involved segmenting both the external information and generated counter-arguments into individual sentences for pairwise comparison, followed by ranking the similarity scores in descending order. Based on manual verification, we established a threshold of 70% similarity as indicative of successful external knowledge integration. Our analysis revealed that Command R+ with external knowledge effectively utilized external evidence in 82% of its generated counter-arguments, while Mistral-7B-Instruct with external knowledge demonstrated such integration in 51% of the cases.

Our manual analysis of similarity scores revealed that, when similarity scores exceed 70%, models directly integrate provided external knowledge, while scores between 65-70% result in partial
Figure 3: Human and LLM-as-a-Judge evaluation results.

<table>
  <tr>
    <th>Model</th>
    <th>BLEU</th>
    <th>ROUGE</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Command R+</td>
    <td>20.35</td>
    <td>18.36</td>
    <td>16.12</td>
    <td><b>86.38</b></td>
    <td>35.30</td>
  </tr>
  <tr>
    <td>Command R+ with external knowledge</td>
    <td><b>20.80</b></td>
    <td><b>18.67</b></td>
    <td><b>16.81</b></td>
    <td>86.15</td>
    <td><u>35.60</u></td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct</td>
    <td>17.36</td>
    <td>15.93</td>
    <td>13.96</td>
    <td>86.23</td>
    <td>33.37</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct with external knowledge</td>
    <td>17.30</td>
    <td>16.58</td>
    <td>14.36</td>
    <td>86.29</td>
    <td>33.63</td>
  </tr>
</table>

Table 2: Results with reference-based metrics; best overall model per metric in bold; best model per family underlined.

Figure 4: Per dimension ranking from manual evaluation.

incorporation mixed with additional information. Notably, this partial incorporation behavior predominantly occurs with sensitive topics such as personal reputation, religion, ethics, economics, or politics, where models tend to generalize rather than use specific factual evidence, leading to lower Opposition dimension scores. Command R+ with external knowledge frequently exhibited this behavior, though interestingly, the same model without external knowledge produced more generalized responses that evaluators often found more plausible than fact-based counterarguments.

The topics of high controversy trigger LLMs to start generation by acknowledging the stance of the input argument, which exhausts the space for including the retrieved evidence, even when explicitly prompted not to do so. Moreover, such counter-arguments were evaluated better than the output that starts the alternative perspective straight from the factual examples.

7 Conclusion

In this study, we examined the effect of incorporating externally dynamically retrieved evidence into LLMs in counter-argument generation using web search as an external knowledge source. Our results on a newly created gold dataset show that, while LLMs with external knowledge improved their counter-argument generation, their reliance on it varies, and in some cases, they generate responses with parametric knowledge that obtained better scores.

We propose LLM-as-a-Judge to automatically
evaluate the quality of counter-arguments with better correlation scores with respect to human judgments than previously used reference-based metrics.

Through qualitative analysis, we found that model behaviour shifts when dealing with sensitive or controversial topics. In these cases, LLMs tend to provide more generalized responses rather than directly integrating factual evidence. Interestingly, such responses were often rated more favourably, suggesting a preference for plausibility and coherence over strict factual accuracy.

Our findings highlight the complexities of integrating external knowledge into LLM outputs. While retrieval-augmented generation (RAG) can enhance factual consistency, models may still prioritize linguistic fluency and alignment with social norms. Future work should focus on refining strategies to ensure that external knowledge is utilized effectively, particularly in contexts that require precise and evidence-based argumentation.

8 Limitations

Our study has some limitations. We focused on two LLMs, both with and without integrated external knowledge, to compute the agreement between human and LLM judgments. Including more models would have significantly strengthened our conclusions. Equally, including other languages should allow for more generalizable results. However, as far as we know, there are no counter-argument generation datasets for languages beyond English. Therefore, one of the short-term objectives of the NLP research community should be to address this glaring gap.

Additionally, due to the effort required for manual assessment, we evaluated only a sample of the generated counter-arguments rather than the entire dataset. Future work could explore more scalable evaluation methods to extend the analysis.

Finally, the LLM-generated counter-arguments may be affected by potential data contamination, where topics and examples of the arguments used in our experiments may overlap with the training data of the LLMs we used. Investigating task contamination is far from trivial, but it should be included in any future work.

Acknowledgments

We are grateful for the provided API credits by the Cohere For AI Research Grant Program6.

We would also like to acknowledge the funding received by the following MCIN/AEI/10.13039/501100011033 projects: (i) DeepKnowledge (PID2021-127777OB-C21) and ERDF A way of making Europe; (ii) DeepMinor (CNS2023-144375) and European Union NextGenerationEU/PRTR; (iii) Disargue (TED2021-130810B-C21) and European Union NextGenerationEU/ PRTR; (iv) ED-HIA (PID2022-136522OB-C22); (v) LOTU (TED2021-130398B-C22) and European Union NextGenerationEU/ PRTR. Anar Yeginbergen’s PhD contract is part of the PRE2022-105620 grant, financed by MCIN/AEI/10.13039/501100011033 and by the FSE+.

References

Milad Alshomary, Shahbaz Syed, Arkajit Dhar, Martin Potthast, and Henning Wachsmuth. 2021. Counter-argument generation by attacking weak premises. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1816–1827.

Milad Alshomary and Henning Wachsmuth. 2023. Conclusion-based Counter-Argument Generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 957–967, Dubrovnik, Croatia. Association for Computational Linguistics.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Preprint, arXiv:2204.05862.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.

Elena Cabrio and Serena Villata. 2018. Five Years of Argument Mining: a Data-driven Analysis. In IJCAI, volume 18, pages 5427–5433.

6https://cohere.com/research/grants
Carlos Carrasco-Ferre. 2024. Large language models are as persuasive as humans, but how? about the cognitive effort and moral-emotional language of llm arguments. arXiv preprint arXiv:2404.09329.

Guizhen Chen, Liying Cheng, Anh Tuan Luu, and Liding Bing. 2024. Exploring the Potential of Large Language Models in Computational Argumentation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2309–2330.

Coherer For AI. 2024. c4ai-command-r-plus-08-2024.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.

Steffen Eger, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2018. Cross-lingual Argumentation Mining: Machine Translation (and a bit of Projection) is All You Need! In Proceedings of the 27th International Conference on Computational Linguistics, pages 831–844.

Iker García-Ferrero, Rodrigo Agerri, Aitziber Atutxa Salazar, Elena Cabrio, Iker de la Iglesia, Alberto Lavelli, Bernardo Magnini, Benjamin Molinet, Johana Ramirez-Romero, German Rigau, Jose Maria Villa-Gonzalez, Serena Villata, and Andrea Zaninello. 2024. MedMT5: An Open-Source Multilingual Text-to-Text LLM for the Medical Domain. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11165–11177.

Martin Hinton and Jean HM Wagemans. 2023. How persuasive is ai-generated argumentation? an analysis of the quality of an argumentative text produced by the gpt-3 ai text generator. Argument & Computation, 14(1):59–74.

Zhe Hu, Hou Pong Chan, and Yu Yin. 2023. Americano: Argument generation with discourse-driven decomposition and agent interaction. arXiv preprint arXiv:2310.20352.

Xinyu Hua, Zhe Hu, and Lu Wang. 2019. Argument Generation with Retrieval, Planning, and Realization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661–2672, Florence, Italy. Association for Computational Linguistics.

Xinyu Hua and Lu Wang. 2018. Neural Argument Generation Augmented with Externally Retrieved Evidence. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 219–230, Melbourne, Australia. Association for Computational Linguistics.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825.

Yohan Jo, Seojin Bang, Emaad Manzoor, Eduard Hovy, and Chris Reed. 2020. Detecting Attackable Sentences in Arguments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1–23, Online. Association for Computational Linguistics.

Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv preprint arXiv:1909.05858.

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. In Forty-first International Conference on Machine Learning.

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. arXiv preprint arXiv:2405.01535.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards General Text Embeddings with Multi-stage Contrastive Learning. Preprint, arXiv:2308.03281.

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Jiayu Lin, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, and Zhongyu Wei. 2023. Argue with Me Tensely: Towards Sentence-Level Counter-Argument Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16705–16720, Singapore. Association for Computational Linguistics.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Allen Roush and Arvind Balaji. 2020. DebateSum: A large-scale argument mining and summarization dataset. Proceedings of the 7th Workshop on Argument Mining, pages 1–7.

Sougata Saha and Rohini Srihari. 2023. ArgU: A Controllable Factual Argument Generator. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8373–8388, Toronto, Canada. Association for Computational Linguistics.

Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2021. Aspect-Controlled Neural Argument Generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 380–396, Online. Association for Computational Linguistics.

Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauchmann, Steffen Eger, and Iryna Gurevych. 2018. ArgumenText: Searching for Arguments in Heterogeneous Sources. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 21–25, New Orleans, Louisiana. Association for Computational Linguistics.

Christian Stab and Iryna Gurevych. 2017. Parsing Argumentation Structures in Persuasive Essays. Computational Linguistics, 43(3):619–659.

Ekaterina Sviridova, Anar Yeginbergen, Ainara Estarrona, Elena Cabrio, Serena Villata, and Rodrigo Agerri. 2024. CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18463–18475.

Shahbaz Syed, Khalid Al Khatib, Milad Alshomary, Henning Wachsmuth, and Martin Potthast. 2021. Generating Informative Conclusions for Argumentative Texts. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3482–3493.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. Preprint, arXiv:2302.13971.

Preetika Verma, Kokil Jaidka, and Svetlana Churina. 2024. Auditing counterfire: Evaluating advanced counterargument generation with evidence and style. arXiv preprint arXiv:2402.08498.

Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein. 2017. Argumentation Quality Assessment: Theory vs. Practice. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 250–255, Vancouver, Canada. Association for Computational Linguistics.

Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the Best Counterargument without Prior Topic Knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241–251, Melbourne, Australia. Association for Computational Linguistics.

Lu Wang, Nick Beauchamp, Sarah Shugars, and Kechen Qin. 2017. Winning on the Merits: The Joint Effects of Content and Style on Debate Outcomes. Transactions of the Association for Computational Linguistics, 5:219–232.

Anar Yeginbergen, Maite Oronoz, and Rodrigo Agerri. 2024. Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11687–11699.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. arXiv preprint arXiv:1904.09675.

Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. Preprint, arXiv:2407.19669.

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. arXiv preprint arXiv:2310.17631.
A  Example of data

Table 3 shows the sequential methodology employed in generating the corpus. Original presents an argument from the publicly available CANDELA corpus that originally is in lowercase and segmented in sentences and tokens. After its reconstruction to a human-readable format, in the Intermediate step, the argument is summarized by means of the language model Llama-3.1-70B instruct. The row Final presents the version after manual elaboration which is made available in this paper.

<table>
  <tr>
    <th>Original</th>
    <td>"we", "should", "n't", "worry", "about", "being", "compassionate", "to", "mexican", "illegal", "immigrants", "the", "same", "way", "we", "do", "n't", "worry", "about", "being", "uncompassionate", "to", "the", "rest", "of", "the", "world", "s", "poor", "i", "am", "specifically", "referring", "to", "poor", "immigrants", "who", "based", "on", "current", "tax", "codes", "will", "take", "far", "more", "in", "benefits", "than", "they", "would", "pay", "in", "taxes", "it", "has", "nothing", "to", "do", "with", "skin", "color", "...", "if", "you", "have", "millions", "of", "white", "people", "suddenly", "all", "working", "manual", "labor", "jobs", "and", "below", "you", "now", "have", "a", "lot", "of", "people", "not", "paying", "many", "taxes", "into", "the", "system", "and", "being", "eligible", "to", "take", "a", "lot", "out", "why", "do", "people", "argue", "we", "need", "to", "be", "compassionate", "when", "with", "that", "same", "logic", "you", "could", "argue", "we", "are", "n't", "being", "compassionate", "for", "not", "all", "living", "a", "minimalist", "life", "and", "sending", "all", "our", "wealth", "to", "africa", "until", "there", "are", "no", "more", "starving", "people", "what", "makes", "mexico", "so", "deserving", "of", "our", "aid", "but", "not", "other", "countries", "logically", "i", "m", "sure", "the", "people", "clamoring", "to", "he", "compassionate", "and", "let", "all", "the", "poor", "immigrants", "in", "i.e.", "making", "the", "immigration", "process", "easier", "or", "amnesty", "realize", "we", "could", "n't", "support", "the", "entire", "world", "so", "why", "is", "mexico", "special", "what", "do", "you", "consider", "our", "breaking", "point", "for", "percentage", "of", "us", "poor", "i", "think", "we", "re", "already", "at", "it", "to", "where", "there", "s", "more", "money", "going", "out", "to", "social", "programs", "and", "tax", "breaks", "than", "there", "is", "coming", "in", "?", "at", "what", "point", "does", "the", "super", "pro", "immigration", "person", "decide", "the", "us", "is", "full", "?"</td>
  </tr>
  <tr>
    <th>Intermediate</th>
    <td>The writer argues that showing compassion to Mexican illegal immigrants is not justified when considering the financial burden they would place on the system, as they would take more in benefits than they would pay in taxes. The sustainability of immigration policies is based on economic impact rather than emotions. Uncertainty about the threshold at which the U.S. would be considered "full" and unable to support more immigrants</td>
  </tr>
  <tr>
    <th>Final (Ours)</th>
    <td>Showing compassion to Mexican illegal immigrants is not justified when considering the financial burden they would place on the system, as they would take more in benefits than they would pay in taxes. This argument is not based on skin color, but rather on the economic impact of a large influx of low-income workers.</td>
  </tr>
</table>

Table 3: Example of the data. Original is the original publicly available data. Intermediate is the summary generated by Llama-3.1-70B. Final is the final version after manual refinement and the data used in the experiments.

B  Examples of arguments excluded due to subjectivity, irrelevance, or lack of verifiability

• if a logical outcome of an action or ideology is undesirable, people will try to find a way to avoid that outcome while still pursuing the action or ideology. They argue that people and societies are generally smart and reasonable enough to know when to stop before reaching an undesirable conclusion.

• if Napoleon had won at Waterloo and completed his conquest of Europe, the continent would have flourished economically, socially, and politically.

• Santa remains a beloved and benign figure in popular culture.

• Eating meat is equivalent to murder, and therefore, meat-eaters feel hypocritical calling out others for immoral behavior. It is not fair to judge others for their moral transgressions, such as racism or rape, while they themselves contribute to animal suffering.

• One should take ownership of their own problems and not place the burden on others to fix them.
C  Snippet of the final version of the dataset

An example of the final version of the data in Table 4.

<table>
  <tr>
    <th>Argument</th>
    <th>Counter-Argument</th>
  </tr>
  <tr>
    <td>[Increasing privilege for everyone is not possible without those who are currently benefiting losing some of their advantages], and that [people only pay lip service to values like equality and fairness without being willing to make real sacrifices.] [When faced with the choice, most people will prioritize protecting their own privileged position rather than fighting for equality.]</td>
    <td>[Relinquishing some privilege can be in one’s self-interest, as it can prevent violence and resentment from those who feel underprivileged.] [Affirmative action and recognizing privilege can lead to a more equal society, where no one has advantages based on skin color, sexuality, or gender.] [A society that promotes universal principles and equality can benefit everyone, including those in positions of power.]</td>
  </tr>
  <tr>
    <td>[Small countries in Europe should unite as a single nation, like the US], and [eventually all small countries should become large, united territories.]</td>
    <td>[The US is a union of states with its own laws and cultures], but [it has a high homicide rate compared to the EU, partly due to clashes between different ethnic and socioeconomic groups.] [Closer economic union between European countries may have advantages and disadvantages], but [a political union may not be beneficial as most countries feel they can govern themselves better than a central European government]. [A separate union for smaller countries could be an option, providing financial leverage and allowing them to maintain their social programs and sovereignty.]</td>
  </tr>
  <tr>
    <td>[Food past recommended daily values should be heavily taxed], as [overeating is a major problem that leads to health issues and costs money in healthcare.] [I propose a system where individuals would be encouraged to eat the proper amount of food, as calculated by a doctor’s visit], and [those who choose to eat excessively would be taxed to support the system and fund welfare programs.]</td>
    <td>[The current food distribution system is flawed because it’s based on wealth, favouring the rich], and [a proposed system to tax food based on consumption would not solve the issue of global hunger], which [is caused by factors like war and poor distribution.] [The world produces enough food to feed everyone], but [over 50% of it is wasted], and [the problem of obesity is not solely caused by food consumption.] Instead of taxing food, [it would be more effective to work on improving the distribution system] and [addressing the root causes of global hunger.]</td>
  </tr>
  <tr>
    <td>[Capitalism is the best economic option], as [it is a free market regulated by the state] that [allows individuals to become self-made millionaires.] [A just and non-corrupt state is necessary for a successful capitalist environment. Communism, on the other hand, interferes with individual freewill and kills incentives], while [a laissez-faire system is unattainable due to human nature.]</td>
    <td>[Government intervention interferes with the free market], which [is contrary to pure capitalism.] [Historically, capitalism has been the best option to incentivize the production of physical goods in a world with abundant natural resources.] However, [in a future where physical work is performed by robots and software only needs to be written once], [capitalism may not be the best option.]</td>
  </tr>
</table>

Table 4: Final argument and counter-argument pair that was obtained as described in Section 3. Text in Blue highlights the claim, evidence in Orange, and examples are in Red. Criteria to identify claims and premises based on Sviridova et al. (2024).
D  Prompts used for query generation, web search, generation with and without external knowledge

Table 5 illustrates the prompts used for each setting of the generation step.

<table>
  <tr>
    <th>Setting</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>Query Generation</td>
    <td>Generate a list of 5 queries for web-search that would help to find information to question the veracity of the given claim and persuade to take the opposing position: {claim}?<br>Provide only questions and nothing else. The answer should be in JSON.</td>
  </tr>
  <tr>
    <td>Web-search</td>
    <td>Answer the question from the following text: {question}?<br>1. Find factual information from different media outlets<br>2. Provide the evidence in a bullet point manner.<br>3. Do not output anything else.</td>
  </tr>
  <tr>
    <td>Generation with external knowledge</td>
    <td>Given the following context: {qacontext}, provide a succinct counter-argument that refutes the following argument using information from the context: {claim}.<br>Provide only the answer and nothing else.<br>Make sure the answer is no longer than 3 sentences.</td>
  </tr>
  <tr>
    <td>Generation without external knowledge</td>
    <td>Generate a succinct counter-argument that refutes the following claim: {claim}?<br>Provide only the answer and nothing else.<br>Make sure the answer is no longer than 3 sentences.</td>
  </tr>
</table>

Table 5: List of prompts used at every stage of the generation. {claim} is a placeholder for the input claim. {question} is a placeholder for the question. {qacontext} is a placeholder for the retrieved context.

E  Per dimension evaluation ranks

In this appendix, we provide evaluation ranks per opposition, factuality, relatedness, specificity, and persuasiveness dimensions for the following LLM-as-a-Judge models: JudgeLM in Figure 5, Prometheus in Figure 6, and Claude in Figure 7. The overall ranks over the whole dataset by LLM-as-a-Judge are shown in Figure 8.

Results, especially those computed using Claude 3.5 Sonnet, align with those obtained in Section 5 with a human-annotated sample of the data. Thus, 8 shows that Command R+ with external knowledge is also ranked first, gold reference, and Mistral-7B-Instruct the worst, while the ranks for Command R+ and Mistral-7B-Instruct with external knowledge vary according to the judge.

![Bar chart showing JudgeLM per dimension ranks for Opposition, Factuality, Relatedness, Specificity, and Persuasiveness across Gold, CmdR+, CmdR+ with external knowledge, Mistral, and Mistral with external knowledge](page_495_1042_495_246.png)

Figure 5: JudgeLM per dimension ranks.
Figure 6: Prometheus per dimension ranks.

Figure 7: Claude per dimension ranks.
![Bar chart comparing LLM-as-a-Judge evaluation results for JudgeLM, Prometheus, and Claude across different models: Gold, CmdR+, CmdR+ with external knowledge, Mistral, Mistral with external knowledge.](page_324_186_800_377.png)

Figure 8: LLM-as-a-Judge evaluation results over the full dataset.
F  Instruction example for human evaluation.

An example of the instructions provided to the human evaluators in the Prolific platform is shown in Figure 9. Firstly, the argument is presented accompanied by a description of the five dimensions (opposition, relatedness, factuality, specificity, persuasiveness) that are evaluated in the paper. Secondly, each of the counter-arguments is presented and the evaluator has to select a value between 1 for unsatisfactory, 2 for moderately satisfactory, and 3 for highly satisfactory in each of the dimensions.

Argument: All high schools should have a mandatory law class that teaches students about laws, their rights, and punishments. This class would serve three purposes:
Informing students of laws to avoid unintentional violations, making them aware of their rights to prevent police abuse, and deterring crime by educating them on punishments.
The budget cuts should be made nationwide to implement this program.

The characteristics to evaluate:
Opposition: measures how much the counter-argument opposes and contradicts the argument example through the expression of an opposing sentiment regardless of the argument’s effectiveness or persuasiveness.
Relatedness: measures the association between the counter-argument and argument example based on contextual or semantic similarity
Factuality: measures how informative, backed with examples, persuasive, or factual a response is.
Specificity: measures how much the counter-argument presents focused and specific arguments that effectively counter the key ideas within the claim example through the use of in-depth arguments, nuanced reasoning, and supporting evidence.
Persuasiveness: measures how much the counter-argument is effective, persuasive or trustworthy with respect to the argument.

There are over 3,000 federal laws, making it difficult to teach enough of the law in a single year * or semester to inform someone of their rights. The complexity of the law, including criminal procedure, requires years of study for law students to grasp. A “know your rights” discussion is valuable, but the philosophy of “cooperate, say nothing, get a lawyer ASAP” does not take a semester or year to teach.

<table>
  <tr>
    <th></th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
  </tr>
  <tr>
    <td>Opposition</td>
    <td>○</td>
    <td>○</td>
    <td>○</td>
  </tr>
  <tr>
    <td>Relatedness</td>
    <td>○</td>
    <td>○</td>
    <td>○</td>
  </tr>
  <tr>
    <td>Factuality</td>
    <td>○</td>
    <td>○</td>
    <td>○</td>
  </tr>
  <tr>
    <td>Specificity</td>
    <td>○</td>
    <td>○</td>
    <td>○</td>
  </tr>
  <tr>
    <td>Persuasiveness</td>
    <td>○</td>
    <td>○</td>
    <td>○</td>
  </tr>
</table>

Figure 9: An example of instructions provided to the human evaluators.

Reviewer paper summaries: This paper proposes integrating external knowledge by generating queries and retrieving factual evidence from the web to enhance the generation of counter-arguments. The effectiveness of this approach is demonstrated through both LLM-based and human evaluations.

This paper explores enhancing counter-argument generation in Large Language Models (LLMs) by integrating dynamically retrieved external knowledge from the web. The authors introduce a new dataset of 150 manually curated argument/counter-argument pairs, designed to balance argumentative complexity and evaluative feasibility, and propose an LLM-as-a-Judge evaluation method that outperforms traditional metrics (e.g., BLEU, METEOR) in correlating with human judgments (Claude 3.5 achieves a Spearman’s ρ=0.82). Experiments with Command R+ and Mistral-7B-Instruct, with and without external knowledge via Cohere’s API, suggest that dynamic retrieval improves counter-arguments’ relatedness, persuasiveness, and factuality. The authors also provide qualitative insights, noting LLMs’ tendency to prioritize fluency over factual accuracy on sensitive topics. Data and code are publicly available, enhancing reproducibility.
This paper presents an interesting idea—dynamic knowledge integration for counter-argument generation and offers a promising LLM-as-a-Judge evaluation method. However some limitations (stated below categorically), also some already identified in the paper weaken its claims.

This paper introduces a counter-argument generation model for debate analysis systems based on Large Language Models. The proposed approach utilises real-time external knowledge to improve the quality of the generated counter-arguments, and introduces a new dataset (CANDELA) for the evaluation of the proposed models. Traditional evaluation metrics for natural language generation, as well as an innovative LLM-as-a-judge method, are proposed for the evaluation of the model, which appears to outperform existing baseline methods.
Summary of strengths: - The idea is clear and well motivated.
- The authors integrated both human evaluation and llm-based evaluation to support their claims.

- The idea of dynamic, real-time web retrieval for counter-argument generation is novel and aims to address a gap in static knowledge integration methods.
- The LLM-as-a-Judge approach shows promise as a scalable evaluation tool, with Claude 3.5 achieving a strong correlation (0.82) with human judgments.
- The curated dataset of concise, three-sentence counter-arguments addresses prior issues with overly verbose or truncated outputs.
- The paper reveals practical insights, such as LLMs prioritising fluency over factual accuracy in sensitive topics like politics, which is valuable for RAG research.
- A comprehensive evaluation combining human judgments, LLM-as-a-Judge, and traditional metrics (BLEU, BERTScore) adds depth to the analysis.

The paper proposes a new dataset (CANDELA) which has the potential to be a useful resource for the Argument Mining community due to its optimal turn length for the arguments and counter-arguments and the apparent quality of the dataset. The results obtained provide interesting evidence on the feasibility of using external knowledge to increase the quality of generative models for argument generation based on LLMs, which might encourage further studies in this direction. Finally, the LLM-as-a-judge approach appears to be effective and highly correlates with human judgement, which may provide a useful alternative to inadequate automated metrics such as BERTScore or ROUGE.
Summary of weaknesses: - The overall contribution of this paper is limited, as it primarily applies a Retrieval-Augmented Generation (RAG) style system to an existing problem and arrives at the well-established conclusion that external knowledge beyond parametric LLM data can be beneficial. While expanding the application of RAG to new domains is still valuable, the paper lacks a clear demonstration of what makes counter-argument generation unique or how external knowledge can be leveraged differently in this context.  

- I disagree with the paper’s stance that philosophical questions do not require external knowledge. On the contrary, I believe this could have been a unique angle to explore how external knowledge influences responses to subjective or philosophical questions. Demonstrating this impact could have strengthened the novelty of the work.  The safety guardrails being triggered is more likely due to the helpfulness/safety tradeoff that arises  mainly in smaller LLMs. 

- The retrieval of factual evidence is not supported by a factuality analysis, leaving unanswered questions about the potential effects of hallucinated or incorrect evidence on the generated arguments. Addressing the reliability of retrieved content would significantly enhance the paper’s credibility.  

- The paper lacks sufficient examples (aside from the first figure) to clearly demonstrate when external knowledge helps or fails to improve counter-argument generation. Including more illustrative cases would provide deeper insights into the strengths and limitations of the proposed approach.  

- I am not convinced by the use of the ranking metric based on aggregated Likert scores, particularly given the inherently subjective nature of the task. Pairwise comparisons, although more computationally expensive, are likely to be more effective and reliable for evaluating argument quality.  
Example in evaluating persuasiveness with LLMS
https://aclanthology.org/2024.findings-emnlp.836.pdf

- The human evaluation section lacks critical information on reliability scores and annotator training. Reporting these details is essential to establish the consistency and validity of human judgments.

- The dataset is too small (150 pairs) and lacks diversity due to manual filtering for “scientific” topics, limiting generalisability.
- Using Llama 3.1-70B for summarisation risks introducing verbosity or safety biases, yet no ablation studies assess its impact.
- The 70% similarity threshold for external knowledge integration is arbitrary and lacks empirical validation.
- Reliance on Cohere’s API for retrieval creates vendor lock-in and potential search algorithm biases; alternative methods (e.g., hybrid search) are unexplored.
- The gold reference counter-arguments is ranked worst in human evaluation, potentially a red flag.
- The evaluation lacks comparisons to RAG baselines or fine-tuned models, leaving the role of retrieval itself unclear.
- Biases in LLM-as-a-Judge or factual inaccuracies in retrieved evidence could potentially be ethical concerns.

The human evaluation study was done on a very small subset of test samples (only 75), which may not be large enough to provide statistically significant results. Some aspects of the dataset (namely the summarization of arguments and counter-arguments and the reconstruction of natural language discussions) was done with AI models, which slightly reduces the soundness of the approach. While the authors claim that the improvement obtained with the aid of external knowledge provide a significant improvement, no statistical significance study is provided, which weakens the claims made by the authors. In the introduction, the authors acknowledge the inadequacy of lexical similarity and semantic similarity metrics (such as ROUGE, BLEU and BERTScore) to evaluate the quality of counter-arguments, but then use these metrics as the main source of comparison between their approach and other approaches in Table 2. This appears counter-intuitive, as the authors don't appear to have a clear stance on the use of these metrics: if the authors believe that they are useful and/or conclusive, they should justify this claim. If, on the other hand, they insist on their claim that they are inadequate for the task (which I agree with), they should possibly refrain from using them, or at the very least acknolwdge their limitations in the caption of Table 2. Incidentally, their results highlighted in Figure 2 provides further evidence of the inadequacy of these metrics, given how little they correlate with human judgement.
Comments, suggestions and typos: - Missing related work in counter argument generation
https://aclanthology.org/2024.inlg-main.8.pdf

- Missing all prompts used and experimental setting (e.g decoding parameters)

- I understand the motivation behind canedella's reconstruction, but I am not sure if you tried to generate counter arguments directly based on the original data and use simpler aggregation of the text. Was the text too noisy?

Could help to:
- Increase to 300-500 pairs, maybe increase topics (e.g., include subjective domains), and report IAA metrics to validate curation.
- Analyse why gold references underperformed (e.g., style mismatch, annotation errors) and refine guidelines or source data accordingly.
- Test multiple retrieval methods (e.g., Bing API, Common Crawl) and quantify biases in Cohere’s results (e.g., source distribution, recency).
- Conduct sensitivity analyses for the 70% similarity threshold and report integration rates across a range (e.g., 60%-80%).
- Compare to static RAG (e.g., Wikipedia-based) and fine-tuned LLMs to disentangle retrieval’s role.
- Assess Llama-3.1-70B’s summarization impact versus human-only curation.

-Page 1 lines 045-050: this paragraph introduces two lines of research within the field of argumentation without specifying which one this work belongs to (presumably, the latter). I would advise the authors to explicitly mention this

-Page 2, lines 083-089: this paragraph should be appropriately backed with a reference in the literature, or removed

-Section 2: related work
The authors might want to elaborate on the human evaluation study that Syed et al. 2021 did and how it is related to their study, since they cite the work. 

-Abstract, and line 134: The authors refer to their dataset as a "manually curated" dataset; however, this is a disputable claim, sine the data has been re-constructed and summarised with LLMs, with humans merely supervising the task. Maybe the authors should make this explicit to avoid any ambiguity in their abstract and introduction that might confuse the reader
Overall assessment scores: 2.5

3

3.5