Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation (2024)

Abstract: With the recent advances of large language models (LLMs), it is no longer infeasible to build an automated debate system that helps people to synthesise persuasive arguments. Previous work attempted this task by integrating multiple components. In our work, we introduce an argument mining dataset that captures the end-to-end process of preparing an argumentative essay for a debate, which covers the tasks of claim and evidence identification (Task 1 ED), evidence convincingness ranking (Task 2 ECR), argumentative essay summarisation and human preference ranking (Task 3 ASR) and metric learning for automated evaluation of resulting essays, based on human feedback along argument quality dimensions (Task 4 SQE). Our dataset contains 14k examples of claims that are fully annotated with the various properties supporting the aforementioned tasks. We evaluate multiple generative baselines for each of these tasks, including representative LLMs. We find, that while they show promising results on individual tasks in our benchmark, their end-to-end performance on all four tasks in succession deteriorates significantly, both in automated measures as well as in human-centred evaluation. This challenge presented by our proposed dataset motivates future research on end-to-end argument mining and summarisation. The repository of this project is available at https://github.com/ HarrywillDr/ArgSum-Datatset

heading: 1 Introduction
text: Crafting arguments for a debate requires a wealth of knowledge, meticulous logical thinking and a high level of command of language (Field, 2017). Recent trends treat this task as the automatic summarisation of a large body of debate-related material to alleviate the effort and time spent on brainstorming by stakeholders (Roush and Balaji, 2020). Most
*Corresponding email: hao.li-2@manchester.ac.uk
of previous work split these processes into multicomponent systems, including context-dependent claim detection (CDCD) (Daxenberger et al., 2017; Reimers et al., 2019; Cheng et al., 2021), claim stance classification (CSC) (Bar-Haim et al., 2017; Trautmann et al., 2020; Toledo-Ronen et al., 2020; Rajula et al., 2022), context-dependent evidence detection (CDED) (Iskender et al., 2021; Ein-Dor et al., 2020; Shnarch et al., 2018; Aharoni et al., 2014) and argument summarisation (AS) (BarHaim et al., 2020a,b; Yamada et al., 2019; Misra et al., 2016). However, thus far, these tasks have been approached in isolation without formulating them into an end-to-end process. This contradicts the natural sequence of actions in the debate preparation process (Rear, 2017), where outputs of previous tasks (e.g. finding relevant evidence) form the basis for the subsequent tasks (e.g. arranging claims and evidence in an argumentative essay).
The only comprehensive automated debating work at present is IBM’s Project Debater (Slonim et al., 2021). However, it is composed of multiple independent components, potentially giving rise to cascading errors, whereby errors made in any of the preceding components might be propagated and amplified in subsequent components (Kleinberg et al., 2007). Currently, there is no dataset supporting the development and evaluation of end-to-end systems. Furthermore, Project Debater lacks evaluative components; its argument quality component only considers how relevant potential candidate texts are to the debate topic.
Earlier works explored the assessment of the argument quality in various dimensions, like convincingness of arguments and evidence (Wambsganss et al., 2020; Gleize et al., 2019; Potash et al., 2019; Habernal and Gurevych, 2016a; Cyra and Górski, 2011) and sufficiency checking (Gurcke et al., 2021; Wachsmuth and Werner, 2020a; Potash et al., 2019; Stab and Gurevych, 2017; Stab, 2017). Recent advances in large-scale language
133

heading: Topic
text: models (LLMs) such as GPT-3.5*, GPT-4 (OpenAI, 2023), LlaMA (Touvron et al., 2023) and Bard (Anil et al., 2023), resulted in the achievement of seemingly human-like generated text according to evaluation dimensions such as fluency and richness (Zhao et al., 2023). As such, the quality of their output falls short with respect to traditional argumentative quality criteria such as convincingness (Habernal and Gurevych, 2016b). At the same time, LLMs still exhibit deficiencies, such as the generation of false information (Azamfirei et al., 2023; Li et al., 2023b). Liu et al. (2023c) proposed a GPT-generated argumentative essay corpus with annotations by human experts to understand the lexical, syntactic and stylistic features of AIgenerated content (AIGC). However, they do not take into account fact-checking the generated texts and assessing the quality of essays in terms of argumentative properties. Such evaluations are crucial in argument mining, as using false evidence to convince others can lead to untenable decisions.
To address these shortcomings, we have developed an end-to-end dataset that cohesively unifies these primary tasks and provides a comprehensive view of debate preparation (illustrated in Figure 1). We provide human annotations for each of the involved tasks as well as explore the performance of SotA LLMs on these tasks in an end-to-end manner, facilitating design and evaluation of automated debate assistant systems, aligned with human preferences. Our proposed dataset supports tasks that are broadly divided into two streams: (1) evidence detection (ED), (2) evidence convincingness ranking (ECR) and (3A) argument summarisation (AS), are concerned with finding evidence to support the arguments and arranging them into a narra-
*https://chat.openai.com
tive. Meanwhile, (3B) argument summarisation ranking (ASR), learning to rank generated summaries from human preferences and (4) learning automated metrics for summary quality evaluation (SQE), are concerned with automating fine-grained evaluation of the generations from a (weak) human supervision signal. In particular, we design ED, ECR and SQE as classification tasks. AS is formulated as a summarisation task, aimed at generating debate scripts. We conduct an extensive empirical evaluation of the selected baselines on the proposed tasks. In addition, we rely on human assessments to verify the quality of the generated tasks, which showed a positive correlation between automated evaluation and human preference. In summary, the main contributions of this paper are:
• A benchmark dataset named Argument Summarisation and Evaluation (ASE) with four sub-tasks which cover the whole debate preparation process, and benchmarks for the proposed tasks with the LLMs.
• Investigation of the feasibility and performance of an-end-to-end debate preparation assistant, evaluated by humans and automatic metrics based on human preference.

heading: 2 Related Work
text: One of the challenges in the field of argument mining is the lack of high-quality annotated arguments to serve as training and test data. Some work has been done to alleviate the situation, focussing on different sub-tasks such as claim identification (Rinott et al., 2015; Levy et al., 2018; Shnarch et al., 2020), stance identification (Bar-Haim et al., 2017; Jochim et al., 2018; Toledo-Ronen et al., 2018), evidence identification (Shnarch et al., 2018; Ein-
Dor et al., 2020), automatic identification of relational properties (Habernal and Gurevych, 2015; Lawrence and Reed, 2017; Li et al., 2020; Liu et al., 2023a,b), argument summarisation and clustering (Misra et al., 2016; Reimers et al., 2019; Ajjour et al., 2019; Syed et al., 2023), key point analysis (Bar-Haim et al., 2020a; Friedman et al., 2021), and debate speech analysis (Mirkin et al., 2018; Lavee et al., 2019; Orbach et al., 2019, 2020). Others contributed to evaluation of argument quality, like convincingness ranking (Gleize et al., 2019; Toledo et al., 2019; Na and DeDeo, 2022), sufficiency checking (Gurcke et al., 2021; Wachsmuth and Werner, 2020b; Potash et al., 2019; Stab and Gurevych, 2017; Stab, 2017) or factchecking (Azamfirei et al., 2023). However, all of these work focus on only one of the abovementioned tasks, making them difficult to integrate due to the different data sources. Similar to our work is the IAM dataset (Cheng et al., 2022), but there, only arguments at the sentence-pair level are considered, building <claims, positions, evidence> tuples, instead of providing a document-level narrative to provide a comprehensive understanding; also, the resulting argument quality is not assessed. In addition, the proposed task focusses on long articles, whereas we focus on shorter, social-media style texts with non-standard structures.

heading: 3 Argument Summarisation and Evaluation Dataset
text: In this section, we discuss the process of data collection and annotation. The annotation process is divided into two main stages, each with two subtasks: Stage 1 includes the classification of evidence and the ranking of the persuasiveness of the evidence; Stage 2 is focussed on the quality of the generated summaries.

heading: 3.1 Data Collection
text: Evidence Collection: We collected 1881 pieces of evidence candidates supporting claims across 31 debate topics. To align well with existing argument mining resources, all debate topics and their underlying claims and non-argumentative texts are taken from the English ArgKP dataset (Bar-Haim et al., 2020a). Human annotators were asked to collect text snippets that are likely to support a given claim under the debate topic. Text snippets were allowed to be reused for multiple claims and are sourced from various open-source online forums, news, as
well as pre-existing datasets. In our dataset, the URL of the original source is clearly indicated for each evidence candidate.
Summary Collection: Each summary is a narrative consisting of (a) all main claims (also known as key points (Bar-Haim et al., 2020b)) for the same stance on the same topic taken from BarHaim et al. (2020a)’s dataset and (b) the evidence to support that claim. We refrain from relying on human annotators to write reference summaries directly, as this may exhibit considerable variability in the outcomes due to differences in writing style or knowledge limitations. Instead, we use LLMs (including GPT-4 (OpenAI, 2023), GPT-3.5*, GPT-3 (Brown et al., 2020), Bard† (Anil et al., 2023), LlaMA-65B (Touvron et al., 2023), GLM-130B (Zeng et al., 2022), Vicuna-13B‡, Alpaca-13B§ and BLOOM-176B (Scao et al., 2022)) to automatically generate the summaries, which are then evaluated by human annotators, both intrinsically (by ranking the summaries by preference) and extrinsically (by assigning scores along established argument quality dimensions).

heading: 3.2 Data Annotation
text: Task 1: Evidence Annotations: Using the Amazon Mechanical Turk (AMT) crowd-sourcing platform¶, we collected gold labels to associate the collected evidence with the arguments. For each piece of evidence, given the debate topic and argument as context, annotators were asked to answer the following two questions: “Would you think the snippet is relevant to the argument?” and “Would you think the snippet supports the argument?”. They were directed to label all evidence pieces relevant to the argument, and only if the answer to both questions was “yes” then it was considered to be a piece of evidence supporting the argument (see example in Appendix E.2). Each piece of evidence was annotated by at least four annotators. The ground truth label was decided by majority vote.
Task 2: Evidence Convincingness Annotations: Following Gretz et al. (2020) and Toledo et al. (2019), annotators were presented with a binary question per argument, asking “In a conversation about the topic, where you can only give a single evidence out of the following two, which one
†The backbone of the Bard at the completion of this work was PaLM2 instead of Gemini
‡https://lmsys.org/blog/2023-03-30-vicuna/ §https://crfm.stanford.edu/2023/03/13/alpaca.html ¶https://www.mturk.com/
would you rather use?”. Texts involved in the ranking are all the text snippets instead of the gold label obtained in Task 1 so that both ground truth-based and pipeline-based setting could be measured. After the annotation process was finished, we applied milp_ranker|| to derive a ranking of all related pieces of evidence for an argument from corresponding pairwise comparisons we get from the annotations; the implementation details can be found in Appendix A. To align with annotated results in Task 1, irrelevant evidence (labelled as 0 in Task 1) was removed from the final ranked lists.
Task 3A: Argumentation Summary: Each topic was divided into two stances separately, to generate summaries using each of the models mentioned in Section 3.1 in different scenarios. Instead of giving definitive quantitative evidence, we instructed the model to select the appropriate evidence(s) to generate summaries, following the intuition that humans choose the most effective evidence to convince the audience (Lipton, 1993), during preparation, instead of relying on the quantity of evidence. However, the exact number of distinct pieces of evidence required to be convincing is an open question (Schwardmann et al., 2022). Specifically, we supply n pieces of evidence from S sources as input, where n is either one, two or all evidence pieces and S is either the gold standard evidence matched with a claim or predicted by the best-performing model on Task 2. We restricted our model choice to instruction fine-tuned models, to minimise the impact of prompt engineering, as performance of non-instruction-tuning models depended heavily on the choice of prompt and in-context-learning examples (e.g., GLM-130B, BLOOM-176B repeated input and LlaMA-65B summarised it into a single sentence).
Task 3B: Summary Ranking based on Human Preference: To provide assistance in training helpful, truthful and harmless tools (OpenAI, 2023), such as a debate assistant, we defined the evaluation of machine-generated summaries as a ranking task aligned with human preferences. In order to provide annotators with consistent assessment criteria and quantitative performance, we ask them to consider HELPFUL, TRUTHFUL, FLUENT and HARMLESS as the dimensions that measure the quality of the generated text Ouyang et al. (2022). We further use SIGNIFICANT, INFORMATIVE, FAITHFHUL and SUFFICIENT as dimensions to measure the
||https://github.com/twoertwein/milp_ranker
quality of the summary as an argumentative text (Li et al., 2023a). Detailed definitions of the dimensions can be found in Appendix B.
The annotators were asked to rate the outputs on a Likert scale ranging from “The Best” (5) to “The Worst” (1) for each of the scenarios, with each score to be assigned only once. Finally, the annotators had to rank all 36 summaries. To obtain fine-grained rankings, we form the data into the ranked instances with at least two values separated by more than 1 and a total length of 4 that are sufficiently distinguished but when assembled together could show absolute ranking (for instance, given a ranking data set A<B<C<D<E<F, we split it into A<B<D; B<C<E; C<D<F; D<E<F). Based on the fact that each annotator’s preference may differ from others as well as the diversity in the models’ outputs, we normalise the performance of the model into scores using the Bradley–Terry model (Guo et al., 2018) and the two-parameter logistic model following the Item Response Theory (IRT) (Embretson and Reise, 2013) separately. The former quantifies the performance of the model on specific settings by pairwise comparisons, while the latter allows for an overall measure of the level of difficulty of the different settings; the detailed explanation and ablation experiments of both models can be find in Appendix C.
Task 4: Summary Quality Annotations: This task requires to rank the outputs and is simultaneously carried out with Task 3. We asked the annotators to assess factual errors, sufficiency (i.e. whether the given text can persuade readers to accept its stance) and logical coherence of each summary when ranking the summaries. They needed to identify the errors as described above and then mark their location.

heading: 3.3 Annotation Statistics and Analysis
text: For the purpose of integrating the components of the argument structure, we split the 31 available topics into 24 topics for training, 4 for tuning hyperparameters and determining early stopping (dev set), and 3 for testing, in line with previous work (Bar-Haim et al., 2020a). The data distribution is shown in Appendix D. In total, 2002 texts in Task 1 were considered relevant to some claim, of which 1456 were labelled as evidence; the number is higher than the 1881 unique evidence pieces, because each piece can support support multiple claims. For Task 2, two pieces of evidence for the same claim were paired together and a total
of 6034 pairs were obtained. The annotation results were quantified into three labels based on the degree of convincingness of the former over the latter, i.e. “1” if the former is more convincing than the latter, “0” vice versa, and “0.5” when both are equally convincing. As shown in Table 14, in Task 3, claims for the same stance on a topic and the evidence supporting them were then summarised into a debate script. Generated by each of the six previously mentioned models according to the six settings, a total of 2232 unique summaries were obtained. The summaries were then ranked by humans along the evaluation dimensions mentioned in Task 4, resulting in 1488 pairwise rankings.

heading: Randomness in Crowdsourcing Annotation

heading: Platforms and Spam Filtering Mechanism:
text: AMT is a widely recognized and extensively used crowdsourcing annotation platform (Paolacci et al., 2010), but ensuring the quality of annotators is challenging (Aruguete et al., 2019). Consequently, we implemented two post-processing measures. On the one hand, to monitor and ensure the quality of the annotations collected, the following measures were employed: (1) Test questions: Inspired by Toledo et al. (2019), a unique set of 100 hidden test questions was designed as an attention check, where one of the options were other text snippets from unrelated topics. These test questions were presented in the same way as the other questions. The work of annotators who failed 20% of the test questions was ignored. (2) Annotator reliability testing: 4% (250) of the annotations were selected for judgement by experts. For each annotator, the average agreement score with other annotators was calculated on the data they annotated, and then the average agreement of all annotators were ranked. If an annotator’s average agreement differed too much (lower than 0.125) from the average agreement scores of other annotators, they were not considered for follow-up annotation**.
ChatGPT as Third Annotator in Task 2: Previous work has demonstrated that ChatGPT can already achieve performance comparable to that of crowdsourcing annotators (Gilardi et al., 2023). Based on this, we first verified the reliability of ChatGPT annotations by annotating all of the test instances mentioned in Annotator reliability testing. ChatGPT’s results were compared with golden labels as well as instances where human annotators
**We release raw annotation data, to facilite subsequent research to estimate gold standards from the noisy crowdsourced labels (Plank, 2022)
reach agreement separately. This ranked ChatGPT third out of 124 human annotators who participated in annotating for Task 2 and reaching a 97% agreement with human annotators in the latter. Thus, ChatGPT was employed as a “third annotator” for all instances that were annotated by an even number of human evaluators with no preference for the results. If ChatGPT’s answer yielded no preference (i.e. “Both evidence 1 and evidence 2 support the claim...”), we considered both pieces of evidence to be equally persuasive and assigned the label “0.5”.
Reliability of Human Annotation: Overall, a total of 1636 human annotators participated in our annotation project. Considering that the evidence was collected after filtering by human experts, the difficulty of Task 1 was low. We hired six experts for this annotation task who participated in an onboarding training with 250 data samples. Finally, four annotators participated in the whole annotation process. We measured Cohen’s Kappa (McHugh, 2012) to assess inter-annotator agreement, reporting an average of 0.6886 across the train/dev/test set, implying that the results are substantially reliable (Munoz and Bangdiwala, 1997). Agreement on Task 2 was significantly lower, presumably because the task depended on the participants’ background and knowledge. Furthermore, a different set of annotators for each annotation resulted in difficulty to obtain meaningful values on Cohen’s Kappa. Therefore, we measure reliability by comparing the pairwise agreement between each two annotators who annotated at least one example together, reporting such an an average agreement of 0.549. We further used ChatGPT with verified reliability as an additional annotator to resolve tie breaks, as mentioned in the previous section. For Task 3, we did a cross-validation for every two annotators that had 25% overlap between their work, resulting in Krippendorff’s α (Krippendorff, 2011) of 0.2474, which suggests that providing rankings is a largely subjective task. For Task 4, Cohen’s
Kappa was measured at 0.8970 for sufficiency and 0.4933 for factuality, respectively. These suggest that the task of identifying factual errors and evaluating argumentative logic within the generated text poses a considerable challenge. Conversely, determining the persuasiveness of debate scripts appears straightforward (see Appendix E for details of the annotation platform, costs and templates).

heading: 4 Baseline Approaches
text: Sentence-pair Classification: We formulate Tasks 1, 2 and 4 as sentence-pair classification tasks, where we employed Flan-T5 (Chung et al., 2022), BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) as baseline models. As an abstract representation of the task ”Does the evidence support the claim”, the input is a concatenation of argument and evidence candidate, and the output is binary, i.e., {1, 0}, where the snippet is considered as evidence for the claim when they are judged to be the same pair. Task 4 (sufficiency and factual error checking, where {1, 0} represent sufficient/factually wrong or not, respectively) follows the same setup. For Task 2, we define it as a multiclass classification task, i.e., given an evidence pair and a claim, “the former is more convincing” is labelled as 1, “the latter is more convincing” is labelled as 0, and “both are equal” is labelled as 0.5.
LLMs with Contrastive Learning: We formulate Tasks 2 and 3 as a contrastive learning task. Conceptually, we consider the evidence closest to the claim in the semantic embedding space as the most supportive candidate for it. For this purpose, we utilise Siamese Neural Networks (including SimCSE-RoBerta (Gao et al., 2021), Sentence-t5 (Ni et al., 2021)) with contrasting loss functions. The inputs are the concatenation of the topic and claim, and the evidence candidate, respectively. The output is the score reflecting how close they are to each other (more details in Appendix F).
Summarisation with LLMs: Inspired by previous work in argument summarisation (Li et al., 2023a), we employed Flan-T5 (Chung et al., 2022) and PEGASUS (Zhang et al., 2020) as summarisation systems. The former is a large model trained on more than a thousand instruction tasks, while the latter has pre-trained objectives tailored specifically for abstract summarisation. The input of the model is in the form of a concatenated pair <stance, topic, claim 1, evidence 1, evidence 2,..., claim 2,
....>. The output will be a single summary. Project Debater: The first end-to-end automatic debating system, Project Debater (Slonim et al., 2021) encompasses multiple components such as claim detection, evidence detection, argument quality assessment and narrative generation. Employing it on our dataset, we can explore the limitations of a component-based SotA system.

heading: 5 Experimental Settings
text: Broadly speaking, we aim to investigate how well the generated data aligns with human standard and the performance of current representative LLMs in end-to-end debate preparation; we also provide a benchmark for future research. Specifically, we ask: (i) How well do representative LLMs perform on tasks? (ii) How well does LLM-generated data correlate with human judgement? (iii) How big is the impact of end-to-end pipeline inputs on model performance? (iv) How good is the baseline performance on each of the subtasks? To answer questions (i) and (ii), we conduct both manual evaluation and automated evaluation by first evaluating LLM outputs by means of human evaluation and then assessing correlations with human judgement and learned metrics. For questions (iii) and (iv), we compare the performance of our proposed baseline approach with both golden input and pipeline input.
To measure the performance on the imbalanced data distribution setting, we measure the Matthews correlation coefficient (MCC) (Guilford, 1954) as well as Macro-F1. The former progressively penalises over-confidence on the majority class, while the latter allocates the same weight to each class regardless of its cardinality in the dataset. For the summarisation task, we employ the widely used ROUGE metric (Lin, 2004), which measures performance by counting n-gram overlap. In addition, BLEURT (Sellam et al., 2020) is used to measure the performance from a semantic similarity perspective. For the ranking task, we consider precision@k, Mean Average Precision@k (MAP@k) (Davis and Goadrich, 2006) and Normalized Discounted Cumulative Gain@k (NDCG@k) (Burges et al., 2005). MAP@k is concerned with precision at every rank up to k, which takes into account both the number of relevant documents retrieved and their order in the rank list, penalising systems that return relevant documents further down the list. Meanwhile,
NDCG gives more weight to the correctness of items at higher positions in the ranked list. Both metrics are widely used to evaluate ranking systems. Further details on parameter settings of baselines can be found in Appendix G.

heading: 6 Benchmarks Results and Analysis

heading: LLMs Generation Aligns with Human Stan-
text: dard, but Weaknesses in Logic and Informativeness persist: Table 1 shows that human evaluations suggest that generated summaries are of high quality. Overall, GPT4 showed the best performance, while Alpaca had the worst performance which is consistent with the results of the BT model based on ranked quantification (shown in Appendix C). Specifically, all models perform well in the TRUTHFUL, HARMLESS, FAITHFUL and REDUNDANT categories, suggesting that LLM outputs are comparable to humans along those dimensions. However, they still lack in dimensions that require logical understanding (HELPFUL and INFORMATIVE).

heading: Similarity to GPT-generated Summaries Cor-
text: relates with Argumentative Quality Assessments: The relative quality of the models is further represented in Table 2 and 3. Using humanevaluated performance of GPT-4 as a reference, ChatGPT performs the best, while Alpaca’s performance drops the most. We further explore the correlation between the human evaluation with Rouge scores (Spearman’s r = 0.75, p < .005) and BLEURT (Spearman’s r = 0.64, p < .005), suggesting that measuring the similarity of out-
puts to GPT4-generated summaries is a feasible proxy-metric representing the evaluation of argument quality. Intuitively, this is possible as GPT4 was assigned nearly full marks in all dimensions during human evaluation. Therefore, we treat its output as the gold standard for grammatical dimensions (e.g., FLUENT and INFORMATIVENESS (Zhao et al., 2023)) in the following experiment that investigates the performance of baseline approaches. Nevertheless, quality assessment still relies on human annotators.

heading: End-to-end Performance Exhibits Significant

heading: Drops, Demonstrating the Challenges of the
text: Task: We select the most powerful model in each of subtask to build the end-to-end pipeline, where the input of each subsequent task is the output of the preceding task. As can be seen from Table 6, the performance of the pipeline has a significant drop compared to the same model with golden inputs, dropping 0.11 points Prec@1 on Task 2 and 16 points ROUGE on Task 3, respectively. This shows that achieving an integrated debating system
that is fully aligned with human preferences is challenging. It should be noted that Project Debater also shows a significant performance drop, indicating that there is still room for further improvement of end-to-end systems, underlining the importance of our collected resources.
Task 1: Evidence Detection Overall, FlanT5-11B achieved the best performance, as shown in Table 5. Specifically, FlanT5-11B and FlanT5-3B perform significantly better than smaller models. This suggests that larger models perform better when faced with label imbalance. FlanT5 also outperforms the encoder-only models of the same size, suggesting that either instructionfinetuning or the encoder-decoder architecture improve performance. Regarding specific classes, all models performed better in predicting evidence, likely because it is the majority label. In contrast to performance reported so far, RoBerta-Large performed best (with Accuracy of 0.52) when predicting non-evidence, and correspondingly it performed worst with an Accuracy of 0.79 when predicting evidence, indicating that it achieved a relatively balanced overall performance. Finally, compared to inference (zero-shot), the models trained on the proposed dataset always perform better, suggesting that the dataset can effectively improve the performance of the models in this task. FlanT5-3B shows comparable overall performance with Project Debater; bigger LLMs could potentially lead to improved performance.
Task 2: Evidence Convincingness Ranking As mentioned in the baseline setting description, Task 2 has been formulated as both a classification and ranking task. Comparing the performance of classification vs contrastive learning, the former outperforms the latter (Table 6). This indicates that the label “0.5” (the same persuasive power
for both) bears important semantics, which is lost incontrastive learning, resulting in lower scores. In detail, it is noticeable that model size does not seem to play a significant role for the contrastive learning approach (c.f. SentenceT5-Large and -3B in Table 7) which might be because the nuanced differences in evidences are hard to pick up by means of semantic similarity. For classification methods, performance scales with size, which is consistent with Task 1 results.
Task 3 Subtask A: Argumentation Summary Here, FlanT5-3B outperformed all other models in all settings, with the remaining models exhibiting comparable performance levels (c.f., Average ROUGE scores of All vs Top-2 in Table 8). This suggests that this particular setting presents the most challenges, potentially due to constraints in input length and the model’s struggle to select relevant evidence. It is worth noting that there is a huge gap between the performance of the model with end-to-end inputs and the model with golden
inputs, which demonstrates the challenges of the task (Shown in Table 4).
Task 3 Subtask B: Argumentation Summary and Ranking Table 9 shows the performance on the proposed baseline models, suggesting that the size of the model does not significantly contribute to succeeding on this task. However, SentenceT5 performs better than SimCSE-RoBERTa-Large, suggesting that embedding two sentences simultaneously has better performance in measuring similarity than embedding sentences separately, presumably due to the fact that the former is better at measuring the relationship between the original and interfering inputs by representing them in the same space than what the latter does, i.e., by representing them separately (Neculoiu et al., 2016).
Task 4: Summary Quality Checking Considering that data sets are imbalanced for both tasks, the smaller models default to predicting the majority class resulting in an MCC score of 0. FlanT5-3B is the only model that performs marginally better albeit with only weak correlation to human judgements. This suggests that the large model exhibits enough robustness to address the challenges posed by imbalanced data and the weak human feedback signal. Specifically, the task of fact-checking appears to be particularly challenging, as all the largesized models exhibit an MCC of 0. This could be
attributed to the scarcity of data labeled as “1”, i.e. factually wrong.

heading: 7 Conclusion
text: In this study, we built an end-to-end argumentative summary and evaluation dataset and established the benchmarks for each task. This process involves annotating the data by human experts mixed with LLMs and applying popular deep learning baseline methods to the new datasets. We open-source our data and benchmark code, enabling researchers to reproduce and enhance the results. In future, we hope to further expand the evaluation dataset to explore the potential of unprocessed real-world data as evidence and fact-checking for its quality.

heading: Limitations
text: Recruiting human subjects for annotation limits the reproducibility of human evaluation. In addition, we recognise that there might be more suitable baseline models, and in this study, just a few of the most advanced models were used. Also, we may employ human experts to complete summaries that are not generated as expected for different reasons to fill in the gaps. Finally, despite being filtered by
human experts, offensive debate content may still be present in the data. It is worth noting, however, that the identification of offensive language is not the aim of this work.

heading: Ethics Statement
text: During the annotation process, we considered the following dimensions to protect our annotators. (1) Consent: To ensure that our participants, i.e., AMT workers, consent to the annotation task, they will be required to review the task guidelines and instructions before starting their work. They can exit the task at any time if they feel uncomfortable. (2) Confidentiality: None of the task participants is known to us, i.e., the entire annotation process is anonymised: no information about the AMT workers is accessible to us, and none of the data being annotated is confidential as the data samples all come from publicly available sources. (3) Safeguarding: We selected debate topics (for annotation) that are used in high school-level debates and argumentative essays. They are unlikely to cause any harmful effects, as they have already been subject to previous scrutiny (Released by KPA dataset (Bar-Haim et al., 2020a)).

heading: Acknowledgements
text: We thank the anonymous reviewers from the ARR December 2023 cycle for their valuable feedback. We would especially like to thank Roy Bar-Haim and Yoav Katz at the IBM Research for helpful discussions, feedback on our work and help with using Project Debater. We would also like to acknowledge the use of the Computational Shared Facility at The University of Manchester. We would like to conclude by thanking all of our hard-working annotators. This work was partially funded by the European Union’s Horizon 2020 research and innovation action programme, via the AI4Media Open Call #1 issued and executed under the AI4Media project (Grant Agreement no. 951911).

heading: A Milp Ranker
text: The milp_ranker formulates the problem that ranking from pairwise comparisons as solving a mixed-integer linear program (MILP). We select it as our ranker because it can handle both the contradictory case between the pairwise comparisons and the equal case between two pieces of evidence. The use of milp_ranker is restricted and therefore an academic license was applied here. To align with annotated results in Task 1, evidence with the label 0 in Task 1 was removed from the ranking.

heading: B Quality evaluation dimensions for summaries
text: • HELPFUL: The output should follow the user’s intention and help users prepare a script for debate that meets their demands
• TRUTHFUL: The output contains accurate information and doesn’t mislead the user.
• HARMLESS: The output should not cause physical, psychological, or social harm to people; or harm to institutions or resources necessary to human wellbeing.
• SIGNIFICANT: Each claim and supporting evidence(s) should stand out and capture a main point.
• INFORMATIVENESS: It should discuss some aspect of the debate topic and be general enough. Any claims or evidence(s) that are too specific or only express sentiment cannot be considered a good candidate.
• FAITHFULNESS: Claims and Evidence should actually express the meaning in the corpus. No conjecture or unfounded claims arise.
• REDUNDANT: Each claim expresses a distinct aspect. In other words, there should be no overlap between the claims.
• FLUENT: tthe generated text constitutes a fluent script that can be presented by humans.

heading: C Bradley–Terry model (BT) and Item Response Theory (IRT)
text: Here, we formalise the BT model into the following:
L(p) = ln ∏
ij
[P (i > j)]wij = ∑
ij
wij ln pi
pi + pj
(1)
pi = ∑ j wijpj/(pi + pj)∑ j wji/(pi + pj)
(2)
where pi is a positive real-valued score assigned to individual i, wij be the number of times individual i beats individual j.
Based on the fact that each annotator’s preference may differ from others as well as the diversity of the models’ generation, we normalise the performance of the model into scores using the twoparameter logistic model following the Item Response Theory (IRT) (Embretson and Reise, 2013). Here, we formalise the task into the following:
P (Xij |θi;αj ;βj) = eDαj(θi−βj)
1 + eDαj(θi−βj) (3)
where θi represent the ability of the ith models , αj represent the discrimination of the j topic, βj represent the difficulty of the j topic, Xij represent the ith model’s response matrix on the topic j and D is a constant and takes the default value of 1.702, where the probability density of the function differs from the normal shoulder curve by less than 0.01.

heading: D Dataset Distribution
text: Table 12 Demonstrate the distributions of the dataset. For Task 1 Evidence Detection, numbers in brackets represent the number of cases labelled as evidence.

heading: E Human Annotators Recruited and Cost

heading: E.1 Overview of Annotation Process
text: We release the annotation task and employ annotators from the Amazon Mechanical Turk crowdsourcing platform directly. The project spends £3160 in total, where each of the data cost $0.2. Any annotator can start working after carefully reading the instructions and can choose to quit at any time.

heading: E.2 Annotation Template
text: We provide a annotation template that we shape the task into bianry classification task in order to simplify the task, example can be seen in Table 13. And annotation template for summarisation ranking task in Table 15.

heading: F Formula of Contrastive learning
text: Analogous to Alshomary et al. (2021), we define the contrastive loss as follows:
L(A,P,N) = max( ∥f(A)− f(P )∥2 − ∥f(A)− f(N)∥2 + α, 0 )
Here, A represent the concatenation of topic and claim, P is the positive input (i.e. convincing evidence), N is the negative input (i.e. non-convincing evidence), α reflects the margin between positive and negative pairs, and f is an embedding.

heading: G Experiment Setting
text: To establish benchmarks for our ASE datasets, we trained the baseline models from the previous section. All of the proposed models were trained and tested on two NVIDIA Tesla A100 80GB GPUs. All classification models were trained for 15 epochs (batch size of 64 (large), 32 (3B) and 8 (11B) respectively), while the summary model was trained for 5 epochs (batch size of 16 (large) and 4 (3B) respectively). The maximum input length for the summary task is 1024 and the max target length is 512. Other parameters are set by default. All contrastive learning models were trained for 10 epochs with the training batch size as 32 for Roberta-large and Setence-t5-large, 8 for Sentence-t5-3B, respectively. The learning rate was set as 3e−5 and the maximum input sequence length was 256. We used Euclidean distance as the measurement between two embeddings in the contrastive loss function with α as 5 by default. The model with the best accuracy on the validation set was taken as the final trained model for inference.

heading: H Project Debater’s output
text: Figure 2 shows the result of IBM Project Debater.

heading: I Ablation Experiment
text: Quality of Settings: Table 11 at Appendix C demonstrates the performance models as judged by humans and interpreted with IRT. Overall, the gold setting is significantly better than the evidence chosen by the baseline model (c.f. Average Golden vs Average Baseline), suggesting that there is scope for improvement on this challenging task. In the gold setting, the scenario that performs best occurs when all evidence is readily accessible (c.f. average All, Best and Top2 scores). This suggests
that the more comprehensive the model, the more effectively it can assess the quality of evidence, especially when the data is of high quality. In such cases, the model benefits from the additional information provided by the inputs. However, when supplied with irrelevant evidence, the models’ performances tend to deteriorate due to the increased presence of noise (compare Top to All).
Performance of Models: When it comes to specific models, as can be seen from the output of BT (From left to right: GPT4, GPT-3.5, GPT3, Bard, Alpaca, Vicuna = {9.2483,9.1009,6.0515,- 1.4266,-1.2101,-1.0872}), GPT4 performed best in all settings followed by GPT-3.5 while Alpaca and Vicuna performed worst, which demonstrates that more powerful models perform better in this task. Bard performs poorly, maybe due to the fact that it will disregard instructions to generate both supporting and opposing text where the source of the information cannot be identified, in the case of supporting evidence provided only.

heading: J Example of whole dataset
text: Table 14 shows the overview of whole dataset
presence of factual errors (e.g. the word justices like in the example is hallucination), and the ranking of quality among all summaries.

heading: Future Work
text: In future, we hope to further expand the evaluation dataset to explore the potential of unprocessed real-world data as evidence and fact-checking for its quality