Title: Exploring the potential of LLMs in Computational Argumentation
Benchmark: LLM-Argumentation
Summary: The authors systematically evaluate large language models (LLMs) on a broad suite of computational argumentation tasks spanning both argument mining (claim, evidence, stance, type) and argument generation (argument construction, summarization). They unify 14 public datasets into six task categories, introduce a new document‑level Counter‑Speech Generation benchmark that jointly exercises mining + generation, and report zero‑/few‑shot results for GPT‑3.5‑Turbo, Flan‑T5/UL2, and Llama‑2 models.
Contribution: - Curated taxonomy of 6 task categories and standardized formats for 14 datasets (≈ 500 examples each).
- New end‑to‑end counter‑speech dataset (250 debate‑speech pairs) measuring holistic argumentative ability.
- First large‑scale zero‑/few‑shot evaluation of multiple LLM families on computational argumentation, with statistical testing and human evaluation.
- Analysis shows LLMs close most supervised gaps with ≤ 10 demonstrations and highlights current metric limitations (e.g., ROUGE vs. semantic quality).
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Argument Comprehension & generation
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given argumentative text, either (i) classify components/relations (claim, evidence, stance, type) or (ii) generate arguments, summaries, or counter‑speeches.
Task item definition: A single row consists of input text (sentence, passage, or full speech) plus gold label(s)/reference text needed for the corresponding sub‑task.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: All 14 datasets are existing public corpora (IBM, IAM, FEVER, MTSD, etc.); authors randomly sample 500 test items each (to equalize effort) and build a new counter‑speech set by matching debate transcripts.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 500 items per legacy dataset; 250 items in Counter‑Speech; total test items ≈ 7.25 k.
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Counter speech generations are evaluated using ROUGE-1/2/L, METEOR, and BERTScore for automatic metrics. Human judges also rate outputs for fluency, persuasiveness (1–5 Likert scale), and coverage of argumentative content (% arguments addressed).
Metric aggregation: Simple Mean
Metric statistics: Mean; McNemar x^2 test for mining
Metric subscores: Yes
Metric subscores detail: Per sub-task/category
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They discuss ROUGE under‑estimating semantic quality and note evaluation limitations.
Taxonomy: Reasoning / 
New taxonomy: General Capability / Natural Language Understanding