Title: MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification
Benchmark: MAFALDA
Summary: The paper introduces MAFALD, a benchmark that provides a unified classification of fallacies and provides a taxonomy. It features manually annotated data with explanations, a tailored annotation scheme, and an evaluation method for subjective NLP tasks. Various language models and human performance are evaluated on fallacy detection and classification in a zero-shot learning setting.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: fallacies in reasoning
Phenomenon defined by authors: Yes
Phenomenon definition:  A fallacy is an erroneous or invalid way of reasoning. A fallacy is an argument where the premises do not entail the conclusion. Sub-elements: Fallacy of credibility, fallacy of logic, appeal to emotion
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a text, detect fallacies and classify them
Task item definition: Level 0: binary classification (fallacy or not), Level 1: groups fallacies into Aristotle’s categories: ‘Pathos’ (appeals to emotion), ‘Ethos’ (fallacies of credibility), and ‘Logos’ (fallacies of logic, relevance, or evidence), Level 2 contains fine-grained fallacies within the
broad categories of Level 1. For instance, under fallacy of credibility, we have specific fallacies such as appeal to tradition, ad populum, and guilt by association.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 9735
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: 3 levels (different granularity)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning