Venue: NAACL 2025

# Fine-grained Fallacy Detection with Human Label Variation

Alan Ramponi, Agnese Daffara, Sara Tonelli\({}^{1}\)

###### Abstract

We introduce Faina, the first dataset for fallacy detection that embraces multiple plausible answers and natural disagreement. Faina includes over 11K span-level annotations with overlaps across 20 fallacy types on social media posts in Italian about migration, climate change, and public health given by two expert annotators. Through an extensive annotation study that allowed discussion over multiple rounds, we minimize annotation errors whilst keeping signals of human label variation. Moreover, we devise a framework that goes beyond "single ground truth" evaluation and simultaneously accounts for multiple (equally reliable) test sets and the peculiarities of the task, i.e., partial span matches, overlaps, and the varying severity of labeling errors. Our experiments across four fallacy detection setups show that multi-task and multi-label transformer-based approaches are strong baselines across all settings. We release our data, code, and annotation guidelines to foster research on fallacy detection and human label variation more broadly.1

Footnote 1: 1}\) Digital Humanities group, Fondazione Bruno Kessler, Italy

\({}^{2}\) Department of Humanities, University of Pavia, Italy

\({}^{3}\) Institute for Natural Language Processing, University of Stuttgart, Germany

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associations for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 762–784


with experiments across setups of increasing complexity and thorough data analyses.

The Faina dataset is the first annotated resource for fallacy detection embracing multiple plausible answers and natural disagreement. It is also the first dataset providing annotations at the fine-grained level of text segments with potential overlaps, accounting for over 11K annotated fallacies by two expert annotators across an inventory of 20 fallacy types. Faina covers public discourse on migration, climate change, and public health issues in social media posts from Twitter, and focuses on Italian, a currently overlooked language in fallacy detection. To account for the complexity axes of the task, we design an evaluation framework that embraces the presence of multiple gold standards that are equally valid, partial span matches with overlaps, and the varying severity of fallacy classification errors.

We conduct experiments across four setups of increasing complexity, from post-level to span-level fallacy detection and using either fallacy macro-categories or the full inventory. Our results and analyses show that multi-task and multi-label transformer-based classifiers are strong baselines for fallacy detection tasks and that current large language models (LLMs) in zero-shot settings are still far from achieving satisfactory performance.

We also provide thorough data analyses, including insights on our multi-round annotation procedure which minimizes annotation errors whilst keeping label variation, and a manual audit of the outputs generated by LLMs. To foster research on fine-grained fallacy detection and broadly on human label variation in span-level tasks, we make our materials available to the NLP community.

## 2 Related Work

The increasing interest in fallacy detection has led to the creation of datasets with different characteristics in recent years (Table 1). Some works focused on the educational domain, either by collecting data through gamification Habernal et al. (2017, 2018) or from online quizzes Jin et al. (2022), whereas others studied transcripts of political debates Goffredo et al. (2022, 2023). Fallacies in news articles have been explored by Jin et al. (2022) and Alhindi et al. (2022) in climate change discourse, and by Musi et al. (2022) for analyzing COVID-19-related misinformation. Some works examined social media content using Reddit comments Habernal et al. (2018); Sahai et al. (2021) or tweets Sheng et al. (2021), with a special interest on political discourse Macagno (2022); Shultz (2024). We also analyze fallacies on social media but tackle previously unexplored topics over a four-year time frame and deal with the Italian language.

Regarding annotation, current datasets mainly provide coarse-grained labels at the text (post, paragraph), snippet, or text pair level. The only exceptions are span-level datasets by Sahai et al. (2021) and Goffredo et al. (2023) which, however, do not foresee overlapping annotations. Inspired by work on propaganda detection Da San Martino et al. (2019, 2020); Paszke et al. (2020) and persuasion techniques detection Piskorski et al. (2023b), our dataset in

\begin{table}
\begin{tabular}{l l c c r r r r} \hline \hline
**Work** & **Langs** & **Genres and domains/topics** & **Scope** & **\#C** & **\#I** & **Multi** & **HLV** \\ \hline Argotario (en) Habernal et al. (2017) & en & \multirow{6}{*}{General} & pair & \$!  & 535 & ✗ & ✗ \\ ChangyHV View Habernal et al. (2018) & en & \multirow{6}{*}{Conference} & vast & text & 1 & 3,622 & ✗ \\ InformalFallaciesSahai et al. (2021) & en & & Politics; religion; veganism; fallacies & span & 8 & 1,708 & ✗ \\ AdHomIntTweets Sheng et al. (2021) & en & \multirow{6}{*}{Unicative} & BLM;MeTo;veganism; remote work & \multirow{6}{*}{Unicative} & \multirow{6}{*}{Unicative} & \multirow{6}{*}{Unicative} & \multirow{6}{*}{Unicative} & \multirow{6}{*}{Unicative} \\ LanguageOfPopulism*Macagno (2020) & en; it; pt; & & & & & & \\ Logic Jün et al. (2022) & en & & General & & & & \\ LogicClimateJün et al. (2022) & en & & Climate change & snippet & 13 & 1,079 & ✗ \\ COVID-19 Musi et al. (2022) & en & & COVID-19 & & & & \\ Climate Alhindi et al. (2022) & en & & Climate change & snippet & 10 & 477 & ✗ \\ ElceDeb@toh16 Goldrebo et al. (2022) & en & & Political debates (US pres. campaigns) & snippet & 14 & 1,628 & ✗ \\ ElecDeb@to0zon Göfflede et al. (2023) & en & & Political debates (US pres. campaigns) & span & 6 & 1,989 & ✗ \\ RuFaal Shultz (2024) & en & & Political discourse (Russian govt.) & text & 13 & 700 & ✗ \\ \hline
**Faina _(ours)_** & it & ✗ & Climate change; migration; public health & span & 20 & 11,064 & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Existing fallacy detection datasets. **Genes**. [EN]news; [EN] educational; [EN]social media/forums; [EN] transcripts. **Scope** of annotation. _pair_: question-answer/post-reply pair; _text_: whole text (e.g., post, paragraph); _snippet_: text excerpt; _span_: sequence of tokens in text. **#C**. Number of fallacy classes. **#II**. Number of _human_-annotated instances. **Multi**. Whether multiple/overlapping annotations for the same pair/text/snippet/span are provided. **HLV**. Whether the dataset includes human label variation. \({}^{\dagger}\): it also includes a negative class whose examples are not counted in #I for fair comparison. \({}^{\ddagger}\): it includes 12K extra examples but are _machine_-annotated. *: Arbitrary dataset name.

 

stead provides span-level annotations with overlaps for a better model transparency. Previous datasets include up to 3.6K _human_-annotated instances Habernal et al. (2018) across 14 fallacies Goffredo et al. (2022). Instead, ours provides 11K human-labeled spans across 20 fallacy types.

Recent work highlighted the importance of considering human label variation Plank (2022), i.e., genuine disagreement Poesio and Artstein (2005), subjectivity and perspectives Aroyo and Welty (2015); Cabitza et al. (2023), and multiple plausible answers Nie et al. (2020) as signal rather than noise. Yet, in fallacy detection all the labels from multiple annotators are typically aggregated Sahai et al. (2021), selectively chosen Musi et al. (2022); Habernal et al. (2017), or adjudicated through discussion or by an expert Jin et al. (2022); Macagno (2022); Goffredo et al. (2022), _inter alia_). More broadly, label variation has been mostly studied at the post or token level, with only few exceptions analyzing span-level disagreement in argument annotation Lindahl (2024); Hautli-Janisz et al. (2022). In our work, we resolve annotation errors while keeping genuine disagreement and multiple plausible answers, propose the first fallacy detection dataset at the span level with parallel annotations, and use human label variation during evaluation.

## 3 Harmonization of Fallacies

Given that different sets of fallacies have been proposed in the literature, we selected our tagset by reviewing fallacy types from previous work (Section 2) and harmonizing them. We thus compiled a list of 41 fallacies and conducted a pilot annotation on 15% of our data (Section 4). This exploratory phase allowed us to homogenize fallacy names and unify those with similar definitions under the same label (e.g., {_Post hoc_, _False cause_} \(\rightarrow\)_Causal oversimplification_), leading to a total of 20 fallacy types. Most fallacy definitions are derived from Musi et al. (2022) but also from Da San Martino et al. (2019), reflecting the persuasive nature of posts in our data. Fallacy definitions are provided below (extended definitions and examples are in Appendix C.2):

1. **Ad hominem** (ah): an excessive attack on an individual or a group;
2. **Appeal to authority** (aa): appealing to an authority figure or a group consensus to support a thesis;
3. **Appeal to emotion** (ae): manipulation of the recipient's emotions in order to win an argument;
4. **Causal oversimplification** (co): the attributed causal relation is simplified and fallacious;
5. **Cherry picking** (cp): choosing evidence which supports a given position, dismissing findings which do not;
6. **Circular reasoning** (cr): the end of an argument comes back to the beginning without having proven itself;
7. **Double** (do): questioning the credibility of someone or something;
8. **Evading the burden of proof** (ep): a position is advanced without any support as if it was self-evident;
9. **False analogy** (fa): two different things or situations are treated equally;
10. **False dilemma** (fd): a claim presenting only two options or sides when there are many;
11. **Flag waving** (fw): playing on strong national feeling (or with respect to a group) to justify or promote an action or idea;
12. **Hasty generalization** (hg): a generalization is drawn from a sample which is not representative or not applicable to the whole situation;
13. **Loaded language** (ll): using words/phrases with strong emotional implications (positive or negative) to influence an audience;
14. **Name calling or labeling** (nc): labeling the object of the propaganda campaign as either something the audience fears, hates, finds undesirable or otherwise loves or praises;
15. **Red herring** (rh): the argument supporting the claim diverges the attention to issues which are irrelevant for the claim at hand;
16. **Slippery slope** (ss): implies that an improbable or exaggerated consequence could result from a particular action;
17. **Slogan** (sl): a brief and striking phrase used to provoke excitement of the audience; 

18. **Strawman** (st): the arguer misinterprets an opponent's argument for the purpose of more easily attacking it, demolishes it, and then concludes that the opponent's real argument has been demolished;
20. **Thought-terminating cliche** (tc): short and generic phrase that discourages meaningful discussion;
21. **Vagueness** (va): words which are ambiguous are shifted in meaning in the process of arguing or are left vague, being potentially subject to skewed interpretations.

We further organize these fallacy types into a taxonomy grouped around three macro-categories which include all the others: _Insufficient proof_, _Simplification_, and _Distraction_, following similar efforts Dimitrov et al. (2024); Tindale (2007). The full taxonomy is presented in Figure 4 and serves for evaluation purposes (Section 5).

## 4Faina Dataset

In this section, we describe the creation of Faina, from data collection (Section 4.1) to data annotation (Section 4.2). We then provide detailed dataset statistics Section 4.3). Data statements Bender and Friedman (2018) are available in Appendix A.

### Data Collection and Sampling

We collect social media posts in Italian (it) that discuss issues pertaining to migration, climate change, and public health using the Twitter APIs.2 To minimize temporal and topic biases, the posts were collected from a large time frame of four years (from 2019-01-01 to 2022-12-31). We filter messages on the aforementioned topics by using a manually curated list of 436 keywords derived from trustable glossaries and manuals and extended to cover all applicable grammatical genders and numbers (see Appendix B for keywords and sources). We then retain posts with \(\geq 5\) tokens3 and select those with the largest number of likes and retweets as in Nakov et al. (2022), therefore focusing on the messages with highest impact to the society. Specifically, to simultaneously mitigate topic, author, and temporal bias in sampling, we keep the top-\(k\) posts (\(k=10\)) for each month and topic, further excluding messages authored by the same user after their most impactful post, and resampling messages until we obtain \(k\) posts for each month-topic combination.4 As a result, we collect 1,440 posts balanced across topics (480 per topic) and time (360 per year) for fine-grained, span-level annotation with overlaps.

Footnote 2: Data was retrieved in February 2023 when the social media platform X was still named Twitter, and when APIs for research purposes were still available for free.

Footnote 3: Indeed, texts with \(<5\) tokens are unlikely to contain argumentation and thus fallacies. For tokenization we used the it_core_news_sm spacy model (v3.5; [https://spacy.io](https://spacy.io)).

### Manual Data Annotation

Span-level annotation with an inventory of 20 fallacy types and potential overlaps is an intrinsically difficult task. It requires annotators to devote significant effort in understanding the nuances of fallacies and master the annotation guidelines, both for spam segmentation and labeling. We thus avoid crowdsourcing which, as also noted by Da San Martino et al. (2019b) for span-level propaganda annotation, is not suitable in this context. We instead devise an annotation protocol that foresees multiple rounds of annotation and discussion among two expert annotators (\(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\)) which allows us to minimize annotation errors whilst keeping signals of human label variation. Details on annotators' profiles are in Appendix A. The annotation proved very challenging and took about 380 person-hours to be completed, discussions included.

Annotation protocol and guidelinesAfter a pilot phase in which we formalize the annotation guidelines and consolidate the label set Section3), we conduct five rounds of annotation and discussion in an increasingly larger number of posts (i.e., 60, 120, 180, 360, 720) balanced across topics. At each round, annotators _i)_ individually located all the text segments expressing fallacies, giving each of them one of the 20 labels from our inventory,5_ii)_ discussed with each other the annotated instances that diverged in the span extent and/or the assigned label, and _iii)_ resolved the cases of disagreement due to errors or attention drops, therefore keeping the naturally-occurring variation in annotation due to multiple plausible answers (e.g., interpretation) and genuine disagreement Plank (2022). The guidelines and annotations from previous rounds have been updated at the end of each round based on the discussion outcomes. After completing all the rounds, the two annotators also revised their 

annotations to ensure that all posts were labeled consistently. We provide the annotation guidelines in Appendix C to foster future work covering other languages and additional annotators.

Inter-annotator agreementWe calculate the inter-annotator agreement (IAA) at each annotation round, both _before_ and _after_ discussion, and considering either span identification or span classification. This allows us to get insights on the whole annotation process besides final results only. Since our dataset includes span annotations of varying length which may also overlap, we avoid using Krippendorff's alpha (\(\alpha\); Hayes and Krippendorff, 2007) or other metrics that are typically applied at the token-level. We instead use \(\gamma\)Mathet et al. (2015) and \(\gamma_{cat}\)Mathet (2017) as implemented in the pygamma-agreement (v0.5.9) package Titeux and Riad (2021) to better account for span-level identification and classification with overlaps. The IAA on the final dataset is \(\gamma=0.6240\) (_span identification_) and \(\gamma_{cat}=0.5445\) (_span classification_), which is comparable to results reported on the related span-level propaganda detection task Da San Martino et al. (2019b). By taking a closer look at the IAA over rounds (Figure 2), we notice that the difference between scores _before_ and _after_ discussion becomes less pronounced over rounds. This indicates that annotators increasingly align to each other, but also that annotating fallacies at the span level inherently calls for multiple discussions. At the same time, we notice that \(\gamma\) and \(\gamma_{cat}\) upper bounds (Figure 2; _green line_) are rather stable over rounds (i.e., \([0.6,0.7]\) and \([0.5,0.6]\) for \(\gamma\) and \(\gamma_{cat}\), respectively), confirming that human label variation can be resolved only partially. Overall, identifying fallacy spans appears to be the main bottleneck for IAA, since finding and assigning a label to a text segment decreases the IAA only minimally (cf. Figure 2; _left_ vs _right_). Finally, we can see that annotators are more resistant to find a consensus after the initial rounds (cf. Figure 2; _fourth_ vs _fifth_ round). We hypothesize that this is due to their increasing understanding of fallacies' nuances. We report the IAA across fallacies in Table 2, in which _Doubt_ and _Slogan_ emerge as the fallacies with highest IAA whereas _Cherry picking_ and _Vagueness_ are the hardest ones to agree on.

### Data Statistics and Analysis

Overall, Faina consists of 11,064 annotated spans (5,532\({}_{\pm 253}\) per annotator) across 58,490 tokens in 1,440 social media posts. We present per-fallacy statistics on all annotations in Table 2 and refer to Appendix D for individual statistics for \(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\).

Fallacy spans have a length of 7.6\({}_{\pm 9.3}\) tokens on average, but length greatly varies across fallacy types. The shortest fallacies are those related to language use (_language fallacies_ in Tindale (2007)) such as _Loaded language_, _Name calling or labeling_, and _Slogan_ (\(<4\) tokens), whereas the longest ones are those commonly referred to as _logical fallacies_ and _fallacies of diversion_Tindale (2007), such as _Strawman_, _Cherry picking_, and _Circular reasoning_ (\(>25\) tokens). The most and least frequent fallacies are _Loaded language_ (1,242\({}_{\pm 178}\)) and _Cir

\begin{table}
\begin{tabular}{l l r c|r} \hline \hline
**Id** & **Fallacy type** & **Spans** & **Length** & \(\gamma_{cat}\) \\ \hline ah & Ad hominem & 319 & 16.0\({}_{\pm 13.3}\) & 0.6651 \\ aa & Appeal to authority & 213 & 6.4 \({}_{\pm 4.3}\) & 0.6147 \\ ac & Appeal to emotion & 2049 & 5.1 \({}_{\pm 4.9}\) & 0.7430 \\ co & Causal oversimplification & 142 & 19.0\({}_{\pm 10.7}\) & 0.5282 \\ cp & Chery picking & 94 & 28.8\({}_{\pm 12.3}\) & 0.3415 \\ cr & Circular reasoning & 20 & 26.8\({}_{\pm 11.0}\) & 0.5397 \\ do & Doubt & 482 & 16.1\({}_{\pm 11.4}\) & 0.7103 \\ ep & Evading the burden of proof & 406 & 16.2 \({}_{\pm 9.9}\) & 0.4335 \\ fa & False analogy & 239 & 22.1\({}_{\pm 13.4}\) & 0.5243 \\ fd & False dilemma & 90 & 15.9\({}_{\pm 11.0}\) & 0.5568 \\ fw & Flag waving & 393 & 4.3 \({}_{\pm 4.9}\) & 0.5735 \\ hg & Hasty generalization & 464 & 11.2 \({}_{\pm 8.0}\) & 0.4980 \\ ll & Loaded language & 2,484 & 2.5 \({}_{\pm 2.7}\) & 0.4365 \\ nc & Name calling or labeling & 1,124 & 2.6 \({}_{\pm 17.7}\) & 0.5566 \\ rik & Red herring & 257 & 13.0 \({}_{\pm 8.5}\) & 0.4378 \\ ss & Slippery slope & 172 & 10.8 \({}_{\pm 6.8}\) & 0.6552 \\ sl & Slogan & 384 & 3.5 \({}_{\pm 3.1}\) & 0.7101 \\ st & Strawman & 109 & 36.3\({}_{\pm 15.4}\) & 0.5570 \\ tc & Thought-terminating cliche & 285 & 5.2 \({}_{\pm 3.0}\) & 0.5305 \\ va & Vagueness & 1,338 & 9.1 \({}_{\pm 8.6}\) & 0.3701 \\ \hline  & **All** & 11,064 & 7.6 \({}_{\pm 9.3}\) & 0.5445 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics and per-class IAA scores (\(\gamma_{cat}\)) across all fallacy types. We report the number of _spans_ and their average _length_ (with standard deviation) at the token level considering all annotations. Individual statistics for both \(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\) are provided in Appendix D.

Figure 2: Inter-annotator agreement (IAA) scores for both span identification (\(\gamma\)) and classification (\(\gamma_{cat}\)) at each annotation round, _before_ and _after_ discussion.

 

_cular reasoning_ (\(10{\pm}2\)), respectively. The number of annotated spans and average token length for each fallacy type varies between annotators. For instance, _False analogy_ has been annotated more by \(\mathcal{A}_{1}\) than by \(\mathcal{A}_{2}\) (147 _vs_ 92 spans), but on average \(\mathcal{A}_{2}\) labeled longer text segments than \(\mathcal{A}_{1}\) for that fallacy type, also with a higher standard deviation (24.1\({}_{\pm 13.8}\)_vs_ 20.8\({}_{\pm 13.0}\) tokens) (Appendix D).

The Faina dataset has dense annotation (3.8\({}_{\pm 0.2}\) spans/post) and overlaps among fallacy spans are also very frequent. In Figure 3 we show the percentage of pairwise overlaps at the token level across fallacy types considering all annotations (individual figures for \(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\) are in Appendix D). The percentage of tokens without overlaps is also shown (Figure 3; _diagonal cells_): for all fallacy types except aa, ae, ep, and sl, at least half of the tokens co-occur with other fallacies' tokens. Among the most frequent overlaps are the fallacies _Thoughtterminating cliche_ and _Appeal to emotion_ (23%\({}_{\pm 2}\)), since emotional words are often used in thoughtstopping discussions.

## 5 Experiments

In this section, we present experiments on fallacy detection using the Faina dataset. We first introduce our experimental setup (Section 5.1) and the approaches we employed (Section 5.2). We then provide results and a thorough discussion, including insights for future work (Section 5.3).

### Experimental Setup

TasksWe cast fallacy detection into different tasks along two dimensions: the annotation unit, i.e., post-level (post) _vs_ span-level (span), and the classification granularity, i.e., coarse-grained with 3 fallacy macro-categories (c; Section 3)6_vs_ fine-grained with all the 20 fallacy types (f). As a result, we deal with four subtasks of increasing complexity (i.e., post-c, post-f, span-c, and span-f) implying the use of different evaluation metrics as detailed in the following.

Footnote 6: For post tasks, we assign to the post the set of unique fallacy span types occurring in the message. For coarse-grained setups, we map each fallacy type to its corresponding macro-categories according to our taxonomy (Figure 4).

Data splitsFor the sake of the experiments, we divide Faina into \(k\)train and test data splits using \(k\)-fold cross validation (\(k=5\)). We use the traindata portions for pretrained models to fine-tune, and use the testportion for evaluation. For model selection, we rely on the trainsplits, further diving them into train(80%) and dev (20%), and selecting the best model configuration based on average performance on dev data portions. Each data split contains annotations for the same posts, at either the post or span level, by both annotators.

Evaluation metricsWe evaluate performance using different flavors of precision, recall, and F\({}_{1}\) score to capture the diverse challenges of each task. For posttasks, in which the set of fallacies expressed in a post must be predicted, we use standard micro-averaged scores. For the more challenging spantasks, in which all text segments expressing fallacies within a post must be identified and classified, we instead adopt precision, recall, and F\({}_{1}\) variants proposed by Da San Martino et al. (2019b) and extend them to operate on tokens.7 We use such variants since they are expressly designed for tasks with spans of varying length that may overlap. Moreover, to account for the severity of labeling errors (e.g., predicting _Red herring_ instead of _Appeal to authority_ is less problematic than predicting _False dilemma_), we compute results using both _strict_ and _soft_ evaluation modes in the span-f task. Concretely, while in strict mode we reward models only if they predict the intended label, in soft mode

Figure 3: Overlap of fallacy annotations in terms of token percentages. Each row indicates the percentage of tokens for a given fallacy type that overlaps with any other fallacy type (_columns_). White cells (_diagonal_) indicate the percentage of tokens for each fallacy type that does not overlap with any other fallacy type. Fallacy overlaps for \(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\) annotations are in Appendix D.

 

we give them partial credit if the predicted label is an immediate parent of the actual label (Figure 4). This is achieved by setting \(\delta=0.5\) as coefficient for partial label matches in the distance function of the metric by Da San Martino et al. (2019b).

Multiple gold standardsFaina consists of multiple parallel annotations (i.e., _multiple views_) that are equally reliable. To equally account for all test set versions while avoiding to favor those with more/less annotations (and thus avoiding to favor models that over/under-predict fallacies), we macro-average scores on individual test sets. The simplicity of this approach makes it suitable for extending evaluation on future test set versions.

### Models

We perform experiments on all tasks using different models in supervised and unsupervised settings.

Supervised modelsSince each data instance in Faina has multiple "ground truths", i.e., parallel labels by annotators \({\mathcal{A}}_{1},{\mathcal{A}}_{2}\in{\mathcal{A}}\), in the supervised setting we aim to jointly model all \(|{\mathcal{A}}|\) annotation versions (hereafter, _views_) to account for human label variation. We therefore adopt a multi-task learning Caruana (1997) approach to leverage signals of different but related _views_ through a shared encoder and \(D\) decoders. Specifically, for post tasks we propose a _multi-view, multi-label_ (MVML) model that uses \(D=|{\mathcal{A}}|\) decoders and thus outputs \(D\) sets of predicted labels, one for each _view_ containing all labels that exceed a threshold \(\tau\). For span tasks, which are sequence labeling problems relying on the bio-tagging scheme, we instead propose a _multi-view, multi-decoder_ (MVMD) model, with a separate decoder for each _view_ and fallacy type \(f\in{\mathcal{F}}\) (i.e., \(D=|{\mathcal{A}}\times{\mathcal{F}}|\)). We give equal importance to all decoders, i.e., computing the multi-task learning loss as \(L=\sum_{d}\lambda_{d}L_{d}\), where \(L_{d}\) is the loss for task \(d\in D\) and \(\lambda_{d}=1\) the corresponding weight. For all tasks, we use widespread models pretrained on Italian data as encoders, namely AlBERTo Polignano et al. (2019) and UmBERTo Parisi et al. (2020), leading to MVML-alb and MVML-umb models for post tasks, and MVMD-alb and MVMD-umb for span tasks. We implement our models using the MaChAmp v0.4 toolkit van der Goot et al. (2021) and adopt default hyper-parameter values (Appendix E).

Unsupervised modelsGiven the challenging nature of our tasks, we further assess classification performance with instruction-tuned LLMs in a zero-shot setting. We experiment with LLaMa-38B Dubey et al. (2024) and Mixtral 8x7B Jiang et al. (2024) to favor reproducibility, as they are freely available and widely used models. Moreover, both include Italian in their pretraining data. We design prompts that describe each task and output format, and either include fallacy definitions (Section 3) or just fallacy names (see Appendix E).8 During model selection, we observe that including definitions increased performance on the dev splits. Therefore, we select _zero-shot_ models _with definitions_ (ZSWD) for testing, i.e., ZSWD-llama and ZSWD-mixtr. Being unsupervised, these models naturally yield a single output for each data instance. We thus compare this output against all \(|{\mathcal{A}}|\) data instance _views_ during evaluation. We use the Hugging Face library and employ default model hyperparameters.

Footnote 8: We tested our prompts in both English and Italian. However, our preliminary experiments showed that prompts in Italian led to inconsistent outputs with regards to fallacy names. We thus employ prompts in English to favor I/O consistency.

### Results and Discussion

Results across all task setups are reported in Table 3 (individual results are in Appendix E). We observe

Figure 4: Taxonomy of fallacy types that we designed for evaluation purposes. Labels below the root (i.e., _Insufficient proof_, _Simplification_, and _Distraction_) represent the fallacy macro-categories for post-c and span-c tasks.

 

that our _multi-view_ approaches (i.e., MVML-* and MVMD-*) yield promising results both in post- and span-level tasks, offering well-performing baselines upon which future approaches can be built. Concerning the encoders, UmBERTo (i.e., *-umb models) appears unstable on fine-grained tasks, while it achieves similar performance to Albertato (i.e., *-alb models) in coarse-grained classification. The difference in performance between the two may be due to the former being pretrained on the Italian portion of the OSCAR corpus Suarez et al. (2019), while AlBERTo is trained exclusively on tweets, and thus is more in line with Faina data. As expected, classification at the span level with fine-grained labels (i.e., span-f) is the hardest task setup, although the MVMD-alb model achieves a F\({}_{1}\) score of 33.3 and 37.0 using _strict_ and _soft_ evaluation modes, respectively. These results are in line with performance on span-level tasks of similar complexity Da San Martino et al. (2019) and can be seen as strong baselines considering that 20 fallacy types are involved. We also observe that high recall is more challenging than high precision, probably because of the presence of multiple labels across partially and fully-overlapping spans.

As regards the second set of experiments using generative LLMs in a zero-shot setting (i.e, ZSWD-* models), results appear unreliable. While we acknowledge that fine-tuning approaches cannot be directly compared with zero-shot classification, our main goal was instead to assess to what extent we can expect to challenge traditional supervised approaches with zero-shot generative models.9 Our results show that a complex task like fallacy detection, which involves capturing lexical, semantic, and even pragmatic aspects of communication, is still far from being addressed with generative models, especially if we aim at embracing human label variation. In addition to the low performance across the challenging span setups (e.g., F\({}_{1}\) of 3.4--5.0 and 4.2--6.5 on the span-f task for ZSWD-llama and ZSWD-mixtr, respectively), LLMs are also more computationally expensive than the other proposed models, making them impractical in scenarios in which computational time is a requirement or large computational resources are not available.

Footnote 9: We also tested LLMs in few-shot settings, but preliminary experiments showed that models tended to replicate the characteristics of the example(s) provided, such as span lengths and the number of fallacies, making the results untursworthy. Future research is still needed to study and mitigate the brittleness of the few-shot approach in similar setups as ours.

To further analyze the behavior of LLMs in fallacy detection, we conduct a manual analysis of the outputs of ZSWD-llama and ZSWD-mixture across all task setups. We sample 50 outputs for each model and setup, for a total of 400 samples. We first audit them according to whether they provide an actual _answer_, _extra instructions_, _both_, or an _empty_ response (Figure 5). Then, we analyze instances where an actual answer is provided, auditing if the output is in the requested format (_format ok_), provides extra _explaining_ations, _wrong labels_, or repetitions (_repeat_) (Figure 6).

For span tasks, ZSWD-llama provides few answers (20--34%; _answer+both_ in Figure 4(a)) compared to ZSWD-Mixtr despite being instrumented, and instead mainly generates prompt continuations (80--96%; _extra instructions+both_ in Figure 4(a)). ZSWD-mixtr appears more robust, producing up to 98% answers (_answer+both_ in Figure 4(a)), of which 92--94% in the requested format (_format ok_ in Figure 5(a)). However, the results obtained with ZSWD-mixtr (Table 3) are just slightly higher than those of ZSWD-llama,

\begin{table}
\begin{tabular}{c l c r r} \hline  & **Model** & **P** & **R** & \multicolumn{1}{r}{F\({}_{1}\)} \\ \hline  & MVML-alb & 80.0\({}_{\pm 1.5}\) & 74.0\({}_{\pm 2.3}\) & **76.8\({}_{\pm 1.6}\)** \\  & MVML-umb & 84.5\({}_{\pm 1.3}\) & 70.1\({}_{\pm 4.2}\) & 76.6\({}_{\pm 2.8}\) \\  & ZSWD-llama & 57.9\({}_{\pm 1.9}\) & 70.0\({}_{\pm 1.9}\) & 63.3\({}_{\pm 1.5}\) \\  & ZSWD-mixtr & 64.7\({}_{\pm 1.6}\) & 45.2\({}_{\pm 1.0}\) & 53.2\({}_{\pm 1.2}\) \\ \hline  & MVML-alb & 63.0\({}_{\pm 2.0}\) & 34.3\({}_{\pm 1.9}\) & **44.3\({}_{\pm 1.9}\)** \\  & MVML-umb & 39.0\({}_{\pm 3.7}\) & 14.6\({}_{\pm 1.6}\) & 21.3\({}_{\pm 2.2}\) \\  & ZSWD-llama & 20.9\({}_{\pm 1.5}\) & 24.3\({}_{\pm 2.3}\) & 22.5\({}_{\pm 1.8}\) \\  & ZSWD-mixtr & 26.0\({}_{\pm 1.8}\) & 18.1\({}_{\pm 1.4}\) & 21.4\({}_{\pm 1.5}\) \\ \hline  & MVMD-alb & 55.2\({}_{\pm 1.7}\) & 51.7\({}_{\pm 2.1}\) & 53.3\({}_{\pm 1.4}\) \\  & MVMD-umb & 59.8\({}_{\pm 1.5}\) & 50.4\({}_{\pm 2.4}\) & **54.7\({}_{\pm 1.5}\)** \\  & ZSWD-llama & 25.3\({}_{\pm 4.2}\) & 7.0\({}_{\pm 0.8}\) & 10.9\({}_{\pm 0.9}\) \\  & ZSWD-mixtr & 31.6\({}_{\pm 1.2}\) & 20.9\({}_{\pm 1.4}\) & 25.1\({}_{\pm 1.2}\) \\ \hline  & _Strict mode_ & & & \\  & MVMD-alb & 47.6\({}_{\pm 1.9}\) & 25.6\({}_{\pm 1.6}\) & **33.3\({}_{\pm 1.4}\)** \\  & MVMD-umb & 57.5\({}_{\pm 5.9}\) & 3.9\({}_{\pm 0.7}\) & 7.3\({}_{\pm 1.3}\) \\  & ZSWD-llama & 4.5\({}_{\pm 0.5}\) & 2.7\({}_{\pm 0.4}\) & 3.4\({}_{\pm 0.3}\) \\  & ZSWD-mixtr & 5.8\({}_{\pm 1.1}\) & 3.2\({}_{\pm 0.5}\) & 4.2\({}_{\pm 0.7}\) \\  & _Soft mode_ & & & \\  & MVMD-alb & 52.2\({}_{\pm 2.0}\) & 28.7\({}_{\pm 1.7}\) & **37.0\({}_{\pm 1.5}\)** \\  & MVMD-umb & 66.3\({}_{\pm 5.5}\) & 4.8\({}_{\pm 0.7}\) & 8.9\({}_{\pm 1.3}\) \\  & ZSWD-llama & 6.4\({}_{\pm 0.6}\) & 4.2\({}_{\pm 0.5}\) & 5.0\({}_{\pm 0.4}\) \\  & ZSWD-mixtr & 8.2\({}_{\pm 1.5}\) & 5.4\({}_{\pm 1.0}\) & 6.5\({}_{\pm 1.1}\) \\ \hline \end{tabular}
\end{table}
Table 3: Test set results for post and span tasks at the _coarse-grained_ (c) and _fine-grained_ (f) level. We report average precision (P), recall (R), and F\({}_{1}\) scores (w/ std dev) across \(k=5\) splits, averaged over all \(|\mathcal{A}|\) test versions. For span-f, we also present scores using both _strict_ and _soft_ modes. Best results are in bold. Results on individual test sets (\(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\)) are in Appendix E.

 

suggesting that while ZSWD-mixtr produces answers in most cases, those are not actually reliable.

As a comparison, we report the results for post tasks in Figure 4(b) and 5(b). We observe that the general trends highlighted for the span tasks still hold also for this setting. Overall, this qualitative analysis indicates that future work is needed for dealing with complex tasks such as fine-grained fallacy detection with LLMs in zero-shot setups.

## 6 Conclusions

We introduced Faina, the first fallacy detection dataset embracing multiple plausible answers and natural disagreement at the fine-grained level of text segments. Faina advances research on human label variation in NLP and opens new avenues for research. Given its multi-topic and multi-year nature, it can be used to benchmark fallacy detection approaches in out-of-domain scenarios and across time. Moreover, our annotation paradigm and guidelines can be applied to cover new languages, topics, and additional annotators. Lastly, Faina can be used for novel work on cross-lingual annotation transfer in which source and target languages are shifted, e.g., from Italian to English.

### Limitations

Our dataset contains tweets in one language. Although the annotation scheme can be easily adopted for other languages, the findings and insights obtained from the experiments may not hold on other data sources and languages. For dataset creation and experiments with human label variation we rely on two annotators. We are aware that more perspectives could have been leveraged using crowdsourcing platforms. However, we opted to involve only expert annotators to prioritize annotation reliability, further providing the full annotation guidelines and details on the annotation protocol to encourage future extensions. The main focus of the paper is on the resource creation and assessment and its fine-grained evaluation; therefore, we employed a limited set of models in our experiments. The performance evaluation of additional models and the fine-tuning of LLMs is a research direction we would like to pursue in future work.

### Ethics Statement

The Faina dataset has been created by paying particular attention to data minimization principles

Figure 5: Analysis of the **raw outputs** generated by LLMs across the four tasks according to whether they contain an actual _answer_, _extra instructions_, _both_ an actual answer and extra instructions, or an _empty_ response.

 

and complying with privacy requirements. Data was collected using the Twitter APIs when they were still freely available for research purposes, and up to one tweet per user has been retrieved for each month and topic. It is therefore impossible to profile users, and this also avoids the users' writing style to interfere with classification performance.

To follow good scientific practices, the Faina dataset is released including only the post texts, with no user information. Indeed, while in the past Twitter datasets were commonly released as a list of tweet IDs, this would make it hard to replicate our work because free Twitter APIs have been discontinued. The dataset is released in its anonymized form, after replacing user mentions, email addresses, phone numbers and URLs with placeholders, as described in Appendix A. To download Faina, it is necessary to fill in an online form declaring compliance with user protection regulations and exclude data misuse.

## Acknowledgments

This work was funded by the European Union's Horizon Europe research and innovation programme under grant agreement No. 101135437 (AI-CODE). We acknowledge also the support of the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by NextGeneration EU.

## References

* Alhindi et al. [2022] Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, and Smaranda Muresan. 2022. Multitask instruction-based prompting for fallacy recognition. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8172-8187, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Arroyo and Welty [2015] Lora Arroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. _AI Magazine_, 36(1):15-24.
* Barbera and Tortone [2012] Elena Barbera and Claudio Tortone. 2012. Glossario O.M.S. della promozione della salute. Technical report, Centro Regionale di Documentazione per la Promozione della Salute (DoRS), Grugliasco, Italy.
* Barretta et al. [2018] Paola Barretta, Piera Francesca Mastantono, and Sabika Shah Povia. 2018. _Linee Guida per l'Applicazione della Carta di Roma: Strumenti di Lavoro per un'Informazione Corretta sui Temi dell'Immigrazione e dell'Asilo_. Associazione Carta di Roma, Rome, Italy.
* Bender and Friedman [2018] Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. _Transactions of the Association for Computational Linguistics_, 6:587-604.
* Cabitza et al. [2023] Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. _Proceedings of the AAAI Conference on Artificial Intelligence_, 37(6):6860-6868.
* Caruana [1997] Rich Caruana. 1997. Multitask learning. _Machine learning_, 28:41-75.
* Da San Martino et al. [2019a] Giovanni Da San Martino, Alberto Barron-Cedeno, and Preslav Nakov. 2019a. Findings of the NLP4IF-2019 shared task on fine-grained propaganda detection. In _Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda_, pages 162-170, Hong Kong, China. Association for Computational Linguistics.
* Da San Martino et al. [2020] Giovanni Da San Martino, Alberto Barron-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 1377-1414, Barcelona (online). International Committee for Computational Linguistics.
* Dan Martino et al. [2019b] Giovanni Da San Martino, Seunghak Yu, Alberto Barron-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grained analysis of propaganda in news article. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5636-5646, Hong Kong, China. Association for Computational Linguistics.
* Dimitrov et al. [2024] Dimitar Dimitrov, Firoj Alam, Maram Hasanain, Abul Hasnat, Fabrizio Silvestri, Preslav Nakov, and Giovanni Ba San Martino. 2024. SemEval-2024 task 4: Multilingual detection of persuasion techniques in memes. In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_, pages 2009-2026, Mexico City, Mexico. Association for Computational Linguistics.
* Dubey et al. [2024]Abhimanyu Dubey, Abhinav Jauhr, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Ecker et al. [2024] Ullrich Ecker, Jon Rozenbeek, Sander van der Linden, Li Qian Tay, John Cook, Naomi Oreskes, and Stephan Lewandowsky. 2024. Misinformation poses a bigger threat to democracy than you might think. _Nature_, 630(8015):29-32.
* Goffredo et al. [2023]Pierpaolo Goffredo, Mariana Chaves, Serenna Villata, and Elena Cabrio. 2023. Argument-based detection and classification of fallacies in political debates.

 

In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 11101-11112, Singapore. Association for Computational Linguistics.
* Goffredo et al. (2022) Pierpaolo Goffredo, Shohreh Haddadan, Vorakit Vorrakittphan, Elena Cabrio, and Serena Villata. 2022. Fallacious argument classification in political debates. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_, pages 4143-4149. International Joint Conferences on Artificial Intelligence Organization.
* Habernal et al. (2017) Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argument meets serious games. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 7-12, Copenhagen, Denmark. Association for Computational Linguistics.
* Habernal et al. (2018a) Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018a. Adapting serious game for fallacious argumentation to German: Pitfalls, insights, and best practices. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Habernal et al. (2018b) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018b. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 386-396, New Orleans, Louisiana. Association for Computational Linguistics.
* Hamblin (2022) Charles Leonard Hamblin. 2022. _Fallacies_. Advanced Reasoning Forum, Socorro, USA.
* Hautli-Janisz et al. (2022) Annette Hautli-Janisz, Ella Schad, and Chris Reed. 2022. Disagreement space in argument analysis. In _Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022_, pages 1-9, Marseille, France. European Language Resources Association.
* Hayes and Krippendorff (2007) Andrew F. Hayes and Klaus Krippendorfff. 2007. Answering the call for a standard reliability measure for coding data. _Communication Methods and Measures_, 1(1):77-89.
* Jiang et al. (2022) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mxtral of experts. _arXiv preprint arXiv:2401.04088_.
* Jin et al. (2022) Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mnimaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 7180-7198, Abu Dhabi, United Arab Eminates. Association for Computational Linguistics.
* Klie et al. (2018) Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEPIION platform: Machine-assisted and knowledge-oriented interactive annotation. In _Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations_, pages 5-9, Santa Fe, New Mexico. Association for Computational Linguistics.
* Latinii et al. (2020) Gianni Latini, Marco Bagliani, and Tommaso Orusa. 2020. _Lessico e Nuvole: Le Parole del Cambiamento Climatico_, 2nd edition. Universita degli Studi di Torino, Turin, Italy.
* Lindahl (2024) Anna Lindahl. 2024. Disagreement in argumentation annotation. In _Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024_, pages 56-66, Torino, Italia. ELRA and ICCL.
* Macagno (2022) Fabrizio Macagno. 2022. Argumentation schemes, fallacies, and evidence in politicians' argumentative tweets--a coded dataset. _Data in Brief_, 44:108501.
* Mathet (2017) Yann Mathet. 2017. The agreement measure \(\gamma\)cat a complement to \(\gamma\) focused on categorization of a continuum. _Computational Linguistics_, 43(3):661-681.
* Mathet et al. (2015) Yann Mathet, Antoine Widlocher, and Jean-Philippe Metivier. 2015. The unified and holistic method gamma (\(\gamma\)) for inter-annotator agreement measure and alignment. _Computational Linguistics_, 41(3):437-479.
* Health (2019) Ministry of Health. 2019. Glossario in materia di liste di attessa. [https://www.salute.gov.it/ings/C_17_publicizationi_2824_ulterioriallegati_uleriorellegato_2_alleg.pdf](https://www.salute.gov.it/ings/C_17_publicizationi_2824_ulterioriallegati_uleriorellegato_2_alleg.pdf). Accessed: 2024-01-01.
* Musi et al. (2022) Elena Musi, Myrto Aloumpi, Elinor Carmi, Simeon Yates, and Kay O'Halloran. 2022. Developing fake news immunity: Fallacies as misinformation triggers during the pandemic. _Online Journal of Communication and Media Technologies_, 12(3):e202217.
* Conference and Labs of the Evaluation Forum_. CEUR-WS.org.
* Nie et al. (2020) Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9131-9143, Online. Association for Computational Linguistics.
* Parisi et al. (2020) Loreto Parisi, Simone Francia, and Paolo Magnani. 2020. UmBERTo: An Italian language model trained with whole word masking. [https://github.com/musicmatchresearch/umberto](https://github.com/musicmatchresearch/umberto). Accessed: 2024-01-01 

Harry Phillips and Patricia Bostian. 2011. _The purposeful argument: A practical guide, brief edition_, 1st edition. Cengage Learning, Boston, USA.
* Piskorski et al. (2023a) Jakub Piskorski, Nicolas Stefanovitch, Valerie-Anne Bausier, Nicolo Faggiani, Jens Lange, Sopho Kharazi, Nikolaos Nikolaidis, Giulia Teodori, Bertrand De Longueville, Brian Doherty, Jason Gonin, Camelia Ignat, Bonka Kotesva, Eleonora Mantica, Lorena Marcaletti, Enrico Rossi, Alessio Spadaro, Marco Verlie, Giovanni Da San Martino, Firoj Alam, and Preslav Nakov. 2023a. News categorization, framing and persuasion techniques: Annotation guidelines. Technical Report JRC132862, European Commission, Ispra, Italy.
* Piskorski et al. (2023b) Jakub Piskorski, Nicolas Stefanovitch, Nikolaos Nikolaidis, Giovanni Da San Martino, and Preslav Nakov. 2023b. Multilingual multifaceted understanding of online news in terms of genre, framing, and persuasion techniques. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3001-3022, Toronto, Canada. Association for Computational Linguistics.
* Plank (2022) Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 10671-10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Poesio and Artstein (2005) Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In _Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky_, pages 76-83, Ann Arbor, Michigan. Association for Computational Linguistics.
* Polignano et al. (2019) Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. Alberto: Italian BERT language understanding model for NLP challenging tasks based on tweets. In _Proceedings of the Sixth Italian Conference on Computational Linguistics_, Bari, Italy. CEUR-WS.org.
* Ramponi (2024) Alan Ramponi. 2024. Language varieties of Italy: Technology challenges and opportunities. _Transactions of the Association for Computational Linguistics_, 12:19-38.
* Ramponi et al. (2024) Alan Ramponi, Camilla Casula, and Stefano Menini. 2024. Variationist: Exploring multifaceted variation and bias in written language data. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_, pages 346-354, Bangkok, Thailand. Association for Computational Linguistics.
* Sahai et al. (2021) Saumya Sahai, Oana Balalau, and Roxana Horincar. 2021. Breaking down the invisible wall of informal fallacies in online discussions. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 644-657, Online. Association for Computational Linguistics.
* Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. "nice try, kiddo": Investigating ad hominems in dialogue responses. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 750-767, Online. Association for Computational Linguistics.
* Shultz (2024) Benjamin Shultz. 2024. An entity-aware approach to logical fallacy detection in kremlin social media content. In _Proceedings of the 2023 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_, page 780-783, Kusadasi, Turkey. Association for Computing Machinery.
* Suarez et al. (2019) Pedro Javier Ortiz Suarez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In _7th Workshop on the Challenges in the Management of Large Corpora_, pages 9-16, Cardiff, UK. Leibniz-Institut fur Deutsche Sprache.
* Tindale (2007) Christopher W Tindale. 2007. _Fallacies and Argument Appraisal_. Critical Reasoning and Argumentation. Cambridge University Press, Cambridge, UK.
* Titeux and Riad (2021) Hadrien Titeux and Rachid Riad. 2021. pygammagreenent: Gamma \(\gamma\) measure for inter/intra-annotator agreement in python. _Journal of Open Source Software_, 6(62):2989.
* UNHCR (2020) UNHRC. 2020. Glossario dei termini. [https://www.unhcr.org/it/wp-content/uploads/sites/97/2020/07/GLOSSARIO.pdf](https://www.unhcr.org/it/wp-content/uploads/sites/97/2020/07/GLOSSARIO.pdf). Accessed: 2024-01-01.
* van der Goot et al. (2021) Rob van der Goot, Ahmet Ustun, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multitask learning in NLP. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_, pages 176-197, Online. Association for Computational Linguistics.
* Van Eemeren and Grootedorstad (2004) Frans H. Van Eemeren and Rob Grootedorstad. 2004. _A systematic theory of argumentation: The pragmadiactical approach_. Cambridge University Press, Cambridge, UK.
* Organization (2021) World Health Organization. 2021. Health promotion glossary of terms 2021. Technical report, World Health Organization, Geneva, Switzerland.

 



## Appendix A Data Statements

We present data statements (Bender and Friedman, 2018) for Faina in the following.

Curation rationale.The dataset consists of anonymized social media messages from Twitter with fallacy annotations at the span level. The posts were collected using search keywords related to migration, climate change, and public health whose use is generally not negatively/positively connoted to minimize the over-representation of specific stances on the topics (see Appendix B). The dataset was created to study fallacious argumentation in social media, to educate about critical thinking, and to encourage research on embracing human label variation in NLP. The dataset is in a CoNLL-like format with individual annotators' labels. Further details on data creation and annotation are in Section 4.

Language varieties.The language represented in the dataset is Italian (ita) in the form of spontaneous written speech. Rare instances (\(<1\%\)) also exhibit code-switching between Italian and local language varieties of Italy (Ramponi, 2024).

Speaker demographic.The corpus consists of anonymized social media posts and therefore user demographics are unknown.

Annotator demographic.The annotators are native speakers of Italian with background in linguistics and NLP. They identify themselves as a woman and a man, with age ranges 20-30 and 30-40. Both have naturally been exposed to public discourse around migration, climate change, and public health issues in the Italian context. They carried out data annotation as part of their work as employees at the host institution.

Speech situation and text characteristics.The interaction is asynchronous and the speakers' intended audience is everyone. The genre of the written texts is social media with a focus on migration, climate change, and public health issues. The posts have been published within a 4-year time period (i.e., from 2019-01-01 to 2022-12-31) and thus temporal biases in the dataset are minimized. The posts have been collected in February 2023.

Preprocessing and data formatting.The posts have been anonymized by replacing user mentions, email addresses, phone numbers, and URLs with placeholders (i.e., [USER], [EMAIL], [PHONE] and [URL], respectively). All emojis have been preserved in the text since they frequently signal _language fallacies_, whereas newline characters (i.e., \(\backslash{n}\), \(\lor{r}\)) have been replaced with single spaces.

## Appendix B Search Keywords

The search keywords have been selected from publicly-available glossaries, manuals, and reports produced by universities, agencies, and associations that deal with migration, climate change, and public health issues. Keyword selection was conducted ensuring to cover various subtopics, and each term/phrase was extended to cover all applicable grammatical genders and numbers. We report the original sources below and refer the reader to Table 4 for the full set of keywords across topics.

MigrationWe use the _United Nations High Commissioner for Refugees_' glossary in Italian (UNHCR, 2020) and the _Guidelines for the Application of the Charter of Rome_(Barretta et al., 2018). The resulting keywords represent the following aspects of migration: _i)_ phenomena, _ii)_ people, and _iii)_ status and hospitality (see Table 4).

Climate changeWe rely on _The Words of Climate Change_, a linguistic manual that includes the definition for over 200 concepts in Italian by 82 authors about climate change across 30 diverse subject areas (Latini et al., 2020). Those include climate change concepts that span environmental, climate, energy, chemical, physical, social, and economic subject areas, among others (see Table 4).

Public healthWe mainly rely on the _Health Promotion Glossary of Terms_ by the World Health Organization (2021) by manually translating terms or finding corresponding translations in the Italian version of the glossary (Barbera and Tortone, 2012). To broaden the scope of the search, we also draw keywords about specific public health areas from the _Glossary on the Subject of Waiting Lists_ by the Italian Ministry of Health (2019), from a glossary on health inequalities,10 and from the "Themes" section of the Italian Ministry of Health website.11 The resulting keywords are in Table 4.





\begin{table}
\begin{tabular}{l l} \hline
**Topic** & **Search keywords** \\ \hline Migration & apolid[e,i]; apploidia; centr[o,i] di accoglienza; centr[o,i] di identificazione ed espulsione; centr[o,i] di permanenza per il rimpatrio; centri di permanenza per i rimpatria; centr[o,i] di parmanenza temporanea; centr[o,i] per il rimpatrito; centri per i rimpatri; corridio[io,i] umanitar[io,i]; domand[a,e] d’asilo; domand[a,e] di asiló; emisrant[e,i]; emigratta[o,i,a,e]; emigrazion[e,i]; espart[io,i]; fattor[e,i] di spinta; immigrant[e,i] immigrar[o,i,a,e]; immigraion[e,i]; its sanguinia; migrant[e,i]; migrator[io,i,ia,ie]; migrazion[e,i]; minor[e,i] stranier[o,i] non accompagnat[o,i];minor[e,i] stranier[a,e] non accompagnat[a,e]; non-refoulemen[t,ts]; permess[o,i] di soggiorno; procedur[a,e] d’asilo; procedur[a,e] di asilo; protezioni[e,i] sussidiari[a,e]; protezioni[e,i] unanimitar[a,e]; push factot[rs]; refoulement[t,ts]; reinselament[o,i]; respingiment[o,i]; richcident[e,i] asilo; rifugialt(o,i,a,e]; rimparti[o,i,j]; rimparti[o,i,a,e]; sdfall[o,i,a,e]; vittim[a,e] della tratta; vitamin[a,e] di tratta \\ \hline Climate & acidificazione del! oceano; acidificationzo degli oceani; aerosol atmósferici[o,i]; allagamento[o,i]; alluvialon[e,i]; alluvolingual[e,i]; ambientalismo di facciata; anidride carbonicica; antropocene; ardidix; bilanc[io,i] climatic[o,i]; inelastic[o,i]; biomcloi[o,i]; biocomstubii[e,i]; biodegradabili[e,i]; biodegradationabitzi; biossolido di carbonio; cambiac[o,i] climate[o,i]; campiancent[o,i] del climax; carbon cost; conbortript; carbon pricing; carbon tax; costoft(i) del carbino; climate change; climate[si,es]; climatic[oa,i,he];climatopia[o,i]; co2; combustibull[e,i] fossil[e,i]; confin[e,i] planetar[io,i]; consumq[o,i] di suolo; crisi climatic[a,he]; deforestzation[e,i]; desalinizzazolen[e,i]; desertificazion[e,i]; diossido di carbonio; disboscament[o,i]; disssalizonic[e,i]; ecological footprint; ecologsimo di faceatica; economi[a,e] circolar[e,i]; efetto serra; emission[e,i]; energi[a,e] runnovable[e,i]; esondazion[e,i]; event[o,i] meteorologic[o,i] estrem[o,i]; fenomen[o,i] meteorologic[o,i] estrem[o,i]; finanza scostenibile; fonto di energira innovabibile; fonti di energia minovability[e,i]; forzant[e,i] radiativ[o,i]; gas serra; gassivste; glacialism[o,i]; glaciazione[e,i]; greenwashing; importa carbonate; improntia di carbonio; imprometa codojcica; innalzzamento de[i,1] mar[e,i]; innalazzimento del livelello[d,i] mar[e,i]; innalazzimento atomosferico; inquaimento del’altmosferic; isol[a,e] di alcarco; islo[a] e urbanda el calore; limit[e,i]; planetar[io,i]; planeterologia; microclimate; moobilia stostiniele; muamen[o,i] climatic[o,i]; olicene; onodatal[a,e] di caldo; onodat[a,e] di calore; paleoclimaria; particellato; particolato; pedoclima; permafrost; permaggelo; prezzo[o,i] del carbonio; prozeizon[e,i] climatic[a,he]; report di ostensibilita; riscaldamento climatico; riscaldamento globale; risch[io,i] climatic[o,i]; scenario[io,i] climatic[o,i]; ciscagitmento[,i] de gihiaccai; sickita; sistem[a,i] climatic[o,i]; sostenibilita ambientale; surirscaldamento climatico; surfiscaldamento globale; svilupp[o,i] sostenibibili[e,i]; tassi[a,e] sul carbonio; transizion[e,i] coolge[a,he]; transizion[e,i] energetic[a,he]; usao [el,i] suolo; utilizzone[e,i] solo; etulizzo@le[i,i] suolo; variabilita climatica[,he] \\ \hline Public & agend[a,e] di prenotaxione; afabetizzzione alla salute; afabetizzzione sanitaria; assistenz[a,e] dominichifar[e,i]; assistenz[a,e] opseddier[a,e]; assistance[a,e] sanitarian[a,e]; assistenza[a,e] sailteri[a,e]; assignenza universalez; azien[a,e] o-spedalier[a,e]; aziendi[a,e] sanitarian[a,e]; bisogn[o,i] sanitary[io,i]; calendar[o,i] diprendozione; carci[o,hi] di malatti; centro unitofica ot ripentazione; citata sam[a,e]; class[e,i] di priori; comportament[o,i] a nischo; comportament[o,i] di salute; copertur[a,e] sematurizi[a,e]; copperur[a,e] unlessurl[e,i]; cur[a,e] incredit[a,he]; curl[a,e] caniarti[a,e]; depegenti[e,i]; degenz[a,e]; determinant[e,i] della salute; determinante[e,i] di salute; dimission[e,i] opedadieri[a,e]; chpositivo[i,i] medic[o,i]; disuguagliano[a,e] di salute; disuguagliano[a,e] nella salute; dibaugalian[a,e] nulla salute; disuguagliano[a,e] sanitarian[a,e]; educazione annlataria;epidemi[a,e]; epidemicmic[o,a,i,he]; epidememiologicalo[a,a,i,he]; equtit di salute; equtità mella salute; equiña[e,e]; sensizcon[e,i] dl ticket; essenzion[e,i] ticket; faftor[e,i] di riskchio; indicator[e,i] di salute; investment[o,i] nella sanita; investiment[o,i] per la salute; investtiment[o,i] per la sanita; isol[a,e] san[a,e]; situtt[o,i] di cura; sitittuo di sanito publica; situtte super

Annotation Guidelines

In this section, we present the guidelines we designed for the annotation of fallacies in Faina. We first introduce the general guidelines (Section C.1), i.e., those concerning neutrality and the identification of spans regardless of fallacy types. We then present fallacy-specific guidelines and extended definitions for each fallacy type (Section C.2).

### General Guidelines

Neutrality in annotationFallacies must be identified on the basis of their argumentative invalidity, without considering the truth or falsity of the statement or whether the position advanced is ideologically agreeable. Any political or personal judgment must be set aside during annotation.

Extent of fallacy spansWe adopt a minimalist approach and annotate the smallest linguistic unit which expresses the fallacy. All overlapping fallacies must be annotated, regardless of negation. Four types of annotation spans can be identified:

1. **Word level.** The fallacy is expressed through a single word, e.g.: "il **becero** profitto" (en: _"the vulgar profit"_) [_Loaded language_];
2. **Phrase level.** The fallacy is the union of more words, e.g.: **"brigate rosese"** (en: _"red brigades"_) [_Name calling or labeling_]. Fallacies of the same type that occur close to each other must be annotated together, e.g.: **"[che razza] di padre [abbietto1]"** (en: _"[what a] [despicable] father!"_) [_Loaded language_];
3. **Clause level.** All the clause contributes to express the fallacy. To highlight the logical passage from the premise to the conclusion, we require the annotation of conjunctions, relative pronouns, and verbs, e.g.: "questo & il primo passo verso la dittatura"** (en: _"this is the first step towards dictatorship"_) [_Slippery slope_]; or "soistieni che bisogna lavorare, **ma non eri propriu tu quello che faceva i festini a casa tau?"** (en: _"You argue that people should work, but weren't you the one throwing parties at your house?"_) [_Ad hominem_];
4. **Sentence/post level.** The fallacy may be expressed through the whole content of the post and usually requires two arguments, e.g.: **"la Dora e piana di acqua, non e vero che c'e il riscaldamento globale!"** (en: _"The Dora is full of water, so it's not true that there is global warming!"_ [_Cherry picking_]. If the premise or conclusion alone is sufficient, only the necessary part must be annotated (see individual fallacy guidelines in Appendix C.2).

Special charactersPunctuation, emojis, emoticons, uppercase letters, and other graphic signs must be annotated when they carry semantic content that influences the argument or contribute to express the fallacy. Emojis and emoticons frequently convey fallacies such as _Appeal to emotion_ and _Loaded language_. Uppercase letters and exclamation points are often used in the expression of the _Loaded language_ fallacy. When annotating punctuation, contiguous marks referring to different fallacies must be annotated separately. Hashtags should be annotated including the "#" character. Some hashtags may simply serve as tags to ease the retrieval of posts about a topic and therefore do not express fallacies, whereas others are often used in _Slogan_ fallacies. Among other symbols, [USER] placeholders must be annotated if they are part of an _Appeal to authority_ fallacy. [URL] placeholders must instead be excluded from annotation.

Claims/arguments made by othersFallacies must be annotated even when part of the arguments are advanced by others. In this case, quotation marks should be excluded from annotation. Instead, reported testimones do not require annotation since they typically convey personal opinions.

Pragmatic strategiesPragmatic strategies like irony can overlap with fallacies as they both involve violations of communicative norms Van Eemeren and Grootendorst (2004). Fallacious reasoning should be annotated in contexts involving irony, following the European guidelines for annotation of persuasion techniques Piskorski et al. (2023).

### Fallacy-specific Guidelines

We here provide the extended definitions for fallacies along with fallacy-specific guidelines (). Examples for each fallacy are provided in Table 5.

Ad dominem (ah)A personal attack to an individual or a group which deviates from the main thesis. It comprises _Abusive_ AH (i.e., when the opponent's character is attacked), _Circumstantial_ AH (i.e., when the opponent is accused of being motivated by personal interest), and _Tu quoque_ (i.e., when there is a contradiction between what the op 

ponent says or does and what they have said or done) (Van Eemeren and Grootendorst, 2004).
* The attack can be addressed to a third person or group. The annotation must only include the attack itself, without the premises.

**Appeal to authority (aa)**The author appeals to an authority or a group consensus to support their thesis, without further evidence. Following Goffredo et al. (2022), we group two fallacies under this label, namely _Appeal to authority_ and _Ad populum/Bandwagon_. This fallacy also includes the cases in which the author appeals to their own authority or opinion.
* The annotation comprises the smallest segment in which the authority is mentioned, the declarative conjunction (or a colon), and additional information about the circumstances, such as the source.

**Appeal to emotion (ae)**It involves the use of negative or positive personal emotions (e.g., shame, indignation, pity, or affection) to intentionally or unintentionally influence the audience. The label also includes the persuasion technique _Appeal to fear/prejudice_ as defined by Da San Martino et al. (2019b).
* This fallacy is a subtype of _Red herring_ as it deviates the attention from the main thesis, regardless of the language in use. In contrast, the subtype _Loaded language_ involves strong language use.

**Causal oversimplification (co)** It involves a simplified and fallacious causal relation. It includes the subtypes _False cause_ and _Post hoc_ as defined by Musi et al. (2022).
* Both the premises and the conclusion must be annotated.

**Cherry picking (cp)**This fallacy consists of choosing evidence to support a thesis while ignoring any other contrary evidence (Musi et al., 2022).
* In contrast to _Hasty generalization_, where the author builds a conclusion based on the evidence, here the author selects evidence that supports a pre-existing conclusion. Both the claim that confirms the thesis and the thesis itself must be annotated.

**Circular reasoning (cr)**An error of circularity: the end of an argument comes back to the beginning without having proven itself (Jin et al., 2022).
* The entire argumentation includes the premises and the conclusion, which usually extend over multiple sentences, and that must be annotated.

**Doubt (do)** It is used to intentionally question the credibility of someone or something (Da San Martino et al., 2019b).
* It usually involves linguistic devices such as question marks, adverbs of doubt (e.g.: "**forse**", en: "_maybe_"; "**magari**", en: "_perhaps_"), adversative conjunctions (e.g.: "**ma**", en: "_but_"; "**pero**", en: "_however_"), conditional conjunctions (e.g.: "se", en: "_if_"), and rhetorical questions (e.g.: "**siamo sicuri che...?**", en: "_are we sure that...?_"; "**perche non...?**", en: "_why not....?_").

Evading the burden of proof (ep)A thesis is advanced without any support as if it was self-evident, meaning that one or more arguments are missing in the argument structure (Musi et al., 2022).
* The fallacy should not be annotated in cases when no evidence could be theoretically provided. URLs can sometimes contain more information about the statement, neutralizing the fallacy.

**False analogy (fa)**This fallacy occurs when two different things or situations are placed on the same level because they are supposed to share similar aspects. The label includes _False analogy_ as defined by Musi et al. (2022) and _False equivalence_ as defined by Phillips and Bostian (2011).
* It can also include lists of entities that are implicitly equated, e.g.: "**migranti e spazzatura**" (en: "_migrants and trash_"). The annotated span must include both/all the concepts being discussed.

**False dilemma (fd)**It presents only two options or sides when there are many (Jin et al., 2022).
* The annotated span must include both the things being presented.

**Flag waving (fw)**This fallacy occurs when the author intentionally plays on a sense of belonging to a country, a group, or an ideology to support an argument, as if waving a flag.
* The fallacy can be frequently found in hashtags. Examples include names of political parties (_e.g.: #FratellidItalia_), or manifestations and organizations (_e.g.: #FindaysForFuture_).

**History generalization (hg)**It occurs when a generalization is drawn from a sample which is too small, not representative, or not applicable to the whole situation if all the variables are taken into account (Musi et al., 2022). It is an example of _Jumping to conclusions_ (Jin et al., 2022) 

We require to annotate the generalization itself, not the premises. In some cases, the generalization just consists in an overly broad statement, where the generalized sample is not expressed.

Loaded language (ll)It involves using words or phrases with strong emotional implications (either positive or negative) to influence the audience Da San Martino et al. (2019).

Cases that are not functional to the argumentation, regardless of intentionality, are not annotated. This fallacy is often found at word level. Frequent devices are swear words (e.g.: "**cazzo**", en: "_shift_"), slang (e.g.: "**sbirri**", en: "_copys_"), evaluative terms (e.g.: "**Tamigerati_", en: "_infamous_"), colloquial expressions (e.g.: "**non frega nulla_**", en: "_they don't care_"), discourse markers (e.g.: "**dai**", en: "_come on_"; "**ovviamente**", en: "_obviously_"), rhetorical strategies (e.g.: "**ma per piacere..._", en: "_oh, please..._"), and repetitions, as well as graphic strategies like emojis, emoticons, hashtags, punctuation (e.g., exclamation marks), or uppercase letters. Instances which do not have inherent connotations but acquire a connotation in the pragmatic context should be evaluated individually by the annotator.

Name calling or labeling (nc)This fallacy involves labeling something or someone positively or negatively to influence the audience, for example associating it with an ideology. We group under this label the two propaganda techniques _Name calling or labeling_ and _Reductio ad Hitlerum_ as defined by Da San Martino et al. (2019).

The annotation includes the target, the label, and the article. In contrast with _Loaded language_, here the linguistic device specifically brings the target back to an ideological or minority group in which they do not identify.

Red herring (rh)The argument supporting the claim diverges the attention to issues which are irrelevant for the claim at hand Musi et al. (2022). It includes the subtypes _Appeal to worse problems_, _Appeal to tradition_, and _Appeal to nature_ as defined in Sahai et al. (2021).

Only the passage that deviates from the thesis must be annotated.

Slippery slope (ss)It implies that an exaggerated consequence could result from a particular action Goffredo et al. (2023).

The annotation must only include the fallacy itself, without the premises. Note that, if the exaggeration is a plausible fact supported by evidence, it should not be annotated.

Slogan (sl)It consists of a brief and striking phrase that is used to provoke excitement of the audience Goffredo et al. (2023).

This fallacy can be frequently found in hashtags. Each hashtag is annotated separately, except when a sequence of related hashtags is found. In contrast to _Flag waving_, it does not specifically plays on a sense of belonging to a group or ideology. Examples include #\(ClimateChangeIsReal\), #RiseForClimate_, #ActOnClimate.

Strawman (st)It consists of distorting someone else's argument and then tearing it down. The arguer misinterprets an opponent's argument for the purpose of more easily attacking it, demolishes it, and then concludes that the opponent's real argument has been demolished Musi et al. (2022).

We require the annotation of the reported argument, which frequently includes _Vagueness_, and its attack.

Thought-terminating cliche (tc)It consists of a short and generic phrase that discourages critical thought and meaningful discussion Da San Martino et al. (2019).

It is usually found at the end of a sentence and involves a final punctuation mark that should be annotated. Phrases created ad hoc by the post author must also be annotated, including corner cases such as "**maazzudo**" (en: "_just guessing_").

Vagueness (va)It is found when ambiguous words are shifted in meaning in the process of arguing or are left vague, being potentially subject to skewed interpretations Musi et al. (2022). The label also includes the _Equivocation_ fallacy as defined by Jin et al. (2022).

When the intentionality of the arguer is not clear, the annotator should evaluate if the content can be misinterpreted and lead to ambiguity. This often occurs with indefinite expressions such as "**molti**" (en: "_many_"), "**quasi**" (en: "_almost_"), "**circxa**" (en: "_about_"), and general extenders such as "**ecc.**" (en: "_etc._"). Vague expressions that do not play a role in influencing the audience and do not lead to potential ambiguity are not considered instances of _Vagueness_, and therefore should not be annotated (e.g.: "**dopo mesii..**", en: "_after months..._") 



**Fallacy type**: _Example_

\begin{tabular}{p{142.3pt} p{856.9pt}} Ad hominem & Gli accoglienti e unanitari francesi non vogliono piu i #migranti della \\  & **#SeaWatch3. La loro parola vale meno di una scoregia.** \\  & _The welcoming and humanitarian French no longer want the fmingrants on the \\  & _#SeaWatch3. Their word is worth less than a far._ \\ Appeal to authority & Lo dice anche la fl Bundesbank, i infmigranti servono ad abbassare i salari di tutti. \\  & _The ffBundesbank also says it, frmigrants serve to lower everyone's wages._ \\ Appeal to emotion & La manifestazione a \#Voghera e arrabbiata, diginitosa e ordintana. Sta girando perutta la città con molti immigati e l’apopggio della sinistura. **Voole giustizia.** \\  & _The demonstration in \#Voghera is angry, dignified, and orderly. It is moving through the city with many immigrants and the support of the left. It wants justice._ \\ Causal oversimplification & Ho fatto laxure l'auto. Dopo mesi di siccita, sono scivuro che ploverva presto. _I had the car washed. After months of drought, I'm sure it will rain soon._ \\ Cherry picking & Questvo rifreschera la memoria a chi l'ha corta e mostrera ai giovani che \\  & Roma ha sempre avuto voragnii e alberatl adopt dop forti biogege. _This video will refresh the memory of those with short memories and reveal to the young that Rome has always had sinkholes and fallen trees after heavy rains._ \\ Circular reasoning & Le leggi sul'immigrazione sono gia cosi fascistic che quando i fascisti prometto di introdurre nuove misure, stanno solo descrivendo le norme fasciste che gais estono o seno applicate da decenni. _Immigration laws are already so fascist that when fascists promise to introduce new measures, they're actually just describing the fascist rules that already exist and have been applied for decades._ \\ Doubt & Nella strage di Lampedusa morirono 368 migranti. L'Italia e l'Europa dissero "mai piu". **Forse ci credevano davvero.** _In the Lampedusa massacre, 368 migrants died. Italy and Europe said "never again". Maybe they really believed it._ \\ Eviding the burden of proof & Trovo geniali **tedesch ei sedesi che censurano i crimini degli immigrati per restare in testa allas classificica dei popoli meno xenofobi.** _I find it ingestion how Germans and Swedes censor immigrants' crimes to stay ahead in the ranking of the least xenobphobic nations_**.** \\ False analogy & Non vi fa sorgere qualche dubbio uno stato che blocca le flnavidaccrociera per preventive il **if**covidi ma fa barcare centinaia di immigrat dialle navi delle **#ONG e dai #barchini?** _Doesn't it raise some doubts about a state that blocks #eruisheships to prevent #covid, but allows hundreds of immigrants to disembark from #ONG ships and #smallboats?_ \\ False dilemma & Queste elezioni saranno uno spartiacque tra chi mette al centre la soctenibilita ambientale e chi difende il combustibile fossile. _These elections will be a turning point between those who prioritize environmental sustainability and those who defend fossil fuels._ \\ Flag waving & Con la forza e l'unita di questo popolo orgoligoso chiediano a ututa la \#Toscana di schierarsi subito. **Uniciamo**, insieme, per la Toscana piu forte e unita!** _With the strength and unity of these proud people, we ask all of #Tuscany to stand with us immediately. Let's win, together, for a stronger and more united Tuscany!_ \\ Hasty generalization & Se Draghi ha mentito agil Italiani sul vaccino, **allowa sta mentendo anche su guerra, clima, transizione ecologica, pmrr, energia, benzine e bollette!** _If Draghi lied to Italians about the vaccine, then he is also lying about the war, climate, ecological transition, pmrr, energy, petrol, and bills!_ \\ Loaded language & Epidemia colposa. **Bomba** in arriov dalla Procura, **valanga** di indagati: **tremano** Speranza e Lorenzin. _Culpable epidemic. **Bombshell** coming from the Public Prosector's Office, avalanche of suspects: Speranza and Lorenzin **trendle**._ \\ \end{tabular}





## Appendix D Additional Dataset Details

In this section, we present additional details about the Faina dataset with regards to individual annotators' labels. Individual summary statistics are presented in Table 6, whereas fallacy overlaps for each annotator are in Figure 7. We also use Variationist (v0.1.4) Ramponi et al. (2024) and calculate the top-\(k\) (\(k=10\)) most informative tokens for fallacy types that have an average span length of \(\leq 10\) tokens (Table 6),12 i.e., those that are mainly related to language use and are therefore characterized by specific lexical choices. Results are in Table 7.

Footnote 12: For the analysis, we set both lowercase and stopwords to True and set the language to Italian (it). We employ normalized, positive, and weighted PMI as a metric.

## Appendix E Additional Experimental Details

In this section, we describe additional details on our experimental settings. All our experiments were run on a single GPU (Tesla V100-SXM2-32GB).

Hyper-parametersFor supervised models, we employ the default MaChAmp van der Goot et al. (2021) hyperparameter values (see Table 8) and fine-tune the models for span and post tasks for 20 and 10 epochs, respectively. The AlBERTo and UmBERTo versions we used are bert_uncased_L-12_H-768_A-12_italian_alb3rt@ and umberto-commoncrawl-cased-v1. For unsupervised models, we use default settings from the Hugging Face library.

\begin{table}
\begin{tabular}{p{59.8pt} p{284.5pt}} \hline \hline
**Fallacy type** & **Example** \\ \hline Name calling or labeling & **Un idiota fasciorazzista** condivide una foto di un giovane immmigrato inventando fossie senza biglietto; foto condivisa 80mila volute con frasi **razzistle**; smentita di Trentialia: “Aveva solo sbagliato poseto”. Siete sololo dei **razzist di merda**. \\  & _A fasci-racisi tidid shares a photo of a young immigrant claiming he was ticketless; photo shared 80k times with racist comments; Trenttalia’s denial: “He just sat in the wrong seat”. You’re just shirty racists_. \\ \hline Red herring & **Siccome pandemia, crsi energetica, crsi climatica, inflazione e tensioni tra Russia e Ucarina non sono sufficienti**, oggi il Corriere ci regala questo articolo  /Trust the professionals of information”._ \\ \hline Slippery slope & [USER] da la colpa delle violenze di Peschiera agili italiani. **Un altro vergognoso gradino verso la voragine della stampa italiana**. \\  & _[USER] blames Italians for the violence in Peschiera. Another shameful step towards the abysys of Italian press_. \\ \hline Slogan & La difresa dei confini e la lotta all’immigrazione incontrollata di massa rimarranno una priorita. **Difendiamo io confinit! @BlocoeNavaleSubito**_The defense of borders and the fight against uncontrolled mass immigration will remain a priority. Let’s defend the borders? #NavalBlockadeNow_ \\ \hline Strawman & **L’incipit del TG1: “Complice ii bel tempo, continuano gli barachi a Lampedusa”. Voigliano il marte a forza 8 cosi c penas lui a maltire i migranti?!**_ _The incipit of TG1: “Thanks to the good weather, landings in Lampedusa continue”. Do we want a force 8 sea so it can get rid of migrants?!_** \\ Thoughst-terminating cliche & Nella UE tornano i muri antimigranti. **Era eraora. In EU, anti-migrant walls come back. It was about time.** \\ \hline Vagueness & Ogni centesimo speso in armi é tolto a sanitià, ricercrea, istruzione e transizione energetica: percio non sono pro-Putin, che in questo **ha fatto una scelta ben precisa**. _Every cent spent on weapons is taken away from healthcare, research, education, and energy transition: therefore, I am not pro-Putin, who has made a very clear choice in this regard._ \\ \hline \hline \end{tabular}
\end{table}
Table 5: Examples of annotated spans for each fallacy type. Annotated spans are indicated in bold and English translations are in italic. Examples have been slightly edited to preserve users’ anonymity.





\begin{table}
\begin{tabular}{l r r|r r|r r} \hline \hline  & \multicolumn{2}{c|}{\(\mathcal{A}_{1}+\mathcal{A}_{2}\)} & \multicolumn{2}{c|}{\(\mathcal{A}_{1}\)} & \multicolumn{2}{c}{\(\mathcal{A}_{2}\)} \\
**Fallacy type** & **Spans** & **Length** & **Spans** & **Length** & **Spans** & **Length** \\ \hline Ad hominem & 319 & 16.0\(\pm_{13.3}\) & 166 & 15.5\(\pm_{13.3}\) & 153 & 16.5\(\pm_{13.3}\) \\ Appeal to authority & 213 & 6.4 \(\pm_{4.3}\) & 120 & 6.4 \(\pm_{4.5}\) & 93 & 6.4 \(\pm_{3.9}\) \\ Appeal to emotion & 2,049 & 5.1 \(\pm_{4.9}\) & 1,022 & 5.7 \(\pm_{5.6}\) & 1,047 & 4.5 \(\pm_{4.0}\) \\ Causal oversimplification & 142 & 19.0\(\pm_{10.7}\) & 87 & 19.4\(\pm_{11.2}\) & 55 & 18.3\(\pm_{10.0}\) \\ Cherry picking & 94 & 28.8\(\pm_{12.3}\) & 52 & 28.2\(\pm_{12.5}\) & 42 & 29.5\(\pm_{12.0}\) \\ Circular reasoning & 20 & 26.8\(\pm_{11.0}\) & 12 & 27.4\(\pm_{10.5}\) & 8 & 25.8\(\pm_{11.8}\) \\ Doubt & 482 & 16.1\(\pm_{11.4}\) & 236 & 16.8\(\pm_{12.2}\) & 246 & 15.4\(\pm_{10.4}\) \\ Evading the burden of proof & 406 & 16.2 \(\pm_{9.9}\) & 195 & 16.3\(\pm_{10.7}\) & 211 & 16.2 \(\pm_{9.2}\) \\ False analogy & 239 & 22.1\(\pm_{13.4}\) & 147 & 20.8\(\pm_{13.0}\) & 92 & 24.1\(\pm_{13.8}\) \\ False dilemma & 90 & 15.9\(\pm_{11.0}\) & 44 & 16.6\(\pm_{12.1}\) & 46 & 15.2 \(\pm_{9.8}\) \\ Flag waving & 393 & 4.3 \(\pm_{4.9}\) & 197 & 4.6 \(\pm_{5.5}\) & 196 & 3.9 \(\pm_{4.2}\) \\ Hasty generalization & 464 & 11.2 \(\pm_{8.0}\) & 241 & 11.8 \(\pm_{8.8}\) & 223 & 10.6 \(\pm_{6.9}\) \\ Loaded language & 2,484 & 2.5 \(\pm_{2.7}\) & 1,064 & 2.8 \(\pm_{3.6}\) & 1,420 & 2.4 \(\pm_{1.6}\) \\ Name calling or labeling & 1,124 & 2.6 \(\pm_{1.7}\) & 561 & 2.5 \(\pm_{1.6}\) & 563 & 2.7 \(\pm_{1.7}\) \\ Red herring & 257 & 13.0 \(\pm_{8.5}\) & 129 & 14.2 \(\pm_{9.3}\) & 128 & 11.8 \(\pm_{7.4}\) \\ Slippery slope & 172 & 10.8 \(\pm_{6.8}\) & 88 & 11.4 \(\pm_{7.5}\) & 84 & 10.2 \(\pm_{6.0}\) \\ Slogan & 384 & 3.5 \(\pm_{3.1}\) & 190 & 3.4 \(\pm_{3.0}\) & 194 & 3.5 \(\pm_{3.2}\) \\ Strawman & 109 & 36.3\(\pm_{15.4}\) & 67 & 34.9\(\pm_{16.1}\) & 42 & 38.5\(\pm_{14.0}\) \\ Thought-terminating cliche & 285 & 5.2 \(\pm_{3.0}\) & 145 & 5.3 \(\pm_{3.2}\) & 140 & 5.2 \(\pm_{2.8}\) \\ Vagueness & 1,338 & 9.1 \(\pm_{8.6}\) & 536 & 9.5 \(\pm_{9.3}\) & 802 & 8.9 \(\pm_{8.2}\) \\ \hline
**All** & 11,064 & 7.6 \(\pm_{9.3}\) & 5,279 & 8.2 \(\pm_{9.9}\) & 5,785 & 7.1 \(\pm_{8.6}\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Per-annotator and overall annotation statistics across all fallacy types. We report the number of _spans_ and their average _length_ (with standard deviation) at the token level. \(\mathcal{A}_{1}\) and \(\mathcal{A}_{2}\) indicate annotation statistics by the individual annotators, whereas \(\mathcal{A}_{1}+\mathcal{A}_{2}\) summarizes the overall ones.



Prompts and technical detailsThe prompt template used across all experiments along with task-specific prompt variables are presented in Table 9. For span tasks, the output format was initially requested in the CoNLL format with the bio-tagging scheme for unique identification of the fallacy segments; however, the models struggled to provide consistent outputs. Therefore, we provide an incremental identifier for each token in the input text (i.e., token id) and request the output following the format [first number-last number = Fallacy Label]. We chose square brackets because we observed that models could easily replicate them. Moreover, they facilitate the retrieval of the portion of the output in which fallacy predictions are actually provided, while disregarding other fallacy mentions across the output. We use regular expressions for extracting predictions from the output and also normalize the predicted fallacy names that clearly refer to the same fallacy type (e.g., we consider both the British spelling _Name calling or labelling_ and the shortened _Name calling_ label as instances of the _Name calling or labeling_ fallacy).

Results on individual test setsIn Table 10, we present the full results obtained by each model across all task setups on individual test sets (i.e., \({\mathcal{A}}_{1}\) and \({\mathcal{A}}_{2}\)), as well as those averaged over all \(|{\mathcal{A}}|\) test set versions (i.e., \({\mathcal{A}}_{1}+{\mathcal{A}}_{2}\)).

\begin{table}
\begin{tabular}{l c} \hline \hline
**Fallacy type** & **Top-\(k\) tokens** \\ \hline Appeal to authority & user, articolo, studio, video, sa, papafrancesco, via, scientifico, intervista, scritto \\ Appeal to emotion & schifo, a, vergogna, b, oumanita \\ Flag waving & italiani, user, fridaysforuture, l, m5s, rfatelliditalia, italiasulserio, \\  & fridayforfuture, piazza \\ Loaded language & lotta, caszo, leggi, merda, caro, invasione, cognioni, combattere, disastro \\ Name calling or labeling & immigrati, immigrazione, migrnenti, clandestini, sinistra, wax, negazionisti, \\  & dittatura, immigato, sanitaria \\ Slogan & primadeldliuvio, italiasulserio, facciamorete, vaccinare, sostenibile, \\  & cambiaehenoninclima, climateactionnow, resistenza, toscana, ivotocaland \\ Thought-terminating cliche & man, so, sapevatelo, eh, capitta, semplice, vince, caio, stop, ragione \\ Vagueness & migranti, immigranti, immigrarzione, lavoro, sanita, governo, politici, politica, \\  & italiani, pandemia \\ \hline \hline \end{tabular}
\end{table}
Table 7: Top-\(k\) (\(k=10\)) most informative tokens for fallacy types with an average length of \(\leq 10\) tokens, i.e., those that are mainly related to language use, considering all the annotations in the dataset (\({\mathcal{A}}_{1}+{\mathcal{A}}_{2}\)).

\begin{table}
\begin{tabular}{l r} \hline \hline
**Hyperparameter** & **Value** \\ \hline Optimizer & AdamW \\ \(\beta_{1}\), \(\beta_{2}\) & 0.9, 0.99 \\ Dropout & 0.2 \\ Epochs & 10 / 20 \\ Batch size & 32 \\ Learning rate & 1e-4 \\ LR scheduler & Slanted triangular \\ Decay factor & 0.38 \\ Cut fraction & 0.3 \\ \hline Task loss weight (\(\lambda\)) & 1 \\ Multi-label threshold (\(\tau\)) & 0.7 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameter values employed for the supervised models in all our experiments.

 

* **Grouping**

_Given an Italian text, your task is to_ $TASK\({}_{\text{DESC}}\)_. The 20 fallacy labels are as follows:_

* **%****FALLACIES + $MACROCAT**
* **%****INSTR\({}_1\) + $__INSTR\({}_2\)**

_There's no limit in the number of fallacies you can find in the text, so it's really important that you identify all possible fallacies._
* **%****INSTR\({}_3\)**

_Here is the Italian text you must analyze +_ $__INSTR\({}_4\)**:**
* **%****INSTR\({}_5\)**

_Now return the output for the provided text._

\begin{table}
\begin{tabular}{l|c c|c|c|c} \hline \hline




\begin{table}
\begin{tabular}{l l c r|r r r|r r|r r} \hline  & & \multicolumn{3}{c|}{\(\mathcal{A}_{1}+\mathcal{A}_{2}\)} & \multicolumn{4}{c|}{\(\mathcal{A}_{1}\)} & \multicolumn{3}{c}{\(\mathcal{A}_{2}\)} \\  & **Model** & **P** & **R** & **F\({}_{1}\)** & **P** & **R** & **F\({}_{1}\)** & **P** & **R** & **F\({}_{1}\)** \\ \hline \multirow{6}{*}{**Spar-f**} & MVML-alb & 80.0\({}_{\pm 1.5}\) & 74.0\({}_{\pm 2.3}\) & **76.8\({}_{\pm 1.6}\)** & 78.8\({}_{\pm 1.9}\) & 70.6\({}_{\pm 3.5}\) & **74.5\({}_{\pm 2.7}\)** & 81.3\({}_{\pm 1.0}\) & 77.3\({}_{\pm 1.1}\) & **79.2\({}_{\pm 0.4}\)** \\  & MVML-umb & 84.5\({}_{\pm 1.3}\) & 70.1\({}_{\pm 4.2}\) & 76.6\({}_{\pm 2.8}\) & 83.4\({}_{\pm 1.2}\) & 67.2\({}_{\pm 4.4}\) & 74.4\({}_{\pm 3.1}\) & 85.6\({}_{\pm 1.3}\) & 73.0\({}_{\pm 3.9}\) & 78.8\({}_{\pm 2.6}\) \\  & ZSWD-llama & 57.9\({}_{\pm 1.9}\) & 70.0\({}_{\pm 1.9}\) & 63.3\({}_{\pm 1.5}\) & 54.8\({}_{\pm 2.0}\) & 69.3\({}_{\pm 1.7}\) & 61.2\({}_{\pm 1.7}\) & 60.9\({}_{\pm 1.8}\) & 70.6\({}_{\pm 2.1}\) & 65.4\({}_{\pm 1.3}\) \\  & ZSWD-mixtr & 64.7\({}_{\pm 1.6}\) & 45.2\({}_{\pm 1.0}\) & 53.2\({}_{\pm 1.2}\) & 63.2\({}_{\pm 1.7}\) & 46.2\({}_{\pm 1.6}\) & 53.4\({}_{\pm 1.6}\) & 66.1\({}_{\pm 1.6}\) & 44.2\({}_{\pm 0.5}\) & 53.0\({}_{\pm 0.8}\) \\ \cline{2-11}  & MVML-alb & 63.0\({}_{\pm 2.0}\) & 34.3\({}_{\pm 1.9}\) & **44.3\({}_{\pm 1.9}\)** & 60.4\({}_{\pm 1.7}\) & 31.2\({}_{\pm 1.4}\) & **41.1\({}_{\pm 1.5}\)** & 65.5\({}_{\pm 2.2}\) & 37.4\({}_{\pm 2.4}\) & **47.6\({}_{\pm 2.4}\)** \\  & MVML-umb & 39.0\({}_{\pm 3.7}\) & 14.6\({}_{\pm 1.6}\) & 21.3\({}_{\pm 2.2}\) & 31.5\({}_{\pm 4.6}\) & 11.5\({}_{\pm 1.8}\) & 16.9\({}_{\pm 2.6}\) & 46.5\({}_{\pm 2.7}\) & 17.7\({}_{\pm 1.5}\) & 25.7\({}_{\pm 1.9}\) \\  & ZSWD-llama & 20.9\({}_{\pm 1.5}\) & 24.3\({}_{\pm 2.3}\) & 22.5\({}_{\pm 1.8}\) & 20.7\({}_{\pm 1.4}\) & 24.7\({}_{\pm 2.1}\) & 22.5\({}_{\pm 1.6}\) & 21.1\({}_{\pm 1.6}\) & 23.9\({}_{\pm 2.5}\) & 22.5\({}_{\pm 2.0}\) \\  & ZSWD-mixtr & 26.0\({}_{\pm 1.8}\) & 18.1\({}_{\pm 1.4}\) & 21.4\({}_{\pm 1.5}\) & 25.7\({}_{\pm 1.9}\) & 18.4\({}_{\pm 1.4}\) & 21.4\({}_{\pm 1.6}\) & 26.3\({}_{\pm 1.7}\) & 17.8\({}_{\pm 1.3}\) & 21.3\({}_{\pm 1.5}\) \\ \hline \multirow{6}{*}{**Spar-f**} & MVMD-alb & 55.2\({}_{\pm 1.7}\) & 51.7\({}_{\pm 2.1}\) & 53.3\({}_{\pm 1.4}\) & 55.8\({}_{\pm 1.2}\) & 50.3\({}_{\pm 2.4}\) & 52.9\({}_{\pm 1.8}\) & 54.6\({}_{\pm 2.1}\) & 53.1\({}_{\pm 1.8}\) & 53.8\({}_{\pm 1.0}\) \\  & MVMD-umb & 59.8\({}_{\pm 1.5}\) & 50.4\({}_{\pm 2.4}\) & **54.7\({}_{\pm 1.5}\)** & 61.8\({}_{\pm 1.5}\) & 49.9\({}_{\pm 2.6}\) & **55.2\({}_{\pm 1.7}\)** & 57.9\({}_{\pm 1.4}\) & 50.8\({}_{\pm 2.1}\) & **54.1\({}_{\pm 1.3}\)** \\  & ZSWD-llama & 25.3\({}_{\pm 4.2}\) & 7.0\({}_{\pm 0.8}\) & 10.9\({}_{\pm 0.9}\) & 25.2\({}_{\pm 4.9}\) & 6.9\({}_{\pm 0.7}\) & 10.8\({}_{\pm 1.0}\) & 25.4\({}_{\pm 3.6}\) & 7.0\({}_{\pm 0.9}\) & 10.9\({}_{\pm 0.9}\) \\  & ZSWD-mixtr & 31.6\({}_{\pm 1.2}\) & 20.9\({}_{\pm 1.4}\) & 25.1\({}_{\pm 1.2}\) & 31.8\({}_{\pm 1.3}\) & 20.9\({}_{\pm 1.0}\) & 25.2\({}_{\pm 0.9}\) & 31.3\({}_{\pm 1.1}\) & 20.8\({}_{\pm 1.7}\) & 25.0\({}_{\pm 1.5}\) \\ \hline \end{tabular}

* _Strict mode_
* MVMD-alb & 47.6\({}_{\pm 1.9}\) & 25.6\({}_{\pm 1.6}\) & **33.3\({}_{\pm 1.4}\)*
* & 47.1\({}_{\pm 2.1}\) & 24.4\({}_{\pm 1.6}\) & **32.1\({}_{\pm 1.3}\)*
* & 48.0\({}_{\pm 1.7}\) & 26.8\({}_{\pm 1.5}\) & **34.4\({}_{\pm 1.4}\)*
* \\  & MVMD-umb & 57.5\({}_{\pm 5.9}\) & 3.9\({}_{\pm 0.7}\) & 7.3\({}_{\pm 1.3}\) & 51.7\({}_{\pm 4.8}\) & 5.0\({}_{\pm 0.9}\) & 9.2\({}_{\pm 1.5}\) & 63.3\({}_{\pm 6.9}\) & 2.9\({}_{\pm 0.6}\) & 5.5\({}_{\pm 1.0}\) \\  & ZSWD-llama & 4.5\({}_{\pm 0.5}\) & 2.7\({}_{\pm 0.4}\) & 3.4\({}_{\pm 0.3}\) & 4.5\({}_{\pm 0.4}\) & 2.6\({}_{\pm 0.4}\) & 3.3\({}_{\pm 0.4}\) & 4.5\({}_{\pm 0.6}\) & 2.8\({}_{\pm 0.3}\) & 3.4\({}_{\pm 0.3}\) \\  & ZSWD-mixtr & 5.8\({}_{\pm 1.1}\) & 3.2\({}_{\pm 0.5}\) & 4.2\({}_{\pm 0.7}\) & 5.9\({}_{\pm 1.