Argument-based Detection and Classification of Fallacies in Political Debates
Authors: Pierpaolo Goffredo
Mariana Chaves Espinoza
Serena Villata
Elena Cabrio
Venue: EMNLP 2023 Main (2023)
Keywords: Fallacy Detection
NLP
Token Classification
Political Debates
Machine Learning
Transformers
Argumantation
TL;DR: Our contribution includes expanding the dataset and using neural network architectures to automatically detect and classify fallacious arguments in political debates.

Abstract: Fallacies are arguments that employ faulty reasoning. Given their persuasive and seemingly valid nature, fallacious arguments are often used in political debates. Employing these misleading arguments in politics can have detrimental consequences for society, since they can lead to inaccurate conclusions and invalid inferences from the public opinion and the policymakers. 
Automatically detecting and classifying fallacious arguments represents therefore a crucial challenge to limit the spread of misleading or manipulative claims and promote a more informed and healthier political discourse. 
Our contribution to address this challenging task is twofold. 
First, we extend the ElecDeb60To16 dataset of U.S. presidential debates annotated with fallacious arguments, by incorporating the most recent Trump-Biden presidential debate. We include updated token-level annotations, incorporating argumentative components (i.e., claims and premises), the relations between these components (i.e., support and attack), and six categories of fallacious arguments (i.e., Ad Hominem, Appeal to Authority, Appeal to Emotion, False Cause, Slippery Slope, and Slogans).  
Second, we perform the twofold task of fallacious argument detection and classification by defining neural network architectures based on Transformers models, combining text, argumentative features, and engineered features.  
Our results show the advantages of complementing transformer-generated text representations with non-text features.

Argument-based Detection and Classification of Fallacies in Political Debates
Pierpaolo Goffredo', Mariana Chaves Espinoza', Elena Cabrio', Serena Villata'
Université Côte d' Azur, CNRS, Inria, I3S, France goffredo @ i3...3............ fr, machaves@ @33.:iii.c....f.... elena.cabrio @ univ-cotedazur.rr serena. serena.villata willata@uivvcoottleee @univ-cotedazurffr
Abstract
move inappropriate with respect to its pragmatic application context (Walton, 1995). Despite their employment in many scenarios (c.g.. online dis- cussion platforms and blogs, TV roundtables), a natural testbed of this form of misleading argu- mentation is political debate. For instance, the ad hominem fallacy, where the plausibility of the ar- gument depends the credentials, personal back- ground and past actions of the speaker, probably one of the fallacy labels that is most often thrown around in political debate. This kind of arguments may sound convincing and has the goal to mislead the audience, persuading it about the validity of the argument. Given the potential nefarious im- pact of these misleading arguments on the society, identifying and classifying fallacious arguments is therefore a main open challenge in Argument Min- ing (AM) (Cabrio and Villata, 2018; Lawrence and Reed, 2019; Lauscher et al., 2022), and in NLP in general.
Fallacies are arguments that employ faulty rea- soning. Given their persuasive and seemingly valid nature, fallacious arguments are often used political debates. Employing these mis- leading arguments in politics can have detri- mental consequences for society, since they can lead inaccurate conclusions and invalid inferences from the public opinion and the pol- icymakers. Automatically detecting and classi- fying fallacious arguments represents therefore a crucial challenge to limit the spread of mis- leading or manipulative claims and promote more informed and healthier political dis- course. Our contribution to address this chal- lenging task twofold. First, we extend the ElecDeb60To16 dataset of U.S. presidential debates annotated with fallacious arguments, by incorporating the most recent Trump-Bidd -idden presidential debate. We include updated token- level annotations, incorporating argumentative components (i.e.. claims and premises), the re- lations between these components (i.e., support and attack), and six categories of fallacious ar- guments (i.e., Ad Hominem, Appeal Author- ity, Appeal to Emotion, False Cause, Slippery Slope, and Slogans). Second, we perform the twofold task of fallacious argument detection and classification by defining neural network architectures based on Transformers models, combining text, argumentative features, and en- gineered features. Our results show the advan- tages of complementing transforner-generateed text representations with non-textual features.
Existing approaches in the literature (Habernal et al., 2017, 2018; Goffredo et al., 2022; Vija- yaraghavan and Vosoughi, 2022; Alhindi et al., 2022; Vorakitphan et al., 2022; Sahai et al., 2021) mainly concentrated on the classification of falla- cious text snippets over a finite set of labels, leaving the challenging issue of identifying the fallacious text snippet and its boundaries under-investigated. In this paper, we tackle this open research ques- tion on dataset of political debates from the U.S. presidential campaigns from 1960 to 2020.
1 Introduction
More precisely, the contribution of this paper is twofold. First, we extend an existing resource of U.S. political debates from the presidential cam- paigns (1960-2016) annotated both with argument components and relations, and six fallacy cate- gories (namely, Ad Hominem, Appeal to Author- ity, Appeal to Emotion, False Cause, Slippery Slope and Slogans). This new resource, named the ElccDeb60to20 dataset, includes now also the debates of the 2020 presidential campaign (Trump-
Fallacious arguments have been firstly defined as defective inferences, i.e., logically invalid types of arguments (Femeren, 2001). More recently, in more pragmatic perspective, fallacious arguments have been defined infringements of performance rules characteristic of a particular ideal type of argu- mentative engagement (Femeren and Grootendorst, 1987) and as illicit dialectical shifts across differ- ent dialogue types, highlighting that the attempted

---

Biden) with all the related annotations. Second, we propose new approach, based on Transformers, to detect the fallacious text snippets in these de- bates, and then classify them along the six fallacy categories. This approach encodes the argument components (i.c., Premise, Claim), the argument relations (i.e., Support, Allack) and the PoS tags to successfully identify and classify fallacious ar- guments. Experimental results show that the pro- posed approach outperforms standard baselines and concurrent approaches with an average fl -core of 0.74, with our proposed model named MultiFusion BERT, on the task of fallacious argument detection and classification.
icant source of inspiration in this area. Recently, researchers have made significant progress iden- tifying and classifying fallacies within discourse. In this section, we discuss the approaches proposed in the literature on the two tasks of fallacy classi- fication (Section 2.1) and fallacy detection (Sec- tion 2.2).
2.1 Fallacy Classification
For instance, Habernal et al. (2017) aimed to im- prove fallacy detection by creating a publicly avail- able software called "Argotario". It serves as an educational gaming platform and a means to gather data from the crowd for annotating fallacy types in everyday arguments. In a subsequent study, Haber- nal al., 2018 released annotated datasets of falla- cious arguments in English and German. They con- ducted experiments using Support Vector Machine and BiLSTM models with German word vectors to classify six fallacious topic types (accuracy=50.9%, macro-Fl scorc=42.1%).
Whilst most of the computational approaches tar- geting fallacious argumentation focus on the pure classification of such nefarious content (Habernal et al., 2017, 2018; Jin et al., 2022: Goffredo et al., 2022; Vijayaraghavan and Vosoughi, 2022; Alhindi et al., 2022), the originality of our contribution is that it proposes, to the best of our knowledge, the first neural architecture to both detect fallacious ar- guments and classify them in political debates, and one of the very few approaches to tackle this task in general, outperforming competing approaches (Vo- rakitphan al., 2022; Sahai et al., 2021).
Jin et al. (2022) proposes an architecture based on a simple classifier that incorporates the struc- tural information of fallacies. They augment a standard pre-trained language model for classifi- cation with template that includes modified as- sumptions and premises based on well-defined masking scheme. They achieve an F1 score of 58. 77% the classification task over 13 classes. They used dataset collected from online teaching materials, specifically designed to teach and test students' understanding of logical fallacies.
The urgency to study fallacies in political dis- course is crucial both from the philosophical and the political perspective. It emphasizes the need scrutinize political arguments for sound reason- ing rather than deception. Understanding logical flaws is a main issue for informed decision-making enabling the recognition and assessment of falla- cious arguments in political discourse. Moreover, examining how fallacies are employed in political debates reveals their strategic role in influencing opinion and diverting attention. This strategic use of fallacies mirrors the subtleties of language, em- phasizing the interplay between rhetoric, philoso- phy, and political communication. Furthermore, offers valuable insights into the dynamics of public discourse and the need for critical analysis (Walton, 1987, 1995).
In more contextual approach, Goffredo et al. (2022) use dataset from U.S. Presidential Elec- tion Debates From 1960 2016. This dataset incorporates information such as the context and argumentative features (i.c., argument components and relations) for each fallacy. The proposed ar- chitecture uses one classifier for each feature in order to calculate the loss. On a sentence classifica- tion task encompassing six fallacy categories, their approach achieves an Fl score of 84%.
To address the identification and classification of fine- fine-grained propaganda tweets, Vijayaragha- van and Vosoughi (2022) proposed an end-to-end wransformers-based approach that considers addi- tional features like context, relational informa- tion, and external knowledge through data aug- mentation. Their dataset consists of approximately 211K tweets annotated with 19 classes (18 propa- ganda types and propaaaagg.. The approach
2 Related Work
Over the years, there has been growing interest in the field of NLP to the detection of fallacies and related phenomena, including misinformation and propaganda (Da San Martino et al., 2020b). The pioneering work of Da San Martino et al. (2019b) on fallacies in newspaper news has been a signif-

---

yielded a 64% F1 score on a text classification task. Alhindi et al. (2022) introduced an instruction- based prompt in multitask configuration using the T5 model to classify fallacies. This method- ology involved leveraging multiple fallacy-base based datasets, including Propaganda (Da San Martino et al., 2019b), Logic (Jin et al., 2022), Argotario (Habernal et al., 2017), Covid-19 (Musi et al., 2022), and Climate (Jin et al., 2022). Their ap- proach enabled the identification of 28 distinct fal- lacies across various domains and genres, facilitat- ing the analysis of model size and prompt selection, and investigating the impact of annotation quality on model performance, potentially supplemented with external knowledge. The results obtained us- ing their T5-large model yielded Fl scores of 4 % for Propaganda, 62% for Logic, 59% for Argotario, 26% for 1ovid-19, and 17% for Climate, respec- tively.
fallacy studies in several ways. Firstly, employ corpus specifically created for fallacy detection in political debates, which has been under inves- tigated in this context even though fallacious ar- gumentation is a main issue in political discourse. Secondly, while the majority of existing studies fo- cus on fallacy classification, in this paper we focus on the fallacy detection task, which is of main im- portance in real-life scenarios where text or speech lacks pre-segmentation and a clear binary classifi- cation into fallacies or non-fallacies. Lastly, our study capitalizes on the annotations of argumen- tation features in our dataset, an element that is often absent in other corpora, allowing us to go beyond pure text-based approaches, and to rely on the argumentation structure.
3 ElecDeb60to20 Dataset
To effectively address the task of detecting and classifying fallacious arguments within political debates, we decided to rely on the ElecDeb60To dataset (Haddadan et al., 2019; Goffredo et al., 2022). It comprises televised debates from U.S. presidential election campaigns spanning from 1960 to 2016. These debates were sourced from the website of the Commission on Presidential Debates which openly provides transcripts of debates broadcasted on television and featuring the prominent candidates for presidential and vice- presidential nominations in the United States. All information on this website is accessible to the public. Considering the most recent presidential election between Trump and Biden occurred in 2020, we expanded the dataset with the transcripts of the debates of this election campaign to include updated annotations, incorporating argumentative components such as Claims and Premises, as well as the relations between these components, i.e.. Support and Attack. As a result of this annotation update, the dataset is renamed as ElecDeb60to202, reflecting the coverage of debates spanning from 1960 to 2020. This updated dataset provides more comprehensive and contemporary collection of fal- lacies in political debates, enhancing the relevance and applicability of the data for further analysis and research in the field.
2.2 Fallacy Detection and Classification Task
Vorakitphan al. (2022) proposes a system capa- ble of automatically identifying propaganda mes- sages and classifying them based on the propa- ganda techniques employed. The system adopts a pipeline approach that firstly detects the text snip- pet containing potential propaganda, and then per- forms classification by leveraging semantic and argumentative features. Two standard benchmarks, NLP4IF' 19 (Da San Martino et al., 2019a) and Se- mEval 20 datasets (Da San Martino et al., 2020a), were used for this propaganda detection and clas- sification task. The binary classification step, uti- lizing BERT, achieved a 72% Fl score, while the sentence-span multi-class classification task (14 classes) achieved 64% micro F1 score using RoBERTa- architecture.
Fallacy detection and token-level classification tasks were also performed in Sahai et al. (2021). They narrowed their study to 8 fallacy types and created corpus of fallacious arguments by anno- tating user comments Reddit. They performed fallacy classification at comment-level and token- level, relying on BERT and MGN (Da San Martino et al., 2019a) models. The inclusion of the conver- sation context, represented by the parent comment or submission tille, was used to enhance the pre- dictions. Fine-tuned BERT with a classification head of linear layer reported the best results on the token classification task (macro F1=53%).
This resource is a valuable benchmark for in- vestigating potential connections between specific
http://wwwbbbbttt..rrr//eeeedddd
lebate-transcripte
http:/twwwb.mmmppppppaaaorrraaaa FallacyDetection
Our present work expands the scope of these

---

argument components and relations that underlie the occurrence of fallacious arguments.
Two annotators, with expertise in computational linguistics, independently annotated the new por- tion of the dataset (Trump vs. Biden debates) by identifying argumentative components, relations, and fallacies. To maintain objectivity and prevent bias, the annotation process for argumentative com- ponents was performed on raw data, without any pre-existing fallacy annotations. This approach was adopted to ensure that the annotation process remains unbiased and free from any preconceived notions related to fallacies. A set of 50 sentences randomly extracted from the debates was annotated to assess Annoottoot Agreement (IAA), and the results, visualized in Table 1, indicate a sub- stantial level of agreement between the annotators.
3.1 Annotated Fallacies
During the annotation process of fallacies within the U.S. political debates of the 2020 presidential election, we rely on the six categories based on the annotation scheme proposed by Da San Martino al. (2019a), the categorization outlined by Wal- ton (1987), and the annotation of the previous de- bates in the first version of the dataset with these fal- lacy categories (Goffredo et al., 2022). Hence, we adopted the following categories: Ad Hominem, Ap- peal to Authority, Appeal to Emotion, False Cause, Slippery Slope, and Slogans. Below, we provide a concise description of each of these six categories.
Measure
Value
When the argument becomes excessive attack an arguer's position (Walton, 1987).
Ad Hominem.
Observed Agreement 0.857
Krippendorff's
0.757
Table 1: IAA agreement over 50 sentences randomly extracted from the debates Trump-Biden.
The unessential loading of the argument with emotional language to exploit the audience emotional instinct.
Appeal to Emotion.
3.3 Statisties and Data Analysis
Appeal to Authority. It occurs when the arguer relies on the endorsement of authority figure group consensus without providing sufficient evidence. It may also involve the citation of non- experts or the majority support their claim.
Table 2 summarizes the Trump vs. Biden's de- bates annotations per category and argumentative features. We tokenized the annotated fallacious guments to compute the average number of words in each category. line with the guidelines of Goffredo al. (2022), Slogans the shortest with 5.0 tokens on average", whereas SlipperySlope was the longest with 20.5 tokens on average.
Slippery Slope. This fallacy implies that an im- probable or exaggerated consequence could result from a particular action.
Category
Frey
AvgTok
Arg. Feature
Freg
False Cause. The misinterpretation of the corre- lation of two events for causation (Walton, 1987).
Hominem
4,6
Claims
1513
AppealtoAuttoriy
18.6
Premise
332
AppealtoEmotion
6,81
Support Rel.
400
Slogan. It is a brief and striking phrase used to provoke excitement of the audience, and is often accompanied by another type of fallacy called ar- gument by repetition.
FalseCause
0
Attack Rel.
112
SlipperySlope Sloguns
20,5
5
Total
232
9,25
Total
2357
Table 2: Distribution of annotated fallacies per cate- gory and argumentative features of Trump vs. Biden's 's debates.
3.2 Annotation Phase
The updated annotations were conducted follow- ing the annotation scheme introduced in Haddadan et al. (2019): Goffredo et al. (2022). Following this approach, each debate was divided into sections, starting with either moderator/panelist an au- dience member asking question on new topic. To facilitate the annotation process, the semantic annotation platform INCEpTION (Klie et al., 2018) was used.
The train and test set split was performed consid- cring the entire new dataset ElccDeb60to20. The training set accounts for 90% of the dataset, while the remaining 10% constitutes the test set. The distribution of fallacy labels is as follows: Appeal- toEmotion (59.94%). AppealtoAuthority (15.20%)
Slogaa is by definition short and striking phrase.

---

Adllominem (13.58%), FalseCause (46.93%). Slip- peryslope (3.97%). and Slogans (2.63%). In the last debate, the most used fallacies are AppealioE- motion and AdHominem, confirming the trend of the previous debates. Behind this strategy, there are many references to the COVID-19 pandemic and some personal issues of the two candidates ex- ploited during the debates. Despite being distinct and unrelated, these topies held significant im- portance and consistently fueled intense debates.
4.1.1 Baselines
BERT + (Bi)LSTM(s)) The simplest models con- sist pre-trained BERT model followed by ci- ther (i) an LSTM layer and dense layer, or (ii) a BiLSTM layer with 0.2 dropout, an LSTM layer, and a dense layer. The weights of the transformer are kept frozen during training. The text serves as input for the transformer, and we extract the last hidden states (i.e., the embedded representa- tion of each token). This output then passed on to the subsequent layers. the case where argumentative features are included in the model, we concatenate the last hidden states of the trans- former with the one-hot-encodie representation of the argument components and relationships. This concatenated feature representation is then fed into the next RNN-based layers. All models used Adam optimizer with default PyTorch parameters.
4 Fallacy Detection
We cast the fallacy detection task as an information extraction problem, where the goal is to identify and classify in the debates the textual snippets cor- responding to the six categories of fallacies anno- tated in the context of a political debate (see Section 3.1 for the list of fallacies and their description).
We rely on the BIONOB data format, and specific tags are assigned to annotate the fallacies, i.e., AdHominem, I-Adlommnnem 3-AppealioAuhority. I-A Aptheattouuhhrrty Emopponn J- AppealtoEmotion, B-FalseCause, I-FalseCause, B- Slpppperssllpe, I-Slipperyslope, B-Slogans, I-Slogans, O.
is transformer- based model relying on bidirectional ap- proach to capture contextual information from surrounding words. We tested two checkpoints: bert base uncased and bert law.lrgg-aasdd-iiietnudooooloooo
The fallacy detection and classification tasks con- sist therefore in assigning one of these thirteen pre- defined labels each token.
is based on a modified transformer architecture with improve- ments like "dc-coupled attention" and "cross- layer parameter sharing" for enhanced language modeling capabilities. The checkpoint used is microsoft/deberta-bbn
To have richer representation of fallacy anno- tations, we build contextual framework that in- cludes the sentence containing the fallacy, well the preceding and following sentences. When the fallacious sentence is the first last in the dialogue, the preceding or following sentence is excluded.
ElectraForrTokeenChsiiieeee novel pre-training method called "dis- criminative pre-training." where a generator and discriminator are trained to enhance the quality of the learned representations. The checkpoint used is: Shadresh-savani/ wawtrraaaabeddiiciiiiie wwwriiiminttriiiini fineeteed-------- con conl103-englist
relies
4.1 Method
To address the above-mentioned tasks, we employ transformer-based architectures in both their ba- sie configuration and specialized configuration designed for token classification", drawing inspira- tion from previous studies on fallacy detection and classification (Da San Martino et al., 2020a; Vorak- itphan et al., 2022: Goffredo et al.. 2022) that have provided empirical evidence of the advantages of text repre-
DistilbertFor DisiileerForTTkkeCClssiiiii TosseClaasificatioon is distilled version of BERT that retains much of its performance while significantly reducing the model size and computational resources quired for training and inference. The check- point used are: distilbert -awee--ass--cseed and Distilbert-base-uccaee
complementing transformer-generan sentations with non-textual features. Moreover, enhance the specialized architecture by including additional argumentative features.
4.1.2 MultiFusion BERT
Relying on the results obtained by the baselines on the fallacy detection task on
'https: / //tgging///u wttp:///////ococcce//eceeeeeeeee basks//ooken_lassffiictttt

---

Model
Avg macro F1 Score
BERT + LSTM
0,4697
BERT + LSTM (comp. and rel. features)
0,5142
BERT + BiLSTM LSTM
0,5495
BERT + BiLSTM LSTM (comp, and rel. features)
0.5614
BertFTC port-bass--ncaase
0.7096
BenFTC dbuut//bert-larg--caad oanaraase--iiintumnm-oooooeeeees
0.7237
DebertaFTC microssof/deeeettt
0.7222
ElectraFTC
0.4033
DistilberFFTC: distilbert--bae-cnne
0.7010
DistilberIFTC distilbert-baas--ueass
0.7047
MultiFusion BERT (comp.. rel. and PoS features)
0.7394
Table 3: Average macro Fl scores for fallacy detection (BIO labels are merged) using different models. The scores are based on average of 3 runs, except for BERT (Bi)LSTM(s) models, which were evaluated using 10 runs. (FTC stands for TForTokenClasssficaiinn
Features
Avg macro
creases the model performances in the context of fallacy classification (Goffredo et al., 2022). Such features can be extracted by specific annotations associated with each fallacy the ElecDDeb60to00 dataset: the type of argumentative component in which the fallacy may be present, and the argumen- tative relations among these different components.
Components
Relationships
PoS
Fl Score
0.6922
0.6922
0.7212
0.7278
0.7166
0.7166
0.7394
The integration of PoS information driven by the observation that certain fallacies exhibit dis- tinctive language patterns that can be more eas- ily discerned using PoS tagging. For example, in the LoadedLanguage fallacy (a subcategory of the AppealloEmottion category), the intensity of sentence is often increased by using emotionally loaded phrases, expressed through the use of timent lexicon, particularly concerning adjectives and adverbs. Similarly, in the AdHominem fallacy, where the focus shifts from attacking the argument to targeting the character, motives, or personal qual- ities of the political opponent, the reference to this opponent is expressed using noun or pronoun and subsequently employs adjectives with negative connotations.
Table 4: Average macro Fl scores for fallacy detection (BIO labels are merged) using MultiFusion BERT and different features. The scores are based on average of 3 runs.
the ElecDeb60to20 dataset reported in Table 35, we select the BertFTC model farre--aset--iintunn--oloolllllo to be included in our proposed architecture. Despite the good performances of BertFTC, we propose enhance its capabilities to detect fallacious text by integrating and "fusing" addi- tional features, namely argumentative components (Claim, Premise), argumentative relations (Support, Attack), and Part-of-Speech (PoS) tags.
Figure illustrates the proposed model, called MultiFusion BERT, for the detection and classi- fication of fallacies in political debates. Multi- Fusion BERT computes logits (L) for each fea- ture by employing specialized TokenForClassi fication Transformer model adapted to the num- ber of labels: 3 for components and relations. and 17 for part-of-speech tags. The architectures for argumentative features for components and rela- tions share the same parameters, enabling us obtain logits for both components and relations. An additional model, based on the number of PoS
Argumentative features were included im- prove the model's understanding of an argument underlying structure, enabling detect when its logical structure is compromised. Since fallacies involve faulty reasoning, we hypothesized that pro- viding this information to the model would be rele- vant. Results in Table 4 provide empirical evidence to support this hypothesis. In particular, it has been shown that including argumentation features in-
"The reported results represent the average performance based on the macro average Fl score.

---

FALLACY
scikit-learn' library. The random seed was 42. The PoS tags were obtained using the spaCy$ library. The maximum tensor size set to 256, ensuring that all necessary text information in- cluded without truncation. The representation of this encoding is showed in the Appendix A.
FALLACY
ARG COMPONENT
RELATIONSHIP
TAGS
BertForToken classification
BertfortokeClasssfictttt
BertForToken Classification
los9/al
IosRconp
1oss
10s8pon
The proposed neural architecture contains 328 million parameters This large parameter count enables the model to capture intricate patterns and dependencies within the data, enhancing its capac- ity for complex information processing and gen- erating more accurate predictions. We evaluated the "MultiFusion" approach on the baseline model that showed the best results. Despite having ap- proximately half of the trainable parameters of BERT (e.g.. 65M for DistilBERT vs. 109M for BERT), BERT outperformed DistilBERT. Conse- quently, we adopted BERT as the architecture for implementing the approach described above. We utilized the Nvidia Quadro RIX 8000 GPU (32 GB) for our experiments. The average runtime was of 21 minutes for training and testing all the configurations of our models.
Joint Loss Learning
ArgMax
Figure MultiFusion BERT with joinhioss approach.
tags (i.e., 17), is used to obtain logits for PoS features. Consequently, distinct losses are com- puted for each model: fallacy loss (loss com- ponent loss (lossemp), relation loss (lossyel), and part-of-speech loss (lo8spos). These individual losses are combined by multiplying them with an arbitrary a value of 0.1, yielding a unified average loss referred to the jointloss (Vorak- itphan et al., 2021). In our study, opted for empirically investigating the optimal alpha value that yielded superior performance, as evidenced by our experiments (see Appendix D for the ex- haustive evaluation). The fack-propagation func- tion incorporates all losses in the following way: jointtoss Q (loss foosaa---sssmms-sssss rlsssrss)
Evaluation
Table 3 presents the results of the tested models for fallacies detection in the political debates. Results are calculated using the macro average F1 metric, considering the following fallacy labels: (i) Ad Hominem, (ii) Appeal to Authority, (iit) Appeal to Emotion, (iv) False Cause, (v) Slippery Slope, (vi) Slogans, and (vūī Other (B and / labels are merged). Despite the relatively smaller size of the dataset and the task complexity, the results ob- tained from the different models are promising. As introduced before, among the baselines, BERT "db- mdz/ saat--aagg-caed-fiiie fiwnetuneddcoool0----ee achieved the best performance. Thus, MultiFu- sion BERT, incorporating argumentative features (components and relations) as well as PoS tags, significantly outperformed the other models (the performance increase with respect to BertFTC is 2.12%.
Nioes
where Nioss denotes the number of losses consid- ered by the model. We conducted an exploration of various values for the a parameter.
4.2 Experimental Setup
All models have been fine-tuned using the ElecDeb60ro20 dataset. The implementation was based on HuggingFace version 4. 30. and on Py- Torch 1.7.0. 0. All models utilize the Adam opti- mizer, with gradient clipping set to 10, dropout of 0.1,, learning rate of 4e- 05 and training batch size of 8 and a test batch size of The training pro- cess consists of 4 epochs, during which the models are fine-tunee tunettnnd and optimized. The dataset was split with 90% for training and 10% for testing, ensur- ing balanced distribution on the fallacy labels using stratification. The partitioning was performed us- ing the train_ands testd__tt___e test_esssllllll spslt function from the
http:///ww.....aeeoo.o//////t/ae split.html
train_test.
" "https://spacy.i
"The standard BenFFCC model around three times smaller, approximatoly 109 million parameters.
index

---

5.1 Ablation Tests
To better analyze the impact of the different fea- tures incorporated in our architecture, we carried out ablation tests. Table 4 presents the results obtained by MultiFusion BERT using all possi- ble combinations of (i) argumentative components, (ii) argumentative relations, and (iii) context PoS tags. Incorporating argumentative components, re- lations, and PoS features individually or pairs resulted in a decline performance compared to the best baseline results (i.e., BertFTC "dbmdz/ bert-large-casee trrge-cased-finn finetumeddccoliii average degradation of 4.35% across the different configurations (excluding the one considering all three features). In contrast, when all three features are included (as described Section 4. 1.2) sig- nificant improvement in model performance ob- served, highlighting the importance of considering all of them together for fallacy detection.
Appostoaudhiiity
The Talerr
Sopars
Antoomen
Appraissuuttoryy Sotttry
Diyner Slogee
with an
Prodicted
Figure 2: Normalized confusion matrix MultiFusion BERT. BIO labels are merged. Normalization per- formed using the number of true elements in each class.
by the model, showing the highest performances (0.89 and 0.84). The definition of "Slippery Slope" revolves around portraying improbable or exagger- ated consequences arising from a specific action, and argumentative components are often used to the cause, well as semantic nuances well cap- tured by the model.
5.2
Error Analysis
Label
precision
recall
fl-score
support
AdHominem
0.99
0.77
0.87
739
AptholtoAuthority
0.90
0.78
0.83
1'049
EmpoionoEmotion
0,82
0.77
0.79
2'224
The results obtained for the other labels are line with those in (Goffredo et al., 2022) for the classification task only. The addition of new fal- lacious examples from the 2020 debates kept un- changed the distribution of fallacies with respect to the previous debates, suggesting that the detection of fallacious snippets remains consistent and stable across different debate contexts.
FalseCause
0.82
0.86
0.84
321
Slipperyslope
0.90
0.88
0.89
332
Slogans
0.00
0.00
0.00
49
O
0.90
0.95
0.93
7'914
accuracy
0.89
12'628
avg
0.76
0.72
0.74
12'628
weighted avg
0.89
0.89
0.89
12'628
Table 5: Classification report of Fallacy Detection and Classification with and / labels merged.
For better understanding of the predictions made by the Multilussio BERT we analyze the normalized confusion matrix visualized in Figure 2. The normalized version is preferred due to the dataset class imbalance. Notably, although the class with the highest Fl score is O (representing non-fallacies), the confusion matrix reveals that the model tends to over-predict instances in this cate- gory. As observed in the column of the predicted O class, false positives are the most prevalent in the non non-fallacious tokens. Moreover, False Cause and Appeal to Emotion are the classes that the models misinterpret the most as non fallacious. In a smaller proportion, the model misclassifies instances of Ap- peal Authority Appeal to Emotion.
Table provides an in de-th analysis of Multi- Fusion BERT's performances on the test set, con- sidering the different target " Notably, the identification of tokens labeled as Slogans exhibits the poorest results, despite being relatively easier recognize for humans. This can be due to the limited presence of examples/tokens in both the training and the test ". In addition, these results point out that recognizing slogans within politi- cal debates involves factors beyond syntactic and argumentative features (mostly semantics and prag- matics). On the contrary, tokens labeled as "Slip- pery Slope" and "False Cause" (with 332 and 321 examples, respectively) much better classified
Table 6 shows a few misclassified fallacy snip- pets. In the first example, the argument misclas- sified as Appeal to Emotion instead of Appel to Authority, because the model is misled by the word
10 A detailed analysis the performances with tokens is provided in Appendix B
A detailed table with the count of cach token can he found
in Appendix C

---

Fallacy snippet
True
Pred.
perform the fallacy detection and classification task. Our results highlight the main role of argumentative features in the correct identification and classifica- tion of fallacious arguments. This approach yields an average performance improvement of 2. 2.12% compared to baseline methods and competing ap- proaches.
fallacy
fallacy
Franklin Roosevelt said in 1932 that the only thing we have to fear is fear itself.
Appeal
Appeal
Authority
Emotion
As the President said the other night, there will always frou- bles in this ol' world, but the United States of America counted on to provide the vision that the world looks for from the United States of America.
Q
Appeal to
Authority
and O
As future research, we intend to delve deeper into fallacious argumentation by integrating knowl- edge in order address more challenging fallacy categories like causal ones, where reasoning and knowledge-based features are required to identify the fallacy. Our further objective is to generate valid arguments from identified fallacious ones and their context. Additionally, challenge we aim to tackle is to explore ways to counter the formal invalidity of fallacious arguments through the gen- eration of new arguments.
But as Admiral Yarnell has said, and he's been supported most military authority, these islands that we're not worth the bones of single American soldier; and I know how difficult it is to sustain troops close to the shove under artillery bombard- ment.
Appeal Authority
to Appeal to Emotion and 0
talking about are
In a place like Chicago, where thousands of people have been killed, thousands over the last number of yeurs, in fact, almost 4,000 have been killed since Borack Ohama became president, overall most 4,00 people Chicago have been killed. We have bring back law and order.
False Cause
False
Cause and
0
7 Limitations
Some limitations of this work require discussion. Firstly, the used training corpus is focused on US political debates, which restricts the applicability of the model to English-language contexts only. Fur- thermore, the imbalanced distribution of labels had noticeable impact on the model's performance and its ability to generalize during prediction. For instance, the label "Slogan" was significantly under- represented compared to other labels, further affect- ing the model's performance. Finally, impor- tant to consider that the GPU requirements, specif- ically the need for Nvidia RTX 8000 with 32GB VRAM, may present limitations on the practical utilization of these models in resource-constrained environments. These limitations highlight the need for further research address the dataset limita- tions with respect to the employed language and the label balance, to improve the model architec- ture, and explore additional strategies to enhance fallacy detection through knowledge injection.
Table 6: Examples misclassification using the best- performing model. Underlined text is to highlight the true label for each token, whereas Bold is for the pre- dicted fallacy and Italic for predicted O tokens.
"fear," which carries an important emotional conno- tation. In the next example, the argument is erro- neously classified as being an Appeal to Authority argument whilst it is not fallacious argument (O). The third example shows another instance where the model confuses Appeal to Authority with Ap- peal to Emotion, while also failing to identify part of the fallacy in general. The third and the fourth examples show where the model partially identifies the correct fallacy or its absence.
6 Conclusion
Existing argumentation schemes (Walton, 1995) to identify flawed and invalid forms of reasoning of- ten fall short when applied to fallacious arguments employed in real-world contexts like political de- bates. To tackle this challenge, the contribution of this paper is twofold. First, we extended the ElecDeb60to16 dataset by incorporating the Trump vs. Biden 2020 presidential debate along with argumentative annotations and fallacies. Second, we proposed and evaluated MultiFusion BERT, transformer-based architecture that combines the debate text, the argumentative features (i.c., com- ponents and relations), and engineered features to
8 Acknowledgments
This work was partly supported by the French gov- ernment, through the 3IA Côte d' Azur Investments in the Future project managed by the National Re- search Agency (ANR) with the reference number ANR-19-P33 MUR-19-P31A-0002 This work was partly sup- ported also by EU Horizon 2020 project A14Media, under contract no. 951911 (https//ai4mmedaa.e)) This work has been partially supported by the ANR project ATTENTION 1-R-21-CE2230007

---

References
pages 4143-4149. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, and Smaranda Muresan. 2022. Multitask instruction- based prompting for fallacy recognition. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8172-818 8187, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ivan Habernal, Raffael Hannemann, Christian Pol- lak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argu- mentation meets serious games. In Proceedings of the 2017 Conference on Empirical Methods in Nat- urul Language Processing: System Demonstrations. pages 7-12, Copenhagen, Denmark. Association for Computational Linguistics.
Elena Cabrio and Serena Villata. 2018. Five years of argument mining: a data-driven analysis. In Proceed- International Joint Con-
ings of the Twenty-Sevent ference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 5427-5433. ijcai.org.
Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018. Adapting Serious Game for Fallacious Argumenta- tion to German: Pitfalls, Insights, and Best Prac- tices. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki. Japan. European Language Resources Association (ELRA).
Giovanni Da San Martino, Alberto Cedren, and
Preslav Nakov. 2019a. Findings of the NLP4IF-2019 shared task on fine-grainee propaganda detection. In Proceedings of the Second Workshop on Natural Lan- guage Processing for Internet Freedom: Censorship. Disinformation, and Propaganda, pages 162-170, Hong Kong, China. Association for Computational Linguisties.
Shohreh Haddadan, Elena Cabrio, and Serena Villata. 2019. Yes, can! mining arguments in 50 years of US presidential campaign debates. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4684-4690, Florence, Italy. Association for Computational Linguisties.
Giovanni Da San Martino, Alberto Barrón-Codeñoo Henning Wachsmuth, Rostislay Petrov, and Preslav Nakov. 2020a. SemEval-2020 task II: Detection of propaganda techniques in news articles. In Pro- ceedings of the Fourfeenth Workshop Semantic Evaluation, pages 1377-1414, Barcelona (online). International Committee for Computational Linguis-
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xi- aoyu Shen, Yiwen Ding. Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. 2022. Logical fallacy detection. arXiv preprint 17X1v:2202.13558.
tics.
Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assissee and snowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrutions. pages 5-9, Santa Fe, New Mexico. Association for Computational Linguistics.
Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeñna Seunghak Yu, Roberto Di Pietro. and Preslav Nakov. 2020b. A survey computa- tional propaganda detection. In Proceedings of the Twenty-Ninth International Joint Conference on Arti- ficial Intelligence, IJCAI-20, pages 4826-4834 4222. Inter- national Joint Conferences on Artificial Intelligence Organization. Survey track.
Anne Lauscher, Henning Wachsmuth, Iryna Gurevych, and Goran Glavas. 2022. Scientia potentia est- on the role of knowledge in computational argumentation. Trans. Assoc. Comput. Linguisties, 10:1392-1422.
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natu- ral language processing (EMNLP-IJCNLP)) pages 5636-5646.
John Lawrence and Chris Reed. 2019, Argument min- ing: A survey, Comput. Linguistics, 45(4)
Elena Musi, Myrto Aloumpi, Elinor Carmi, Simeon Yates, and Kay O'Halloran. 2022. Developing fake news immunity: fallacies misinformation triggers during the pandemic. Online Journal of Communica- tion and Media Technologies, 12(3).
Frans H. Van Eemeren. 2001. Fallacies. Critical con- cepts in argumentation theory, page 135-164.
Frans H. Van Eemeren and Rob Grootendorst. 1987. Fallacies in pragma-dialectical perspective. Argu- mentation, 133):283-301.
Saumya y Sahai, Oana Balalau, and Roxana Horincar. 2021. Breaking down the invisible wall of infor- mal fallacies in online discussions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 644 657, Online. Association for Computational Linguistics.
Pierpaolo Goffredo, Shohreh Haddadan, Vorakit Vorak- itphan, Elena Cabrio, and Serena Villata. 2022. Fal- lacious argument classification in political debates. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22,

---

Prashanth Vijayarahhvvan and Soroush Vosoughi. 2022. TWEETSPIN: Fine-gr Fine-grained propaganda detection in
"And there is system In fact, it's starting apart not becouse of lawsuls - though they are problem, and John Edwards and are committed fixing them because of don't Ameticans, Children country
CCOML, PROK VORD, DET. HOUL,
ADP
ROUN,
social media using multi-view representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, pages 3433-3448, Seattle, United States. Association for Computational Linguistics.
Encoding
Vorakit Vorakitphan, Elena Cabrio, and Serena Villata. 2021. "Don't t discuss": Investigating Semantic and Argumentative Features for Supervised Propagandist Message Detection and Classification. In RANLP 2021 Recent Advances in Natural Language Pro- cessing, Varna / Virtual, Bulgaria.
input_ids: [0, 16, 467, 11,-,
attention_mask 1, 1, 1,
offset__appinng [[0,0], [0,3], [0,5], [0.2], [0,0]]
labels: [-100, 12, 12, 1280,118,6.72
7.7.7.7.74 7.7.7.7.77777.. 7.7.
7.7.7...7. 7.777.7 187,7-
Vorakit Vorakitphan, Elena Cabrio, and Serena Villata. 2022. Protect: A pipeline for propaganda detection and classification. In CLiC-it 2021 IItalian Confer- ence on Computational Linguisties.
7,,77,
-180,
12, 12, 12.1221110 100, 12, 12,
12,
-100,
-189]
100, 2, 2.
[-100, 2, 2,
-880,
2
2,2,2,2, 2,
2,
2,2,
2,
-100, 2,
2,
-180,
169]
D.N. Walton. 1987. Informal Fallacies: Towards a The- ory of Argument Criticisms. Pragmatics & beyond companion series. J. Benjamins Publishing Company.
rels: 11, 1, 1,1,1,1,
108,
1,
188. 1, 1,111,
1,1,1,11, 1,1,, 1,,,,,
1,,,,,
1, 1108, 1,1, -188)
1,8,
1,0,1,,1111188
D.N. Walton. 1995. Pragmatic Theory of Fallacy. Studies in rhetoric and communication. University of Alabama Press.
[4,
7, 13, 1,
3,5,
3,
12,4, 13, 7,,
11, 12,
1,
3,
7.7. 12,
100
100
A Encoding
A representation of the dataset's encoded example is illustrated the Figure 3, demonstrating how is prepared for input into the architecture for token label prediction. Notably, the argument features align with the offset mapping approach employed by the tokenizer during tokenization.
Figure 3: Encoded example of single item of the dataset ElecDeb60to20.
Label
precision
recall
fl-score
support
B-AdHominem
1.00
0.19
0.31
B--ApealloAAuhhri
0.75
0.60
30
A-ApppaatoEmotion
0.72
0.51
120
B-FalseCause
0.75
0.33
0.46
Each argumentative feature is represented by tensor of length 256 (the maximum token length of 216 is used) filled with label IDs (0: None, Clain/Support, 2: Premise/Atrack) up to the max- imum length. The same process is applied to the Part-of- Speeech tensor, where each tag is converted into its corresponding ID (0: ADJ, 1: ADP, 2: ADV, 3: AUX, 4: CCONJ, 5: DET, 6: INTJ, 7: NOUN, 8: NUM, 9: PART, 10: PRON, 11: PROPN, 12:
B-Slipperyslope
0.33
0.12
0.18
8
B-Slogans
0.00
0.00
0.00
5
I-AdHommien
0.98
0.79
0.88
712
AutpoalloAuthority
0.90
0.78
0.84
1'019
Apppealto@inotiun
0.81
0.78
0.79
2'104
I-FalseCause
0.81
0.87
0.84
312
S-Slippeyyslppe
0.89
0.89
0.89
324
I-Slogans
0.00
0.00
0.00
44
0.90
0.95
0,93
7'914
accuracy
0.88
127628
macro weighted avg
0.68
0.51
0.56
127628
0.88
0.88
0.88
12'628
PUNCT, 13: SCONJ, 14: SYM, 15: VERB, 16: X).
Table 7: Classification report of fallacy entity classifica- tion with BIO labels.
B Detailed Model Performance
In this appendix section, a comprehensive classifi- cation report of the best-performing model is pre- sented, considering all BIO labels and all the three features (urgumentative components, argumenta- tive relations and PoS tags). This report provides a thorough assessment of the model performance, offering valuable insights into its accuracy and ef- fectiveness in recognizing and classifying different categories to every token.
Token Distribution
Table 8 shows the distribution of the BIO tokens among the training and test set.
D Analysis of Different a Values
MultiFusion BERT's loss function defined as jointioes a tlosssen toss tlosspos)
Nisss

---

Label
Train
Test
Total
B-AdHominem
243
27
270
B-AppealtoAuthorig
272
30
302
BooopoolloEmootion
1'073
120
1'193
B F-FaleCCuuse
84
9
93
B S-Siipperys
71
8
79
B-Slogans
47
5
52
I-AdHominem
4'855
712
5'567
L-AppralloAuthoriiy
8'707
1'019
9'726
EmopooaltEmotionn
17'408
2'104
19'512
I-FalseCause
3'076
312
3'388
I-Slipperyslope
2'487
324
2'811
I-Slogans
254
44
298
74'493
7.914
82407
Total
113'070
12'628
125'698
Table 8:
Distribution of BIO fallacy tags among the
train test set.
Features
Avg macro Fl Score
Components
Relationships
PoS
0.6922
0.6922
0.7212
0.1
0.7278
0.7166
0.7166
0.7394
0.7054
0.7054
0.7214
0.3
0.6889
0.7160
0.7160
0.7084
0.7057
0.7057
0.6817
0.5
0.7366
0.7054
0.7054
0.7070
Table 9: MultiFusion BERT's average macro F1 scores for fallacy detection using different features and values of the parameter. The scores based average of 3 runs. and / labels were merged.
where Nioss denotes the number of losses consid- ered by the model. We conducted an exploration of various values for the a parameter. Table shows that the impact of the parameter varies depending on the features used the model. That is, none of the values of a yields a generalized improvement in the macro Fl score across all feature combinations.


Reviews:
[{"review_id": "TSgdRJHt70", "reviewer": "Reviewer_CQr9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments.\nWe agree on the fact that having a bigger dataset would strengthen even more our findings, and this is also why we extended the ElecDeb60To16 dataset by incorporating the most recent Trump-Biden presidential debates. The size of the considered dataset - consisting of approximately 1,600 fallacy examples, is well within the range of existing fallacy corpora (e.g. Musi et al (2022), Sahai et al (2021), Jin et al (2022)), making it a suitable benchmark to prove the robustness of the experimental validation. \nMoreover, the ElecDeb60To16 is the only available dataset - and therefore the largest we could use - for political debates, which has been proven to be the main context in which fallacies are employed to mislead the audience.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper exhibits a well-organized structure, systematically presenting each aspect of the research. The major claims are not only articulated clearly but are also substantiated by comprehensive experimental results, lending credibility to the findings. The authors introduce a novel model architecture that is both intriguing and relevant to the field of natural language processing (NLP). The innovative design contributes to a marked improvement in classification performance, demonstrating superiority over a wide array of currently popular methods. This comparative analysis strengthens the paper's position within the existing body of work. Furthermore, the authors undertake a meticulous analysis of the model's errors, providing insights into the underlying causes and potential areas for further refinement. They also engage in a thoughtful discussion of the model's limitations, showcasing a balanced perspective. This critical examination not only adds depth to the paper but also guides future research directions. Overall, the paper represents a significant contribution to the field of NLP, with a blend of originality, rigorous experimentation, and thoughtful analysis. It is a commendable piece of research that may inspire further exploration and development in the domain", "reasons_to_reject": "The proposed paper's primary limitation lies in its reliance on a relatively small dataset for the evaluation of the method. Such a limited scope of data undermines the robustness of the experimental validation and raises concerns about the generalizability of the findings. This lack of comprehensive evaluation decreases the paper's potential contribution to the field and calls for a more thorough investigation before it can be considered for acceptance", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents two main contributions. First, it extends an existing resource, the ElecDeb60to20 dataset, which includes annotated U.S. political debates from presidential campaigns (1960-2016) with argument components, relations, and six fallacy categories (Ad Hominem, Appeal to Authority, Appeal to Emotion, False Cause, Slippery Slope, and Slogans). The extension incorporates the 2020 presidential campaign debates (Trump-Biden) with all corresponding annotations. Second, the paper introduces a novel approach using Transformers, named MultiFusion BERT, to detect and classify fallacious text snippets in these debates across the six fallacy categories. By encoding argument components (Premise, Claim), relations (Support, Attack), and PoS tags, the approach successfully identifies and classifies fallacious arguments. Experimental results demonstrate that the proposed method outperforms standard baselines and concurrent approaches, achieving an average F1-score of 0.74. The originality lies in proposing the first neural architecture specifically for detecting and classifying fallacious arguments in political debates, outdoing competing methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5aJ4K0a6VX", "reviewer": "Reviewer_YLEV", "summary": "", "questions": "Regarding the limitations, you mentioned the aspect of the smaller model, which I also think is very important. I wondered if a smaller model like \"distilbert\" enriched with the presented additional features in joint training could also give good results. Have you also tested your ModelFusion with smaller models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments.\n1) We agree on the importance of exploring fallacies and the impact of such arguments in the field of political research. If our work is accepted at EMNLP, we will dedicate part of the extra-page to add such discussion, which is our current work in collaboration with colleagues studying philosophy of language and political argumentation. \n\n2) Concerning the features selection (argumentative features and Part-of-Speech):\ni) Argumentative features have been included to improve the model's understanding of an argument's underlying structure, enabling it to detect when this structure is compromised. Since fallacies involve faulty reasoning, we hypothesized that providing this information to the model would be relevant. Results in Table 4 provide empirical evidence to support such hypothesis.      \nii) The integration of  PoS information was driven by the observation that certain fallacies exhibit distinctive grammatical patterns that can be more easily discerned using PoS tagging. For example, in the \"loaded language\" fallacy (a subcategory of the \"appeal to emotion\" category)  the intensity of a phrase is often increased by using emotionally loaded phrases, expressed through the use of adjectives and adverbs. Similarly, in the ad hominem fallacy, where the focus shifts from attacking the argument to targeting the character, motives, or personal qualities of the arguer, it is usual to reference someone using a noun or pronoun and subsequently employ adjectives with negative connotations.\n\n3) Concerning  reproducibility: In Section 4.2 we detail the experimental setup: in particular, we detail the model parameters as well as the libraries version used during the training and evaluation phases. We will make both the dataset and the source code available through a dedicated GitHub repository upon paper acceptance. \n\n4) Regarding the discussed architectures, we employ transformer-based architectures in both their basic configuration and in a specialized configuration, in both \"cased\" and \"uncased\" variations, as well as both standard and \"large\" configurations. Section 4.1.1  summarizes the  architectures chosen for evaluation, among which the DistilBERT model is included. We decided to evaluate the \"MultiFusion\" approach on the baseline model that yielded the most favorable outcome. Despite DistilBert having approximately half the number of trainable parameters compared to BERT (65 million for DistilBert vs. 109 million for BERT), the performance of BERT was superior. As a result, we selected BERT as the architecture to which we applied the approach introduced in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The authors provide a solid motivation for their task of fallacy detection at the token level within the context of political debates.\n* The most crucial aspect of the paper is the extension of a relevant dataset with fine-grained additional information at the token-span level. This extensive data augmentation enhances the study's comprehensiveness and allows for more nuanced explorations. It also broadens the scope of potential applications, fostering an environment for future research and advancements in the field.\n* The authors have proposed a novel approach, compared to multiple baselines, to execute the task of fallacy detection. This new method is sound, offering a fresh perspective on addressing such a complex problem. Moreover, introducing the concept of \"fusing\" additional features is an exciting development and has been shown to enhance the model's performance moderately.", "reasons_to_reject": "* The study holds the potential to deliver even greater insight if it could allocate more focus toward the field of political research, especially the effects and consequences of such arguments. Adding this focus could make the discussion richer and wider.\n* The new method mentioned in the paper (ModelFusion) could be discussed in more detail. Understanding the rationale behind the authors' selection of features such as PoS over other (meta-)aspects could add another layer of depth to this work. It appears that a grounded motivation, in terms of political debates and similar features, is somewhat lacking in the paper. Currently, the paper is strong on its focus on NLP aspects. Adding more context on how these decisions relate to the domain of political debates etc. could make it even more comprehensive.", "questions_to_authors": "Regarding the limitations, you mentioned the aspect of the smaller model, which I also think is very important. I wondered if a smaller model like \"distilbert\" enriched with the presented additional features in joint training could also give good results. Have you also tested your ModelFusion with smaller models?", "paper_topic_and_main_contributions": "This paper focuses on the extension of an existing corpus of U.S. political debates, covering English transcripts from 1960 to 2016, by extending it with additional annotations of argument components, relations, and a unique classification of six fallacy categories, comprised of 2357 features. Another key contribution is the development of a novel methodology named MultiFusion. This approach leverages transformer models to detect fallacious text snippets by integrating the learning of argument components, part-of-speech tags, relationships, and fallacy tags. The proposed method has demonstrated robust performance with an f1 macro score of .74. In totality, the paper is well-written, easy to follow, and methodically structured, presenting a solid new layer of information that contributes to the research community.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EIJAX7n5iS", "reviewer": "Reviewer_fe3c", "summary": "", "questions": "1. **Alpha Parameter Selection:** What is the rationale behind setting alpha to 0.1? In general, alpha might be considered as a coefficient to assign different weights to various losses (e.g., Loss1 -> 0.5, Loss2 -> 0.4, Loss3 -> 0.1, where the sum is 1). Could you provide more insight into this choice?\n2. **Final Target Clarification:** Could you clarify the final target in your classification with respect to Figure 1? Understanding this aspect could provide a clearer picture of the model's objectives and the overall methodology.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. \nConcerning the permissions for data collection: we would like to clarify that the debates that constitute the ElecDeb60To20 dataset were sourced from the website of the Commission on Presidential Debates, which openly provides transcripts of debates broadcasted on television and featuring the prominent candidates for presidential and vice-presidential nominations in the United States. All information on this website is accessible to the public. Also, the annotations of this dataset were carried out by annotators with a background in computational linguistics, that followed established ethical guidelines and best practices throughout the annotation process.\n\nQ1. Alpha Parameter Selection: Our methodology draws inspiration from the work of Vorakitphan et al. (2021), who applied this approach with an alpha value of 0.5. In our study, we opted for empirically investigating the optimal alpha value that yielded superior performance, as evidenced by our experiments (elaborated upon comprehensively in Appendix D).\n\nQ2. The primary objective of our work is to automatically identify segments within texts or debates and classify them into one of six specified labels corresponding to the initially mentioned fallacies. Illustrated in Figure 1 is the model's architecture designed to realize this objective, achieved through the incorporation of various inputs: the BIO tags associated with the fallacies, POS tags, Tags corresponding to the category of argumentative components (Claim, Premise, None), Tags representing the type of argumentative relationship (Attack, Support, None). Subsequently, the loss for each input is computed, and these individual losses are combined into a unified \"joint loss\" by applying an alpha coefficient to the average of the preceding losses. This approach facilitates the synthesis of multiple input sources into a cohesive training objective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Relevance to Contemporary Issues:** The task of fallacy detection and classification holds substantial importance in the current landscape of NLP. It's especially significant in contributing to broader subjects like ethics in NLP and the nuanced detection of misinformation.\n- **Potential Community Impact:** The introduction of the ElecDeb60to20 dataset is a commendable effort that may act as a catalyst for further research and innovation in the community.\n- **Modeling and Experimentation:** The modeling approach and conducted experiments appear robust and compelling. The design illustrated in Figure 1 offers a simplistic yet effective way to fuse three different losses. The use of the BIO (Beginning, Inside, Outside) setting adds to the transparency of the models' decision-making process, reflecting a thoughtful design choice.", "reasons_to_reject": "While the paper's strengths are many, one area of concern is the lack of detailed information regarding the Institutional Review Board (IRB) permissions for data collection. Clear documentation and adherence to ethical standards are crucial, and the absence of this information might raise questions about the dataset's collection methodology.\n", "questions_to_authors": "1. **Alpha Parameter Selection:** What is the rationale behind setting alpha to 0.1? In general, alpha might be considered as a coefficient to assign different weights to various losses (e.g., Loss1 -> 0.5, Loss2 -> 0.4, Loss3 -> 0.1, where the sum is 1). Could you provide more insight into this choice?\n2. **Final Target Clarification:** Could you clarify the final target in your classification with respect to Figure 1? Understanding this aspect could provide a clearer picture of the model's objectives and the overall methodology.\n", "paper_topic_and_main_contributions": "This paper focuses on the identification and study of fallacies within political debates. It makes a significant contribution by introducing a new dataset, ElecDeb60to20, along with a novel model that could potentially have a far-reaching impact on this subfield.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adds argumentation related annotations to an existing dataset on US political debates. No additions include argument component and relation annotations as well as 6 common logical fallacies.  It also presents MultiFusion BERT, a model to identify text snippets with a logical fallacy, achieving .74 F1.

Reviewers appreciated the strong motivation and the resource the authors construct, as well as a novel model. The paper would benefit from a more detailed description of the model with justifications for the design choices. Also, either expanding to wider domains or providing more detailed discussions within the political science domain would be helpful.
Decision: Accept-Main