SemEval-2023 Task 3: Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup
Year: 2023

Abstract:
We describe SemEval-2023 task 3 on Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multilingual Setup: the dataset, the task organization process, the evaluation setup, the results, and the participating systems. The task focused on news articles in nine languages (six known to the participants upfront: English, French, German, Italian, Polish, and Russian), and three additional ones revealed to the participants at the testing phase: Spanish, Greek, and Georgian). The task featured three subtasks: (1) determining the genre of the article (opinion, reporting, or satire), (2) identifying one or more frames used in an article from a pool of 14 generic frames, and (3) identify the persuasion techniques used in each paragraph of the article, using a taxonomy of 23 persuasion techniques. This was a very popular task: a total of 181 teams registered to participate, and 41 eventually made an official submission on the test set.

heading: 1 Introduction
text: The widespread use of Internet and the advances in Internet-related technologies paved the way to easily create direct communication channels between information producers and consumers, potentially leaving the latter exposed to manipulative, propagandistic, and deceptive content. Given the potentially huge audience that can be reached through online channels, major public events and debates revolving around relevant topics could be influenced as a result. Therefore, there is an ever-growing need to develop automated tools supporting experts in analysing the news ecosystem and identifying large-scale manipulation attempts, and facilitating the study of how different events, global topics, and policies are being embraced by media in various countries, in order to carry out cross-country analysis and to gather knowledge on the ways how media informs public opinion, i.e., what aspects
are being highlighted and linked to a topic, what pros and cons are mentioned, the way opinions are conveyed, and what rhetorical devices, i.e., logical fallacies and appeal to emotions, are used to support flawed argumentation, potentially leading to manipulation.
To foster research in this direction, there have been several shared tasks asking to detect the use of specific propaganda techniques in text, as well as the specific span of each instance. This includes the NLP4IF-2019 shared task on Fine-Grained Propaganda Detection (Da San Martino et al., 2019), SemEval-2020 task 11 on Detection of Persuasion Techniques in News Articles (Da San Martino et al., 2020a), SemEval-2021 task 6 on Detection of Persuasion Techniques in Texts and Images (Dimitrov et al., 2021b), and WANLP-2022 Shared Task on Propaganda Detection in Arabic (Alam et al., 2022).
Our task is an extension of the above ones and introduces several novelties. First, it is multilingual, covering nine languages. Second, it adds additional dimensions for better news understanding, i.e., framing and news genre detection. Finally, our taxonomy of persuasion techniques is an extension compared to previous inventories, and it contains 23 techniques organised in a two-tier hierarchy.

heading: 2 The Tasks
text: The shared task comprises three subtasks:
Subtask 1 (ST1): News Genre Categorization. Given a news article, determine whether: (a) it is an opinion piece, (b) aims at objective news reporting, or (c) is satirical.1 This is a multi-class classification task at the article level.
Subtask 2 (ST2): Framing Detection. Given a news article, identify one or more frames used in
1A satirical piece is a factually incorrect article, with the intent not to deceive, but rather to call out, ridicule, or expose behaviours considered ‘bad’. It deliberately exposes realworld individuals, organisations and events to ridicule.
the article from a pool of 14 generic framing dimensions (introduced in Card et al. (2015)): Economic, Capacity and resources, Morality, Fairness and equality, Legality, constitutionality and jurisprudence, Policy prescription and evaluation, Crime and punishment, Security and defense, Health and safety, Quality of life, Cultural identity, Public opinion, Political, External regulation and reputation. This is a multi-class multi-label classification task at the article level.
Subtask 3 (ST3): Persuasion Techniques Detection. Given a news article, identify the persuasion techniques used in each paragraph of the article. The pool of persuasion techniques consists of 23 techniques, and is an extension of the taxonomy introduced in Da San Martino et al. (2019); Dimitrov et al. (2021b)2. This is a multi-class multilabel classification task at the paragraph level. The persuasion techniques are grouped into six main categories: Attack on reputation: The argument does not address the topic, but rather targets the participant (personality, experience, deeds) in order to question and/or to undermine their credibility. The object of the argumentation can also refer to a group of individuals, an organization, an object, or an activity. Justification: The argument is made of two parts, a statement and an explanation or an appeal, where the latter is used to justify and/or to support the statement. Simplification: The argument excessively simplifies a problem, usually regarding the cause, the consequence or the existence of choices. Distraction: The argument takes the focus away from the main topic or argument to distract the reader. Call: The text is not an argument, but an encouragement to act or to think in a particular way. Manipulative wording: the text is not an argument per se, but uses specific language, which contains words or phrases that are either non-neutral, confusing, exaggerating, loaded, etc., in order to impact the reader emotionally.
Figure 1 gives an overview of the two-tier persuasion techniques taxonomy.

heading: 3 Related Work
text: This section discusses prior work related to each of the subtasks of the shared task.
2the second paper has five additional techniques with respect to the previous one

heading: 3.1 News Genre Categorization
text: Rashkin et al. (2017) developed a corpus with document-level annotations into four classes (trusted, satire, hoax, and propaganda), annotated using distant supervision. Horne and Adali (2017) studied the relationship between fake news, real news, and satire with focus on style. They found that fake news is more similar to satire than to real news. Golbeck et al. (2018) developed a dataset of fake news and satire stories and analyzed and compared their thematic content. Satire was also one of the categories in the NELA-GT-2018 dataset (Nørregaard et al., 2019), as well as in its extended version NELA-GT-2019 (Gruppi et al., 2020).
The set up of our shared task is different, and focusing on distinguishing between objective news reporting vs. opinion piece vs. satire.

heading: 3.2 Framing Detection
text: Framing is a strategic device and a central concept in political communication for representing different salient aspects and perspectives for the purpose of conveying the latent meaning about an issue (Entman, 1993). It is important for news media as the same topics can be discussed from different perspectives. There has been work on automatically identifying media frames, including annotation schemes and datasets such as the Media Frames Corpus (Card et al., 2015), systems to automatically detect media frames (Liu et al., 2019; Zhang et al., 2019), large-scale automatic analysis of New York Times Articles (Kwak et al., 2020), and a semi-supervised approach to detecting frames in online news sources (Cheeks et al., 2020).
In our shared task, we adopt the frame inventory of the Media Frames Corpus.

heading: 3.3 Persuasion Techniques Detection
text: Work on persuasion detection overlaps to a large extent with work on propaganda detection, as there are many commonalities between the two.
Early work on propaganda detection focused on document-level analysis. Rashkin et al. (2017) predicted four classes (trusted, satire, hoax, and propaganda), labeled using distant supervision. BarrónCedeno et al. (2019) developed a corpus with two labels (i.e., propaganda vs. non-propaganda) and further investigated writing style and readability level. Their findings confirmed that using distant supervision, in conjunction with rich representations, might encourage the model to predict the
source of the article, rather than to discriminate propaganda from non-propaganda.
An alternative line of research focused on detecting the use of specific propaganda techniques in text, e.g., Habernal et al. (2017, 2018) developed a corpus with 1.3k arguments annotated with five fallacies that directly relate to propaganda techniques. A more fine-grained analysis was done by Da San Martino et al. (2019), who developed a corpus of news articles annotated for 18 propaganda techniques, considering separately the task of technique spans detection and classification. They further tackled a sentence-level propaganda detection task, and proposed a multi-granular gated deep neural network. Subsequently, the Prta system was released (Da San Martino et al., 2020c), and improved models were proposed addressing the limitations of transformers (Chernyavskiy et al., 2021), or looking into interpretable propaganda detection (Yu et al., 2021). Finally, there is work addressing the detection of use of propaganda techniques in memes (Dimitrov et al., 2021a), the relationship between propaganda and coordination (Hristakieva et al., 2022), and work studying COVID-19 related propaganda in social media (Nakov et al., 2021a,b). See (Da San Martino et al., 2020b) for a survey on computational propaganda detection.
Several shared tasks on propaganda detecting in text were also organized. SemEval-2020 task 11 on Detection of Persuasion Techniques in News Articles (Da San Martino et al., 2020a) focused on news articles, and asked to detect the text spans where propaganda techniques are used, and to predict their type (14 techniques). Closely related to that is the NLP4IF-2019 task on Fine-Grained Propaganda Detection (Da San Martino et al., 2019), which asked to detect the spans of use in news articles of each of 18 propaganda techniques. The SemEval-2021 task 6 on Detection of Persuasion Techniques in Texts and Images focused on 22 propaganda techniques in memes (Dimitrov et al., 2021b), while WANLP’2022 shared task asked to detect the use of 20 propaganda techniques in Arabic tweets (Alam et al., 2022). Here, we extend and redesign the above annotation schemes.

heading: 4 The Dataset
text: This section provides a brief description of the dataset, whereas detailed guidelines, definitions and examples are provided in a separate technical report (Piskorski et al., 2023).
We collected articles in nine languages: English, French, German, Georgian, Greek, Italian, Polish, Russian, and Spanish published in the period between 2020 and mid-2022, and revolving around various globally discussed topics, including the COVID-19 pandemic, abortion-related legislation, migration, Russo-Ukrainian war, some local events such as parliamentary elections, etc. We considered both mainstream media and “alternative” media sources that could potentially spread mis/disinformation. For the former, we used various news aggregation engines, e.g., Google News3 and Europe Media Monitor4, etc., which cover sources with different political orientation, whereas for the latter, we used online services such as MediaBiasFactCheck5 and NewsGuard.6 We extracted the article texts either using Trafilatura (Barbaresi, 2021) or, in few cases, ad hoc procedures.
We annotated each text for genre, framing, and persuasion techniques using the taxonomy described in Section 2. While genre and framing were annotated at the document level, we annotated the persuasion techniques at the span level. We had about 40 annotators, who were either media analysts, disinformation specialists or NLP experts, most of which had prior experience in performing linguistic annotations. All annotators were either native or near-native speakers of the language they annotated for. We used the INCEpTION (Klie et al., 2018) platform for carrying out the annotations. The annotation interface for an example document using INCEpTION is shown in Figure 2.
As regards English, we exploited the texts from Da San Martino et al. (2019), but the annotations
3https://news.google.com 4https://emm.newsbrief.eu 5https://mediabiasfactcheck.com 6https://www.newsguardtech.com
for persuasion techniques have been slightly modified in order to match the extended taxonomy, most notably Whataboutism included two meanings: distracting from the main argument and calling the hypocrisy of the speaker; the latter meaning is now covered by the technique Appeal to Hypocrisy. Moreover, we added annotations for framing and news genre.
Each document was annotated by at least two annotators. Once the individual annotations for a document have been accomplished, a curator (an experienced annotator) with the help of annotators
consolidated the final annotation. The consolidation consisted of: (a) merging complementary annotations (tagged only by one annotator), (b) deciding whether overlapping annotations are to be kept as they are (multi-labels) or joined into a singlelabeled annotation, and (c) carrying out global consistency analysis. The detailed description of the annotation and the consolidation process are described in a detailed technical report (Piskorski et al., 2023). In order to assess the annotation quality, we computed the Inter-Annotator Agreement (IAA) using Krippendorf’s α: the value was 0.342, which is lower than the recommended threshold of 0.667, but we should note that this value represents the agreement before the consolidation, and as such, it is more representative of the consolidation difficulty rather than of the quality of the final consolidated annotations. Actually, we used IAA to allocate consolidation roles and to eliminate low-performing annotators.
We further studied the IAA by ranking the annotators by their performance with respect to the ground truth on the subset of documents they annotated. We then split them into two groups, top and low, based on the median micro-F1 scores. Their respective α scores were 0.415 and 0.250. Finally, we considered the value of α of the group of annotators, based on Italian, the only language with two curators, achieving 0.588, which is lower but close to the recommended value.
The annotated data, consisting of 2,049 documents in total, were divided into train, dev, and test datasets, whose high-level statistics are provided in Table 1. Georgian, Greek, and Spanish-annotated data was used only for testing (surprise languages). Table 2 shows the distribution of articles per language in terms of genre. Detailed statistics about the fine-grained persuasion techniques are shown in Table 17 in Annex A.

heading: 5 Evaluation Framework

heading: 5.1 Evaluation Measures
text: Subtask 1 is a multi-class classification problem. We used macro F1 as the official evaluation measure, but we also computed micro F1. Subtask 2 is a multi-label multi-class classification problem. We used micro F1 as the official evaluation measure, but we also computed macro F1. Subtask 3 is a multi-label multi-class classification problem. We used micro F1 as the official evaluation measure. The official score was computed
using the 23 fine-grained persuasion technique labels. We also computed macro F1.

heading: 5.2 Task Organization
text: The shared task was run in two phases: Development Phase: initially, only training and development data were made available to the participants, and no gold labels were provided for the latter. The participants competed against each other to achieve the best performance on the development set. They could make an unlimited number of submissions, and the best score for each team, regardless of the submission time, was shown in real time on a public leaderboard. Test Phase: in the second phase, the gold labels for the development and the test sets were released, and the participants were given a week to submit their final predictions on the test set. It is important to note that the test data contained news in three additional languages, i.e., Georgian, Greek, and Spanish, which were not known upfront to the participants (surprise languages). The participants could again submit multiple runs, but they would not get any feedback on their performance. Only the latest submission of each team was considered as official and was used for the final team ranking. Overall, 41 teams made official submissions to all the tasks, where 27, 22, and 22 teams submitted results for ST1, ST2, ST3, respectively. Moreover, 13, 14, and 14 teams submitted results for all languages for ST1, ST2, ST3, respectively.
The results for the development and the test phases are available on the leaderboard7 page. After the competition was over, we left the submission system open for the test dataset for post-shared task evaluations and to monitor the state of the art for the different subtasks across the languages.

heading: 6 Participants and Results

heading: 6.1 Subtask 1: News Genre Categorization
text: The full results for Subtask 1 are shown in Tables 3 and 4 (surprise languages). We used a linear SVM8 with class balancing, trained on 5-char n-grams as a baseline (highlighted in blue in the tables). Table 5 shows an overview of the approaches. Almost all participants used transformers. The scarcity of the annotated data was dealt with either by combining the datasets for all languages, e.g., via multilingual
7https://propaganda.math.unipd.it/ semeval2023task3/leaderboard.php
8https://scikit-learn.org
language models or by automatic translation, or by looking for similar datasets in the literature; ensemble methods have also been very popular.
Below, we give a short description of the system papers that were top-ranked for at least one language.
SinaAI (EL) used multilingual languages models, XLM, mBERT and LABSE, and ensembles thereof. They further used data augmentation by selecting 30% of the sentences of each document to create new synthetic examples.
DSHacker (ES) created synthetic texts for each class using the OpenAI GPT-3 Davinci language model. Each language was augmented by approximately 500 articles per genre, producing roughly 13,500 artificially generated articles. Then, they fine-tuned a single XLM-RoBERTalarge on the
original and the augmented data.
FTD (PL): They experimented with monolingual and multilingual models, ensembles, additional data, and uncertainty estimation. For Russian and English, they fine-tuned models pretrained on the FTD dataset for genre classification. For English, they added 1,000 reporting texts from Gigaword. For Polish and German, their best results were achieved by fine-tuning a monolingual Polish BERT and a monolingual German Electra, respec-
tively. For the other languages, their best systems used multilingual BERT, XLM-RoBERTa, or ensembles thereof. In all cases, they truncated the input to the first 510 tokens. They further upsampled the data to balance the distribution between the classes (the results without upsampling were low).
Hitachi (IT, RU) augmented the dataset for subtask 1 by collecting labelled examples from similar datasets. They pretrained (XLM-)RoBERTa in multi-task (one language, subtasks 1 and 2), multilingual (one subtask, all languages), and multilingual multi-task (subtasks 1 and 2 in all languages) settings. Besides using the single models, they report experiments with ensembles of base models and different hyper-parameter values.
MELODI (EN) fine-tuned the domain-specific
language model trained on English data, POLITICS, on the English input articles and on the articles in all other languages, which were automatically translated. In addition, in order to use whole articles as input, they used a sliding window and aggregated each window representation using mean-pooling. They also tested other multilingual approaches, such as XLM-RoBERTa, and were able to process long documents (Longformer), which were generally less effective.
UMUTeam (FR, DE) used a multilingual model based on XML-RoBERTa, which was fine-tuned on all languages at once and a sentence transformer model to extract the most important chunk of text. The input data was truncated to 200 tokens with 50 overlaps using the sentence-transformer model to obtain the subset of text most related to the article’s title.
SheffieldVeraAI (DE) deployed an ensemble of three fine-tuned mBERT models and one mBERT model with a bottleneck adapter. All used bertbase-multilingual-cased. The pool of training data was also extended by integration additional “satire” resources for English. The final predictions were drawn as a majority-voting predicted class.

heading: 6.2 Subtask 2: Framing
text: The full results for subtask 2 on framing classification are provided in Table 6 and 7 (surprise languages). We used linear SVM trained using word unigrams and bigrams as a baseline (highlighted in blue in the tables). Table 8 shows an overview of the approaches. Since the models were all
transformer-based, what differentiated the participating systems were once again the pre-processing and the data augmentation techniques. The vast majority of teams trained their systems on all languages and used ensembles.
MarsEclipse (IT, RU, FR, DE, PL): This team used a multi-label contrastive loss for fine-tuning XLM-RoBERTa using SimCLR and SimCSE and adapting the loss function to a multilabel setup.
mCPT (ES): This team used a two-phase training procedure of a transformer model, first by pretraining jointly on all the languages and then by fine-tuning for each language. In both phases, a multi-label contrastive loss was used.
SheffieldVeraAI (EN, EL, KA): The team achieved the best average rank score over all the languages. They used two different ensembles of MUPPET large and of XLM-RoBERTa large with adapters and task-adaptive MLM pretraining on the train+dev+test data. Their data was preprocessed and truncated. The models were trained both with and without class weighting.

heading: 6.3 Subtask 3: Persuasion Techniques Detection
text: The full results for subtask 3 on persuasion techniques detection are given in Tables 9 and 10 (surprise languages). We used linear SVM trained using word uni-grams and bigrams as a baseline (highlighted in blue in the tables). Table 11 shows an overview of the approaches used by the partic-
ipating systems. The big picture is very similar to the previous subtasks: multilingual transformer models were used by all participants, and what differentiated the approaches was again the preprocessing and data augmentation strategies, for example, a few teams made use of the span-level annotations.
APatt (EN): The team combined different finetuned transformer models (XLNet, RoBERTa, BERT, ALBERT, and DeBERTa) through a weighted average. For English, they weighted the predictions of the models to give higher importance to certain models.
KInITVeraAI (IT, RU, DE, PL, EL, KA): This team performed overall the best, using a fine-tuned XLM-RoBERTa-large transformer model trained on all the input data. They carefully adjusted the prediction threshold for each language using a principled approach. They truncated the input, and also found that preprocessing did not impact the quality much.
NAP (FR): The team presented an approach combining predictions of several models in an ensemble, which differ in three main aspects: a) training data, b) model architecture, and c) input format to the model. They leveraged translation as data augmentation strategies using available MarianMT models. Model architectures included XLM-RoBERTa models, Adapters, SetFit, and linguistically-informed heuristics for underrepresented techniques which were fine-tuned on different combinations of original and augmented data. They fine-tuned models on both paragraphand span-level information.
TeamAmpa (ES): The team used different oversampling strategies, data truncation, and monolingual and multilingually trained models, combined in an ensemble for the English Task 3 data. The surprise languages were handled using the multilingual model only, which where trained using XLM-R on all languages with oversampling, for one of these languages the team ranked first.

heading: 6.4 Aggregated results
text: Tables 12-14 report the average micro F1 scores of the teams who, for each subtask, submitted solutions for multiple languages: the 6 for which we provided training data (6L), the 3 surprise ones (3L), all of them (9L). Results are ranked by decreasing value on all.

heading: 7 Conclusions and Future Work
text: We presented SemEval-2023 Task 3 on Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup. The task attracted a lot of attention: 181 teams registered for the task, 41 teams eventually madevan official submission on the test set, and 32 teams also submitted a task description paper.
In future work, we plan to further increase the data size, cover additional languages, and explore different ways of evaluation of the persuasion technique detection, e.g., by changing the focus (sentence- and text span-level evaluation).

heading: 8 Limitations
text: Dataset Representativeness Our dataset covers a range of topics of public interest (COVID-19,
climate change, abortion, migration, the RussoUkrainian war, and local elections) as well as media from all sides of the political spectrum. However, it should not be seen as representative of the media in any country, nor should it be seen as perfectly balanced in any specific way.
Biases Human data annotation involves some degree of subjectivity. To mitigate this, we created a comprehensive 60-page guidelines document, which we updated from time to time to clarify newly arising important cases during the annotation process. We further had quality control steps
in the data annotation process, and we have been excluding low-performing annotators. Despite all this, we are aware that some degree of intrinsic subjectivity will inevitably be present in the dataset and will eventually be learned by models trained on it.

heading: Acknowledgments
text: We are greatly indebted to all the annotators from different organisations, including, i.a., the European Commission, the European Parliament, the University of Padua, and the Qatar Computing Research Institute, HBKU, who took part in the anno-
tations, and notably to the language curators whose patience and diligence have been fundamental for the quality of the dataset. We are also thankful to Nikolaidis Nikolaos for the preparation of the baseline models and for sharing various ideas.
Part of this work was supported by IDKT Fund TDF 03-1209-210013: Tanbih: Get to Know What You Are Reading.

heading: A Supplementary Corpus Information
text: A.1 Statistics
This section contains additional statistical information related to the corpus.
Table 15 provides the statistics for genre for all languages. One can observe that opinion and satire are the most and least populated classes across the languages respectively.
Analogously, Table 16 shows the number of different framing dimensions per language. Political and Security and Defense framings constitute the two most frequent ones across all languages. The distribution of the different framings varies across the languages though.
Finally, Table 17 reports the exact count of finegrained persuasion techniques per language for the entire dataset. The two most frequent techniques irrespective of the language are Loaded Language and Name Calling-Labelling, which account for 18.5% and 23.7% of the dataset, trumping by several order of magnitude the lower populated classes. They are followed by Casting Doubt (12.5%), Questioning the Reputation (7.6%), Appeal to Fear-Prejudice (4.8%), and ExagerationMinimisation (4.7%). These six classes together represent 71.8% of the entire dataset.

heading: B Participant Systems
text: In the following we list the systems of all participants who submitted a system description paper. The team name used for the submission is in bold; in case the team used a different name on the leaderboard, it is appended in parentheses; the list of subtasks the team participated in is also given in brackets; in case the team was ranked first for at least
a subtask-language pair, the list of all such pairs where it is ranked first is given; a list of keywords; and finally, a short description of the system.
ACCEPT [ST2] (Heinisch et al., 2023) (Keywords: XLM-RoBERTa, ConceptNet) They used an ensemble combining XLM-RoBERTa with static multilingual and monolingual word embeddings; for the latter, they translated the non-English-texts to English using Google Translate. They further experimented with external common sense knowledge graphs, specifically ConceptNet.
APatt [ST3] (Purificato and Navigli, 2023) (first for: ST3:EN) (Keywords: XLNet, RoBERTa, BERT, ALBERT, DeBERTa ) They used an ensemble of pre-trained language models fine-tuned on the propaganda dataset: BERT, RoBERTa, ALBERT, XLNet, DistilBERT, and HerBERT. Whenever more LLMs for the same language were available, their output is combined through a weighted average.
Appeal for attention (AppealForAtt) [ST3] (Amihaesei et al., 2023) (Keywords: XLMRoBERTa-Large, WordNet, data augmentation) They focused on data augmentation techniques. They translated the datasets from each language into all other languages using the DeepTranslator API, and they extracted synonyms from WordNet to generate new sentences. Finally, they trained XLM-RoBERTa-large on that augmented data.
BERTastic [ST2] (Mahmoud and Nakov, 2023) (Keywords: mBERT, XLM-RoBERTa) They used cross-lingual transformers, mBERT and XLMRoBERTa, using different orderings of the languages when doing fine-tuning. They further used test data augmentation via translation of both the training and the test sets.
CLaC (CLAC) [ST3] (Costa et al., 2023) (Keywords) RoBERTa augmented the dataset translating examples from other languages, focusing on articles having least represented techniques in order
to balance the dataset. They used a RoBERTa-base model trained on the English language and made predictions on the other languages by first translating the text into English. They report better F1 scores with such approach on French and German than using Large Language Models trained directly on the target language.
DSHacker [ST1, ST3] (Modzelewski et al., 2023) (first for: ST1:ES) (Keywords: XLMRoBERTa-large, GPT3-Davinci, sequence classification, text generation, Ensemble Learning, XGBoost, Logistic Regression, LightGBM, BERT, Data Augmentation, Summarization) created, for ST1, synthetic texts for each class using the OpenAI GPT-3 Davinci language model. Each language was augmented by approximately 500 articles per genre, producing roughly 13,500 artificially generated articles. Single XLMRoBERTa-large model was trained using the original and augmented data. For ST3 they developed for Polish an ensemble consisting of three one-vs-rest classifiers: eXtreme Gradient Boosting, Logistic Regression and Light GradientBoosting Machine with HerBERT embeddings and various stylometric features using StyloMetrix library. For all other languages BERT-based pre-trained models were deployed and they used summarization applied to longer paragraphs. Furthermore, training data was augmented through machine translation utilizing the DeepL API.
FramingFreaks [ST1, ST2] (Baumann and Deisenhofer, 2023) (Keywords: SVM, Logistic Regression) classified texts by splitting them into subwords and then using these tokens as input to a Support Vector Machines for ST1 and to Logistic regression for ST2.
FTD [ST1, ST2, ST3] (Lepekhin and Sharoff, 2023) (first for: ST1:PL) (Keywords: SLMRoBERTa, multilingual BERT, Electra, monolingual BERT-based models, fine-tuning, uncertainty estimation, ensembles) focused on ST1, where they experimented with monolingual and multilingual models, ensembles, additional data, and uncertainty estimation. For Russian and English, they finetuned models pre-trained on the FTD dataset for genre classification. For English, they added 1,000 reporting texts from Gigaword. For Polish and German, their best results were achieved by fine-tuning a monolingual Polish BERT and a monolingual German Electra, respectively. For the other languages, their best systems used multilingual BERT,
XLM-RoBERTa, or ensembles thereof. In all cases, they truncated the input to the first 510 tokens. They further upsampled the data to balance the distribution between the classes (the results without upsampling were low). For English they further, experimented with uncertainty estimation, and showed that replacing the model predictions that have high uncertainty with the majority class on the training data was helpful on the dev set. For ST2, for each language, they used the model and the setup that worked best for ST1, and just retrained it on the ST2 data.
HHU [ST1] (Billert and Conrad, 2023): (Keywords: XLM-RoBERTa, Adapters, AdapterFusion) used an Adapter-based configuration: Using XLM-
RoBERTa as a base, they stacked first a languagespecific adapter and then a task-specific adapter on top of it. Moreover, they augmented each dataset by translating the articles from the datasets in the other languages.
Hitachi [ST1, ST2] (Koreeda et al., 2023) (first for: ST1:IT, ST1:RU) (Keywords: XLM-RoBERTa, RoBERTa ) augmented the dataset for ST1 by collecting labelled examples from similar datasets. They pretrained (XLM-)RoBERTa in multi-task (one language, ST1 and ST2), multilingual (one subtask, all languages) and multilingual multi-task (ST1 and ST2 in all languages) settings. Besides using the single models, they report experiments with ensemble of base models with different hyper-
parameters. JUSTR00 [ST1, ST2] (Al-Qarqaz and Abdullah, 2023): (Keywords: LongFormer, BERT, RoBERTa, mBERT, XLM-RoBERTa, BigBird experimented with many state-of-the-art transformer-based language models, both monolingual and multilingual. Their top performing model is based on a transformer called “Longformer”.
kb [ST1, ST3] (Baraniak and Sydow, 2023) (Keywords: BERT, Bert, hierarchical learning, multitask learning) tackled ST3 by training a BERT model to identify the start of a text fragment with a technique, then the first index of predicted span was used to get the BERT embedding for classification.
KInITVeraAI (KInIT) [ST3] (Hromadka et al., 2023): (first for: ST3:IT, ST3:RU, ST3:DE, ST3:PL, ST3:EL, ST3:KA) (Keywords: XLMRoBERTa large and base, mBERT base, monolingual RoBERTa base and large, monolingual BERT base, distilBERT, language model fine-tuning with different layer freezing strategies) used a fine-tuned XLM-RoBERTa-large transformer model trained on all the input data. They carefully adjusted the prediction threshold for each language using a principled approach. They truncated the input, and also found that pre-processing did not impact the quality much.
MarsEclipse [ST2] (Liao et al., 2023): (first for: ST2:IT, ST2:RU, ST2:FR, ST2:DE, ST2:PL) (Keywords: XLM-RoBERTa, mBERT, SimCSE, SimCLR) used a multi-label contrastive loss for finetuning pre-trained language models in a multilingual setting. They followed the general architecture of SimCLR and SimCSE to do contrastive learning, but modified the contrastive loss to make it fit for a multi-label setup. This yielded very competitive results for ST2, and this was the winning system for five of the languages.
mCPT (PolarIce) [ST2] (Reiter-Haas et al., 2023) (first for: ST2:ES) (Keywords: paraphrasemultilingual-MiniLM-L12-v2, contrastive pretraining) used a two-phase training procedure of a transformer model, first by pre-training jointly on all the languages and then by fine-tuning for each language. In both phases, a multi-label contrastive loss was used.
MELODI [ST1] (Devatine et al., 2023): (first for: ST1:EN) (Keywords: Translation + POLITICS (RoBERTa)) fine-tuned the domain-specific language model trained on English data, POLITICS, on the English input articles and on the arti-
cles in all other languages automatically translated. In addition, in order to use whole articles as input, they used a sliding window approach and aggregated each window representation with mean pooling. They also tested other multilingual approaches, such as XLM-RoBERTa, and approaches able to process long documents (Longformer), which were in general less effective.
MLModeler5 [ST1, ST2] (Khanchandani et al., 2023) (Keywords: RoBERTa, ALBERT) provided a solution for English only. For ST1 they pre-trained the RoBERTa, ALBERT and other deep learning models using the original training data in English and translated versions of the data in other languages and performed NLP augmentation using NLPAUG library on it. For ST2 a similar-in-nature approach was used.
NAP [ST3] (Falk et al., 2023) (first for: ST3:FR) (Keywords: XLM-RoBERTa (base and large), setfit, adapters, translation and backtranslation of paragraphs) presented an approach combining predictions of several models in an ensemble, which differ in three main aspects: a) training data, b) model architecture, and c) input format to the model. They leveraged (back)translation as data augmentation strategies using available MarianMT models. Model architectures included XLM-RoBERTa models, Adapters, SetFit, and linguistically-informed heuristics for underrepresented techniques which were fine-tuned on different combinations of original and augmented data. They fine-tuned models on both paragraphand span-level information.
NL4IA [ST3] (Pritzkau, 2023) (Keywords: RoBERTa) used RoBERTa and exploited the span level annotations framing ST3 as a token-level classification one, but report better results when treating the subtask as a sequence classification one.
NLUBot101 [ST1,ST3] (Liu et al., 2023) (Keywords: mDeBERTa) built, for ST3, a solution on top of mDeBERTa NLI model and exploit crosslingual data augmentation. The performance could be improved through the exploitation of the expanded definitions of the persuasion technique guidelines from the official annotation guidelines vis-a-vis the usage of single words or phrases. Their system achieved the highest macro F1 score for the English language.
QCRI (QCRITeam) [ST1, ST2, ST3] (Hasanain et al., 2023) (Keywords: XLM-RoBERTa, French Europeana BERT, Gottbert-base, Italian BERT,
HerBERT) used, for all subtasks, data augmentation and then fine-tuned a BERT model specifically for each language, in addition to fine-tuning XLMRoBERTa on all languages at once.
QUST [ST1,ST2,ST3] (Jiang, 2023): (Keywords) XLM-RoBERTa Their model is build on top of XLM-RoBERTa, which is fine-tuned with the pre-calculated class weights and sample weights to combat the imbalanced data. The class weights are multiplied by the loss to make the model focus more on the minority class. The sample weights are combined with a weighted sampler to resample the distribution of the training batch. In addition, two types of fine-tuning strategies, the task-agnostic and the task-dependent, where the latter proved to help the multilingual model to learn the shared information between subtasks. The submitted system achieves the second best result for Italian and Spanish (zero-shot) in ST1.
ReDASPersuasion [ST3] (Qachfar and Verma, 2023) (Keywords: XLM-RoBERTa) uses XLMRoBERTa as a backbone model, incorporating language agnostic features, computed over the articles translated using Google translation. Such features target specific techniques, including sentiment- and polarity- based features targeting appeal to fear and slogans, indefinite pronouns indicative of exaggeration and minimisation, a profanity language detection to capture loaded language. XLM-RoBERTa has been proved to be a powerful multilingual pretrained language model compared against other models like Multilingual BERT (M-BERT).
SheffieldVeraAI (vera) [ST1, ST2, ST3] (Wu et al., 2023) (first for: ST1: DE, ST2:EN, ST2:EL, ST2:KA) (Keywords: mBERT, adapters, text preprocessing, upsampling, XLM-Roberta, Pfeiffer Adapters, MUPPET, Task-adaptive Pre-training, RoBERTa, class weighting) deployed an ensemble of three fine-tuned mBERT models and one mBERT model with a bottleneck adapter for ST1. All used bert-base-multilingual-cased. For the finetuned mBERT models, they pre-processed the data by filtering out non-informative sentences. The pool of training data was also extended by integration additional “satire” resources for English. In the cases where the length of the tokenised article was more than 512 tokens, an equal number of sentences from the beginning and the end of the article was selected until the size of 512 tokens in a concatenated text is reached. The final predictions were drawn as a majority-voting predicted class
For ST2 they used two different ensembles of MUPPET large, and of XLM-R with adapters and task-adaptive MLM pre-training on the train+dev+test data. Their data was pre-processed and truncated. The models were trained both with and without class weighting.
For ST3 they trained a monolingual RoBERTaBase model for English and a multilingual mBERTcased model for the remaining languages. They used class weighting to account for class imbalance. They also experimented with augmenting data through translation, which improved the performance for the surprise languages.
SinaAI (SinaaAI) [ST1, ST2, ST3] (Sadeghi et al., 2023) (first for: ST1:EL) (Keywords: XLM, mBERT, LaBSE) used multilingual languages models such as XLM, mBERT and LaBSE, which they combined in an ensemble. For ST1 and ST2, they further used data augmentation by selecting 30% of the sentences of each document to create new synthetic examples.
TeamAmpa [ST2, ST3] (Pauli et al., 2023) (first for: ST3:ES) (Keywords: RoBERTa, XML-R, ensemble models) used different oversampling strategies, data truncation, and multilingual and monolingually trained models, combined in an ensemble for the English data. The surprise languages were handled using the multilingual model with oversampling on English data and data from lowrepresented classes.
TheSyllogist [ST2] (Mohammed Afzal and Nakov, 2023): (Keywords: BERT) participated in ST2, and experimented with zero-shot transfer: translating the data for all languages into English (using Google Translate), and then training and applying an English system. They used fine-tuned BERT (bert-base-uncased) with mean-pooling.
UM6P [ST1, ST3] (Alami et al., 2023): (Keywords:Longformer, RoBERTa, GCN) fine-tuned Longformer and RoBERTa transformers for both ST1 and ST3. They further added a graph convolution network, and a classifier based on the number of sentences in each document. Finally, they used an ensemble to combine the predictions of these models.
UMUTeam [ST1, ST2] (Pan et al., 2023) (first for: ST1:FR, ST1:DE) (Keywords: Sentence transformers, XML-RoBERTa) used a multilingual model based on XML-RoBERTa, which they finetuned on all languages at once and a sentence transformer to extract the most important chunk of text
for ST1 and ST2. They further truncated the input data to 200 tokens with 50 tokens of overlap using the sentence-transformer model to obtain the subset of text most related to the article title.
UnedMediaBiasTeam [ST1, ST3] (RodrigoGinés et al., 2023) (Keywords: XLM-RoBERTa, bert-base-multilingual-cased) solutions are based on two-stage fine-tuned multilingual models. For ST1 they exploit the media bias detection datasets called BABE and MBIC and XLM-RoBERTa model fine-tuned in two stages: first with the BABE and MBIC datasets, and later with the data provided for the task. For ST3 a similar approach is deployed, where instead of training a single model in two phases, two models are trained and the cascading inference is carried out.
Unisa [ST1, ST3] (Bangerter et al., 2023) (Keywords: DistilBert, SHAP) built solutions on top of DistilBert and leverage the application of the eXplainable Artificial Intelligence (XAI) method, Shapley Additive Explanations (SHAP). In ST1, data augmentation was exploited through translation data to the target language (English) on top of which the model was trained with only the first 512 tokens of the articles being considered as input to the model. SHAP was used to understand what was driving the model to fail so that it could be improved.
In ST3, a re-calibration of the Attention Mechanism is realized by extracting critical tokens for each technique. XAI is exploited for countering the overfitting of the resulting model and attempting to improve the performance when there are few training samples. First, a binary model first processes a new incoming paragraph to predict whether it contains any persuasion attempt. If the text is predicted to be propaganda, it is compared with SHAP Vocabularies previously created, which represent the most important words associated with each persuasion technique. Such comparison defines the additional input to pass to the final multi-class model trained to focus on the span that identifies the text that characterizes the persuasion technique.
UTB-NLP (UTBNLP) [ST1, ST2] (Cuadrado et al., 2023) (Keywords: ) used a feature-based representation: they extracted noun phrases and represented them as tf-idf vectors; they considered several features specific for each of the three classes of ST1, such as psycholinguistic, writing style, readability, structural characteristics, conceptual embeddings and argumentation-based features.
In addition, they used SMOTE to oversample the minority classes.
ST2 was tackled by collecting extra texts from Wikipedia related to the frames, pre-processing them to create a frame-related lexicon and then to use it to create a bag-of-words representation for each input article.

heading: Future Work
text: In future work, we plan to further increase the data size, cover additional languages, and explore different ways of evaluation of the persuasion technique detection, e. g. , by changing the focus (sentence- and text span-level evaluation)