QCRI at SemEval-2023 Task 3: News Genre, Framing and Persuasion Techniques Detection using Multilingual Models
Year: 2023

Abstract:
Misinformation spreading in mainstream and social media has been misleading users in different ways. Manual detection and verification efforts by journalists and fact-checkers can no longer cope with the great scale and quick spread of misleading information. This motivated research and industry efforts to develop systems for analyzing and verifying news spreading online. The SemEval-2023 Task 3 is an attempt to address several subtasks under this overarching problem, targeting writing techniques used in news articles to affect readers’ opinions. The task addressed three subtasks with six languages, in addition to three “surprise” test languages, resulting in 27 different test setups. This paper describes our participating system to this task. Our team is one of the 6 teams that successfully submitted runs for all setups. The official results show that our system is ranked among the top 3 systems for 10 out of the 27 setups.

heading: 1 Introduction
text: Monitoring and analyzing news have become an important process to understand how different topics (e.g., political) are reported in different news media and within and across countries. This has many important applications since the tone, framing, and factuality of news reporting can significantly affect public reactions toward social or political agendas. A news piece can be manipulated on multiple aspects to sway readers’ perceptions and actions. Going beyond information factuality, other aspects include objectivity/genre, framing dimensions inserted to steer the focus of the audience (Card et al., 2015), and propaganda techniques used to persuade readers towards a certain agenda (Barrón-Cedeno et al., 2019; Da San Martino et al., 2019a).
News categorization is a well studied problem in the natural language processing field. Recently, research attention has focused on classifying news by factuality (Zhou and Zafarani, 2020; Nakov et al.,
2021), or other related categorizations such as fake vs. satire news (Low et al., 2022; Golbeck et al., 2018). However, there have been efforts towards other classification dimensions. Card et al. (2015) developed a corpus of news articles annotated by 15 framing dimensions such as economy, capacity and resources, and fairness and equality, to support development of systems for news framing classification. Moreover, identifying propagandistic content has gained a lot of attention over several domains including news (Barrón-Cedeno et al., 2019; Da San Martino et al., 2019a), social media (Alam et al., 2022) and multimodal content (Dimitrov et al., 2021a,b).
The SemEval-2023 Task 3 shared task aims at motivating research in the aforementioned categorization tasks, namely: detection and classification of the genre, framing, and the persuasion techniques in news articles (Piskorski et al., 2023). It targets multiple languages including English, French, German, Italian, Polish, and Russian to push the research on multilingual systems. Moreover, to promote development of language-agnostic models, the task organizers released test subsets for three surprise languages (Georgian, Greek, and Spanish).
Our proposed system is based on fine-tuning transformer based models (Vaswani et al., 2017) in multiclass and multi-label classification settings for different tasks and languages. We participated in all three subtasks submitting runs for all nine languages, which resulted in 27 testing setups. We experimented with different mono and multilingual transformer models, such as BERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020; Chi et al., 2022) among others. In addition, we also experimented with data augmentation.
The rest of the paper is organized as follows. Section 2 gives an overview of related work. In section 3, we present the proposed system. In section 4, we provide the details of our experiments.
1237
Section 5 presents the results for our official runs, and finally, we conclude our paper in section 6.

heading: 2 Related Work

heading: 2.1 News Genre Categorization
text: Prior works on automated news categorization have focused on various aspects such as topic, style, how news is presented or structured, and intended audience (Einea et al., 2019; Chen and Choi, 2008; Yoshioka et al., 2001; Stamatatos et al., 2000). News articles have also been categorized based on their factuality and deceptive intentions (Golbeck et al., 2018). For example, fake news is false and the intention is deceive where satire news is also false but the intent is not deceive rather to call out, ridicule, or expose behavior that is shameful, corrupt, or otherwise “bad”.

heading: 2.2 Propaganda Detection
text: Propaganda is defined as the use of automatic approaches to intentionally disseminate misleading information over social media platforms (Woolley and Howard, 2018). Recent work on propaganda detection has focused on news articles (Barrón-Cedeno et al., 2019; Rashkin et al., 2017; Da San Martino et al., 2019b, 2020), multimodal content such as memes (Dimitrov et al., 2021a,b) and tweets (Vijayaraghavan and Vosoughi, 2022; Alam et al., 2022). Several annotated datasets have been developed for the task such as TSHP-17 (Rashkin et al., 2017), and QProp (Barrón-Cedeno et al., 2019). Habernal et al. (2017, 2018) developed a corpus with 1.3k arguments annotated with five fallacies (e.g., red herring fallacy), which directly relate to propaganda techniques. Da San Martino et al. (2019b) developed a more fine-grained taxonomy consisting of 18 propaganda techniques with annotation of news articles. Moreover, the authors proposed a multigranular deep neural network that captures signals from the sentence-level task and helps to improve the fragment-level classifier. An extended version of the annotation scheme was proposed to capture information in multimodal content (Dimitrov et al., 2021a). Datasets in languages other than English have been proposed. For example, using the same annotation scheme from (Dimitrov et al., 2021a), Alam et al. (2022) developed a dataset of Arabic tweets and organized a shared task on Arabic propaganda technique detection. Vijayaraghavan and Vosoughi (2022) developed a dataset of
tweets, which are weakly labeled with different fine-grained propaganda techniques. They also proposed a neural approach for classification.

heading: 2.3 Framing
text: Framing refers to representing different salient aspects and perspectives for the purpose of conveying the latent meaning about an issue (Entman, 1993). Recent work on automatically identifying media frames includes developing coding schemes and semi-automated methods (Boydstun et al., 2013), datasets such as the Media Frames Corpus (Card et al., 2015), systems to automatically detect media frames (Liu et al., 2019a; Zhang et al., 2019), large-scale automatic analysis of news articles (Kwak et al., 2020), and semi-supervised approaches (Cheeks et al., 2020).
Given the multilingual nature of the datasets released with the task at hand, our work is focused on designing a multilingual approach for news classification for the three subtasks of interest.

heading: 3 System Overview
text: Our system is comprised of preprocessing followed by fine-tuning pre-trained transformer models. The preprocessing part includes standard model specific tokenization. Our experimental setup consists of (i) monolingual (∗mono): training and evaluating monolingual transformer model for each language and subtask; (ii) multilingual (∗multi): combining subtask specific data from all languages for training, and evaluating the model on task and language specific data; (iii) data augmentation (∗aug): applying data augmentation on language specific training set, then training a monolingual model using augmented dataset, and evaluating it on the test set. This has been applied for each subtask.

heading: 3.1 Data Augmentation
text: Data augmentation is an effective way to deal with class imbalance issues or to increase the size of the training dataset or increase within-class variation. Typically, textual data augmentation has been done by upsampling techniques such as SMOTE (Chawla et al., 2002), however, that approach is applied to the vector representation. Very recently, some useful strategies are introduced for textual data augmentation (Feng et al., 2021), which range from rule-based approaches to modelbased techniques. Wei and Zou (2019) proposed a set of token-level random perturbation operations
including random insertion, deletion, and swap, which have been employed in several studies (Feng et al., 2021; Alam et al., 2020).
We used such approaches with contextual representation from transformer models in this study. These include (i) synonym augmentation using WordNet, (ii) word insertion and substitution using BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b) and DistilBERT (Sanh et al., 2019). More details on the implementation of these approaches can be found in the following data augmentation package.1

heading: 4 Experiments
text: In this section, we describe the tasks and datasets used during experiments and provide implementation details for our models.

heading: 4.1 Task and Dataset
text: The SemEval-2023 Task 3 is composed of 3 subtasks for each language:
1. News Genre Categorization (subtask1): Given a news article in a particular language, classify it to an opinion, news reporting, or a satire piece. This is a multiclass classification task at the article level.
2. Framing Detection (subtask2): Given a news article, identify the frames used in the article. This is a multi-label classification task at the article level. This task includes 14 frames/labels such as economic, capacity and resources, morality, and fairness and equality.
3. Persuasion Techniques Detection (subtask3): Given an article, identify the persuasion technique(s) present in each paragraph. This is a multi-label classification task at the paragraph level. This task includes 23 techniques/labels such as loaded language, appeal to authority, appeal to popularity, and appeal to values.
The task organizers released three subsets (train, development and test) of data per language of the six main languages for each subtask. Further details and statistics can be found in (Piskorski et al., 2023). Starting with the six train subsets, we apply three methods to acquire new versions of these train subsets:
1https://github.com/makcedward/nlpaug
1. Train subset splitting: we randomly split each of the train subsets into 80-20 splits to acquire training and validation subsets for each subtask and each language. As will be shown in the following subsection, our models were re-trained using different random seeds. The validation set is used to select the random seed leading to the best model.
2. Multilingual dataset construction: to support our multilingual training setup, we combine the training subsets resulting from the previous step for all languages to create a multilingual training subset. We apply the same approach to the validation subsets.
3. Data augmentation: for each of our generated training splits, we apply data augmentation to it and use the resulting datasets to train a monolingual model for each subtask and each language.

heading: 4.2 Implementation Details
text: We use HuggingFace (HF) library (Wolf et al., 2020) on top of PyTorch framework (Paszke et al., 2017) as our base and source of all the pre-trained language models. Since different random initialization can considerably affect the model performance, we train the model for each language with k different random seeds.
For all experiments, we use Adam optimizer (Kingma and Ba, 2015) with the learning rate of 2×10−5. In setting other parameters of the models, we distinguish between subtask1 and subtask2 that operate on the document level, and subtask3multi/aug that works at the paragraph level and has a much larger training subset. Only for
subtask3multi/aug, the number of epochs=5, k=5, maximum sequence length=256, and batch size=8. For all remaining training setups and subtasks, the number of epochs=10, k=10, maximum sequence length=512, and batch size=4.
For each of the three training setups described in section 3, the models trained using k seeds for a language are evaluated over our validation subset using the official evaluation measure for the corresponding subtask. The model with the best performance is then applied to the development set. Eventually, the training setup that has the best performance on the development subset will be used to generate the official run for the corresponding subtask and test language. As for the “surprise” test languages, we use the model trained on the multilingual training subset with the best performance on the multilingual validation subset.
For our multilingual training setup, we opt to use XLM-RoBERTa (Conneau et al., 2020). As for all
other setups, we used per-language monolingual pre-trained models listed in Table 1.

heading: 5 Results
text: The results for our official runs per subtask are shown in Tables 2, 3 and 4. For each subtask, we compare our official runs to two baselines: the top run in each test language, and the baseline as reported by the task organizers.
We observe that the multilingual models are generally the best performing models across all tasks. On average, the performance of the system was best for subtask3 with a slight average ranking difference compared to subtask2. Another interesting observation is that although subtask3 has much larger train subsets , since it operates on the paragraph level, this did not improve the average system ranking across languages when compared to subtask2. The results also clearly show the robustness of our model across languages and subtasks, as it
managed to be among the best 3 runs for 10 out of the 27 test subsets, and it was among the top 5 runs for 15 of them.
Results over subtask1 and subtask3 showed that our proposed system had a strong cross-lingual transfer ability when training the model on multilingual data and testing it on unseen languages (Georgian, Greek and Spanish).

heading: 6 Conclusion
text: In this paper, we presented our experiments and findings on news genre categorization, framing and persuasion techniques detection on multiple languages, which was a part of SemEval-2023 Task 3 shared task. The task includes 27 test setups for three subtasks and nine test languages. Our team successfully submitted runs for all setups. We proposed a system that is based on fine-tuning transformer models in multiclass and multi-label classi-
fication settings. We experimented with different mono and multilingual pre-trained models, in addition to data augmentation. From the experimental results, we observed that our multilingual model based on XLM-RoBERTa performs better across all tasks, even on unseen languages.
Our future work includes domain adaptation and further exploration of data augmentation techniques.

heading: Acknowledgments
text: This publication was made possible by NPRP grant 14C-0916-210015 The Future of Digital Citizenship in Qatar: a Socio-Technical Approach from the Qatar National Research Fund.
Part of this work was also funded by Qatar Foundation’s IDKT Fund TDF 03-1209-210013: Tanbih: Get to Know What You Are Reading.
The views, opinions, and findings presented in this paper are those of the authors alone and do not necessarily reflect the views, policies, or positions of the QNRF or any other affiliated organizations.

heading: Future Work
text: This publication was made possible by NPRP grant 14C-0916-210015 The Future of Digital Citizenship in Qatar: a Socio-Technical Approach from the Qatar National Research Fund. Part of this work was also funded by Qatar Foundation’s IDKT Fund TDF 03-1209-210013: Tanbih: Get to Know What You Are Reading. The views, opinions, and findings presented in this paper are those of the authors alone and do not necessarily reflect the views, policies, or positions of the QNRF or any other affiliated organizations