CLaC at SemEval-2023 Task 3: Language Potluck RoBERTa Detects Online Persuasion Techniques in a Multilingual Setup
Year: 2023

Abstract:
This paper presents our approach to the SemEval-2023 Task 3 to detect online persuasion techniques in a multilingual setup. Our classification system is based on the RoBERTabase model trained predominantly on English to label the persuasion techniques across 9 different languages. Our system was able to significantly surpass the baseline performance in 3 of the 9 languages: English, Georgian and Greek. However, our wrong assumption that a single classification system trained predominantly on English could generalize well to other languages, negatively impacted our scores on the other 6 languages. In this paper, we provide a description of the reasoning behind the development of our final model and what conclusions may be drawn from its performance for future work.

heading: 1 Introduction
text: The SemEval-2023 shared task 3 (Piskorski et al., 2023) focuses on 3 subtasks: detecting the news genre (subtask 1), the framing (subtask 2) and the persuasion techniques (subtask 3) used in online news across different languages. Training data provided by the organizers was collected from news and web articles between the years 2020 and 2022 in six different languages, namely English, French, German, Italian, Polish, and Russian. During the testing phase of the shared task, three additional surprise languages were included only as testing data, Georgian, Greek and Spanish, to explore the language-agnostic robustness of the trained models.
In this paper, we focus on the detection of persuasion techniques used to influence the reader of these online news as part of subtask 3 (Piskorski et al., 2023). We did not participate in the news genre categorisation (subtask 1) and framing detection (subtask 2). In total, 23 different persuasion techniques were annotated at the paragraph-level
for each article across all the training languages. Such techniques include loaded language, guilt by association and the straw man fallacy. A more detailed description of each persuasion technique is available in the annotation guidelines of the shared task (Piskorski et al., 2023).
We applied a RoBERTa (Liu et al., 2019) based model trained predominantly on the English training dataset enriched with specific instances from the other training languages to the detection of persuasion techniques across all languages. Section 3 provides an overview of our final classification pipeline. In Section 4, we provide a detailed description of the experiments and data augmentation techniques that led to our final model decision. However, our assumption that a predominately English trained model would be able to generalize well and even outperform models trained on languages specific to the prediction task was proven wrong by the official leaderboard results of the shared task as further discussed in Section 5.
With the exception of English and two of the three surprise test languages, Greek and Georgian, where we were able to attain significantly higher scores than the baseline, our final model either performed worse or only slightly better than the baseline. All of the code used in the implementation of the models described in this paper is made available on GitHub1.

heading: 2 Background
text: The massively widespread of misinformation to deliberately influence opinions and beliefs through the use of rhetorical and psychological persuasion techniques, commonly referred to as fake news, has become a major societal problem capable of even swinging elections (Zellers et al., 2019). It has, thus, become of paramount importance to develop systems capable of automatically detecting
1https://github.com/CLaC-Lab/ Persuasion-Techniques-Detection
1613
such persuasion techniques (Da San Martino et al., 2019).
With the advent of the Transformer (Vaswani et al., 2017) architecture and ensuing pre-trained large language models, such as BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020), the automatic detection of misinformation in online news has become feasible with high performance and hence an active area of research (Zellers et al., 2019). The detection of propaganda techniques in news articles was in fact the goal of the SemEval2020 shared task 11 (Da San Martino et al., 2020).
Based on the approach followed by the best participants at the SemEval-2020 shared task 11 (Jurkiewicz et al., 2020) and the recent successes of Transformer based pre-trained large language models, we based our system for the SemEval2023 shared task 3 (Piskorski et al., 2023) on RoBERTa (Liu et al., 2019).

heading: 3 System Overview
text: The goal of the task is to identify all persuasion techniques (if any) employed at the paragraph-level in online news articles. Therefore, our model takes as input an entire paragraph and classifies it as exhibiting none, one or multiple persuasion techniques as output. Figure 1 shows an overview of the classification pipeline we employed for this subtask. As Figure 1 shows, our classification system is composed of a RoBERTa-base2 model with a feedforward layer of 23 nodes, one for each of the annotated techniques. We used a sigmoid activation function at the output layer and we considered the entire task a 23-class multi-label binary classification problem.
Since the RoBERTa-base model was pre-trained on the English language exclusively, for all the other training languages we first had to translate the datasets into English before feeding them to our model. We did this using the Googletrans3 API. We then augmented the datasets of each language (see Section 4.2). Note that for each language we could also have used a different RoBERTa based model pre-trained on that specific language, but instead we opted to first translate the different languages into English and then use RoBERTa-base. This decision was motivated by our experiments with the development set (see Section 4.3).
2https://huggingface.co/roberta-base 3https://py-googletrans.readthedocs.io/en/
latest

heading: 4 Experimental Setup
text: Before opting for our final classification system configuration, we experimented with different augmentation techniques, we considered different language models and we explored training our models with specific language datasets to classify the persuasion techniques across all languages. In this section we will describe in detail our experiments and the decision process that led to our final model.

heading: 4.1 Data Splits
text: Data for each language was made gradually available during three different phases of the shared task. In the first phase, participants had only access to the annotated training datasets of English, French, German, Italian, Polish, and Russian; in a second phase, participants had additional access to the annotated development datasets of the same languages; and in the third and final phase, the annotated test datasets with hidden labels of the same languages and of the surprise Georgian, Greek and Spanish were made available.
During the first phase, we randomly divided the training datasets into three different splits: 70% for training, 10% for validation and 20% for testing. During the second phase, we randomly split 80% of the original training datasets for training and 20% for validation, while using the newly available development datasets for testing. Note that the development set of each language was around 25-35%
the size of its corresponding training dataset. This meant that for each language we still had around 70% of the combined data for training, 10% for validation and 20% for testing. Finally, during the third and final phase, we merged both training and development datasets into single datasets for each language, which we then randomly split 80% for training and 20% for validation, and used the new test datasets with hidden labels to submit our final classification predictions.
Throughout all phases, we used the same set-up: we used a batch size of 8 and kept the training to a maximum of 10 epochs with an early stop call after 2 epochs without improvement on the validation set. Since we applied a sigmoid activation function at the output layer of the multi-label binary classification system, we used a binary cross entropy loss function across all experiments.

heading: 4.2 Data Augmentation
text: As described above, each language had its own training, development and test datasets. However, the distribution of persuasion techniques was not the same across languages. In fact, certain techniques were even missing in some languages. Figure 2a shows the distribution of the original English training dataset. As shown in the figure, the dataset includes no instances of the Appeal to Time, Consequential Oversimplification, Appeal to Values and Questioning the Reputation techniques. Conversely, there is an unbalanced over-representation of certain techniques, such as Loaded Language and Name Calling Labelling. This was a recurring problem across all datasets.
Therefore, in order to have a representative number of samples of all persuasion techniques, we first enriched each dataset with instances of missing or under-represented persuasion techniques by translating the instances from other languages into the original dataset language. This approach was motivated by the already successful use of automatically translated data in subjective-oriented natural language processing tasks (Banea et al., 2008).
We considered instances to be under-represented if they appeared less than 5% of the total number of labelled instances. However, due to the multi-label nature of the task, some of the newly added instances were also jointly labelled with already over-represented techniques in the original datasets. Thus, after augmenting the original datasets, we randomly dropped instances of the
over-represented techniques that did not co-occur with the under-represented ones. We considered instances to be over-represented if they appeared more than 20% of the total number of labelled instances in the augmented dataset.
Finally, we dropped all instances without a label from all the datasets used for training our models. These were the mostly over-represented instances across all languages and the shared task did not evaluate our ability to predict them. Figure 2b shows the final distribution obtained after augmenting the original English training dataset.

heading: 4.3 Model Selection
text: During the model selection stage, we considered different models in the BERT (Devlin et al., 2019) family. We considered only these models because of their ability to learn a bidirectional representation of a sentence given their pre-training objective. This makes them particularly well-suited for the downstream task of detecting persuasion techniques at the paragraph-level. This is different from traditional LSTM recurrent neural networks (Hochreiter and Schmidhuber, 1997) that see the words one after the other and from autoregressive models such as in the GPT (Radford et al., 2018) family.
As shown in Table 1, we experimented with four different models. We considered RoBERTabase on all languages after translating them to English (see Figure 1), as well as the French based CamemBERT-base4 (Martin et al., 2020) on the French language only, the German based GermanBERT-base5 on the German language only and, finally, the Multilingual-BERT-base6 natively on all languages. All four models were trained and tested on the same language datasets after being enriched with instances of their missing or underrepresented techniques from the other languages (see Section 4.2). They were trained exclusively on data from the training datasets and tested on the separately released development datasets for their corresponding training language (see phase 2 in Section 4.1).
Based on the results shown in Table 1, the RoBERTa based model outperformed all other models. Surprisingly, it even outperformed the CamemBERT based model on French texts and the German-BERT on German texts. Based on these results, we further experimented with other variations of RoBERTa; namely, RoBERTa-large7 and XLMRoBERTa-base8 (Conneau et al., 2020). However, these larger models took significantly more time to train and did not provide substantial improvements in performance, so they were ruled out from further experiments and we chose RoBERTa-base as our final model (see Figure 1).
4https://huggingface.co/camembert-base 5https://huggingface.co/
bert-base-german-cased 6https://huggingface.co/ bert-base-multilingual-cased 7https://huggingface.co/roberta-large 8https://huggingface.co/xlm-roberta-base

heading: 4.4 Training Language Selection
text: After selecting our final model, we additionally explored if we could train a single model on a specific language and then generalize it to predict persuasion techniques on all other languages after translating their testing data into the training language. These results are shown in Table 2. Row-wise, the table shows the performance of a model trained on a given language training dataset across the different development datasets of the different languages. Column-wise, the table shows how well the models trained on different languages were able to perform on a given language development dataset.
As Table 2 shows, our RoBERTa-base model performed particularly well across all languages when trained on English and it performed especially well on both Latin languages, French and Italian. Based on these results, we assumed that there could be an etymological explanation behind the structure of the English language that led to these results.

heading: 5 Results
text: Based on the results of Table 1 and Table 2, for our official submission, we trained our single RoBERTa-base classification system described in Section 3 on the merged English training and development datasets following the augmentation techniques discussed in Section 4.2. We then translated all of the provided test datasets into English before generating our final classification predictions on each language. The official performance results of our system are shown in Table 3, along with the baseline score and the score obtained by the best performing system on each language.
As Table 3 shows, the preliminary results obtained with the training and development datasets
(see Table 2) which led us to develop an Englishonly trained model, were not representative of the official performance obtained with the final test datasets of the shared task. Considering only our results on the languages for which a training set was provided, the model was able to significantly outperform the baseline only on English. In fact, it performed worse than the baseline in French, German, Italian and Russian. This could be a result of different label distributions between the training datasets and the final test datasets. However, the assumption that a predominantly English trained model could generalize well to the task of detecting persuasion techniques across different languages than training a model natively on the same language of the predictive task, turned out to be wrong.
On the three surprise languages, our classification system performs substantially better than the baseline on Georgian and Greek, while only slightly better on Spanish. Given that we did not have access to any training dataset in the Georgian
and in the Greek family of languages, these results may indicate that although our assumption was wrong, it may not have been entirely unfounded. Thus, a model trained in a Latin language, such as French or Italian, could have perhaps led to better results in the Spanish language.

heading: 6 Conclusion
text: In this paper we described our approach to the SemEval-2023 Task 3 to detect online persuasion techniques in a multilingual setup. We applied a single classification system based on the RoBERTabase model trained predominantly on the English language to detect persuasion techniques across 9 different languages. Based on the results obtained with the development set, we assumed that our English trained model could generalize well to every other language and, hence, we did not develop language specific models. Given the actual test set, the model was only able to significantly outperform the baseline for English and the two surprise, Georgian and Greek, for which no previous training data was available.
In the future, we plan to explore the use of other language models, such as ALBERT (Lan et al., 2019), and to evaluate how our final classification system would have performed if trained on the same language of the prediction task. We also plan on exploring a cascading model of classification systems for each persuasion technique on a one-vsall scenario. Finally, given the different structure of the different languages, we would also like to investigate which techniques are more frequently classified correctly and which techniques are more frequently classified incorrectly on each language. This may lead to practical insights on how subjective information may be lost using automatic machine translation.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their feedback on the previous version of this paper. This work was financially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

heading: Future Work
text: In the future, we plan to explore the use of other language models, such as ALBERT (Lan et al. , 2019), and to evaluate how our final classification system would have performed if trained on the same language of the prediction task. We also plan on exploring a cascading model of classification systems for each persuasion technique on a one-vsall scenario. Finally, given the different structure of the different languages, we would also like to investigate which techniques are more frequently classified correctly and which techniques are more frequently classified incorrectly on each language. This may lead to practical insights on how subjective information may be lost using automatic machine translation