SheffieldVeraAI at SemEval-2023 Task 3: Mono and multilingual approaches for news genre, topic and persuasion technique classification
Year: 2023

Abstract:
This paper describes our approach for SemEval2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the highest mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensembles: a monolingual RoBERTa-MUPPETLARGE and an ensemble of XLM-RoBERTaLARGE with adapters and task adaptive pretraining. For Subtask 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the remaining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques. 1

heading: 1 Introduction
text: With the rise of opinion-manipulating news and misinformation surrounding COVID-19, elections and wars, the task of propaganda and hyperpartisan detection has received much attention over the last five years. Since 2019, various SemEval tasks have addressed detecting hyperpartisan (Kiesel et al., 2019), sarcasm (Abu Farha et al., 2022), and persuasion techniques in textual and multimodal data (Da San Martino et al., 2020; Dimitrov et al., 2021). This task (Piskorski et al., 2023) can be seen as an extension of the latter two tasks, suggesting an expanded ontology of persuasion techniques and addressing other related aspects of persuasion, such as satire, opinionated news, and framing detection.
#Equal contribution, listed randomly. 1Our code is available at https://github.com/
GateNLP/semeval2023-multilingual-news-detection
The three subtasks presented in this shared task are the detection of: 1) genre: opinion, objective reporting or satire; 2) framing techniques: 14 multilabel frames; 3) persuasion techniques: 23 multilabel techniques, which can be grouped into 6 high-level classes.
The data consists of labelled training and development sets in English, French, German, Italian, Polish and Russian, and unlabelled test sets in the same languages plus three zero-shot languages: Spanish, Greek and Georgian.
The main contributions of this paper are twofold: 1) evaluation of the viability of monolingual versus multilingual models for each of the subtasks; and 2) presentation of the models which ranked first in four subtask-language pairs, top three in 16 subtasklanguage pairs, and were within the top 10 for all.
Our approaches for the three subtasks differ, therefore we present each subtask separately in sections 3-5 respectively. An overview of the techniques used in each subtask is shown in Table 1.

heading: 2 Background
text: Fine-grained propaganda technique classification was first introduced by Da San Martino et al. (2019), who suggested a multi-granularity network, where the lower and higher granularity tasks refer to the fragment and sentence-level classification respectively. Other state-of-the-art approaches to this task used an ensemble of RoBERTa models with class weighting, where some models perform a semi-supervised task of span detection (Jurkiewicz et al., 2020) and an ensemble of 5 different transformer models (Tian et al., 2021), namely BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), XLNet (Yang et al., 2019), DeBERTa (He et al., 2020) and ALBERT (Lan et al., 2019).
Framing detection specifically has been addressed primarily for political news, with the models exploring unsupervised probabilistic topic models combined with autoregressive distributed-lag
1995
models (Tsur et al., 2015), finetuning BERT (Liu et al., 2019a) and multilingual BERT (mBERT) (Akyürek et al., 2020). The latter system is the closest to our task since it explores the multilabel multilingual setting and the effect of translating texts for use in monolingual models. However, it uses article headlines instead of the full texts as the classification data. The authors found that English BERTBASE uncased trained on translated data and tested on the data in the target language often outperforms the multilingual model. We perform similar comparison experiments for all three subtasks of this shared task.
Wang and Banko (2021) performed a series of experiments comparing monolingual and multilingual approaches for hate speech detection and sentiment analysis and found that different tasklanguage combinations favour either monolingual and multilingual settings. The authors also concluded that data augmentation in the form of translation and task-adaptive pretraining (TAPT) (Gururangan et al., 2020) helps to further improve the results.
Another important task addressed in fake news detection is satire detection, with the methods ranging from convolutional neural networks (CNNs) (Guibon et al., 2019) to adversarial training (McHardy et al., 2019) and BERT-based architectures with long-short-term memory (LSTM) (Pandey and Singh, 2022; Liu and Xie, 2021) and CNN (Kaliyar et al., 2021) layers on top.
Bottleneck Adapters Adapters (Houlsby et al., 2019; Bapna and Firat, 2019) represent a family of techniques aimed at improving parameter efficiency in finetuning by freezing a pretrained model and inserting low-dimension adapter modules within each layer. Houlsby et al. found that, despite training only 3.6% of the parameters compared to a full model, performance only decreased by 0.4%, while Bapna and Firat found that adapters produced comparable, or in some cases better, results. Particularly relevant for our task is He et al. (2021)’s finding that adapter-based tuning of LLMs is particularly effective for low-resource and crosslingual tasks. For our system, we used two different
configurations of the bottleneck adapter modules: 1) the original Houlsby et al. bottleneck configuration, which places adapter modules after the multi-head attention block and feed-forward block of each transformer layer; 2) the Pfeiffer et al. configuration, which places adapter modules only after the feed-forward block of each layer.
Chalkidis et al. (2021) also found that for XLMR on the MultiEURLEX dataset, training bottleneck adapters outperforms traditional full finetuning and improves zero-shot cross lingual capability. Similarly to this task, the MultiEURLEX dataset was used for multilingual multilabel classification, though it is significantly larger than the data in this task (covering 23 languages and classifying hierarchically from 21 to 567 labels).

heading: 3 System Description for Subtask 1

heading: 3.1 System Overview
text: The system consisted of an ensemble of four models, comprising 1) three mBERT models each finetuned using the organiser training set and 2/3 of the development set; 2) one frozen mBERT model with a finetuned Houlsby adapter2. The ensemble predictions were decided by majority vote, with rare tie cases handled by selecting the model with the best validation performance.
Full Finetuning mBERTBASE (Devlin et al., 2019) was finetuned on a shuffled combination of all languages. Different to previous approaches (Wu and Dredze, 2020; Adebara et al., 2020), we chose the epoch with the best validation performance per-language, instead of overall, since the best overall epoch is not necessarily the best for a given language3. By using three identicallyconfigured models in the ensemble, the data sacrificed for model selection can be rotated between them, so overall no data is truly unseen.
Adapter Model A Houlsby bottleneck adapter was applied to a pretrained mBERTBASE model, with a reduction factor of 8 (i.e. d = 96), using the AdapterHub (Pfeiffer et al., 2020a) framework.
2Using a data split described in section 3.2 3See Appendix C.1
The mBERT model parameters were frozen, so only the adapter and classification head parameters were trained.
Data Preprocessing Since task data is obtained from webpages, it often contains unwanted content, such as hyperlinks, account handles, dates and author biographies. We applied the preprocessing described in Appendix B to remove this content.
Long Article Truncation The organiser annotation instructions indicate that even human annotators find it difficult to distinguish opinion from reporting and satire. This is due to subtle differences in how opinionated direct speech could be balanced or reported on. The instructions also mention that single opinionated sentences, which would trigger the opinion genre, often appear at the end of an article. Given the limit on input length for BERT models, for the articles that were longer than 512 tokens, we sequentially selected sentences from the beginning and the end of the article, preserving the original sentence order, until the length of 512 tokens was reached.
External Satire Due to the lack of satire data, our training set was supplemented with 203 Englishlanguage satire articles from Golbeck et al. (2018).
Data Oversampling The training data is severely imbalanced, with less than 6% of articles annotated as satire and 18% of articles annotated as reporting. Although the external satire data improved the balance for English, the performance of the satire class in other languages still remained inadequate. To address this, we performed oversampling for both satire and opinion classes by repeating random oversampling without replacement on the original data for a given language and class until the classes were balanced. For English satire class, we applied the same approach but oversampled the external satire data mentioned earlier rather than the original training set. We also compared the effectiveness of the oversampling approach with the class weighting approach in our experiment, and the results showed a slight advantage for the oversampling approach on average.

heading: 3.2 Experimental Setup
text: For the final submission, mBERT transformer models were finetuned on the organiser training set and a part of the development set for 30 epochs with the learning rate of 1e-5, AdamW optimiser (ε=1e-8)
and ReLU activation function. The organiser development set was split into three parts, stratified by label and language. We then finetuned three models, using 13 of the development set as a test and merging 23 of the development set with the training data and shuffling the dataset. The held-out part of the development set was used to identify the language-specific best checkpoints for each model. We utilised the checkpoint with the best overall F1macro on the held-out set to make predictions on the surprise languages. As described in section 3.1, all articles were preprocessed and the training data was oversampled for satire and reporting classes of each language.
Adapter models were trained on the combined organiser and development set (resplit 80% train, 10% validation, 10% held-out test, stratified by label), for 20 epochs with the learning rate of 1e-4, AdamW optimiser (ε=1e-8) and Tanh activation function in the classifier, and selected the checkpoint with the highest overall validation F1macro score. The above preprocessing and oversampling were not used, and articles were truncated at 512 tokens.
After submission, we conducted additional experiments using the organiser training and development sets for consistency. For monolingual models, all articles in the training set and the external satire were translated with Google Translate into the language of each monolingual model in question. Due to the character length limitation, particularly long articles were translated sentence-by-sentence.

heading: 3.2.1 Results and Reflections
text: The final submission results of the ensemble are listed in Table 2.
The final ensemble results achieved a higher F1macro score than in the supplementary multilingual results in all languages, except Polish. In English, the ensemble achieved F1macro score 25% higher than the single mBERT transformer or adapter model. It should be noted that the final models were trained on both training and development data. However, since the development sets
are only ≈ 13 the size of the training sets, the difference in the amount of the training data was not dramatic.
In the absence of gold standard labels for the test set, it is difficult to analyse why the model achieved a high score in Georgian, despite being zero-shot. However, our ensemble predictions suggest that there is likely to be no satire articles in the Georgian test set, which was consistently the most difficult class to detect.
Table 3 shows the differences between monolingual and multilingual versions of adapter bottleneck and transformer models, evaluated against the organiser development set. The multilingual transformer models always perform better than monolingual ones, while for 4 out of 6 languages, adapter bottleneck models benefit from the monolingual setup. This may be due to using a fixed reduction factor across all languages. Interestingly, the mBERT model demonstrates the best average result in English for both transformer and bottleneck adapter models. For Italian, XLM-R yields the best results for both transformer and adapter bottleneck models. It is also notable that the results for English are by far the worst across all the models, possibly because the models are overly focused on capturing semantic meaning and are not as effective in genre classification.
Even though transformer XLM-R demonstrated significantly better results than transformer mBERT for Italian and German, these differences were only marginal in our main setting where the validation set was smaller, while the marginally better results for Russian were not observed at all. Given the above observations and the fact that XLM-R yielded higher F1macro fluctuations, sometimes reaching 10%, we opted for the mBERT model as our main submission.

heading: 3.3 Post-competition Findings
text: Since, in our final submission, all languages were evaluated without translation (including the three surprise languages), a natural question we wanted to explore after the competition was whether translating texts into a different language for evaluation (the ‘translate-test’ approach) would have yielded better results.
We selected the checkpoints that, during training, achieved the highest validation F1macro for each individual language. These ‘language-optimal’ checkpoints were then used to evaluate transla-
tions of the other test sets. For example, using the ‘French’-optimal checkpoint, we translated all tests sets into French and made predictions.
Surprisingly, we found that the translate-test English → Russian and Italian → French each improved the F1macro performance by 1% on English and Italian respectively, while French → Russian improved by over 6%.
Two out of the three surprise languages, Spanish and Greek, also benefited from being translated into other languages and tested using the corresponding best checkpoints. The Spanish → English setting showed particularly striking increase in F1macro , from 68.7 to 81.7, which is also 21% higher than the score of the winning team for Spanish and is 20% above the other post-competition results. Except for German, translating the Greek test set into the other 5 main languages and testing using the corresponding checkpoints also provided significant improvements in the range of 5%-13%, which is over 1% above the result of the winning system and is the current leaderboard-best.

heading: 4 System Description for Subtask 2

heading: 4.1 System Overview
text: Two systems were used for submission, depending on the language. For English and the three surprise languages, we used a monolingual English ensemble of 3 RoBERTa-MUPPETLARGE models. For the remaining languages (French, German, Italian, Russian, Polish), we used a multilingual ensemble of 3 XLM-RLARGE models (with adapters and task-adaptive pre-training (TAPT)).
An overview of the two models is shown in Table 4. A key difference between these two systems is that the monolingual MUPPET models were trained using traditional finetuning of all parameters, whereas the XLM-R models 1) underwent task-adaptive pre-training; and 2) were finetuned using Pfeiffer bottleneck adapters.
For both systems, we trained our models jointly on articles in all languages (using English translations for our monolingual model). This meant that we produced a single monolingual or multilingual system that was able to make predictions for all languages. We chose this approach of joint training across all languages in order to maximise the number of examples seen for each class, since the dataset for Subtask 2 is quite small, particularly when split by language. Our early experiments showed that this approach was superior to training
models on the individual articles in each language.

heading: 4.1.1 The English Monolingual System
text: Full Finetuning Our monolingual system used a finetuned RoBERTa-MUPPETLARGE (Aghajanyan et al., 2021) ensemble. RoBERTa-MUPPET improves on its baseline RoBERTa counterpart by adding an additional ‘pre-finetuning’ stage of multitask learning. We opted not to use adapters, because our cross validation experiments showed this worsened performance (see Appendix Table 13).
English Translations Because RoBERTaMUPPET is a monolingual model, we translated all articles into English for training, and used them for finetuning alongside the original English articles. We performed inference on non-English languages by translating the articles into English: a ‘translate-test’ approach.

heading: 4.1.2 Multilingual System
text: We used XLM-RoBERTaLARGE (Conneau et al., 2020) for our multilingual model. Our system uses two techniques to improve performance: TAPT and adapter layers.
Task-adaptive Pre-training We performed taskadaptive pre-training on the entire XLM-R model, following the approach suggested by Gururangan et al. (2020). Masked-language modelling was
performed, using all available articles (including the organisers’ development and test sets). We trained for 60 epochs with a learning rate of 1e-4 and batch size of 128 (for full hyperparameters, see Appendix Table 11).
TAPT could alternatively be performed by freezing the base model and training the adapters with an MLM objective. Despite being a faster approach, this has been found to sometimes decrease performance (Kim et al., 2021).
Adapters Our multilingual system used the Pfeiffer bottleneck adapter configuration, with a reduction factor of 8, which for XLM-RLARGE corresponds to a bottleneck hidden size of 128.
Although using adapters did result in slightly improved performance, we found that their main advantage came from their low parameter number, which allowed for faster training and more experimentation.

heading: 4.2 Ensemble
text: Predictions made by our ensembles were decided by a majority vote. Each ensemble consisted of 3 individual models (MUPPET models for monolingual; adapter-finetuned XLM-R + TAPT for multilingual). Within each ensemble, two models were trained with class-weighting, and one-without.
Class Weighting Class weighting helped to account for class imbalance by balancing the impact of under- and over-represented classes. When calculating the loss, the logit for each class was multiplied by a class weight that was inversely proportional to the frequency of that class in the dataset.
Overall F1micro scores were similar for models with or without class weights. Class-weighting did help to improve performance on less frequent frames (such as Cultural Identity and Public Opinion), but at the expense of more frequent classes (such as Political). Additionally, class weights
were problematic in the joint language setting, causing varying performance across languages while maintaining similar overall F1micro . (A comparison for XLM-R is provided in the Appendix Table 13.) For this reason, we chose to use a mix of classweighted and non-class-weighted models for our ensembles in order to reduce the variance of our final systems.

heading: 4.3 Experimental setup

heading: 4.3.1 Data Preprocessing
text: For both monolingual and multilingual models, we cleaned and preprocessed the article text using a set of steps described in appendix B and truncated it to the first 512 tokens. For monolingual English models, we used Google Translate to produce English translations.

heading: 4.3.2 Data Split
text: For subtask 2, we merged the organiser training and development sets, and used 3-fold cross-validation (stratified by language) to identify the best model configurations. We then produced final models by training on the entire training and organizer development set (rather than a 2/3 fold).
Although this meant we did not have a validation set to judge the final models that went into our ensemble, it enabled training on all available data, which was important due to the small size of the dataset.

heading: 4.3.3 Hyperparameters
text: We finetuned our monolingual models for a fixed 20 epochs using a learning rate of 3e-5 (warm-up ratio 0.1; linear decay), a batch size of 8, and the AdamW optimiser.
We used the same hyperparameters for adapter finetuning, except we raised the learning rate to 1e4. In general, adapters require a higher learning rate than traditional finetuning, and this is reflected by the findings of Chalkidis et al. (2021) for a similar task.

heading: 4.3.4 Cross-validation Findings and Language Selection
text: Table 5 displays a condensed summary of our crossvalidation results (for full version, see Appendix Table 13).4 5
4This table shows the average performance of individual models trained during cross-validation, and not the performance of any ensembles.
5The Overall F1micro column refers to the F1micro of the entire validation fold and not the mean F1micro score across
For monolingual models, MUPPETLARGE achieves an F1micro of 70.4 on English, outperforming the RoBERTa baseline by 2 points. Similarly, our XLM-R + TAPT + Adapters demonstrates small but consistent improvements over the multilingual XLM-R baseline across most languages.
When comparing across monolingual and multilingual models, we see that for English, XLM-R models are unable to compete with the performance of monolingual MUPPET. (They are, however, able to match the performance of their monolingual counterpart RoBERTa). In contrast to this, the multilingual models generally demonstrated better performance on non-English languages. This is reflected by the overall F1micro scores: MUPPET’s 63.5 vs XLMR+TAPT+Adapter’s 64.2. Based on these results, we decided to use MUPPET for English and XLM-R for other languages.
For the unseen languages, we decided to use the monolingual ‘translate-test’ approach based on additional holdout experiments that indicated better MUPPET performance. Although this decision enabled us to achieve 1st place on the evaluation test set for Greek and Georgian, our post-competition findings (section 4.5) discovered that submitting our multilingual model may have achieved even better results.

heading: 4.4 Results and Reflections
text: The scores and positions of our model are shown in Table 6.
The strong performance of our monolingual model, which achieved first place in 3 out of 4 languages submitted, suggests that the ’translatetest’ approach is competitive for performing multilingual classification, especially for a zero-shot cross-lingual scenario.
Although our multilingual model also performed well, the competition results suggest that even better performance may be achieved by applying our monolingual approach to other languages i.e. for each language, training a native monolingual model with translations.
Article truncation poses a limitation on our findings, since news frames can potentially be located in parts of the article that are unseen by the model. One of important future extension of these experiments would be to apply long-document processing techniques to this task.
languages.
Monolingual English EN DE FR IT PO RU Overall F1micro RoBERTa-Large 68.4 ± 2.0 63.5 ± 2.0 57.9 ± 2.9 60.9 ± 0.2 65.8 ± 3.4 54.5 ± 2.7 63.6 ± 0.1 MUPPET-Large 70.4 ± 2.0 62.1 ± 3.7 59.0 ± 0.9 58.3 ± 1.5 65.7 ± 0.9 52.9 ± 1.7 63.5 ± 0.7
Multilingual

heading: 4.5 Post-competition Findings
text: Is a ‘translate-test’ monolingual model really better than a multilingual model? After the competition ended, we wanted to compare the performance of our two ensembles across all test set languages, as each system had only been submitted for a subset of languages.
Surprisingly, in contrast to our cross-validation experiments, we found that for English, our multilingual system outperformed our monolingual submission: 58.475 (multi) vs 57.895 (mono). The multilingual system also performed better on two of the surprise languages: Greek (58.0 vs 54.63) and Spanish (52.023 vs 50.829). In contrast to the findings of Xenouleas et al. (2022), who found translation-based approaches “vastly outperform cross-lingual finetuning with adapters”, this suggests that the two approaches are competitive with each other. It is difficult, however, to draw firm conclusions from this finding due to the small size of the unseen test set as well as the impact of TAPT. For more details, see Appendix D.

heading: 5 System Description for Subtask 3

heading: 5.1 System Overview
text: In subtask 3, our focus was not only to maximize the overall F1micro , but also to ensure a balanced
model performance across all classes6. Due to the highly imbalanced nature of the 23 classes in subtask 3, achieving a balanced model performance across all classes is challenging. To address this issue, similarly to subtask 2, we explored crosslingual training and implemented class weighting. We also explored an oversampling technique, but it did not provide any additional benefit compared to class weighting and increased the training time.
In contrast to the organiser baseline, which discards paragraphs without a label, we assigned them a class vector of zeros and included them in the training set. This method lead to a significant improvement in performance across all languages7.
We evaluated the performance of the models on unseen languages by training multilingual models while holding out each language. However, the performance significantly decreased for every language when held out. Therefore, we translated the articles of the three unseen languages into English for the final predictions.
Our final models were a RoBERTaBASE for the submission of English and translated unseen languages, and an mBERTBASE for the remaining languages.

heading: 5.2 Experimental Setup
text: RoBERTaBASE was finetuned for 20 epochs only on English data8, with a batch size of 32, truncated to 256 tokens, and AdamW optimizer with a learning rate of 0.00005, 20% of the training steps as linear warm-up and weight decay of 0.1. A classification threshold of 0.4 was used.
mBERTBASE was finetuned on all languages combined and shuffled, with the same hyperparameters as the RoBERTaBASE model, except a batch size of 16.
We conducted an experiment to explore cross-
6We further discuss this in appendix E.3 7A comparison of these approaches are in appendix E.1. 8Unlike subtask 2, we finetuned the model only with the original English dataset without adding other translated languages.
lingual training by comparing monolingual vs. multilingual models for each language with fixed hyperparameters, across three runs. The monolingual models were finetuned and validated on only one language, using either RoBERTa-Base for English or mBERT for the other languages, while the multilingual version was an mBERT finetuned on all the languages and validated on each language separately.
For the final submissions, we merged the training and development sets and finetuned the models without validation data. The model with the lowest training loss across all epochs was selected as the best model. We used the random seed that produced the best F1micro in the previous development set experiments while training the final model.

heading: 5.2.1 Results
text: Table 9 presents the final leaderboard results for subtask 3. Although the main metric for this subtask is F1micro, we highlight that our placings for subtask 3 improve considerably when measuring through F1macro, placing top 5 in all languages except the three surprise languages and first place for Italian and French, as shown in Table 16 in the Appendix.
Results for the monolingual and multilingual experiments on the official development set are displayed on Table 7, where we report the F1micro for each language. Scores are reported over three runs on different random seed initialisations. Notice that English is the only language in which the monolingual model outperforms the multilingual version.
Table 8 shows the F1micro for the zero-shot experiments, in which a multilingual model was finetuned on all languages except the one being evaluated. By comparison with Table 7, we see that zero-shot drastically hinders the performance on all languages. For this reason, we decided to translate
the test sets for the three surprise languages into English and perform inference using the English monolingual model. We were unable to experiment with translating into different languages other than English due to time constraints.

heading: 6 Conclusion
text: We presented three systems aimed at solving three subtasks within SemEval-2023 Task 3. Our systems applied a variety of state-of-the-art techniques including adapters and TAPT, and consistently achieved a high rank across all available languages, including zero-shot low-resource languages. We additionally presented an analysis of the viability of monolingual vs multilingual approaches for each subtask, and found that the results of the comparison vary depending on the subtask. For subtask 1, multilingual transformer models demonstrate better average performance than monolingual models with translations. A similar effect was observed for subtask 2 and subtask 3 where multilingual settings achieved better performance than monolingual ones in all languages except English. We found the impact on bottleneck adapters to be unpredictable across tasks – despite performing on average better for monolingual models in subtask 1, they were more beneficial for multilingual models in subtask 2 (and hindered
monolingual performance). Finally, we presented post-competition findings, which suggest that subtask 2 would have benefited from a zero-shot prediction using multilingual models, while subtask 1 could have achieved much better results with the ‘translate-test’ approach. Further analysis of this will be possible when test labels are released.

heading: 7 Acknowledgments
text: This work has been co-funded by the European Union under the Horizon Europe vera.ai (grant 101070093) and Vigilant (grant 101073921) projects and the UK’s innovation agency (Innovate UK) grants 10039055 and 10039039.

heading: A Language Models Used
text: The models used in each subtask are shown in Table 10.

heading: B Article Preprocessing
text: Articles were preprocessed with the following steps, for all languages:
• add a full stop at the end of the title. • remove duplicate sentences directly following
each other; • remove the @ symbol from twitter handles; • remove hyperlinks to websites and images;
English articles were further preprocessed: • remove lines indicating the possibility to share
the article on different social media platforms; • remove sentences suggesting the user to take
part in online polls, comments, or advertisements; • remove sentences indicating the terms of use; • remove sentences indicating the licenses and
containing phrases such as ‘reprinted with permission’, ‘posted with permission’ and ‘all rights reserved’; • remove sentences relating to the article author biographies

heading: C Subtask 1
text: C.1 Language-specific performance after each epoch on development set
Figure 1 shows F1macro scores on the held-out development set for one of the finetuned transformer models. As can be seen, Polish reaches its best performance quite early on, while German and Russian need more than 17 epochs to achieve the best score.

heading: D Subtask 2
text: Hyperparameters for Subtask 2 TAPT are shown in Table 11. The detailed cross-validation results are shown in Table 13.
D.1 Post-competition Findings Table 12 shows the full set of post-competition results. The multilingual model outperforms our official monolingual submission for test set English, Greek, and Spanish, but is much worse for Georgian. Conversely, our monolingual model does better than our official multilingual submission for French and Italian. This suggests that neither monolingual nor multilingual models are consistently better than the other across all languages.
Traditional zero-shot cross-lingual experiments focus on finetuning a multilingual model on a single source language and testing in a target language. In our situation, we jointly finetune on multiple languages, which encourages the model to retain multilingual representations, even for languages not in its training, thus improving its zero-shot cross-
lingual capabilities. This may help to explain the improved performance of our multilingual model. However, it is important to note that these results are not representative of true zero-shot classification, since our multilingual model did perform taskadaptive pre-training on articles from the surprise languages. Unfortunately, because the organisers have not released labels for the test set, we are unable to perform error analysis. As mentioned in the main section, the small size of the test set also makes it difficult to draw firm conclusions on whether translate-test is better than multilingual zero-shot classification.
Monolingual English EN DE FR IT PO RU Overall F1micro RoBERTa-Large 68.4 ± 2.0 63.5 ± 2.0 57.9 ± 2.9 60.9 ± 0.2 65.8 ± 3.4 54.5 ± 2.7 63.6 ± 0.1 MUPPET-Large 70.4 ± 2.0 62.1 ± 3.7 59.0 ± 0.9 58.3 ± 1.5 65.7 ± 0.9 52.9 ± 1.7 63.5 ± 0.7 MUPPET-Large + Adapters 68.0 ± 1.0 59.5 ± 1.6 54.5 ± 1.9 58.0 ± 0.7 61.9 ± 2.1 51.0 ± 3.7 61.1 ± 0.9
Multilingual Models

heading: E Subtask 3
text: E.1 Training With vs. Without Non-Labelled Examples
In the combined training set across all the languages, there are 9,280 paragraphs that do not have a label. Although it is expected that this also occurs on the test set, the organizer’s baseline approach discards these train samples, so it never explicitly trains on unlabelled examples. Table 14 displays the sizes of the train set for each language without adding non-labelled examples vs. adding them. Table 15 shows the F1micro results of both approaches, with means and stds computed over three random seed initialisations. Note that adding the non-labelled examples contributes to a considerable increase in performance for all languages, particularly English, which is also the language that had the biggest increase in train set size.
E.2 Development Set Fine-grained Results Table 17 shows the fine-grained results for the English official development set. Results are obtained from the best random seed over three runs. Although Appeal_to_Time, Appeal_to_Values, Consequential_Oversimplification and Questioning_the_Reputation classes do not have
a single example in development set, there are six other classes in which we also obtain 0.0 F1-Score, namely Appeal_to_Hypocrisy, Appeal_to_Popularity, Obfuscation-VaguenessConfusion, Red_Herring, Straw_Man and Whataboutism, although together they account for only 5% of the development set. The three biggest classes, Loaded_Language, Name_CallingLabeling and Doubt account for 29%, 15% and 11% of the development set, respectively, thus having a large impact on F1micro .
E.3 Full Leaderboard Results Table 16 shows our full final submission scores and placements according to both F1micro and F1macro . As we previously point out in section 5.1, we aimed towards a model capable of identifying all the 23 classes, thus having high F1macro , even though the main metric for the subtask is F1micro . We believe that a realistic application of a model for this particular label scheme should not disregard under-represented classes, otherwise they should simply be removed from the label scheme. Although our placings according to F1macro are considerably higher, we acknowledge that because the main metric for the subtask is not F1macro , other teams’ submissions are likely not focusing on maximizing it, thus making their scores lower on average.

heading: Future Work
text: One of important future extension of these experiments would be to apply long-document processing techniques to this task. languages. Monolingual English EN DE FR IT PO RU Overall F1micro RoBERTa-Large 68. 4 ± 2. 0 63. 5 ± 2. 0 57. 9 ± 2. 9 60. 9 ± 0. 2 65. 8 ± 3. 4 54. 5 ± 2. 7 63. 6 ± 0. 1 MUPPET-Large 70. 4 ± 2. 0 62. 1 ± 3. 7 59. 0 ± 0. 9 58. 3 ± 1. 5 65. 7 ± 0. 9 52. 9 ± 1. 7 63. 5 ± 0. 7
Multilingual