Hitachi at SemEval-2023 Task 3: Exploring Cross-lingual Multi-task Strategies for Genre and Framing Detection in Online News
Year: 2023

Abstract:
This paper explains the participation of team Hitachi to SemEval-2023 Task 3 “Detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup.” Based on the multilingual, multi-task nature of the task and the low-resource setting, we investigated different cross-lingual and multi-task strategies for training the pretrained language models. Through extensive experiments, we found that (a) cross-lingual/multi-task training, and (b) collecting an external balanced dataset, can benefit the genre and framing detection. We constructed ensemble models from the results and achieved the highest macro-averaged F1 scores in Italian and Russian genre categorization subtasks.

heading: 1 Introduction
text: As we pay more and more attention to the socially influencing problems like COVID-19 and the Russo-Ukrainian war, there has been an increasing concern about infodemic of false and misleading information (Piskorski et al., 2023). In particular, cross-lingual understanding of such information is becoming more important due to polarization of political stances, economical decoupling and echo chamber effect in social media. To that end, Piskorski et al. (2023) put together SemEval-2023 Task 3 “Detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup.” The shared task aims to analyze several aspects of what makes a text persuasive and to foster development of building blocks for multilingual media analysis.
Creating annotated data for media analysis is time consuming thus we cannot assume that we can obtain training data of enough quality and quantity. To tackle the problem, we investigated and compared strategies for multilingual media analysis under a low-resource setting. Through ex-
∗ Equal contribution
tensive experiments, we found that (a) cross-lingual/multi-task training, and (b) collecting an external balanced dataset, can benefit the genre and framing detection. We constructed ensemble models from the results and participated in genre categorization (subtask 1) and framing detection (subtask 2) in six languages, where we achieved the highest macro-averaged F1 scores in Italian and Russian subtask 1.

heading: 2 Task Definition and our Strategy
text: SemEval-2023 Task 3 aims to analyze several aspects of what makes a text persuasive. It offers three subtasks on news articles in six languages: German (de), English (en), French (fr), Italian (it), Polish (pl) and Russian (ru). There are three additional languages (Georgian, Greek and Spanish) without training datasets (i.e., participants need to perform zero-shot language transfer).
Subtask 1: News genre categorization Given a news article, a system has to determine whether it is an opinion piece, it aims at objective news reporting, or it is a satire piece. This is multi-class document classification and the official evaluation measure is macro average F1 score (macro-F1) over the three classes. Subtask 2: Framing detection Given a news article, a system has to identify what key aspects (frames) are highlighted the rhetoric from 14 frames (see (Card et al., 2015) for the taxonomy and definitions). This is multi-label document classification and the official evaluation measure is micro average F1 score (micro-F1) over the 14 frames. Subtask 3: Persuasion techniques detection Given a news article, a system has to identify the persuasion techniques in each paragraph from 23 persuasion techniques. This is multi-label paragraph classification.
The target articles are those identified to be
1702
potentially spreading mis-/disinformation and are collected from 2020 to mid-2022. They revolve around widely discussed topics such as COVID19, migration, the build-up leading to the RussoUkrainian war, and some country-specific local events such as elections.
We observed that the numbers of articles are limited for the relative large label space and there exist considerable overlaps of articles between subtask 1 and 2 (see Appendix A.1). Hence, we decided to investigate if models trained on multiple languages or another subtask can benefit the target task in this low resource setting (Section 5). Since subtask 1 and 2 conveniently share the task format, we opted to participate in subtask 1 and 2 in all the six languages.
We also noticed that the English training dataset exhibits significantly different label distribution to other languages and it is unbalanced. Hence, we decided to collect additional external dataset for English subtask 1 in a wish to improve task performance in English and to help with other languages through cross-lingual training (Section 3).

heading: 3 External Data for English Genre Categorization
text: In a preliminary analysis of the English subtask 1 dataset, we found that label distribution is quite unbalanced and it is different in the training and the development data. Therefore, we did not make any assumption about the distribution of the test data and decided to increase the number of rare labels in order to create a new, balanced dataset for English genre categorization. First, we undersampled articles from the training dataset for subtask 1 such that the numbers of articles for each label are equal, i.e., ten articles for each label.
We referred to a survey on fake news detection datasets (D’Ulizia et al., 2021) and checked a total of 27 datasets to see if they can be converted to subtask 1 dataset format using the following criteria: Label similarity We checked whether the labels
Satire Reporting Opinion Dataset name O E C O E C O E C Total Original 10 0 0 41 0 0 382 0 0 433 Augmented (small) 10 75 0 10 75 0 10 75 0 255 Augmented (large) 10 75 31 41 75 0 41 75 0 348
O: Official dataset, E: External, existing datasets, C: Collected by us
Table 2: Number of articles from different sources in the original and our augmented datasets
defined by an external dataset are close to subtask 1. For example, we focused on whether they used identical label names, such as “satire”. Text similarity We checked if the text type of a dataset is similar to subtask 1, such as whether they use news articles. Task similarity We checked whether the task setting employed by a dataset is a method of classifying them into different classes rather than, for example, scoring them with a scale of 1 to 5.
After these checks, we adopted the Random Political News Data (Horne and Adali, 2017) which contains 75 articles for each of three labels. We added the total of 225 articles to the sampled 30 original articles and constructed the Augmented (small) dataset which contains 255 articles in total.
Since Horne and Adali (2017) disclose the news media from which the data was collected, we independently collected around 1,000 additional articles from the sources shown in Table 1. However, we found in a preliminary experiment that overloading the dataset with external sources did not improve the performance. Hence we sampled 31 satire articles from the collected data and sampled more articles from the original dataset. This resulted in Augmented (large) with 348 articles altogether. The final compositions of the augmented datasets are summarized in Table 2.
Since Horne and Adali (2017) considered English articles only, we were only able to obtain external data for English subtask 1. Nevertheless, the augmented data might be able to benefit nonEnglish and subtask 2 datasets through pretraining on the augmented English dataset.

heading: 4 Cross-lingual Multi-task Transformers
text: We utilized pretrained language models (PLMs) in a simple sequence classification setup (Devlin et al., 2019). We employed XLM-RoBERTa large1 (Conneau et al., 2020) and RemBERT1 (Chung et al., 2021). For English, we also utilized RoBERTa
1https://huggingface.co/{xlm-roberta-large, rembert, roberta-large}
large1 (Liu et al., 2019) for single-language multitask training. In order to allow multi-task training, we added a classifier head for each subtask2 on top of each model’s [CLS] token followed by softmax for subtask 1 and sigmoid for subtask 2. Hence, each model shares most of the parameters for the two subtasks. We used Transformers library (Wolf et al., 2020) for the implementations.
In multi-task training we simply took sum of losses from two tasks. Since there exist articles that only have either of subtask 1 or 2 labels, we ignore predictions for missing labels from loss calculation. For cross-lingual training, we concatenate all articles. All parameters are shared for the same subtask in different languages. For preprocessing, we concatenated all sentences from each article and tokenized them using the default tokenizer for each PLM. We truncated articles whose tokens do not fit onto each model’s maximum context size.

heading: 5 Exploring Cross-lingual Multi-task Strategies
text: It is empirically known that further fine-tuning a model trained in a multi-task or cross-lingual setting on the target downstream task/language improves its performance (Koreeda et al., 2019). Also, models tend to require different hyperparameters for different training paradigms or languages. Hence, we decided to explore different multi-task and cross-lingual strategies through a series of random hyperparameter searches (Figure 1).
First, we ran a random hyperparameter search in cross-lingual and/or multi-task settings (Stage I). Regarding the resulting Stage I models as an additional hyperparameter, we ran another random hyperparameter search to optimize the choice of the pretraining paradigm along with other hyperparam-
2dropout→linear→tanh→dropout→linear for (XLM-) RoBERTa, and dropout→linear for RemBERT
eters (Stage II). Finally, we construct an ensemble for each language-subtask pair from all models in Stage I and II using their performance in the development dataset.
Unlike more sophisticated hyperparameter search methods, this approach has an advantage that we can compare and evaluate different training paradigms post hoc.
The choice of the subtask 1 English datasets (Section 3) is also incorporated as an additional hyperparameter. The hyperparameter search spaces are shown in Appendix A.2.
We used the official development dataset for each language-subtask pair in order to calculate and compare the performance of each model.

heading: 5.1 Stage I Training
text: In Stage I, we fine-tuned PLM in three settings. (1) Multi-task (30 hyperparameter sets for each
language = 180 models) (2) Cross-lingual (50 hyperparameter sets for each
subtask = 100 models) (3) Cross-lingual multi-task (50 hyperparameter
sets = 50 models) Hence, we trained 330 models in Stage I.

heading: 5.2 Stage II Training
text: Stage I results in three groups of models that have been trained on each language-subtask pair. For example, “en subtask 1” in Figure 1 has incoming arrows from (1) multi-task (“en subtask 1+2”), (2) cross-lingual (“de+en+fr+it+pl+ru subtask 1”), and (3) cross-lingual multi-task (“de+en+fr+it+pl+ru subtask 1+2”). We also utilize vanilla PLMs for Stage II training (see the arrow from RoBERTa).
For each language-subtask pair, we picked four models from each group, resulting in 12 models for each language-subtask pair. The four models were
chosen whose macro-F1, micro-F1, ROC-AUC or mAP was the best in the development dataset for the target language-subtask pair. This means that the same model can be chosen multiple times (e.g., a model which was the best in macro-F1 and microF1). We did not remove the duplicates in that case — such model will be sampled twice as much as a model which was the best only in a single metric.
Regarding these Stage I models and vanilla PLMs as an additional hyperparameter, we carried out Stage II random hyperparameter search on each language. We sampled Stage I models three times more than PLMs, so that all groups (i.e., the four arrows entering “en subtask 1” in Figure 1) are sampled equally. We trained 50 models for each language-subtask pair (50 models × 6 languages × 2 subtasks = 600 models).

heading: 5.3 Ensembling
text: Finally, we created an ensemble for each languagesubtask pair from the results of hyperparameter search. In a rare case, fine-tuning the model on the downstream task can degrade the performance. Hence, we also considered the Stage I models for the ensemble.
We implemented multiple ensemble methods. Due the scarcity of the development data, the results tend to be unstable. Hence, we manually chose the best one for each language-subtask pair while monitoring multiple leave-one-out metrics on the development dataset. This allowed us to
choose models that are not only overfitting to a single metric. The details of ensembling are described in Appendix A.3.

heading: 6 Results

heading: 6.1 Subtask 1: News Genre Categorization
text: Excerpts from the official leaderboards (Piskorski et al., 2023) for subtask 1 are shown in Table 3. We were the first place in Italian and Russian and within top threes in French and Polish. In an unofficial ranking of mean macro-F1 of six languages, we were the first place (Table 4).
In Figure 2, we show macro-F1 for the development dataset of all the models considered for the ensemble construction. In all six languages, the models fine-tuned from cross-lingual and/or multi-lingual pretraining tend to perform better (i.e., have better median macro-F1) than the single language/task models trained from PLM (“PLM → Stage II”). This shows that cross-lingual multi-task training was overall useful for genre categorization. In most cases, fine-tuning Stage I models in Stage II yields better results than the vanilla Stage I models.
The breakdown of the performance based on how each model was pretrained in Stage I is also shown in the Figure 2. The results are mixed as to which Stage I pretraining paradigms were useful to the Stage II downstream performance. In German, French and Italian, cross-lingual pretraining tends to be more beneficial than multi-task pretraining. In English, Polish and Russian, multi-task pretraining tends to be more beneficial. Interestingly, the combination of the both was never the best option in any language.
We analyzed the effect of incorporating external, balanced datasets for English subtask 1 (Figure 3). When directly fine-tuning PLM in Stage II, we
0.1 0.2 0.3 0.4 0.5 0.6 0.7 macro F1
PLM Stage II
Stage I Stage II
Original Aug. (small) Aug. (large)
Figure 3: The effect of different training datasets in the development dataset for English subtask 1
see that models using either of external datasets tend to be considerably better than ones trained on the original training data. For both conditions, we can see that Augmented (large) tend to perform better than Augmented (small). This shows that obtaining a balanced dataset is important for news genre categorization.

heading: 6.2 Subtask 2: Framing Detection
text: Excerpts from the official leaderboards (Piskorski et al., 2023) for subtask 2 are shown in Table 5. We were third to fifth places in all but Russian where we obtained the ninth place.
In Figure 4, we show micro-F1 for the development dataset of all the models considered for the ensemble construction. As in subtask 1, the models fine-tuned from cross-lingual and/or multi-lingual pretraining tend to perform better than the single language/task models fine-tuned directly from PLM. This suggests that cross-lingual multi-task training was also useful for framing detection.
In all languages in subtask 2, cross-lingual Stage II pretraining tends to result in better micro-F1 than multi-task models (Figure 4). We suspect that this lies in the difference in the linguistic nature of two subtasks; Framing can be determined by lexical semantic to some extent, hence transfers well across different languages with multilingual transformers. On the other hand, distinguishing the genre requires
capturing language-specific pragmatics which may be the reason why it did not transfer between languages as effectively as subtask 2.

heading: 7 Related Work
text: There exist several studies on multilingual and/or multi-task learning in the context of media analysis. Uyangodage et al. (2021) applied cross-lingual training to multilingual false information detection and showed that cross-lingual training results in on par performance to monolingual training. Alam et al. (2021a,b) annotated Tweets in multiple languages with multiple choice questions regarding COVID-19 disinformation. They fine-tuned mBERT model (Devlin et al., 2019) in a crosslingual setup and in a multi-task setup, and found that the results are mixed in terms of their benefits. The SemEval 2023 Task 3 dataset (Piskorski et al., 2023) exhibits significantly different properties to the datasets previous works used and we found that cross-lingual and/or multi-task learning help in this dataset. We also carried out more thorough experiments than the previous works, which might have been the key to the improvement.
There also exist different approaches for multilingual and low-resource settings in broader domains. Reimers and Gurevych (2020) proposed a method for training multilingual sentence embeddings and demonstrated its effectiveness in 50+ languages. In a low-resource setup, Heinisch et al. (2022) used a data augmentation approach that retrieves additional data from related datasets by automatically labeling them. We only explored BERT-based approach in this work, but we wish to explore other approaches for cross-lingual multi-task learning in the future.

heading: 8 Conclusion
text: In our participation to SemEval-2023 Task 3, we investigated different strategies for multilingual genre and framing detection. Through the extensive experiments, we found that collecting an external balanced dataset can help genre categorization. We also find that cross-lingual and multi-task training can help both genre and framing detection and found that cross-lingual training is more beneficial for framing detection. We constructed ensemble models from the results and achieved the highest macro-F1 in Italian and Russian genre detection.
For future work, we will investigate the effect of cross-lingual multi-task training on zero-shot language transfer (Greek, Spanish and Georgian subtasks that we did not participate), as well as
the effect on and benefit from training models on persuasion techniques detection (subtask 3).

heading: Acknowledgements
text: We used computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by the National Institute of Advanced Industrial Science and Technology (AIST) for the experiments.
We would like to thank Gaku Morio and Yuichi Sasazawa for their valuable feedbacks. We would also like to thank Dr. Masaaki Shimizu for helping us with the computational resources.

heading: A Appendix
text: A.1 Data Analysis
Numbers of articles in each subtask and their overlaps are summarized in Table 7. We can see that the numbers of articles are limited for the relative large label space and there exist considerable overlaps of articles between subtask 1 and 2.
Readers should refer (Piskorski et al., 2023) for the details on the datasets.
A.2 Details of Hyperparameter Search
As described in Section 5, we carried out the extensive experiments as a series of hyperparameter searches. In this section, we will list and describe
all the hyperparamter search spaces of Stage I and II training.
The hyperparameter search spaces of Stage I training are listed as in the following: • Cross-lingual multi-task: Table 8 • Cross-lingual Table 9 and 10 • Multi-task: Table 11 and 12
The hyperparameter search spaces of Stage II training are listed in the following: • Subtask 1 in English: Table 13 • Subtask 1 in all other languages: Table 14 • Subtask 2 in English: Table 15 • Subtask 2 in all other languages: Table 16
We introduced a loss weighting technique and introduced it as an additional hyperparameters. Since label distributions are highly skewed in subtask 1, we weight losses for each label Ll (l ∈ {satire, opinion, reporting}) by wl (i.e., L′l = wl · Ll) such that they are inversely proportional to the count of each label cl (i.e., wl ∝ 1/cl)
while adding up to 1 (i.e., ∑
l wl = 1).
wl = hmean(csatire + copinion + creporting)
cl ,
where hmean is harmonic mean. For subtask 2, we carried out the classification in both a single multi-label classification and multiple, separate binary classifications. We have also regarded the choice of the classification method as a hyperparamter and incorporated this into the random search in Stage II (“classwise”).
A.3 Details of Ensemble Construction
We created an ensemble for each language-subtask pair from the results of hyperparameter search. As outlined in Section 5.3, we implemented multiple ensemble methods and manually chose the best one for each language-subtask pair. Here, we show the details of ensemble construction and selection on each subtask.
A.3.1 Ensembles for Subtask 1 For subtask 1, we implemented three ensemble methods: Top one We choose the best model with the best macro-F1 in the development dataset. Top 3 average We picked three models based on the macro-F1 score in the development dataset.
We take an average of the output probabilities (i.e., scores after softmax). Bootstrap bagging We greedily add models to average ensemble with replacement until the score no longer improves or the ensemble size reaches five. We use the minimum F1 score of all the classes. This idea of trying to improve the worstclass performance was inspired by distributionally robust optimization.
We also considered different pools of candidate models to construct the ensembles from. Specifically, we either created an ensemble from (i) all the models from Stage I and II, and (ii) the models only from Stage II (i.e., only models that are fine-tuned on the target language).
Due to the highly imbalanced label distribution, especially the lack of satire pieces in the training dataset, we introduced two postprocessing methods at the ensembling stage: Probability reweighting We multiply the probability for satire label by a value, followed by renormalization of the probabilities by their sum. Heuristics-based relabeling The model tend to output the opinion label more than other labels in English, even though we balanced the training dataset (Section 3). Hence, we converted a portion
of predicted opinion labels to other labels with heuristics. First, we selected documents whose words consist of more than 0.8% CARDINAL named entities3. This is based on the intuition that both satire and reporting pieces tend to utilize numbers to be more persuasive. We then relabeled the selected documents with satire label if it contains “!” or “?”. Otherwise, they were relabeled with reporting label.
The model pools and the postprocessing methods were also considered as an option and we compared all the combinations of the ensemble methods, the
3We used spaCy (https://spacy.io/) for named entity recognition.
Language
Label English French German Italian Polish Russian
Capacity and resources Top3 (AP)* Top3 (ROC) Top1 (F1)* Top5 (ROC)* Top3 (ROC) Top3 (F1)* Crime and punishment Bagging* Top5 (F1) Bagging Top3 (F1) Top3 (ROC)* Top5 (F1)* Cultural identity Top5 (ROC)* Top1 (F1)* Top5 (F1) Top5 (F1)* Top5 (F1)* Top3 (AP)* Economic Bagging* Top1 (F1) Top5 (ROC) Top5 (F1)* Top3 (AP)* Top3 (F1) External regulation and reputation Top3 (F1)* Top5 (F1)* Top1 (F1) Top3 (AP) Top3 (AP) Top3 (ROC)* Fairness and equality Top3 (AP) Bagging* Top3 (AP) Top3 (AP)* Top3 (F1) Top5 (F1) Health and safety Top5 (ROC) Top5 (ROC)* Top3 (F1)* Top5 (ROC)* Top3 (ROC) Top5 (F1) Legality, constitutionality and jurisprudence Bagging Top3 (ROC)* Top3 (F1) Top5 (F1)* Top5 (F1)* Top5 (F1)* Morality Top5 (AP)* Top3 (AP)* Top3 (F1)* Top5 (F1) Top1 (F1) Top3 (AP)* Policy prescription and evaluation Stacking* Top3 (AP)* Top5 (ROC)* Bagging* Top3 (ROC)* Top3 (AP)* Political Top5 (F1)* Top3 (AP) Top3 (AP) Top3 (F1) Top3 (ROC)* Top3 (AP) Public opinion Top3 (F1)* Top5 (F1)* Top1 (F1)* Top3 (F1) Top5 (F1) Top3 (AP) Quality of life Top3 (AP)* Top1 (F1) Top5 (ROC)* Top3 (F1) Top3 (ROC) Top3 (ROC)* Security and defense Bagging* Top3 (ROC) Top3 (AP) Top5 (F1) Top5 (F1)* Top5 (ROC)
* Chooses models from both Stage I and II. Otherwise, models are chosen only from Stage II. See Appendix A.3.2 for the details about the ensemble methods.
Table 18: The selected ensemble method for each language in subtask 2
model pools and the postprocessing methods. Due the scarcity of the development data, the results tend to be unstable. Hence, we manually chose the best ensemble type for each language with following criteria while monitoring leave-oneout metrics on the development dataset.
• We try to choose model with a good class score balance (i.e., good macro-F1) and good general classification abilities (good mAP and ROCAUC).
• Unless the difference is unbearably large, we tried to avoid top one model as it can be unstable.
After choosing the best ensemble method, model pool and postprocessing method for each language, we recreated the ensemble using the whole development dataset. The selected ensemble method for each language is shown in Table 17.
A.3.2 Ensembles for Subtask 2
For subtask 2, we implemented nine ensemble methods: Top one We choose the best model with the best macro-F1 in the development dataset. Top n average We picked n models based on the score in the development dataset. We take average of the output probabilities (i.e., score after the sigmoid). We adopted ranking by (1) F1 score with n = 3, (2) average precision score with n = 3, (3) ROC-AUC score with n = 3, (4) F1 score with n = 5, (5) average precision score with n = 5, and (6) ROC-AUC score with n = 5. Bootstrap bagging Same as subtask 1 but we optimize F1 score. Stacking ensemble We fit lasso regression classifier on the development dataset (C = 1.0), regarding probability from each model as a feature.
As in subtask 1, we considered different pools of
candidate models to construct the ensembles from (Appendix A.3.1).
We manually chose the best ensemble method and model pool for each language-label pair in the same way as subtask 1. The selected ensemble method for each language-label pair is shown in Table 18.

heading: Future Work
text: For future work, we will investigate the effect of cross-lingual multi-task training on zero-shot language transfer (Greek, Spanish and Georgian subtasks that we did not participate), as well as
the effect on and benefit from training models on persuasion techniques detection (subtask 3)