MELODI at SemEval-2023 Task 3: In-domain Pre-training for Low-resource Classification of News Articles
Year: 2023

Abstract:
This paper describes our approach to Subtask 1 "News Genre Categorization" of SemEval2023 Task 3 "Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup", which aims to determine whether a given news article is an opinion piece, an objective report, or satirical. We fine-tuned the domain-specific language model POLITICS, which was pre-trained on a large-scale dataset of more than 3.6M English political news articles following ideologydriven pre-training objectives. In order to use it in the multilingual setup of the task, we added as a pre-processing step the translation of all documents into English. Our system ranked among the top systems overall in most language, and ranked 1st on the English dataset.

heading: 1 Introduction
text: The News Genre Categorization subtask of SemEval-2023 Task 3 (Piskorski et al., 2023) is a classification task that aims to identify the genre of a news article by considering three classes: opinion, reporting, satire. News genre categorization is important because it helps readers to quickly identify the type of content they are reading. It also helps to ensure that readers are not misled by the content they are consuming. The organizers provided articles in six languages for the training phase (English, French, German, Italian, Polish, and Russian) and three surprise languages were revealed for the evaluation on the test sets (Spanish, Greek and Georgian). It is therefore a multilingual task that includes small datasets during training and zero-shot classification for surprise languages.
Our main strategy relies on the fine-tuning of POLITICS (Liu et al., 2022), a state-of-the-art language model for political ideology prediction and stance detection, pre-trained on more than 3.6M English news articles. Based on this, a wide variety of strategies were tested, including a sliding
window to cover long documents, pooling methods, adapted loss functions, data splitting and hyperparameters tuning. Moreover, as POLITICS is monolingual, and to take advantage of the largescale pre-training in the multilingual setup, we propose as a pre-processing step to translate all articles from other languages into English. Other interesting strategies have also been considered and tested but have been less successful on the development set, including inherently multilingual models such as XLM-RoBERTa (Conneau et al., 2020) or longsequence transformers (Beltagy et al., 2020).
We ranked 1st on the English test set with a Macro-F1 score of 78.43 (+16.8 points above the second ranked system), but were less successful on the other languages with an average of 6th place (out of about 20 systems, depending on languages). We draw several observations from our participation in this task: - the dominance of large-scale pre-trained and domain-specific language models over more adapted but not massively pre-trained approaches; - the need to further develop multilingual language models and datasets rather than having to rely on low performing translation systems for complex tasks; - the importance of considering the whole text in the case of long documents, whereas most language models are limited to the first 512 subtokens.

heading: 2 Background
text: News genre categorization is a multi-class, singlelabel classification problem that aims to predict the type of content of a news article according to 3 classes (Piskorski et al., 2023):
Opinion An article is an opinion piece when it expresses the writer’s opinion on a topic. It is usually written in a persuasive style and is meant to influence the reader’s opinion on the subject.
108
Reporting News reporting is the genre that focuses on providing factual information. It aims to inform readers about the world around them and to provide an objective account of events.
Satire Satire is a form of writing that uses exaggeration, absurdity and obscenity to mock and ridicule people, organizations or events. It is not meant to deceive, but to expose and criticize behavior that is wrong or immoral. Satirical pieces often mimic real articles, using irony to provide humor.

heading: 2.1 Datasets
text: The input data for this task are news articles in plain text format, with their title being on the first line. During the first stage, a training split and a development split were provided for each of the 6 known languages: English (en), French (fr), German (ge), Italian (it), Polish (po), and Russian (ru). The annotations were given only for the training sets at first, then also for the development sets when the test sets were released. For the evaluation on the test sets, 3 surprise languages were revealed which involves zero-shot classification (no train or dev sets): Spanish (es), Greek (gr) and Georgian (ka). Given the limited number of news articles available per language, this is a kind of few-shot learning task. This problem is also characterized by a strong class imbalance, especially for the satire genre which is represented only a few dozen times across the datasets. Dataset statistics are summarized in Table 1. The distributions of classes vary a lot between corpora, between train, development, and probably test sets, given the observed performance differences between dev and test. This means there was a (blind) domain-adaptation aspect to the task.

heading: 2.2 Related Work
text: Substantial efforts have been made to address the problem of media analysis on a multitude of aspects such as bias analysis (Hamborg et al., 2019), summarization (Eyal et al., 2019), text categorization (Pérez-Rosas et al., 2018; Karimi and Tang, 2019) or propaganda techniques (Da San Martino et al., 2020). Recently, Liu et al. (2022) proposed POLITICS, a pre-trained language model over RoBERTa (Liu et al., 2019), fine-tuned on a large corpus of English news to address both political ideology prediction and stance detection. Their approach has proven to be the most efficient on a wide range of well known datasets for these tasks such as
Hyperpartisan (Kiesel et al., 2019), Allsides1 (Baly et al., 2020) or BASIL (Fan et al., 2019).

heading: 3 System Overview
text: Our main strategy is based on Liu et al. (2022)’s domain-specific pre-trained language model POLITICS which has recently demonstrated great success on a variety of news articles ideology and stance prediction datasets. It has several advantages for this task: (i) it was massively pre-trained on more than 3.6M English news articles, (ii) it relies on the comparison of articles on the same story written by media of different ideologies, (iii) it demonstrated its robustness in few-shot learning scenarios. Although predicting political ideology is not the same as detecting the genre of an article, our assumption is that these two notions overlap as in both cases there is a linguistic shift in the way information is conveyed. We can also observe similarities between the genre reporting and articles from the Center political class, or between the genre opinion and Left or Right leaning articles.
1https://www.allsides.com

heading: 3.1 Data Translation
text: POLITICS was trained solely on articles from English-language media, whereas in the multilingual configuration proposed for this task, we have to classify articles in 9 languages, 3 of which were unknown during training. Because of this strong constraint and the impossibility of re-training POLITICS in a multilingual configuration, we decided to resort to translation into English. By translating all texts into English, we end up with an augmented English training set that can be used with POLITICS. This solution represents an additional cost due to the pre-processing step, and a loss of information that depends on the quality of the translation system. Several translation models were compared based on performance, language coverage, and accessibility, including GoogleTranslate,2 DeepL3 and OPUS-MT (Tiedemann and Thottingal, 2020), resulting in choosing GoogleTranslate as the most appropriate one. DeepL was the best performing system but had accessibility limitations due to its pricing for handling large amounts of data, we had to fall back on GoogleTranslate which was the second best performing and freely available system.

heading: 3.2 Fine-tuning
text: For the classification model, we fine-tuned POLITICS on the English dataset augmented with translations. It works by adding a 2-layer perceptron on top of the standard [CLS] classification token representation to predict the distribution over class labels. POLITICS is based on RoBERTa and, like
2https://translate.google.com 3https://www.deepl.com/
most language models, is limited with respect to the size of the input text, here at most 512 subtokens. News articles are on average much longer (Figure 1), thus truncating the first 512 subtokens would result in a significant loss of information. Rather than truncating the text, we use a sliding window of size 512 with an overlap of size 64, and aggregated the information by mean pooling. That is, we encode the first 512 tokens of the document, then the next 512 with an overlap of 64, and so on, until we reach the end of the document, then we calculate the mean of all these representations to obtain the final representation of the document.

heading: 3.3 Alternative Approaches
text: Other approaches of interest for this task were also considered, but proved to be less successful during the training phase on the development split. In particular, on English, POLITICS showed much higher performance than the other models considered and was on average better on the other languages. Thus, we have favored this model for the evaluation although a posteriori we can see that it is less efficient in certain cases (e.g. on surprise languages or on French/Russian, Table 3).
XLM-RoBERTa (Conneau et al., 2020) The multilingual version of RoBERTa (Liu et al., 2019), a state-of-the-art model in natural language processing, pre-trained on 2.5TB of CommonCrawl data containing 100 languages and including the 9 ones covered by this task. Similar to POLITICS, we used a sliding window to consider the entire
article.
Longformer (Beltagy et al., 2020) Transformerbased model designed to process longer sequences, up to 4096 tokens, making it suitable for news article classification, and pre-trained on a variety of large-scale generic datasets, including Wikipedia, BooksCorpus, and CommonCrawl. No sliding window was introduced with this model, as the input length covers the vast majority of texts, cf. Figure 1.

heading: 4 Experimental Setup
text: First, the train, dev and test split articles for each language are translated using Google Translate. Next, we aggregate the train and dev splits for each language to create an augmented English dataset containing 1234 articles in the training split and 358 in the development split, on which we train a single English model.
We built on Liu et al. (2022)’s implementation,4
and the HuggingFace transformers library (Wolf et al., 2020). Hyperparameters were set using grid search on the augmented development split for each of the models considered (Table 2). The hyperparameters for POLITICS and XLM-RoBERTa were found to be the same. We also conducted a control experiment on the sliding window to assess its effectiveness by evaluating the same model trained only on the first 512 tokens of the articles. Control experiments and alternative approaches (see table 3
4https://github.com/launchnlp/POLITICS
were performed on the same set of hyperparameters. Since there is a strong class imbalance in the training set, we weighted the cross entropy on the class distribution, which turned out to give us better performance.
Regarding alternative approaches, we fine-tuned xlm-roberta-large5 using a sliding window on the same set of hyperparameters as POLITICS (Table 2). Longformer was trained using longformer-base-40966 and the hyperparameters given in Table 2.
The official evaluation measure used is MacroF1. Micro-F1 is also given as a secondary evaluation measure on the official leaderboard.

heading: 5 Results
text: Table 3 summarizes the results obtained by our main strategy, the control experiments, the baseline proposed by the task organizers and the alternative approaches. Our main strategy (POLITICS) obtains the best results for 3 of the 9 languages, with a special distinction for English on which it particularly stands out (+9 points than the second best approach we tested). This shows the benefits of large scale in-domain pre-training on English news articles, but also the limitations of translation. For French, the score is surprisingly low, which was not as pronounced in our experiments on the development split. From the control experiment, we can confirm the interest of the sliding window
5https://huggingface.co/xlm-roberta-large 6https://huggingface.co/allenai/
longformer-base-4096
rather than just truncating the article, with important gains on most training languages. Interestingly, for the surprise languages, removing the sliding window leads to much better results, which shows that in a zero-shot context, introducing too much unseen language-specific information confuses the model. Regarding the alternative approaches, the results obtained by Longformer confirm the previous observations on the importance of considering the whole article, with results close to or even better than POLITICS outside English and without any in-domain pre-training. Furthermore, the XLM-RoBERTa multilingual model performs well against POLITICS without the need for translation, but the lack of in-domain pre-training results in a significant performance drop, especially for English.
These results should be taken with a grain of salt for the following reasons: test sets are small (30-70 instances) hence the huge variance between models and across languages. This problem is compounded by the chosen evaluation : Macro-F1 scores equalize the contributions of all classes to the final scores, meaning that one instance from the minority class classified one way or the other could swing the evaluation disproportionately. It would probably be informative to evaluate accuracy, and per class metrics, but gold labels were not provided for the test sets. This also makes it hard to estimate the distribution shift between train, development and test sets, although it is already apparent there are large differences between train and dev sets.

heading: 6 Conclusion
text: We propose to take advantage of in-domain pretraining for detecting the genre of news articles. We fine-tuned the large-scale English language model POLITICS using sliding windows on the multilingual corpus translated into English. Our various experiments have shown the effectiveness of indomain pre-training as well as the importance of approaches adapted to the processing of long documents. We ranked 1st on the English dataset while being among the top systems overall.

heading: Acknowledgements
text: Nicolas Devatine’s work is supported by the SLANT project (ANR-19-CE23-0022). This work was partially supported by the ANR (ANR-19PI3A-0004) through the AI Interdisciplinary Institute, ANITI, as a part of France’s “Investing for
the Future — PIA3" program. This work is also partially supported by the AnDiaMO project (ANR21-CE23-0020). Chloé Braud and Philippe Muller are part of the programme DesCartes and are also supported by the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.

heading: Future Work
text: This work was partially supported by the ANR (ANR-19PI3A-0004) through the AI Interdisciplinary Institute, ANITI, as a part of France’s “Investing for
the Future — PIA3" program. This work is also partially supported by the AnDiaMO project (ANR21-CE23-0020). Chloé Braud and Philippe Muller are part of the programme DesCartes and are also supported by the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme