BERTastic at SemEval-2023 Task 3: Fine-Tuning Pretrained Multilingual Transformers – Does Order Matter?
Year: 2023

Abstract:
The naïve approach for fine-tuning pretrained deep learning models on downstream tasks involves feeding them mini-batches of randomly sampled data. In this paper, we propose a more elaborate method for fine-tuning Pretrained Multilingual Transformers (PMTs) on multilingual data. Inspired by the success of curriculum learning approaches, we investigate the significance of fine-tuning PMTs on multilingual data in a sequential fashion language by language. Unlike the curriculum learning paradigm where the model is presented with increasingly complex examples, we do not adopt a notion of “easy” and “hard” samples. Instead, our experiments draw insight from psychological findings on how the human brain processes new information and the persistence of newly learned concepts. We perform our experiments on a challenging news-framing dataset that contains texts in six languages. Our proposed method outperforms the naïve approach by achieving improvements of 2.57% in terms of F1 score. Even when we supplement the naïve approach with recency fine-tuning, we still achieve an improvement of 1.34% with a 3.63% convergence speed-up. Moreover, we are the first to observe an interesting pattern in which deep learning models exhibit a human-like primacyrecency effect.

heading: 1 Introduction
text: Deep learning models are state-of-the-art (SOTA) in many fields including natural language processing (NLP). In NLP, the current SOTA models (Wang et al., 2019) are based on transformers, which are deep-learning models with attention mechanism (Vaswani et al., 2017). While many transformers are monolingual, there has been increased research and public interest in multilingual transformers (Doddapaneni et al., 2021). Notably, pretrained transformers, require huge amounts of training data, no matter the domain (Devlin et al., 2019; Dosovitskiy et al., 2021).
Consequently, this results in a substantial carbon footprint (Strubell et al., 2019), which is against global sustainability objectives.1 There are many approaches to address the AI carbon footprint concerns ranging from using more carbon-efficient energy sources to applying more efficient AI models and training algorithms. Curriculum learning (CL) encompasses a specific class of efficient training strategies for deep learning models. On the one hand, the naïve approach of fine-tuning pretrained deep learning models on downstream tasks involves feeding them mini-batches of randomly sampled subsets of the available training data. On the other hand, in curriculum learning, the idea is to finetune the model with a sequence of progressively more challenging examples. This is motivated by and mimics the way humans learn, where we start with simpler concepts and gradually build up more complex ones. Curriculum learning research (Soviany et al., 2022) shows that such a strategy helps the model achieve better performance and converge faster.
1https://sdgs.un.org/goals
58
Contributions Following the success of CL approaches, and inspired by cognitive science, we propose an approach to fine-tuning PMTs on multilingual data. We investigate the significance of doing this in a sequential fashion language by language. Unlike CL where the model is presented with increasingly complex examples, we do not adopt a notion of “easy” and “hard” examples. Instead, our experiments draw insight from psychological findings on how the human brain processes new information and the persistence of newly learned concepts (Murdock, 1962).
We perform our experiments on a multi-label text classification dataset (Piskorski et al., 2023) that contains text in six languages: English, French, German, Italian, Polish, and Russian. Our proposed method outperforms the naïve approach by achieving an F1 score gain of 2.57%. Even when we supplement the naïve approach with recency finetuning, it achieves an F1 score gain of 1.34% with a 3.63% average convergence speed-up. Moreover, we observe an interesting pattern in which deep learning models exhibit a human-like primacyrecency effect, which is also commonly referred to as the serial-position effect (Murdock, 1962). The effect describes the human tendency to remember the first and the last items in a list more accurately than the ones in the middle. Our contributions are as follows:
• We propose and evaluate fine-tuning PMTs on multilingual data in a sequential fashion language by language.
• We find that a deep learning model exhibits a human-like primacy-recency effect.
• We compare the performance of PMTs finetuned on monolingual data versus multilingual data.
• We examine the use of translation for data augmentation and analyze the performance of monolingual versus multilingual pretrained language models.

heading: 2 Background

heading: 2.1 SemEval Task Description
text: We perform our experiments on data from the second subtask of task 3 of SemEval-2023 on “Detecting the Genre, the Framing, and the Persuasion Techniques in Online News in a Multi-Lingual Setup” (Piskorski et al., 2023).
This is a challenging task and it is more nuanced than mere topic classification (Card et al., 2015), e.g., while the topic of a news article may be COVID-19, the framing could be from an economic, political, or/and health perspective(s). In concrete terms, the news framing subtask can be formulated as a multilabel text classification problem. Given a news article, Ti = [w1, ..., wNi], where wi denotes the i
th token, and Ni denotes the number of tokens in Ti, the goal is to learn a mapping f ∶ T → S where S = [a1, ..., aM] and aj ∈ True, False denotes whether Ti contains the j th framing label and M denotes the total number of framing labels, which is fourteen in this subtask. Table 1 shows in detail the train/dev/test splits per language as provided by Piskorski et al. (2023). The dataset has many challenges:
• The classes are imbalanced (Figure 2a).
• The frames’ proportions across languages are unequal (Figure 2b).
• Most examples, namely 80.51%, contain more than 512 tokens, which is the maximum sequence length for BERT-like models.
• The number of examples per language is small and so is the collective dataset size.
• Three of the test languages are surprise languages for which no training or development data was provided.

heading: 2.2 Related Work
text: Curriculum Learning A large body of research on CL (Soviany et al., 2022) investigates efficient training strategies for deep learning models. A core idea in CL research is the notion of “easy” and “hard” examples.
Qualifying the difficulty of an example requires creating a complexity evaluation method, which could be challenging. Previous evaluation methods in NLP ranged from simply looking at sentence length (Cirik et al., 2016) to examining specialized linguistic features (Jafarpour et al., 2021).
Primacy-Recency Effect Nikishin et al. (2022) showed that deep reinforcement learning models exhibit a primacy bias. Wang et al. (2023) applied a machine learning model to study primacypeek-recency effect in human subjects. Both papers, however, do not tackle, nor do they observe a primacy-recency effect in deep learning models. To the best of our knowledge, we are the first to propose and to evaluate fine-tuning PMTs on multilingual data in a sequential fashion language by language. We are also the first to observe the human-like primacy-recency effect in deep learning models in general and in PMTs in particular.

heading: 3 System Overview
text: We perform all of our experiments using two transformer-based models, XLM-R with 278M parameters (Conneau et al., 2020) and uncased BERT with 110M parameters (Devlin et al., 2019), using the setup depicted in Figure 3. We illustrate in the upcoming subsections the different training strategies we adopt in our experiments.

heading: 3.1 XLM-R-BL: Baseline
text: The baseline XLM-R-BL is the pretrained XLMR model without fine-tuning (i.e., it has a randomly initialized classification head). Note that “BL” stands for “baseline.” This baseline will be helpful later on to examine how fine-tuning for a particular language contributes to the performance for other languages.

heading: 3.2 XLM-R-S: Naïve Approach of Shuffling
text: We denote by XLM-R-S the naïve approach of shuffling all languages and randomly sampling during fine-tuning of XLM-R-BL. Here, “S” stands for “shuffling.”

heading: 3.3 XLM-R-S-FT*: Shuffle then Fine-Tune on the Target Language
text: For a fair comparison of the recency effect contribution of the sequential training approach (section 3.4) against the naïve approach (section 3.2), we fine-tune XLM-R-S six times once on each of the six target languages. Here,“FT” denotes “finetuning,” and the asterisk denotes a wild-card. We have six target languages, and thus this means we independently fine-tune XLM-R-S six times, once on each of the six respective target languages, resulting in six distinct models. For example, one of the six models that fall under XLM-R-S-FT* is XLM-R-S-FT-EN, which denotes XLM-R-S finetuned on English.

heading: 3.4 XLM-R-O*: Fine-Tune Sequentially Language by Language
text: Fine-tuning XLM-R-BL sequentially by imposing an ordered structure on our batches based on the language the examples belong to yields XLM-RO*, where “O” denotes “order,” and, again, the asterisk is a wild-card.
We have six languages, and thus the number of possible fine-tuning sequences is 7! = 720 from which we select six:
• O1 =[English, French, German, Italian, Polish, Russian] • O3 =[Italian, Russian, English, Polish, German, French] • O5 =[German, French, Russian, English, Italian, Polish]
The remaining three sequences O2, O4, and O6 are the reversed counterparts of O1, O3, and O5, respectively. Observe that the six sequences were selected so that each of the six languages appears once either at the start or at the end of a sequence.

heading: 3.5 BERT-EN and BERT-TR: Data Augmentation via Translation
text: We investigate the performance of monolingual against multilingual models by fine-tuning BERT on English training data to obtain BERT-EN. We also investigate translation using Google Translate as a means for data augmentation by translating training data from all six languages to English then fine-tuning BERT to obtain BERT-TR where “TR” denotes “translation.” Needless to say, we also translate test data to English before inference. Note that we find that BERT-TR outperforms all other approaches on the dev set, and thus our official submission on the test set is based on BERT-TR.

heading: 3.6 Contribution of Each Language to All Other Languages
text: We study how fine-tuning XLM-R-BL on any single one of the six languages affects the performance of the model on all the other languages, which we did not fine-tune the model on. We use XLM-R-FR, for instance, to denote fine-tuned XLM-R-BL on only French data.

heading: 4 Experimental Setup
text: For all of our experiments, we split the task’s training set using stratified sampling2 to deal with class imbalance. We adopt a 70/30 train/dev split, and we use the official development set as a test set. In sections 3.2, 3.3, and for BERT-TR from section 3.5, we do the split on the combined dataset from all languages, while for the other sections, we do this for each language independently.
2https://github.com/trent-b/ iterative-stratification
We use an NVIDIA Quadro RTX 6000 GPU to fine-tune our models. We use WordPiece tokenizer for BERT and SentencePiece tokenizer for XLMR.3 After tokenization, we truncate the input to 512 tokens. The CLS contextualized embeddings from either BERT or XLM-R are fed to a fully-connected neural network for multilabel classification with a binary cross entropy loss for each of the fourteen labels. We use a learning rate of 1e−5, and a batch size of 16. We report micro-F1 scores which is a suitable metric for examining fine-tuning sequence ordering impact and is also the task’s official metric. We also optimize for the threshold used in the decision functions after the Sigmoids for all models across all experiments. That is, after finetuning, we use the training split to find the optimal threshold in terms of micro-F1 score.
We fine-tune epoch by epoch and converge with early stopping with a patience of three epochs. Here, it is important to explain that for XLM-R-SFT*, we first fine-tune XLM-R-BL on the shuffled dataset and converge with early stopping to obtain XLM-R-S. Next, we fine-tune XLM-R-S on any of the six target languages, say, English, again with early stopping, to obtain XLM-R-S-FT-EN. This process is repeated to obtain five models that cover the five remaining languages.
As for XLM-R-O1, we fine-tune XLM-R-BL on English with early stopping, then take the resulting model and fine-tune on French with early stopping, and so on all the way till Russian to obtain XLM-RO1. In other words, XLM-R-O1 involves six finetuning steps with six early stopping convergences. The same methodology is repeated for all other XLM-R-O* models with the only difference being the different order of languages when fine-tuning.

heading: 5 Results

heading: 5.1 Order Matters
text: Looking at Table 2, we see that XLM-R-S-O* has the best overall performance achieving an average F1 score gain of 2.57% and 1.34% over XLM-R-S and XLM-R-S-FT*, respectively. Note that XLMR-S-FT* outperforms XLM-R-O* only on German. Even though XLM-R-S-FT* exhibits the recency effect by virtue of fine-tuning XLM-R-S on each target language, we can see that the recency effect of XLM-R-O* is stronger. Not only does XLMR-O* yield superior performance, but it also converges 3.63% quicker as shown in Table 3.
3https://huggingface.co/
We also make a striking observation where PMTs exhibit a human-like primacy-recency effect as shown in Table 4 and in Figure 1.

heading: 5.2 Data Augmentation via Translation
text: Recall that training data for three of the test languages is not available (Table 1). For this reason, we explore translation as a form of data augmentation, and report our results in Table 5. We find that this approach yields better performance than PMTs, so the system we adopt for the task submission is BERT-TR. We fine-tune BERT on English
data only to study the effect of data augmentation and notice a ∼5% gain in performance in BERT-TR over BERT-EN.

heading: 5.3 Contribution of Each Language to All Other Languages
text: In Table 6, we see that fine-tuning XLM-R-BL on any of the six languages benefits all other languages. We also observe that French and Russian seem to benefit negligibly more from outof-domain English and Italian fine-tuning, respectively, than from in-domain fine-tuning.

heading: 6 Conclusion and Future Work
text: We are the first to illustrate the significance of finetuning PMTs sequentially language by language. We show that this not only yields sizable performance gains of 2.57% over the naïve approach, but it also converges 3.63% faster when supplemented with recency fine-tuning. Moreover, we are the first to observe a human-like primacy-recency effect in deep learning models in general and PMTs in particular. We also perform other experiments to study data augmentation via translation, and we study how fine-tuning on any single language benefits all other languages.
In the future, we plan to experiment with larger datasets, more languages, multiple runs with different random seeds, different downstream tasks, and more variants of PMTs.
This work has the potential to give us a better understanding of the inner workings of deep learning models by drawing insight from psychology. It also has the potential to improve the way deep learning models in general, and PMTs in particular are pretrained and fine-tuned.
In our work, we looked at only six out of 720 possible sequences of languages for fine-tuning. It would be interesting to investigate other sequences. For example, it is interesting to study keeping languages from the same family closer in the sequence or farther apart. It would also be interesting to frame this research as a CL problem and define a notion of “easy” and “hard” languages.

heading: Acknowledgements
text: The authors would like to thank Timothy Baldwin for his valuable insights and helpful discussion.

heading: Future Work
text: In the future, we plan to experiment with larger datasets, more languages, multiple runs with different random seeds, different downstream tasks, and more variants of PMTs. This work has the potential to give us a better understanding of the inner workings of deep learning models by drawing insight from psychology. It also has the potential to improve the way deep learning models in general, and PMTs in particular are pretrained and fine-tuned. In our work, we looked at only six out of 720 possible sequences of languages for fine-tuning. It would be interesting to investigate other sequences. For example, it is interesting to study keeping languages from the same family closer in the sequence or farther apart. It would also be interesting to frame this research as a CL problem and define a notion of “easy” and “hard” languages