Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning
Year: 2023

Abstract:
Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.

heading: 1 Introduction
text: Standard supervised NLP methods perform well when training on enough data from a uniform distribution. However, they fail to retain knowledge learnt in the past when sudden shifts occur in training data distributions. This effect of dropping performance on data from past distributions is commonly referred to as catastrophic forgetting (McCloskey and Cohen, 1989; de Masson D’Autume et al., 2019; Biesialska et al., 2020a), where stability or preservation of knowledge is traded off for increased plasticity or the ability to acquire new knowledge. To tackle this issue, continual learning (CL) methods were proposed under various settings, such as limited compute or ability to store past data (Lopez-Paz and Ranzato, 2017; de Masson D’Autume et al., 2019). The data shifts commonly studied are obtained by training over a sequence of non-iid partitions (Chaudhry et al.,
∗ These authors contributed equally. † The work was done while at Bloomberg.
2018), different tasks (Jin et al., 2021), or by training on various domains such as in task-oriented dialogue (Madotto et al., 2021), named entity recognition (Monaikul et al., 2021), part-of-speech (Liu et al., 2020), and intent detection (Wu et al., 2021).
Lifelong learning is key to the success of deployed multilingual systems, enabling the system to incorporate annotated data for new languages as they become available without costly retraining and redeployment of the entire system. This sequential availability of data for new languages is a common case of training data shift (see Figure 1 for the task setup). Yet, the effect of catastrophic forgetting was not yet systematically studied for multi-lingual models with multiple diverse languages. M’hamdi et al. (2022) study continual learning in a crosslingual setting limited to just six languages. The cross-lingual abilities of pre-trained models were found to drop when performing fine-tuning for a target language (Liu et al., 2021), although applying continual learning approaches can effectively reduce the magnitude of the effect (Lopez-Paz and Ranzato, 2017).
In this paper, we systematically study the effect of catastrophic forgetting and mitigation strategies in a massively multilingual setting covering up to 51 languages on three different tasks. We start by quantifying the extent to which forgetting happens when languages are presented to the model in sequence, identifying an up to 16% F1 drop compared to training using all the data mixed. Next, we propose LR ADJUST, a simple, yet effective, method
768
to preserve the learned knowledge by adjusting the learning rate over time to alleviate the knowledge overwriting from the new language and preserve the previous learned knowledge. This method is orthogonal to continual learning methods and thus can be handily combined with any of these. We find that across three different CL methods, LR ADJUST helps further reduce the gap between a fully trained model and the CL setup. We conduct analysis on the aspect of cross-lingual transfer in backward and forward directions to measure the influence of the CL on previous tasks and its ability in zero-shot learning respectively. Finally, we conduct analyses on the effects of catastrophic forgetting when first training on multiple languages jointly and when using a curriculum learning approach informed by language similarity.

heading: 2 Massively Multilingual Continual Learning

heading: 2.1 Task Setup
text: We define a curriculum of T tasks as an ordered set of data sets D = {D1, D2, ..., Dt, ..., DT } and model θt, where Dt is the data set with task t. In this case, the task is a distinct language. The weights of model θ are updated continuously θt+1 ← f(θt, Dt) by minimizing the log-likelihood over data set Dt via gradient updates.

heading: 2.2 Inter-task Learning Rate Adjustment
text: We propose LR ADJUST, a simple and effective method to adjust the learning rate when we start training on a new task. Our intuition is that models are susceptible to catastrophic forgetting when we provide a higher learning rate, so the learning rate should be toned down with time to ensure the preservation of the learned knowledge and to reduce the effect of overwriting the weights with the new knowledge. Learning rate adjustments have been studied in the context of incremental learning (Cavalin et al., 2009; Khreich et al., 2012) and for efficient optimization using schedules (Ge et al., 2019). Concretely, the new learning rate is lowered every time as the following: lrt = max(lrmin, lrt−1 ∗ γ), with a weight γ, where γ < 1 and lrmin is the minimum learning rate. The method is detailed in Algorithm 1.

heading: 2.3 Continual Learning Method
text: We experiment with the following continual learning approaches:
Algorithm 1 Inter-task Learning Rate Adjustment (LR ADJUST) Require: An ordered list of tasks D = {D1, D2, ..., Dt, ..., DT } Require: γ: learning rate adjustment coefficient, lrt: learning rate at time t, θ: model weights, lrmin: minimum learning rate
1: Randomly initialize the classifier on θ 2: for all Dt ∈ D do 3: Adjust learning rate to lrt =
max(lrmin, lrt−1 ∗ γ) 4: Compute∇θLDt(fθ) using Dt 5: θt+1 ← θt − lrt∇θLDt(fθ) 6: end for
• Experienced Replay (de Masson D’Autume et al., 2019) uses an episodic memory to store seen training data in memory and retrieve it from memory for fine-tuning. We schedule the replay step to be run every few iterations. During the replay step, we retrieve the data and fine-tune the model using the retrieved data. The number of stored data is constrained to ensure efficient memory use. And, we take a uniform distribution of samples across all labels. • Averaged GEM (A-GEM) (Chaudhry et al., 2018) also utilizes an episodic memoryM and is a more efficient implementation of GEM (LopezPaz and Ranzato, 2017) that computes the gradient constraints and minimizes the loss as follows:
Lt(θt,M) ≤ Lt−1(θt−1,M), where loss Lt is constrained to be lower or equal to the loss Lt−1. • Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) minimizes the following loss:
Lt(θ) = Lt(θ) + ∑
i
λ 2 Fi(θt − θ∗t−1)2,
where Fi is the Fisher information matrix, λ is a coefficient that sets how important the old task is compared to the new one, and θ∗t−1 is the previous learned weights. The Fi is pre-computed after each task is completed and we incorporate the loss to the training on each gradient update.

heading: 3 Experimental Setup

heading: 3.1 Data sets
text: We use a multilingual natural language understanding data set, MASSIVE (FitzGerald et al., 2022)
and a multilingual named entity recognition (NER) data set, WikiAnn (Rahimi et al., 2019). The MASSIVE data set consists of two tasks: intent classification and slot filling with 51 languages. The WikiAnn data set consists of 40 languages. We adopt the data splits from the original papers.

heading: 3.2 Methods
text: Our model architecture is an encoder-only multilingual model XLM-RBASE (Conneau et al., 2020) with a classification layer for each task. All parameters in the model are updated during the training. The full hyper-parameters are listed in Appendix A. The language order for the experiments are listed in Appendix B. We experiment with the following approaches: • MULTI: A single multilingual model is trained on
data mixed from all languages. This represents an upper bound to CL methods, as there are no memory or data sequencing constraints. • MONO: A separate model is trained on all the supervised data for each language and applied to all inputs in that language. • VANILLA: A single model is trained by sequentially presenting data from each language. The language order is selected randomly. • CL Methods: We run REPLAY, A-GEM, and EWC to train a single model on data from each language presented sequentially. • CL Methods + LR ADJUST: We run the CL methods with the learning rate adjustment method described in Section 2.2.

heading: 3.3 Metrics
text: We measure the ability of cross-lingual transfer using CL metrics adapted from Lopez-Paz and Ranzato (2017). We define a matrix R ∈ RT×T , where Ri,j denotes the test score performance of the model on task tj when training the last sample from task ti. We formally define the metrics as:

heading: 3.3.1 Cross-lingual Forward Transfer (CFT)
text: This metric represents the ability to perform zeroshot learning by evaluating on the test data from tasks/languages that are unseen in training. We formally define the metric as:
CFT = 1 T − 1 T−1∑
i=1
X̄i,
X̄i = 1 T − i T∑
j=i+1
Ri,j ,
where X̄i is the average performance of the languages that will be seen in the future (t>i).

heading: 3.3.2 Cross-lingual Backward Transfer (CBT)
text: This metric measures the influence of learning a task ti on the performance of the previous tasks. We formally define the metric as the following:
CBT = 1 T − 1 T−1∑
i=1
RT−1,i −Ri,i.
CBT practically measures the effect of catastrophic forgetting of past tasks after adding a new task to the model.

heading: 4 Results and Analysis
text: Figures 2, 3 and 4 show performance numbers across the languages seen to that point in train-
ing for the three data sets. Each point on the graph shows the average and standard deviation of the F1 scores, obtained over 5 runs with different seeds. For the CL experiments, the seed also controls the order of languages in training. This can lead to higher variance across runs – compared to mixing the same data across runs – because the forgetting effect depends on the language order. Table 1 shows the forward and backward transfer for the three data sets.
Results show the following: Catastrophic forgetting is present in multilingual continual learning, with performance dropping quickly when training on languages sequentially (VANILLA). This method converges to between 4 – 17 F1 lower than mixing the same data and training a full model every time (MULTI). We also see that this effect is present even when the performance of monolingual models is close to that of multilingual models, as in the case of WikiAnn. Training a full model on all languages (MULTI) always performs best, outperforming training on one language at a time (MONO) sometimes substantially (MASSIVE-Slot - 10 F1, MASSIVE-Intent - 4 F1), highlighting the importance of cross-lingual transfer and preserving information from past seen languages in a continual learning setup.
Continual learning methods generally help dampen the impact of forgetting. For example, in WikiAnn, the REPLAY and A-GEM CL methods reduce the backward transfer from 16.90 to 11.87 and 10.75 respectively, albeit EWC does not substantially improve the performance relative to the VANILLA
method. LR ADJUST, the learning rate adjustment scheme, further reduces the gap to the multi-task model significantly and consistently across languages when combined with any of the CL methods. For example, on the WikiAnn dataset, the backward transfer is reduced from 16.90 to just 3.59 and 3.79 for the A-GEM and REPLAY methods respectively, making multilingual CL feasible. Further, we see that using CL methods alone results in a continuous drop in performance as more languages are added, while adding LR ADJUST stabilizes average performance after the first few languages, resulting in a flatter curve.
Finally, we see that the patterns of improvement hold when studying cross-lingual forward transfer, which quantifies the zero-shot model performance on languages unseen in training. The continual learning approaches improve over sequential training (e.g. +4.45 on WikiAnn) and using LR ADJUST in addition further boosts performance (e.g. +9.25 for VANILLA, +5.83 for REPLAY on WikiAnn). This shows that the resulting models were able to retain essential and generalizable information for the task that is more universal across all languages.

heading: 4.1 Multi-task Training vs. Catastrophic Forgetting
text: We conduct additional experiments to understand whether initially training on multiple languages at once can reduce the severity of catastrophic forgetting in the CL setup when new languages are added. We run a multi-task training on the first k languages, where k is 10 or 30, and then, we run the remaining languages sequentially on the WikiAnn data set. As shown in Figure 5, the model is more
robust to forgetting when it is exposed to multi-task training with more languages with higher final average scores at the final task, but the graphs shows the performance still drops dramatically after being exposed to the first new language fed sequentially.

heading: 4.2 The Role of Language Order
text: To investigate the role of the language order on CL, we decide to reorder the language list by using heuristics. We start with two languages from the same family, as listed in Ethnologue (Eberhard and Gary, 2019), and add all languages from the same family one by one, then switch to a new language family and continue the same process. We conjecture that seeing a similar language at an interval will allow a more effective cross-lingual transfer. Figure 5 (LANGUAGE ORDER) displays the results, which indicate that performance does not improve after we manually select the languages, and its performance is similar to random ordering (VANILLA).

heading: 5 Related Work
text: CL aims to learn effectively over iterations by leveraging information from previously learned tasks (McCloskey and Cohen, 1989). CL has been applied towards mitigating catastrophic forgetting in many computer vision tasks (Lopez-Paz and Ranzato, 2017). Broadly speaking, CL methods can be classified (Biesialska et al., 2020b) into rehearsal (re-using training examples from prior task) (Rolnick et al., 2019; de Masson D’Autume et al., 2019), regularization (guide consolidation via additional loss) (Kirkpatrick et al., 2017), memory (Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2018), and architectural methods (using taskspecific parameters) (Rusu et al., 2016; Madotto et al., 2021).

heading: 6 Conclusion
text: We present the first study of catastrophic forgetting in a massively multilingual setting involving up to 51 languages on named entity recognition and natural language understanding tasks. We investigate continual learning methods and present a learning rate scheduling method that is simple yet effective in reducing the effects of catastrophic forgetting. Furthermore, we show that this method is effective across multiple continual learning methods. Finally, we provide analysis and further insights into the dynamics of catastrophic forgetting.

heading: Acknowledgments
text: We are grateful to Abhinav Singh and Shuyi Wang for feedback on a draft of this manuscript.
Limitations
The experiment in this paper is limited to three common CL methods: Replay, A-GEM, and EWC. Due to compute resources, we experiment with XLMRBASE and do not compare with other variants or larger base models. We acknowledge that the MASSIVE data set varies in coverage across language groups and evaluation could over-represent certain linguistic families (Pikuliak and Simko, 2022).
Ethics Statement
In our experiments, we use publicly available data sets with permissive licenses for research experiments. We do not release new data or annotations as part of this work. There are no potential risks.

heading: A Hyper-parameters
text: In all experiments, we run with five different seeds {42, 52, 62, 72, 82} using a V100 32GB GPU and each run takes up to a week to finish.
A.1 WikiAnn Table 2 shows the hyper-parameters used in the experiments using WikiAnn dataset.
A.2 MASSIVE-Slot Table 3 shows the hyper-parameters used in the experiments using MASSIVE-Slot dataset.
A.3 MASSIVE-Intent Additionally, we run the CL setup on MASSIVEIntent dataset and the results are shown in Figure 4. Table 4 shows the hyper-parameters used in the experiments using MASSIVE-Intent dataset.

heading: B Language Order
text: We randomly shuffle the language order for each seed. Tables 5 and 6 show the language order we use in the experiments for WikiANN and MASSIVE datasets, respectively.

heading: C Geographical Information of Languages
text: Table 7 shows all languages’ language families and subgroups on NusaX and MASSIVE datasets.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Page 5. Section Limitations
A2. Did you discuss any potential risks of your work? Not applicable. No potential risks
3 A3. Do the abstract and introduction summarize the paper’s main claims? Yes, on the first page (abstract and section 1 introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We put the computing infrastructure and GPU hours on Appendix A.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3 Experimental Setup and Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Figure 2 and Figure 3, we provide the average and std
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We formally define the metric as:
CFT = 1 T − 1 T−1∑
i=1
X̄i,
X̄i = 1 T − i T∑
j=i+1
Ri,j ,
where X̄i is the average performance of the languages that will be seen in the future (t>i)