Venue: NAACL 2024

# Leitner-Guided Memory Replay for Cross-lingual Continual Learning

Meryem M'hamdi

Microsoft

meryemphhamdi@microsoft.com &Jonathan May

Information Sciences Institute

University of Southern California

jonmay@isi.edu

Work was conducted while the first author was a Ph.D. student at the University of Southern California.

###### Abstract

Cross-lingual continual learning aims to continuously fine-tune a downstream model on emerging data from new languages. One major challenge in cross-lingual continual learning is catastrophic forgetting: a stability-plasticity dilemma, where performance on previously seen languages decreases as the model learns to transfer to new languages. Experience replay, which revisits data from a fixed-size memory of old languages while training on new ones, is among the most successful approaches for solving this dilemma. Faced with the challenge of dynamically storing the memory with high-quality examples while complying with its fixed size limitations, we consider Leitner queuing, a human-inspired spaced-repetition technique, to determine what should be relayed at each phase of learning. Via a controlled set of quantitative and qualitative analyses across different memory strategies, we show that, just like humans, carefully picking informative examples to be prioritized in cross-lingual memory replay helps tame the stability-plasticity dilemma. Compared to vanilla and strong memory replay baselines, our Leitner-guided approach significantly and consistently decreases forgetting while maintaining accuracy across natural language understanding tasks, language orders, and languages.

## 1 Introduction

Cross-lingual continual learning is a machine learning paradigm aimed at continually adapting a downstream model to datastreams drawn from different languages [20]. Naive approaches to cross-lingual continual learning involve training a new model from scratch each time a new language is available or training jointly over all languages, which can be inefficient and even inaccessible. Faced with an overwhelming stream of languages, modelers turn to continual learning techniques to adapt models such that maximal learning from data is achieved when available data is temporally limited. The consequence of a finite data buffer limitation on a potentially infinite data source is _catastrorophic forgetting_[37]. Catastrophic forgetting exemplifies the stability-plasticity dilemma [45, 6, 8]: It is inherently hard to preserve the previously acquired knowledge (stability) while learning novel information (plasticity).

Various continual learning approaches have proposed to mitigate catastrophic forgetting by either restricting entire sets of parameters from changing [9, 10, 11, 12], designing language-specific model components [11, 20], or replaying a fixed buffer memory from previously seen languages [21, 22]. m'hamdi et al. shin2020learning show that memory-based approaches are more robust than other approaches in taming the stability-plasticity dilemma. Moreover, they are more scalable than other approaches, such as model expansion, which grows in complexity as a function of the underlying downstream architecture.

Experience replay (ER) [2] is a cognitively inspired memory-based approach that reinforces previously seen experiences similar to the process of memory consolidation in biological systems [14]. As more languages are incorporated into the datastream, fitting examples from new languages into a fixed-size memory buffer becomes more challenging. This invites a critical question: How can we dynamically come up with informative memory examples to keep for each language?

In this paper, we propose a human-inspired approach for learning what to replay at each phase of cross-lingual continual learning. We hypothesize

that in such a setup at the beginning, most data is difficult but as training progresses some data becomes well-learned and informative. We surmise that reducing the forgetting of previously learned examples requires using a strategy of alternately learning new difficult examples along with reinforcement of well-learned examples. To design cross-lingual memory, we leverage Leitner queues, a cognitive technique that has been used for strategically planning what to review in humans (Leitner, 1974; Reddy et al., 2016) and for determining informative and spurious data in self-training non-continual learning applications Amiri et al. (2018); Amiri (2019). Our Leitner-guided memory sampling policy is a dynamic language-agnostic skill rating system which selects candidates for inclusion into memory according to how well they are learned (Figure 1). We analyze memory design attributes that contribute to reducing cross-lingual continual learning forgetting and evaluate on typologically diverse benchmarks ranging in difficulty.1

Footnote 1: Our code is available at [https://github.com/meryemhhandi1/x-continuous-learning/tree/main/humanlearn](https://github.com/meryemhhandi1/x-continuous-learning/tree/main/humanlearn).

We summarize our contributions as follows:

1. We are the first to formalize a human-inspired solution based on Leitner queues to guide cross-lingual memory replay (SS2.3).
2. We show that our Leitner-inspired approach for selecting memory replay items reduces forgetting without sacrificing transfer learning gains (SS4.1).
3. We provide a fine-grained analysis over different language orders and languages showing that our approach is consistently and robustly beneficial (SS4.2).
4. We provide a qualitative analysis that investigates the usefulness of data as a function of its learnability (SS4.3).

## 2 Methodology

In this section, we start by describing our ER approach adapted to cross-lingual continual learning (SS2.1). Then, we explain the mechanism for determining skill ratings based on Leitner queues (SS2.2). After that, we explain how we use this Leitner-based skill rating system to guide memory storage and update in cross-lingual ER (SS2.3).

### Cross-lingual Experience Replay

We follow the same setup for cross-lingual continual learning and ER defined by M'hamdi et al. (2023). The general continual learning paradigm assumes access is limited to a portion of the data at any phase. The cross-lingual continual learning process consists of sequentially fine-tuning a model on a cross-lingual datastream in multiple phases. A cross-lingual datastream \({\cal D}_{1\cdots N}\) is a set of N distinct labeled datasets sampled from different languages one at a time. Each dataset \({\cal D}_{i}\) is drawn from a

Figure 1: An overview of Leitner-guided memory replay for multi-phase cross-lingual continual learning: On top of a cross-lingual datastream, we build a skill rating system to continually guide the memory population and update. Skill ratings are scores from \(1\) to \(5\) obtained from Leitner queues; a higher score reflects greater learnability. At the end of each phase, the skill ratings on the main data items from the phase language are used to choose what goes in the memory, and the skill ratings of data items already in the memory are re-evaluated to determine if they can remain. Our approach is language-agnostic in the sense that it uses Leitner queues to determine which examples to keep in the memory at the end of each phase irrespective of the language.

 

single distinct language \({\mathbf{\ell}_{i}}\in\mathscr{L}=\{{\mathbf{\ell}_{1},\mathbf{\ell}_{2}\cdots\mathbf{\ell}_{N}}\}\). Each phase \(\mathcal{P}_{i}\in\mathcal{P}_{1\cdots N}\) is a stage in cross-lingual continual learning where the model gets fine-tuned on a dataset \(\mathcal{D}_{i}\) for a number of epochs. The ER approach is implemented as follows: At the end of each phase (except the last one) \(\mathcal{P}_{i}\in\mathcal{P}_{1\cdots N-1}\), we choose some data from \(\mathcal{D}_{i}\) to add to a memory buffer \(\mathcal{M}\) of fixed size \(|\mathcal{M}|\). In later phases \(\mathcal{P}_{j}\) after \(\mathcal{P}_{i}\), we replay from \(\mathcal{M}\), which contains memory data drawn from \(\mathcal{D}_{<j}\) interleaved with the main loss on data drawn from \(\mathcal{P}_{j}\).

### Leitner-based Skill Rating System

We draw inspiration from Leitner queues [3], a method of prioritization originally conceived of as a strategy for human memorization and later used in machine learning applications [5]. The key prioritization insight we leverage is that of _demonstrated mastery_. That is, items in a (training) data set may be rated by the degree to which they have been mastered by the learner. We instantiate this by associating a rating \(r\) to each training data item \(d\), and changing \(r(d)\) based on a model \(m\)'s ability to correctly classify \(d\) during training. Let \([s,e]\) be the acceptable rating range, let \(r_{m^{\prime}}(d)\) be the rating for \(d\) according to some previous model \(m^{\prime}\), and let \(\phi_{m}(d)\in\{-1,1\}\) indicate that model \(m\) classified \(d\) {incorrectly, correctly}, respectively. Then

\[r_{m}(d)=\max(\min(r_{m^{\prime}}(d)+\phi_{m}(d),e),s)\]

Thus, \(r\) is raised when \(d\) is correctly classified and lowered when it is misclassified, subject to the acceptable range. In this work, we set \([s,e]=[1,5]\), following established practice [4, 6].

### Leitner-Guided Cross-lingual Experience Replay (_LER_)

We explore the use of \(r(d)\) to determine whether or not to include \(d\) in \(\mathcal{M}\). At the start of phase \(\mathcal{P}_{i}\), by convention, for all \(d\in\mathcal{D}_{i}\cup\mathcal{M}\), we set \(r_{0}(d)\), the initial rating, to \(s\). At the end of each epoch within the phase, we update \(r\) for each data item in \(\mathcal{D}_{i}\) and \(\mathcal{M}\) according to the model \(m\) at that point in training. At the end of \(\mathcal{P}_{i}\), we use \(r\) values to form the new \(\mathcal{M}\), selecting \(\frac{|\mathcal{M}|}{i}\) items from \(\mathcal{D}_{i}\) and \(|\mathcal{M}|-\frac{|\mathcal{M}|}{i}\) items from the current \(\mathcal{M}\) according to one of two strategies:

* _LER (Easy)_: Highest-rated items are prioritized.
* _LER (Hard)_: Lowest-rated items are prioritized.

Our approach, which selects data from \(\mathcal{D}_{i}\) inversely proportional to \(i\), enables the fixed and limited \(\mathcal{M}\) to contain an even distribution of samples from all \(\mathcal{D}_{<i}\) thus seen, militated by the relative learning difficulty of different phase datasets. Our approach is language-agnostic as we continue to rank the examples to keep in the memory regardless of their language and based solely on their degree of learnability.

## 3 Experimental Setup

We start by presenting the different baselines and model variants used to compare between different experimental scenarios (SS3.1). We then describe the benchmark datasets and their base models (SS3.2) along with the multilingual datasreams (SS3.4) that we focus on in this evaluation. More implementation details, such as the hyperparameters, number of parameters used, and runtime for different models can be found in Appendix A.

### Baselines & Model Variants

BaselinesBefore delving into different variants of Leitner-guided memory replay, we consider the following baselines:

* _No ER_. This is our lower-bound naive sequential fine-tuning baseline. This sequentially fine-tunes on datasets sampled from one language at a time \(\mathcal{D}_{i}\in\mathcal{D}_{1\cdots N}\) without using any experience replay.
* _Balanced_. This is an experience replay approach adapted from Lopez-Paz and Ranzato (2017), which allocates equally sized buffers balanced across language. At the end of each phase \(\mathcal{P}_{i}\), \(|\mathcal{M}|/(N-1)\) examples are randomly picked from \(\mathcal{D}_{i}\) and added to \(\mathcal{M}\).
* _Random_. This is a more realistic experience replay approach, adapted from Riemer et al. (2019), which randomly samples and updates \(|\mathcal{M}|\) from \(\mathcal{D}_{<i}\) at the end of each phase \(\mathcal{P}_{i}\). Other techniques have been proposed to produce memory exemplars such as mean of features [8], K-Means clustering [1], and prototypical networks [10]. However, we don't explore those techniques since they either under-perform _Random_ (reservoir sampling) [1] or their contribution severely depends on the order of training sets experimented with 

Model VariantsWe design the following model variants on top of _LER_. The research question we analyze here is: Does dynamically prioritizing easy elements help mitigate forgetting more than hard elements or vice versa? Our analysis evaluates the aggregated effectiveness of different strategies used for memory construction. This consists of _LER (Easy)_ and _LER (Hard)_, which use easy and hard examples, in terms of their learnability, to fill and update the memory, respectively.

### Benchmarks & Base Models

We conduct experiments on two datasets commonly used in natural language understanding literature, covering different typologically diverse languages and requiring different levels of reasoning: multilingual task-oriented dialog (_MTOD_) and multilingual question answering (_MQA_).

_MTOD_This is a multilingual goal-oriented system focusing on the natural language understanding module. This module consists of two subtasks, namely intent detection and slot filling. For _MTOD_ evaluation, we use two multilingual task-oriented dialog datasets: _MTOP_Li et al. (2021) and _MultiATIS++_Xu et al. (2020). _MultiATIS++_ covers \(18\) intents and \(84\) slots on average per language from one domain. _MTOP_ covers \(117\) intents and \(78\) slots from \(11\) domains. We choose _MTOP_ and _MultiATIS++_ since they are among the large-scale datasets available for task-oriented dialog covering typologically diverse languages. We use the same architecture as in Castellucci et al. (2019) to jointly learn intent classification and slot-filling subtasks. M-BERT Devlin et al. (2019) is used to encode each input sentence. On top of the \([CLS]\) representation of the sentence, we use a linear layer plus Softmax to predict its intent class. We use a sequence labeling layer in the form of a linear layer plus CRF Lafferty et al. (2001) to predict slot labels in BIO annotation. We optimize jointly over the sum of intent and slot losses. For evaluation, we use accuracy and F1 scores to evaluate intent classification and slot filling, respectively.

_Mqa_This is a multilingual span-based question-answering task that extracts the answer token span to a question given a defined context. To ensure a challenging and trustworthy evaluation for _MQA_, we choose TyDiQA Clark et al. (2020), which is a translation-free realistic information-seeking benchmark. We follow the same pre-processing and architecture as in Hu et al. (2020). Specifically, we concatenate the input question (after prepending it with a \([CLS]\) token) and the context as a single packed sequence separated by a \([SEP]\) token and feed that to M-BERT. Then, the embeddings of the context are fed to a linear layer plus Softmax to compute the probability that each token in the context is the start or end token of the answer span. We optimize for the joint loss over the start and end tokens predictions. Complying with Hu et al. (2020) evaluation, we use F1-score macro-averaged over examples.

Table 1 shows the statistics per language and split for _MTOP_, _MultiATIS++_, and _TyDiQA_ datasets.

### Evaluation Metrics

Our primary analytical tool is _forgetting_, which measures the degree to which a learned skill is lost when a model is trained on out-of-language data. Lower forgetting is better, while negative forgetting indicates the model has improved due to out-of-language training. We also show _final performance_, which is simply a metric's value after all phases of continual learning. We follow the same formulation of cross-lingual continual learning evaluation protocols as M'hamdi et al. (2023). Let \(R\) be some success metric for evaluating a task and \(R_{i,\leq j}\) be the evaluation on the test set for language \(\mathbf{\ell}_{i}\) fine-tuning on \(\mathcal{D}_{\leq j}\). For a more succinct analysis of interference versus transfer tradeoff, we focus on the following two metrics:

* **Forgetting (F \(\downarrow\)).** We compute forgetting averaged _over_\(\mathcal{D}_{2\cdots\leq N}\) as follows: \[F =\frac{1}{N-1}\sum_{j=2}^{N}F_{\leq j},\] (1) \[F_{\leq j} =\frac{1}{j-1}\sum_{i=1}^{j-1}F_{i,\leq j},\]

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \multicolumn{1}{p{56.9pt}|}{} \\
**Dataset** & **Language** & **Train** & **Dev** & **Test** \\ \hline  & English & \(15,667\) & \(2235\) & \(4386\) \\ _MTOP_ & German & \(13,424\) & \(1815\) & \(3549\) \\  & Hindi & \(11,330\) & \(2012\) & \(2789\) \\  & Thai & \(10,759\) & \(1671\) & \(2765\) \\ \hline  & English & \(4488\) & \(490\) & \(893\) \\ _MultiATS++_ & French & \(4488\) & \(490\) & \(893\) \\  & Chinese & \(4488\) & \(490\) & \(893\) \\  & Turkish & \(578\) & \(60\) & \(715\) \\ \hline  & Indonesian & \(5131\) & \(571\) & \(565\) \\ \multirow{2}{*}{_TyDiQA_} & Russian & \(5841\) & \(649\) & \(812\) \\  & Swahili & \(2479\) & \(276\) & \(499\) \\  & Telugu & \(5006\) & \(557\) & \(669\) \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of _MTOP_, _MultiATIS++_, and _TyDiQA_ per language and split.

 

where \(F_{\leq j}\) is the average forgetting that resulted at the point of training on \(\mathcal{D}_{j}\). \(F_{i,\leq j}=\max_{k\in[1,j-1]}R_{i,\leq k}-R_{i,\leq j}\). \(F_{i,\leq j}\) is the degree to which performance on \(\mathcal{D}_{i}\) has degraded by continuing to train on \(\mathcal{D}_{\leq j}\) instead of stopping before including \(\mathcal{D}_{j}\).
* **Final performance (FP \(\uparrow\)).** This is the final performance at the last phase \(\mathcal{P}_{N}\) averaged over all datasets \(\mathcal{D}_{\leq N}\) : \[FP=\frac{1}{N}\sum_{i=1}^{N}R_{i,\leq N}.\] (2)

### Datastreams

We design a balanced set of distinct language permutations, following the cross-lingual continual learning evaluation paradigm established by M'hamdi et al. (2023). Formally, for a given set of \(N=4\) languages, we sample a subset of \(N\) language permutations \(P\subset\mathfrak{S}(\mathscr{L})\) where each language appears exactly once in each permutation. Table 2 shows the language permutations we consider for different downstream benchmarks. For example, the first permutation in _MTOP_ is chosen such that the languages are ordered from high-resource to low-resource languages. The last permutation uses the opposite order. Then, we curate the two other permutations such that each language appears exactly once in each place across all permutations as if we are solving a 4x4 Sudoku. This ensures that our evaluation metrics are well-balanced when the average is taken over the 4 permutations taking into account the forgetting on different languages with equal contribution.

## 4 Results & Analysis

In this section, we provide an extensive analysis to demonstrate the effectiveness of our Leitner-guided cross-lingual experience replay approach. We present both a summary of the test performance based on the best epoch given Dev data split performance and over each epoch throughout different training stages (SS4.1). Then, we present a more fine-grained analysis, shedding light on the language orders and languages for which our Leitner-based skill rating system is particularly helpful (SS4.2). Last but not least, we present a qualitative analysis of different categories of skill ratings and what makes ruling out hard examples useful (SS4.3).

### Average Performance

In Table 3, we compare between different Leitenre-guided memory selection strategies and baselines for _MTOP_, _MultiATIS++_, and _TyDiQA_ benchmarks in terms of their forgetting. We start by showing their forgetting on the test data averaged over different language orders based on the best-performing model on the Dev data split. Compared to _No ER_ baseline, all _ER_ approaches: _Balanced_, _Random_, and _LER_ variants are beneficial in reducing forgetting, irrespective of the strategy followed in memory storage and update. It is clear that the forgetting gap between _No ER_ and _ER_ approaches is more pronounced for _MTOP_ and _MultiATIS++_ tasks than it is for _TyDiQA_. We conjecture that this is due to the formulation of _TyDiQA_ as a span-based question-answering task. The latter employs a simple token classification model, which is less challenging than joint optimization over classification and sequence modeling objectives in _MTOD_. The gains are even more pronounced for _MTOP_, whose ontology covers more domains, intents, and slots than that of single-domain _MultiATIS++_. Among _MTOP_ subtasks, slot filling has a higher overall forgetting than intent detection. The implication of all of these findings is that forgetting is more pronounced, and our technique more crucial, when tasks are more difficult.

By keeping a balanced memory across languages, _Balanced_ could have the benefit of making sure to revisit all languages assuming knowledge of the total number of languages involved in the continual learning. However, using a balanced memory across languages _Balanced_ doesn't lead to lower forgetting than picking a random memory across languages _Random_. This could be because _Bal

\begin{table}
\begin{tabular}{l|c|l} \hline \multicolumn{1}{l|}{**Dataset**} & \# & **Order** \\ \hline  & 1 & English\(\rightarrow\)German\(\rightarrow\)Hindí\(\rightarrow\)Thai \\ _MTOP_ & 2 & German\(\rightarrow\)English\(\rightarrow\)Thai\(\rightarrow\)Hiñoli \\  & 3 & Hinid\(\rightarrow\)Thai\(\rightarrow\)Eighash\(\rightarrow\)German \\  & 4 & Thai\(\rightarrow\)Hindí\(\rightarrow\)German\(\rightarrow\)English \\ \hline  & 1 & English\(\rightarrow\)French\(\rightarrow\)Turkish\(\rightarrow\)Chinese \\ _MultiAITIS++_ & 2 & French\(\rightarrow\)English\(\rightarrow\)Chinese\(\rightarrow\)Turkish \\  & 3 & Turkish\(\rightarrow\)Chinese\(\rightarrow\)English \\  & 4 & Chinese\(\rightarrow\)Turkish\(\rightarrow\)French\(\rightarrow\)English \\ \hline  & 1 & Russian\(\rightarrow\)Indoonesian\(\rightarrow\)Tehug\(\rightarrow\)5wahili \\ _TyDiQA_ & 2 & Indonesian\(\rightarrow\)Russian\(\rightarrow\)Swahili\(\rightarrow\)Telugu \\  & 3 & Telbug\(\rightarrow\)Swahili\(\rightarrow\)Russian\(\rightarrow\)Indonesian \\  & 4 & Swahili\(\rightarrow\)Telugu\(\rightarrow\)Indonesian\(\rightarrow\)Russian \\ \hline \end{tabular}
\end{table}
Table 2: Language permutations for _MTOP_, _MultiATIS++_, and _TyDiQA_ 

anced_ picks a balanced amount of examples per language, exposing the model to less diversity compared to _Random_. This could also show the need to continuously update the diversity of memory to make room for higher-quality examples in continual learning. _LER (Easy)_ stands out as one of the most successful strategies in reducing forgetting and beating both experience replay baselines. _LER (Easy)_ reaches the lowest forgetting with reductions of \(1.76\), \(0.64\), and \(0.46\) in forgetting of F1 score for slot filling compared to _LER (Hard)_, _Balanced_, and _Random_ respectively. Results on _MultiATIS++_ and to some degree _TyDiQA_ confirm the consistent superiority of _LER (Easy)_ and the inferiority of the _LER (Hard)_ approach.

For the remaining analysis, we focus on _MTOP_, shedding more light on the added value of _LER_ approaches compared to the best-performing _ER_ baseline, _Random_. Figures 2a and 2b show the learning curves of different models on slot filling in terms of forgetting and final performance, respectively. Throughout the training, _LER (Easy)_ is consistently more effective than _Random_ and _LER (Hard)_ in minimizing forgetting while improving final performance, thus taming the interference versus transfer tradeoff. _LER (Easy)_ can converge and stabilize at a low forgetting score earlier in training. On the other hand, the _LER (Hard)_ strategy exacerbates the forgetting problem as training proceeds. This shows that replaying easy examples is consistently more effective than revisiting hard ones that the model is struggling with. Our Leitner-based skill rating system provides a dynamic measure that keeps selecting pertinent instances as language exemplars in constructing the memory replay.

### Fine-grained Language Analysis

Figures 3 and 4 show a fine-grained analysis of forgetting between different models across different language orders and languages, respectively.2 For each language order and language, we report Test results for the best-performing model based on Dev data split. Overall, we observe that _LER (Easy)_ consistently outperforms _LER (Hard)_ and _Random_ across different language orders and languages. Certain language orders such as Thai\(\rightarrow\)Hindi\(\rightarrow\)German\(\rightarrow\)English [(4)] and Hindi\(\rightarrow\)Thai\(\rightarrow\)English\(\rightarrow\)German [(3)] have more forgetting than others. The languages that benefit the most compared to _Random_ are Hindi and

Figure 2: Average forgetting and final performance of slot filling for different model variants compared to the _Random_ baseline averaged over different language orders. The lower the forgetting and the higher the final performance the better.

 

German whereas the gains for Thai and English are more minimal. _LER (Easy)_ manages to bridge that gap in forgetting, keeping it within a low range.

### Discussion

In this part, we conduct a qualitative analysis to complement our conclusion from our quantitative analysis that choosing training data for the memory that is easy to learn is more beneficial than choosing data that is not easily learned. To dig deeper into why ruling out harder examples from the memory is beneficial, we look more closely at the characteristics of those hard cases among training data. We define an _intractable_ example as an example whose skill rating never gets promoted and stays throughout training. At the other end of the spectrum is a _confident_ example whose skill rating converges to \(5\) and never gets demoted after that.

In Figure 5, we report percentages of intractable and confident training data in _MTOP_ for each language, averaged over all phases and language orders. We notice that for each language, 70% or more examples are confident. Thus, the _Random_ approach to memory selection is unlikely to differ all that much from the _LER (Easy)_ approach, at least in intent detection for _MTOP_. For other tasks with lower rates of easy examples, it might not be straightforward to pick easy examples with a random approach. We also observe a trend where the more high-resource the language is, the less likely its examples are intractable and the more likely its examples are confident. Thai, which has the highest percentage of intractable examples, is the most low-resource in _MTOP_. This explains why _LER (Easy)_ is much more beneficial than _Random_ for Thai and Hindi compared to English and German for intent classification in Figure 9 (Appendix B).

As an exemplar, we focus now on English data, specifically concentrating on training data analysis from the end of the first phase in language order English\(\rightarrow\)German\(\rightarrow\)Hindi\(\rightarrow\)Thai. To understand what makes an example particularly intractable, we define the following categories:

* **Low-resource (LR)**: A training instance is considered low-resource if the number of training instances per its intent label is below \(10\). For English, there are \(137\) training instances per intent on average. This makes low-resource labels fall within the \(25\)% percentile.
* **Difficult to disambiguate (DD)**: This is the case if the true class is among the most similar to the predicted class. We let the \([CLS]\) token representation of a sentence be its representation. We then compute the centroid of the sentence representations per class label. For each label, we determine its most similar predicted classes based on the \(5\) nearest neighbors.
* **Poorly-defined (PD)**: Unlike low-resourced and difficult to disambiguate examples which are au

Figure 4: Fine-grained analysis of forgetting of slot filling over different languages. Best (lowest) results for each language are highlighted in **bold**.

Figure 5: Percentages of examples that never get promoted past skill rating \(1\) (Skill never promoted) and those that converge to the maximum skill rating \(5\) (Converged to max skill) per language averaged over different language orders. More statistics on the proportion of different languages in the memory at the end of training using different approaches are shown in Appendix C.

Figure 3: Fine-grained analysis of forgetting of slot filling over different language orders as defined in Table 2. Best (lowest) results for each language order are highlighted in **bold**.

 

tomatically determined by their labels, we inspect here case by case for poorly-defined sentences. We define a poorly-defined example as any sentence that doesn't make sense to be attributed to a certain label. This could be due to a mismatch or lack of commonsense in the way the ontology was defined for certain labels.

We show in Figure 6 some statistics of different categories of intractable examples. Most intractable examples are either DD, LR, or both. Inspecting confident examples reveals that no LR or DD examples are encountered among them. This demonstrates that our Leitner-guided approach _LER (Easy)_ can detect such hard categories and rule them out. By imposing a more fine-grained skill rating system, our Leitner-guided memory replay approach provides a more confident approach to determine which labels the model is struggling with more than relying on prediction loss Amiri et al. (2018). The skill rating system adds information that prediction loss alone does not. In fact, only \(27\)% of the English examples that have skill ratings between \(2\) and \(4\) (neither intractable nor highly confident) are wrongly predicted at the end of the first phase. Those unstable examples are part of the selection of _LER (Hard)_, so not prioritizing such examples is beneficial.

In Table 4, we provide some examples of different categories of wrongly predicted labels. We observe inconsistencies in those examples. Those that are specifically DD are so close to being picked as representative examples of certain classes, which can only confuse the learner. For example, while "Pause the current timer and delete." is supposed to be classified as _timer:PAUSE_TIMER_, this label is far from being comprehensively descriptive of the sentence intent. Its predicted label _timer:DELETE_TIMER_ is not wrong either, as it detects the intent to delete, which is the second part of the example. We suspect that reinforcing the learning using difficult cases can only mislead the learner. In Figure 7, we show a t-SNE projection of the centroids of different intent label representations. We highlight in that figure the most common DD labels whose representations are indistinguishable in the vector space. Some of those labels like GET_STORIES_NEWS and GET_DETAILS_NEWS are to the human eye also DD, which could be an artifact of how the intent ontology was defined. Our Leitner-guided strategy _LER (Easy)_ rules them out, favoring examples the learner is more confident about with class labels that correspond to more clearly separable representations.

## 5 Related Work

Continual learning work inspired by human-like learning falls into several categories. Notable categories include spaced-repetition Smolen et al. (2016); Amiri et al. (2019); Feng et al. (2019); Klasson et al. (2023), mechanisms of sleep Ball et al. (2020); Mallya and Lazebnik (2018); Schwarz et al. (2018), and reactivation of memories Hayes et al. (2020); van de Ven et al. (2020). Leitner queues, one of the most famous spaced repetition techniques, started garnering attention for machine learning recently. However,

Figure 6: Distribution of different categories of intractable examples in the English data.

 

most of the work is focused on scheduling when to review data in non-continual learning setups. Amiri et al. (2017) show the sample efficiency of a human-inspired memory model to determine when to review each item as a function of the difficulty of the item and the strength of the network. Klasson et al. (2023) propose a Monte Carlo tree search approach for memory replay. More work Amiri et al. (2018, 2019) demonstrates the effectiveness of Leitner queues at determining spurious data and confident labels for self-training applications. Our work, by contrast, is the first to test for the effectiveness of Leitner queue-based skill ratings in mitigating forgetting in cross-lingual continual learning. Some studies, such as He et al. (2021), have looked into analyzing the forgetting problem from the perspective of knowledge transfer and context sensitivity from pre-training to fine-tuning. Fine-grained analyses such as Luo et al. (2023) look at different types of general knowledge in large language models that are saliently forgotten and at the role that model scale plays in forgetting. To the best of our knowledge, no previous work has analyzed forgetting in the cross-lingual continual learning context.

## 6 Conclusion

In this paper, we formulate a human-inspired experience replay approach specifically for cross-lingual continual learning. We propose a Leitner-based skill rating system to dynamically populate and update the memory with high-quality items. Our approach can deal with the stability-plasticity dilemma better than random selection, especially for complex tasks and consistently over languages and language orders. The implications of this analysis include a recipe for how to incorporate aspects of human learning in the design of memory replay in cross-lingual continual learning.

### Limitations

Other Variants of Leitner QueuesIn this paper, we have focused on Leitner queues as an approach to guide the process of memory storage and update. In future work, other variants of Leitner queues or other approaches based on human learning theories could be explored. For example, more fine-grained approaches based on theories of how languages get forgotten to model the retention curve as a function of the task difficulty, review periods, and strength of the model could be investigated. This could help us understand how the process of forgetting works and when to schedule revisions accordingly to circumvent that.

Fine-grained Analysis of Task DifficultyIn this paper, we evaluate on a representative set of natural language understanding tasks. We prove that our approach benefits challenging tasks more consistently. However, we don't closely investigate if there is a correlation between the difficulty of a task and the effectiveness of our Leitner-based skill rating approach on it. For such an analysis to be possible, we need a principled way to define what makes a task more difficult naturally or to simulate that synthetically. We leave a systematic fine-grained analysis over more downstream tasks ranging in difficulty for future work.

#### Other Scenarios of Cross-lingual Continual Learning

Given the myriad ways to define realistic setups, the endeavor of defining a more principled realistic model of annotation scenario, following some logic or pattern, warrants its own dedicated investigation. In this study, we chose to focus on a common realistic scenario encountered by many users: a model is used by a user in one language, and then later on, the user experiences the need to incorporate more languages one at a time. Aside from being an extreme adaptation case and thus a useful stress test of our technique, a realistic language phase scenario would be a company's strategic decision to enter into a new language/market due to an influx of customer data. While exploring broader scenarios of continual learning -- where each phase may entail training on multiple languages repeatedly -- holds

Figure 7: t-SNE visualization of centroids of different intent labels highlighting some ambiguous labels indistinguishable in the embedding space.

 

potential, such exploration lies beyond the scope of this study.

#### Experimenting with Other Language Orders

In this work, we experiment with a carefully curated and balanced set of language orders. In future work, it would be worth investigating the extent to which models forget when using other possible language orders. It is also not clear if there is a deterministic linguistic or non-linguistic factor that can make a particular language order languages more susceptible to forgetting than others. Hence, a meticulous and comprehensive analysis needs to be conducted to unravel the intricacies of the phenomenon of forgetting and its underlying triggers in the context of cross-lingual continual learning.

Experimenting with Memory ScaleTo ensure a consistent comparison between different models, we maintain a fixed memory size per benchmark throughout our analysis. Naturally, as the memory size grows, we anticipate the need to be discriminative in memory selection to be diminished. Specifically, since most examples are not intractable (not difficult to learn), a larger memory capacity increases the likelihood of a random approach selecting higher-quality, well-learned examples. Conversely, we anticipate that experimenting with even smaller memory budgets may not yield optimal results, as there must be adequate memory size for the experience replay approach to significantly outperform the vanilla baseline. In future work, interested readers can explore more low-resource and sample-efficient approaches, such as meta-learning, on top of our approach.

## Acknowledgements

We would like to extend our thanks to CUTELABNAME labmates and USC NLP Group for helpful discussions and all anonymous reviewers and meta-reviewers for their valuable feedback. This material is partially based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes, notwithstanding any copyright annotation therein.

## References

* Amiri [2019] Hadi Amiri. 2019. Neural self-training through spaced repetition. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 21-31, Minneapolis, Minnesota. Association for Computational Linguistics.
* Amiri et al. [2017] Hadi Amiri, Timothy Miller, and Guergana Savova. 2017. Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2401-2410, Copenhagen, Denmark. Association for Computational Linguistics.
* Amiri et al. [2018] Hadi Amiri, Timothy Miller, and Guergana Savova. 2018. Spotting spurious data with neural networks. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2006-2016, New Orleans, Louisiana. Association for Computational Linguistics.
* Ball et al. [2020] Philip J. Ball, Yingzhen Li, Angus Lamb, and Cheng Zhang. 2020. A study on efficiency in continual learning inspired by human learning. In _NeurIPS 2020 Workshop on BabyMind_.
* Carpenter and Grossberg [1987] Gail A Carpenter and Stephen Grossberg. 1987. Art 2: Self-organization of stable category recognition codes for analog input patterns. _Applied optics_, 26(23):4919-4930.
* Castellucci et al. [2019] Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019. Multi-lingual intent detection and slot filling in a joint bert-based model. _ArXiv preprint_, abs/1907.02884.
* Chaudhry et al. [2019a] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019a. Efficient lifelong learning with A-GEM. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Chaudhry et al. [2019b] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, P Dokania, P Torr, and M Ramzato. 2019b. Continual learning with tiny episodic memories. In _Workshop on Multi-Task and Lifelong Reinforcement Learning_.
* Clark et al. [2020] Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. _Transactions of the Association for Computational Linguistics_, 8:454-470.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of 

the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Feng et al. (2019) Kanyin Feng, Xiao Zhao, Jing Liu, Ying Cai, Zhifang Ye, Chuansheng Chen, and Gui Xue. 2019. Spaced learning enhances episodic memory by increasing neural pattern similarity across repetitions. _The Journal of Neuroscience_, 39:2741-18.
* Hadsell et al. (2020) Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. 2020. Embracing change: Continual learning in deep neural networks. _Trends in Cognitive Sciences_, 24:1028-1040.
* ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII_, volume 12353 of _Lecture Notes in Computer Science_, pages 466-483. Springer.
* He et al. (2021) Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2021. Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1121-1133, Online. Association for Computational Linguistics.
* Ho et al. (2023) Stella Ho, Ming Liu, Lan Du, Longxiang Gao, and Yong Xiang. 2023. Prototype-guided memory replay for continual learning. _IEEE Transactions on Neural Networks and Learning Systems_, pages 1-11.
* Hu et al. (2020)Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 4411-4421. PMLR.
* Isele and Cosgun (2018) David Isele and Akansel Cosgun. 2018. Selective experience replay for lifelong learning. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_, pages 3302-3309. AAAI Press.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillamne Desjardins, Andrei A. Rusu, Kiearan Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Darshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. _Proceedings of the National Academy of Sciences_, 114(13):3521-3526.
* Klasson et al. (2023) Marcus Klasson, Hedvig Kjellstrom, and Cheng Zhang. 2023. Learn the time to learn : Replay scheduling in continual learning. _Transactions on Machine Learning Research_.
* July 1, 2001_, pages 282-289. Morgan Kaufmann.
* Leitner (1974) S. Leitner. 1974. _So lernt man lernen_. Herder.
* Li et al. (2021) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2950-2962, Online. Association for Computational Linguistics.
* Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 6467-6476.
* Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. _ArXiv preprint_, abs/2308.08747.
* Mallya and Lazebnik (2018) Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to a single network by iterative pruning. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 7765-7773. IEEE Computer Society.
* McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower, editor, _Psychology of Learning and Motivation_, volume 24, pages 109-165. Academic Press.
*hamdi et al. (2023)Meryem M'hamdi, Xiang Ren, and Jonathan May. 2023. Cross-lingual continual learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3908-3943, Toronto, Canada. Association for Computational Linguistics 

Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7654-7673, Online. Association for Computational Linguistics.
* Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: Incremental classifier and representation learning. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_, pages 5533-5542. IEEE Computer Society.
* Reddy et al. [2016] Siddharth Reddy, Igor Labutov, Siddhartha Banerjee, and Thorsten Joachims. 2016. Unbounded human learning: Optimal scheduling for spaced repetition. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_, pages 1815-1824. ACM.
* Riemer et al. [2019] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2019. Learning to learn without forgetting by maximizing transfer and minimizing interference. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Ritter et al. [2018] Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. Online structured laplace approximations for overcoming catastrophic forgetting. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada_, pages 3742-3752.
* Schwarz et al. [2018] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Haadsell. 2018. Progress & compress: A scalable framework for continual learning. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 4535-4544. PMLR.
* Shin et al. [2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 2990-2999.
* Smolen et al. [2016] Paul Smolen, Yili Zhang, and John Byrne. 2016. The right time to learn: Mechanisms and optimization of spaced learning. _Nature Reviews Neuroscience_, 17:77-88.
* van de Ven et al. [2020] Gido van de Ven, Hava Siegelmann, and Andreas Tolias. 2020. Brain-inspired replay for continual learning with artificial neural networks. _Nature Communications_, 11:4069.
* Wolczyk et al. [2021] Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. 2021. Continual world: A robotic benchmark for continual reinforcement learning. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_, pages 28496-28510.
* Xu et al. [2020] Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for cross-lingual NLU. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5052-5063, Online. Association for Computational Linguistics.
* Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, volume 70 of _Proceedings of Machine Learning Research_, pages 3987-3995. PMLR.

## Appendix A Implementation Details

We specify below more implementation details, such as hyperparameters and datasets, in addition to the runtime and number of parameters of different models.

### Hyperparameters

For all experiments, we use M-BERT (bert-base-multilingual-cased)3 with \(12\) layers as our pretrained multilingual Transformer-based encoder model. Consistent with M'hamdi et al. (2023) and Hu et al. (2020) for _MTOP_ and _TyDiQA_, respectively, we use the Adam optimizer Kingma and Ba (2015), fixing the learning rate to \(3e-5\) for all experiments for a fair comparison. M'hamdi et al. (2023) perform a manual hyperparameter search over the range \([1\times 10^{-4}\), \(3\times 10^{-4}\), \(1\times 10^{-5}\), \(3\times 10^{-5}]\) to choose the most optimal learning rate based on Dev data split performance. For _TyDiQA_, those hyperparameters are chosen based on Hu et al. (2020). For _MultiATIS++_, we perform a manual search over the same learning rates range and find that \(3\times 10^{-5}\) performs comparably to other learning rates. So, we fix a learning rate of \(3\times 10^{-5}\), \(\epsilon=1\times 10^{-8}\), \(\beta_{1}=0.9\), \(\beta_{2}=0.99\) in the optimizer for a fair comparison for all experiments. For _TyDiQA_ experiments, we find it helpful when a scheduler with linear decaying learning rates is used. We use batch sizes of \(4\), \(16\), and \(4\) for _MTOP_, _MultiATIS++_, and 

_TyDiQA_, respectively. In all baseline models _Balanced_ and _Random_ and Leitner-guided _ER_ (\(LER\)) model variants, we choose a fixed memory proportion to \(20\)% of the training data from each benchmark. Based on that, we fix \(|{\cal M}|\) memory size to \(10{,}105\), \(500\), and \(500\) for all _MTOP_, _MultiATIS++_, and _TyDiQA_ experiments, respectively. We also fix the sampling frequency from the memory to every 10 minibatches. For all experiments, we run for \(10\) epochs maximum and pick the best model based on Dev data split. We use the same seed across all experiments to report the mean results. We also fix a seed of \(42\) for the random initialization of Numpy, Random, and Torch libraries over all experiments. All experiments are run on the same computing infrastructure using 1 NVIDIA A40 GPU of \(46{,}068\) MiB of memory CUDA version 11.6 and Pytorch version 1.13.1.

### Dataset License

_MTOP_ dataset has been released by Facebook under Creative Commons Attribution-ShareAlike 4.0 International Public License. _MultiATIS++_ and _TyDiQA_ datasets have been released under the Apache License, which allows the use, modification, and distribution of the dataset.

### Runtime

We show in Table 5 the runtime of different approaches and baselines for one single language order on _MTOP_. This runtime includes both the costs of training and evaluation. Our _LER_ only incurs 3 hours more than _No ER_ approach, with most of it spent calculating the skill rating at the end of each epoch.

Table 6 compares between the number of parameters of models used for different downstream benchmarks. Task-oriented dialog benchmarks (_MTOP_ and _MultiATIS++_) require more parameters and thus are more challenging compared to span-based question answering (_TyDiQA_).

## Appendix B More Results

Figures 9 and 9 show a fine-grained analysis of the forgetting of intent classification over different language orders and languages, respectively.

## Appendix C Proportion of Memory per Language

Figure 10 shows a fine-grained analysis of the language distribution of the memory at the last phase after training for each language order in _MTOP_. We compare between different distributions using Leitner queues approaches against the random approach. While _Random_ maintains a close to uniform per language distribution in different language orders, _LER (Easy)_ and _LER (Hard)_ make informed decisions of which proportions to pick from each language depending on its learnability.

\begin{table}
\begin{tabular}{l|l} \hline \hline
**Model** & **\# Parameters** \\ \hline _MTOP_ & \(178{,}081{,}402\) \\ _MultiATIS++_ & \(178{,}036{,}139\) \\ _TyDiQA_ & \(177{,}264{,}386\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Fine-grained parameter analysis per benchmark.

Figure 8: Fine-grained analysis of forgetting of intent classification over different language orders as defined in Table 2. Best (lowest) results for each language order are highlighted in **bold**.

\begin{table}
\begin{tabular}{l|l} \hline \hline
**Model** & **Total Runtime** \\ \hline _No ER_ & \(6\) hrs \(23\) min \(20\) sec \\ _Balanced_ & \(6\) hrs \(45\) min \(22\) sec \\ _Random_ & \(6\) hrs \(25\) min \(25\) sec \\ _LER_(easy/hard) & \(9\) hrs \(23\) min \(13\) sec \\ \hline \hline \end{tabular}
\end{table}
Table 5: Fine-grained runtime analysis per model for one single language order on _MTOP_.

Figure 9: Fine-grained analysis of forgetting of intent classification over different languages. Best (lowest) results for each language are highlighted in **bold**.

 

* [25] M. C. Bader, A. Krizhevsky, and S. Bengio. Learning from deep learning with large-scale features. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1083-1094, 2016.
* [26] J. Dosovitskiy, L. Krausek, and I. Sutskever. Learning representations by jointly learning a generative model. _arXiv preprint arXiv:1706.03018_, 2017.
* [27] R. Girshick et al., "Very good results for image recognition." in _IEEE Transactions on Image Processing Systems_, vol. 10, no. 1, pp. 1-10, Jan. 2016.
* [28] Y. Jia, X. Liu, Z. Zhang, and W. Sun. Deep residual learning for image classification. In _NIPS conference on computer vision_, volume 1, pages 1-10, 2016.
* [29] H. Li, T. Lin, and G. Yuille. Unsupervised learning via latent representation. In _ICLR 2016 Workshop on Machine Learning_, pages 1-10, 2016.

[MISSING_PAGE_POST]