Evaluating Online Continual Learning with CALM
Authors: Germán Kruszewski
Ionut Teodor Sorodoc
Tomas Mikolov
Venue:  (2021)
Keywords: online continual learning
catastrophic forgetting
benchmark
language modelling

Abstract: Online Continual Learning (OCL) studies learning over a continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn “on-the-wild”. Yet, commonly available benchmarks are far from these real world conditions, because they explicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose a new benchmark for OCL based on language modelling in which input alternates between different languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based on compositions of experts. Finally, we introduce a simple gating technique that learns the latent similarities between different inputs, improving the performance of a Products of Experts model.

Under review as a conference paper at ICLR 2021
EVALUATING ONLINE CONTINUAL LEARNING WITH CALM
Anonymous authors Paper under double-bling review
ABSTRACT
Online Continual Learning (OCL) studies learning over continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn "on-the-wild" Yet, commonly available benchmarks are far from these real world conditions, because they ex- plicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose new benchmark for OCL based on language modelling in which input alternates between differ- ent languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based compositions of experts. Finally, intro- duce a simple gating technique that learns the latent similarities between different inputs, improving the performance of Products of Experts model.
INTRODUCTION
Machines, like humans, can learn to perform multiple different tasks from feedback alone (Caru- ana 1997). On the other hand, humans, but not machines, can benefit from settings in which tasks presented repeatedly for multiple trials before switching the next one (Flesch et al.) 2018), whereas machines require examples to be presented in a shuffled (i.i.)) order to learn effectively. Otherwise, they suffer from an effect known as "catastrophic forgetting" ence" (McCloskey & Cohen 1989 Ratclif 1990). While there has been a considerable amount of work focused on solving this problem, an endeavour that also goes by the name of 'Continual', "Inreemental' or 'Life-long' Learning. large part of it is evaluated on settings in which there is an explicit delimitation signal for every new task presented to the model (Kirkpatrick et al. 2017] Zenke et al. 2017| (Sodhani et al.) 2018), Serra et al.] 2018: Lopez-Paz & Ranzato [201
"catastrophic interfer-
Fernando Rusu Rusu et al.) 2016 Li & Hoiem [018]] Aljundi et al. 2017 Adel al. Titsias et al. 2020 Ebrahimi et al. 2020 von Oswald et al.) 2020 Li et al. 2020; Yoon et al.
let al.
2017] Lee al. [217]]
2020
20200. j. However, humans do not need any such signalling at all. Consider, for example, the case
of child growing up in multi lintill environment. Even though it is not entirely clear whether the child would rely on environmental cues (for instance, the identity of the speaker) to distinguish different input languages not (De Houwer (017), any mechanism must be necessarily inferred from the context. Moreover, even the concept of "task" could be vacuous, as it could be represented by shifting data distributions (Lesort al. 2020).
Even though the emerging field of Online Continual Learning PParsi & Lomonaco 2020 A Aljundi
al., 2019a) or Task-Free Continual Learning (Aljundi et al. 2019b] Lee et al.) 2020) has started to propose solutions to these problems, commonly available benchmarks make assumptions that are far from the real world conditions, such as lacking latent similarity structure on the data stream (c.g. orthogonal permutations of an image pixels) or assuming temporal independence between different examples (e.g. an image of a chair can be classified as "chair" independently of any previous exam- ples). Consider, instead, the challenge of natural language learning which requires making sense of a highly correlated and temporally interdependent data stream. We argue that the notable scarcity of benchmarks featuring temporally correlated sequences of examples, with short and long-term dependencies, latent similarities between different classes of examples, and no explicit delimitation when transitioning between different classes has left a blind spot in the Online Continual Learning community, which we address here. Moreover, almost none of the commonly used benchmarks deals with language, further limiting the amount of research that extends to this modality.

---

Under review as a conference paper at ICLR 2021
Here, we make a two-fold contribution towards studying online continual learning in neural networks in a linguistic setting. First, we bring CALM (Class-Agnostic Continual Language Modelling)) to the community, continual language modelling evaluation framework containing text that alternates between different classes of input (e.g. different languages or domains) with latent similarities to which the models could adapt. We introduce two variants. The first is character- based language modelling benchmark featuring five different languages that randomly switch be- tween one another. The second one is a word-based language modelling task, where the text al- ternates between four different domains. No segmentation signal is given when a switch happens, thus requiring models learn to adapt to these changes. We also propose novel metrics that capture the impact of catastrophic forgetting in an online learning setting by measuring how efficiently can models adapt to class switches. In line with Aljundi et al. (2019b), we note that when a distribution shift occurs, neural network that suffers from catastrophic forgetting will display a spike in the loss signal, even when the distribution had been observed in the past (sce Figure lak. Thus, we propose catastrophic forgetting metrics based on characterizing the size of these peaks. The benchmark is provided as a Python library that can be easily imported into a Py Torch project Second, we eval- uate multiple baselines based on expert architectures and propose novel albeit simple mechanism that we call plastic gates, which we show to improve the performance of Products of Experts. Our post-hoc analysis shows that this mechanism is effective in producing gating strategy that helps to circumvent catastrophic interference while also uncovering latent similarities in the input classes.
2 RELATED WORK
The field of Continual Learning. Incremental Learning or Lifelong Learning has grown to encom- pass a large body of work, which is better summarized in respective reviews (Parisi et al. 2019] Lesort et al. 2020). An overwhelming majority of this work concerns image classification prob- lems or object recognition. Some evaluation datasets are derived from traditional machine learn- ing datasets by manipulating the input examples in more or less artificial ways -like Permuted MNIST (Kirkpatrick et al. 2017) or Rotated MNIST (Lopez-Paz & Ranzato 2017 while others keep examples unchanged but present them in specific non-i..d. order, like for instance, iCIFAR- 100 (Rebuffi et al. (017) or split-MNIST (Zenke et al., 2017). All of these datasets comprise single-input classification problems in which there are no temporal dependencies nor correlations between two consecutive examples. To better approximate the conditions of real-world experiences. Fanello et al. (2013). Lomonaco Maltoni (2177) and Roady et al. (2020) introduced Corld,,,, CORe50, and Stream-51 respectively, which comprise short videos of objects from different angles (further including naturalistic scenes in the latter case). These datasets address the problem of cor- related examples, but not of temporal dependencies, which we do address in this work. [ et al.) (2020) and de Masson d' Autume et al. (2019) proposed the only benchmarks dealing with language that we know of, in which the former adopts a sequence to sequence paradigm study incremental learning of new vocabulary items on simplified or artificial datasets, while the latter adapted existing text classification and QA benchmarks analogously to dhove-mentionel work in image classifica- tion. Our work instead uses naturalistic textual data containing natural latent similarities between distributions that can drive information transfer forgetting.
and
By and large, work directed to address catastrophic forgetting in neural networks presumes the existence of task identifier signal different learning units. However, recent work has aimed at lackling catastrophic forgetting even in conditions in which no task boundaries are provided (Aljundi
al. 2019b Lee et al. 2020)) going under the name of "Task-Free Continual Learning" or "Online
Continual Learning" (Parisi & Lomonaco, 2020 Aljundi et al. 2019a). Of these works, Ally|Aljundi al. (2019b) (0199) uses naturalistic data to classify actors appearing in soap-opera episodes (Aljundi et al.
2016), while others resort to artificially modified datasets like split or permuted MNIST. Here, we complement this resource with a text-based benchmark for Task-Free Continual Learning, while arguing for more work on more naturalistic non-i.i.d. datasets.
Another aspect of Continual Learning deals with how models are evaluated. Most often, this is done by measuring accuracy on a dedicated test set (Lopez-Paz & Ranzato 2017 Díaz- Rodríguez et al.) 2018 Hayes et al. 2018, Chaudhry et al. 2018 de Masson d' Autume et al. 2019). However this
'Code and materials included in the supplementary materials will be made publicly available upon accep- tance.
2

---

Under review as a conference paper at ICLR 2021
evaluation protocol is tailored for batch learning conditions, in which model is fit to training dataset, and then stops learning. Here, instead, we argue in favour of situated evaluation protocols adapted to far-from aut-from-quuiibrium learning conditions (Holland 1992) by adopting an Online Learning
framework (Hoi et al. 2018), which is also known as the prequential approach (Dawid 1984 Gama
et al. 2013)
On the modelling side, this work explores Mixture of Experts (Jacobs et al. 1991) and Product of Experts (Hinton 1999) architectures. Variations thereof are at the base of many architectural proposals for addressing catastrophic forgetting (Rusu et al. 2016 Li & Hoiem 2018 Aljundi et al.) 2017
Lee et al., 2020). However, often they are accompanied by other mechanisms, such as the growth of new modules, freezing of weights or generative modelling of the input. Here we examine the simplest enactments of these architectures and propose an pasy-o--mmplemnnt gating mechanism which can be learned online and provides a strong baseline for more complex architectures.
Finally, our study falls within the line of language modelling using neural network models (Bengio et al. 2003; Mikolovv et al. 2010)). In this context, adaptation to the recent past has been studied in the context of cache models (Grave et al., 2017: Merity et al. (017). There, non-parametric model deals with capturing higu-frey recent statistics while a parametric model captures the more stable aspects of the distribution. These solutions, however, are not well-adapted for cases in which the whole distribution changes over time. Moreover, language modelling is generally studied in train-test split, where a model is fitted to the training data and asked to generalize over the unseen test data. Here, instead, study how a model can adapt incoming data in online fashion.
3 THE CALM BENCHMARK
We designed a benchmark for evaluating Online Continual Learning algorithms having in mind the following three desiderata: 1) naturally correlated sequential data, 2) task agnosticism and 3) temporally situated evaluation. Paarisi & Lomonaco (2020) discusses the first two. The first requires that on the one hand, data is observed in potentially infinite data stream with high-dimensional, non-stationary, and temporally correlated examples. The second, that learning systems should not be fed external task boundaries to help them learn in these conditions. Furthermore, we also introduce third desideratum, by which we ask models to be evaluated in-situ on each example presented to the model, following the classical Online Learning setting (Hoi et al. 2018]] [ahoo et al. 2018). We thus propose an Online Continual Learning benchmark featuring a language modelling objective where the data stream can switch between different distributions. Because switches are not announced to the model, this is a SSingle-Incrementee "Takk" or "No task label" scenario under the framework proposed by Lesort et al. (2020).
More precisely, consider sequence of observations Ia E X' that are fed to a model M parametrized by O, which makes the prediction ÿt E y. Then, the true target yt y will be revealed and the loss L, = L(je, V1) is observed and later used to compute the model's 's performance from given time S until time T as the average loss in that span T.- T-S Et=s Ia for evaluation purposes. Only after reporting the loss can the model be trained based the received feedback, preventing data leakage. In practice, these examples are presented as mini-batches (X,, Y,) E x x ) containing b parallel streams, and chunked into small windows of length w for efficiency considerations related to the training of neural networks (Parisi & Lomonaco 2020
The data stream is composed of N sequences of consecutive batches of length Ti, T2,...... Tn. and starting at positions S. Σ}= Tj. In turn, each of these sequences belong to one of n different classes [Dr.... D,,], presented in random order.
To characterize the effect of forgetting we note that a model that becomes disadapted to given distribution will display spike the loss after the stream switches to this distribution, even if has been observed before (see Figure Tap. For model to be resilient to catastrophic forgetting, it must adapt quickly to every new distribution, which can be characterized by the height and width of these peaks. We thus propose the following metrics to complement the standard online performance:
Loss after switch: Tracks the loss for the first l times-steps after switch occurs to quan- +k
tify the height of the peak. Formally, L@ss N/4 Σ T.S.
3

---

Under review as a conference paper at ICLR 2021
Recovery time after switch: Counts the number of time-steps that it takes the model to reach the mean loss observed for the last seen sequence of the current class. this way, we can quantify the length of the peak.
3.1 DATASET
In this work, we created two datasets for CALM. One is character-level and multilingual, whereas the other word-level and multi-domain. Both benchmarks feature conflicting learning signals when moving between domains or languages, making the learning systems susceptible to catas- trophic forgetting.
For our first dataset (MultiL.ngg Lingual and character-based), we propose language modelling bench- mark in which incoming text data can alternate between different languages. This benchmark is character-based because there would hardly be any forgetting at the word level, as the word dis- tributions hardly share any support. Concretely, we build on parts of the news corpus developed for the 2009 Workshop of Machine Translation Hallison-Burche et al. 2009). We extracted text from five languages: English, French, Spanish, German, and Czech (containing 1.8B, 572M, 160M, 715M and 439M characters, respectively) because they all have similar character sets, while also showing interesting linguistic variability. In particular, they belong to three different Indo-Puropean branches: Romance (French and Spanish), Germanic (English and German), and Slavic (Czech). As a consequence, there is a latent similarity structure between the different classes that models could learn to recognize. Compared to earlier multilingual corpora (Kawakami et al. 2017), our dataset was carefully constructed to include only linguistically valid characters, in order to prevent non- linguistic noise from interfering with our experiments. For this, we removed all lines from the input that contained characters appearing less than 100 times on the full corpus. The resulting character vocabulary consists of 211 characters.
The second dataset is an English word-level MultiDomain dataset. For this, we used four different source corpora: news (same as above), europarl (Koehn 2005), the British National Corpus (Con- sortium et al.) 2007). and Wikipedia (Merity et al. 2017). They each have 300M, 54M, 100M and 101.4M tokens, respectively. In contrast with the previous dataset, word-level is the most appropri- ate choice here, differences between the distributions at the character level would be too nuanced to drive any forgetting. We kept in the vocabulary the top 25K words for each corpus, which after merging yielded a vocabulary size of 58K words. Samples from all source corpora are included in the appendix.
We then created the final MultiLingual and MultiDomain corpora by joining N = 100 different frag- ments evenly distributed among the different classes (languages or domains) with lengths sampled from a (truncated) exponential distribution: T; Exp(A). Thanks to this distribution's memoryless- ness property, it is virtually impossible to estimate when the next switch is going to happen. While we do not constrain switches to occur at word or sentence boundaries, but rather after an integer number of sequences of length the noise introduced at transition points for this reason is rela- tively mild and does not affect the Eistribution-alternafii and further analysis become considerably simplified by removing the need to handle variable-length input. We constructed two different variations with shorter or longer fragments. For MultiLingual, constructed IM and I0M-characters- and A= = 100k characters, respectively. For MultDomain we followed the same procedure, extract- ing 100 alternating sequences with mean lengths of À = 10k and X = 20k, for total of 1M and 2M words. These relatively modest sizes allow for faster iteration and exploration of different models, while still allowing us to observe forgetting (or lack thereof) dynamics in the studied models. To facilitate further research, we release a Python ( a data iterator for both datasets in which a researcher can experiment with different variations by picking parameters N and X.
nature of the dataset. In benefit, training
corpora with expected fragment lengths of X = 10k
4 BASELINE MODELS
To endow CALM with simple and yet, strong baselines, we explored architectures based on (Weighted) Product of Experts or PoE (Hinton 1999) and Mixture of Experts or MoE (Jacobs) et al. 1991 Eigen al. 2013), henceforth generically denoted expert architectures. Thanks to
Available htttt://ancnnmm ize
4

---

Under review as a conference paper at ICLR 2021
combining predictions from different experts, they can potentially learn different parts of the latent distributions. Moreover, gating weights can avert catastrophic forgetting on the individual experts by modulating the learning signal, making them excellent candidate to model Online Continual Learning problems. Indeed, while variations thereof have been explored before (see Section 2. here emphasize simplicity would befit baseline models, yet not neglecting performance.
In the standard implementation of expert architectures, mixture weights are produced by a third "gating" module function of the current inputs. While this gating model could quickly adapt to changes in the environment, learning to do so is far from trivial in a continual learning setup, sometimes requiring pre-trate pre-training to distinguish input classes (Aljundi et al., 2017). The problem comes from the fact that the gating network must learn a latent classifier to pick the experts best adapted the current class, but classes are observed non-i.i.d. as long sequences of examples from one class at a time. Thus, the gating network can easily settle for constant function for any given current class, which only changes when examples of different class start to be observed, making experts vulnerable to catastrophic forgetting. In order to alleviate this issue and make experts more stable, we propose plastic gates, by which the gates are fast-adapting parameter values that are trained on recent experience.
More formally, an expert architecture is composed of a set of modules M = (M with parameters OM, ........ θM,,
used to compute unique prediction as follows. When an input (with target y) is observed, it is fed to all modules Mı...n, obtaining log-linear outputs y()) Mi (r) 5.() = Mz(x). Then, an additional vector of mixture weights E I" is used to combine them. This vector is computed by a separate gating module w = G(xr with parameters Oc, jointly trained with the rest of the network. The output of the full model is then linear
combination of the individual modules outputs Y =1) ...... "..)).....yy) 5(")] weighted by w after or before normalizing, depending on whether the model is MoE or PoE:
MoE (Mo)(w)- Σ softmax(w), (softmax (())
PoB = softmax (YTw)
Note that in contrast to MoE, PoE are more efficient to compute because they do not require to normalize the output of each individual model. Once the loss is computed on mini-batch (X,, Y) and kept for evaluation (see Section 3), all sub-networks G and M are trained for one or more gradient steps to reduce this loss, and the system moves to the next mini-batch.
Plastic Gates Rather than learning a gating network, which can be challenging, we propose to continually learn the gating coefficients that best fit the recent experience:
Wi+1 arg min ((Y; Yt)
w
In practice, we perform (hyperparameter) number k of gradient descent steps the above objec- tive to allow for some regularization of the gates over time.
Parametrization for Language Modelling We instantiate the expert modules M₃ to be double-
layered LSTM networks (Hochreiter & Schmidhuber 1997). with predictions y hht1" (0) = LSTM; (z,, h (5)) For the regular gating network, we use single-layer LSTM network. That is, Wt. h4 1 LSTM(r,, hτ).
EXPERIMENTS
We explored the performance of different baseline models while they made single pass the CALM datasets. Following standard practice, rather than reporting the cross-entropy loss, we use the perplexity each time step, given by eap(Lr). Furthermore, we allowed the models learn over the first half of the datasets without being evaluated, and only start computing metrics on the second half. Otherwise, we use the measures discussed in Section 3 to track models' performance, namely, average perplexity (ppl), average perplexity for k = 10 batches after a switch (ppl@sw) and recovery time after a switch (rec).
Note that the since Pot linearly combines the logits is effectively computing a geometric combination of each individual module's unnormalized probabilities: exp()""F) I[; exp(y./)
5

---

Under review as a conference paper at ICLR 2021
We explored models featuring different degrees of modularization, varying their hidden size vectors to make them all have an approximately equal total number of parameters. On one extreme, we had large two-layers LSTM network. Next, we considered standard PoE and MoE models with mixture weights computed by an LSTM gating network, plus their plastic weights variants (+PW), described in Section 4 Moreover, trained ensemble models (Ensemble), which equivalent to MoE where all mixture weights are 1 for all n modules. We studied both a more centralized network composed of 5 modules and larger hidden dimensionality (marked with 5) and more dis- tributed network with 30 modules but with smaller hidden sizes (marked with 30). As reference points (but not as real contenders), we also trained independent LSTMs (Ind. LSTM), one for each class, which enabled us to compare the performance of our model to a situation where there is no forgetting from conflicting learning signals, but also where there no possibility of transferring learned representations across possibly related domains. Furthermore, we compare Mixture-of- Softmax (MoS) model (ang et al. 2018). in which multiple softmax layers are combined to extract the predictions from single LSTM module. While were also interested in applying state-of-the art online continual learning methods (Lee et al. [0200] Aljundi et al., (019a). having these systems being designed for image classification datasets they would require non-trivial adaptations signif- icantly departing from the original models, which would limit any possible conclusions we could draw. Similarly, experimented extensively on validation data with Transformer models (Vaswani et al. 2017). However, due to these models sensitivity to dataset size and learning rates scheduling schemes which have been studied extensively for batch-learning (Popel & Bojar 2018), but not for these far-from-equilibbe
(Holladd) 1992) conditions, their performance was worse than expected. We give detailed account of our attempts the appendix and leave a study on how to adapt these models for Online Continual Learning for future work.
We controlled the number of model parameters to remain constant for each of the MultiLingual (about 21M parameters) and the MultiDomain (about 600M parameters) experimental setups. (The difference size explained by the larger vocabulary sizes in the latter.) For this, we adjusted the hidden dimensionality of different models accordingly, which, together with all explored hyperpa- rameters, are reported in the appendix. We kept the size of the incoming batches fixed at w = 20 and 10 for all models and used PyTorch (Paszke al. 2017) with the standard implementations for the underlying models.
5.1 RESULTS
MultiLingual
MultiDomain
À
10k
À
100k
À
10k
À
20k
ppl
ppl
@pl@sw
rec
ppl
ppl@sw
rec
ppl
@pl@sw
rec
ppl
ppl @
rec
Ind. LSTM
7.1
7.16
1.15
4.7
4.73
1.18
356
349
1.11
295
292
1.15
Large LSTM
7.78
10.4
6.82
4.86
8.58
18.9
352
106
3.61
457
619
6.56
MoS
8.13
10.6
6.6
5.43
10.3
19
343
443
4.6
298
409
6.08
Ensemble 5
8.84
11.3
7.41
5.6
10.2
24.7
418
519
3.89
317
411
4.83
MoE
8.65
10.9
7.11
5.55
9.86
21
125
524
3.76
335
439
4.94
MoE+PW
8.74
11.1
7.2
5.58
10
23.3
446
557
3.94
331
4.63
PoE 5
7.68
10.1
7.06
5.32
9.79
25.5
297
389
5.18
404
505
4.47
PoE+PW
7.2
8.46
3.67
5.02
7.54
14.9
320
361
2.82
270
322
3.35
Ensemble 30
11.9
14.8
8.08
7.05
14.2
30.9
509
623
3.72
391
511
5.14
MoE 30
11.1
13.7
7.54
6.89
13.7
30
539
651
3.47
436
572
4.97
MoE+PW 30
11.2
13.8
7.97
6.92
13.7
29.7
555
675
3.49
419
561
5.43
PoE 30
7.96
10.7
7.33
5.17
9.9
24.8
315
375
3.89
297
389
5.18
PoE+PW 7.41
9.17
4.76
5.04
9.03
285
316
2.68
241
287
3.54
Table 1:
Average perplexity (ppl). perplexity for 10 batches after switch (ppl @sw), and recovery time after switch in batches (rec) for both datasets per mean sequence length (A).
Results are averaged over ten different runs and reported in Table Standard deviations are reported on the Supplementary Materials.

---

Under review as a conference paper at ICLR 2021
Catastrophic Forgetting
MuttiLingual
MultiDomain
PoE
PPE 30
PoE+PW PrE4PW 5
PoE-4WW5
by cn hunte
predinemee deplow
Number belicee alter switch
Namber batchhe
(
(b)
(c)
Toac
0.51 0.110.01443 0.43 0.5
Crech
Syanhh
0.76
0.6
0.50
Spanish
0.73 O. 66
French
Touch
0.7
English
GGrman
Cock
Cooch
Spanish
French
English
(d)
Figure I: (a) CALM: model's loss (L) is tracked as it observes text pertaining to different classes, while catastrophic interference provokes spikes in this signal. (b-c) Mean cross-entropy for the first 15 batches after a switch averaged over all occurrences in MultiL.ngual and MultiDomain, respectively, under different random seeds. (d) Mixture weights produced by the PoE+PW 30 model on multilingual data (A = 10k). (e) Correlation coefficients between mixture weights corresponding to different languages for the PoE+PW 30 model collected during the last 100 batches (A = 10k).
We begin by observing that higher values of correspond to lower perplexities, as expected from the fact that these corpora with longer sequence lengths are also proportionally larger in total length.
Second, we note that Ensemble and MoE systems with 5 modules and larger hidden vectors out- performed models with 30 modules and smaller hidden dimensionality, but this is not the case for PoE, which show comparable performance between the two variants or even the opposite trend. Furthermore, the PoE's performance considerably better than the former two, which can be at- tributed to a combination of multiple factors. On the one hand, we note that, in agreement with previous work (Shen et al. )019), MoE models are often sensitive to Winner-Takes-AN (WTA) ef- fect in which only one single expert gets trained at the end. Thus, models with larger dimensionality per module can benefit from having a larger capacity. However, also ensembles show comparable performance, showing that this effect is not only caused by single module being trained. Per- haps, more important is the fact that, as hypothesized by Hinton 1999), PoE can use their capacity to learn complementary parts of the distribution, and thus it makes smaller difference for them whether there are a few high-capacity modules or many of them, but with smaller capacity.
Next, we note that while PW does not strongly alter performance on MoE architectures, as expected from the WTT effect influencing these models, they significantly improve the vanilla PoE. counter- parts, confirming the effectiveness of the proposed mechanism in this task. This observation holds not only for overall perplexity but also in terms of the metrics quantifying adaptation efficiency at class switches (ppl@sw and rec). Indeed, Figures Ib and Ic show this fact in more detail, by rep- resenting the mean cross-entropy of each different model for the 15 batches occurring immediately after a switch. As we can see, the PoE+PW model shows a large spike on the first batch because its adaptation mechanism that depends on this error signal has not kicked in yet. However, in the subsequent batch, its performance increases sharply outperforming comparable models.
In comparison to monolithic LSTM model, PoE and PoE+PW models perform on-par Multi- Lingual (although with better adaptation records), and better on MultiDomai.. In the latter case, we can observe that the version with 30 modules yields better performance than the one with just One possible explanation is related to word-level language modelling being a higher rank problem than character-level language modelling, and thus it can be better fitted by combining the judgements from multiple lower-rank experts (Yang et al. 2018). This explanation also consistent with the comparatively better performance of the MoS model.
7

---

Under review as a conference paper at ICLR 2021
Finally, we note that the model with Independent LSTMs for each class performs best MultiLin- gual, but it is outperformed by a large margin on MultiDomain. We note that this model does not suffer from forgetting when switching classes but also misses the training signal from transferable training data. As consequence, it has an edge on Ingua,, which switches between classes that have considerably different statistical properties, but not on MultiDomain where the differences between the classes are much more nuanced. All in all, this shows that while, in consonance with previous results (Dhar& & Bisazza 2018), there is little room for transferring knowledge in the Mul- tiLiinuall case, the MultiDomain setting provides plenty of opportunities for transferring knowledge across each domain. Thus domain danain-agnostic systems can benefit from them.
5.2 ANALYSIS
Next, turned analyze the gating strategies acquired by the more successful models to under- stand whether they have captured the latent similarities between classes and how might they help them in coping with catastrophic forgetting. For this, we focused on the PoE+PW 30 modules oper- ating on the MultiLingual dataset (A = 10k) because its 30-dimensional gate vectors can represent more nuanced similarities.
Figure ld shows a heatmap of the mixture weights as the model processes different language se- quences. High absolute values represent the activation of a module, regardless of whether these are negative or positive values. It can be seen that upon language switches, the model reconfigures itself to a different set of mixture weights that are maintained more or less consistently within the sequence. Furthermore, we note that modules that receive mixture weights close to 0 are protected from forgetting, as this gating value is also multiplied to the module's gradients. Moreover, we hypothesize that modules are protected even when their corresponding weight is set to the oppo- site sign (see, for instance, module 16 on English and Spanish), because the incoming training data serves negative training data, namely, something not-to-be to-be-peeee cae-bee-ppedee predicted. Thus, this should not affect what the model does predict when used with a positive weight. Instead, this allows for dual use of the modules, encoding information both when it is weighted positively and negatively.
Finally, recall from Section 3.1 that the languages in our MultiL.nguua dataset are derived from dif- ferent linguistic families with latent similarity structure. To uncover whether the learned latent similarities reflect this structure, we computed the correlations between the mixture weights pro- duced while processing the last 100 batches of each class. The results are displayed in Figure [e] and show that the similarities are indeed well-reflected in the gating values. Notably, we observe that Czech seems to be using the most distinct set of modules. Spanish and French correlate quite strongly in the modules they use, and while English also correlates with French, it also does so with German, with the latter correlating to a lesser extent with the other languages. Indeed, applying a simple hierarchical clustering algorithm over this matrix recovers the underlying linguistic families!
6 CONCLUSIONS
In this paper, we have introduced the class-agnostic continual language modelling task (CALM), together with a Python library with MultiLingual and MultiDomain datasets, which allows multiple parameter configurations and can also be easily adapted to different corpora. We expect that will foster more empirical work on continual learning in a language-centred setup in which there natural latent similarity structure between different tasks. We have argued that in addition to measuring the overall performance of models in an online learning fashion, their susceptibility to catastrophic forgetting can be observed in terms of adaptation speed to changes in the input class, and proposed measures to capture it. Finally, we have evaluated multiple simple baselines to serve as references for future work on this benchmark and introduced a simplification of the gating strategy for a Product of Experts, which improves its performance significantly by allowing it to distribute effectively different distributions across different experts so that the resulting system can act as a strong baseline for future work on this task.
While addressing catastrophic forgetting is still a major challenge for Online Continual Learning, it is by means the only one. In the future, we would like to understand how learning systems can also bootstrap on their knowledge to improve their learning skills, so that they will not only be able to acquire knowledge from different sources in seamless way but also get better at it as they go.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tameem Adel, Han Zhao, and Richard E Turner Continual learning with adaptive weights (claw). In Proceedings of ICLR, 2020.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Who's 's that actor? automatic labelling of actors in series starting from imdb images. In Asian Conference on Computer Vision, pp. 467-483. Springer, 2016.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In Proceedings CVPR, 7120-121177111 7120-7129, 2017.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 1189--1866/ 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of CVPR, pp. 11254-1113 11263, 2019b.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. neural probabilistic language model. Journal of machine learning research, (17eb),1137-1155,,000
Chris Callison- Philipp Koehn, Christof Monz, and Josh Schroeder. Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT '09, pp. 1-28, 2009.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arXiv:
2018.
BNC Consortium al. The british national corpus, version 3 (bne xml edition). Distributed by Oxford University Computing Services on behalf of the BNC Consortium, 5(65);6, 2007.
A Philip Dawid. Present position and potential developments: Some personal views statistical theory the prequential approach. Journal of the Royal Statistical Society: Series A (General), 147(2): 278-290, 1984.
Annick De Houwer. Bilingual language acquisition. The handbook of child language, pp. 219-250, 2017.
Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems, pp. 13143-13152, 2019.
Prajit Dhar and Arianna Bisazza. Does syntactic knowledge in multilingual language models transfer across languages? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP, pp. 374-377, 2018.
Natalia Díaz Rodríguez. Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don't Don't forget. there is more than forgetting: new metrics for continual learning. arXiv preprint arXiv:1810.13166 2018.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. In Proceedings of ICLR, 2020.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint 1312.43 2013.
Sean Fanello, Carlo Ciliberto, Matteo Santoro, Lorenzo Natale, Giorgio Metta, Lorenzo Rosasco, and Francesca Odone. icub world: Friendly robots help building good vision data-sets. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 700-705, 2013.

---

Under review as a conference paper at ICLR 2021
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 1rX1v:1701.08734 2017.
Timo Flesch, Jan Balaguer, Ronald Dekker, Hamed Nili, and Christopher Summerfield. Comparing continual task learning in minds and machines, Proceedings of the National Academy of Sciences, 115(44):103113-E0333 2018.
João Gama, Raquel Sebastião, and Pedro Pereira Rodrigues. On evaluating stream learning algo- rithms. Machine Learning, 90(3):11-3346, 2013.
Edouard Grave. Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online lan- guage modeling with open vocabulary. In Advances in Neural Information Processing Systems, pp. 6042-6052, 2017.
Tyler L Hayes, Ronald Kemker, Nathan D Cahill, and Christopher Kanan. New metrics and exper- imental paradigms for continual learning. Proceedings of CVPR Workshops, pp. 2112-2115, 2018.
GE Hinton. Products of experts. In Proceedings of ICANN, volume pp. 1-6. IET, 1999.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive survey. arXiv preprint arXhh:1802.028777 2018.
John H Holland. Complex adaptive systems. Daedalus, 121(1):17-30, 1992.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom.
Learning to create and reuse words in open- vocabulary neural language modeling. In Proceedings of ACI. 2017, pp. 1492-1502, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, pp. 20161 1835, 2017.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pp. 79-86, 2005.
Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woy Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching, In Advances in Neural Information Processing Systems, pp. 4652 4662, 2017.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. Proceedings of ICLR 2020, 2020.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Dodr-Rodr' Continual learning for roboties: Definition, framework, learning strategies, op- portunities and challenges. Information Fusion, 58:52-68, 2020.
Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny. Compositional language continual learning. In Proceedings of ICLR, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-299) 2947, 2018
Vincenzo Lomonaco and Davide Maltoni. Core50:
dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pp. 17-26, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
10

---

Under review as a conference paper at ICLR 2021
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Proceedings ICLR 2017 (Conference Track), 2017.
Tomáš Mikolov, Martin Karafíát, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. Recurrent neural network based language model. Eleventh annual conference of the international speech communication association, 2010.
German I Parisi and Vincenzo Lomonaco. Online continual learning on sequences. In Recent Trends in Learning From Data, pp. 197-221. Springer, 2020.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Martin Popel and Ondfej Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1):3370,, 2018.
Roger Rateliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Ryne Roady, Tyler L. Hayes, Hitesh Vaidya, and Christopher Kanan. Stream-51: Streaming clas- sification and novelty detection from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 228-229, 2020
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 04671, 2016.
Doyen Sahoo, Quang Pham, Jing Lu, and Steven C.H. Hoi. Online deep learning: Learning deep neural networks on the fly. In Proceedings of IJCAI, pp. 2660-2666, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of ICML, pp. 4555 4564, 2018.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc' Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of ICML, volume 97 Proceedings of Machine Learning Reseurch, pp. 5719-5728. PMLR, 2019, URL httlhttp///rrcceeee/ ngs m www.s/ssese...... / / /t///shnnn Iecccc.ttt html
Shagun Sodhani, Sarath Chandar, and Yoshua Bengio. Toward training recurrent neural networks for lifelong learning. Neural Compulation, 0(0):1-34, 2018.
Michalis Titsias, Jonathan Schwarz, Alexander G G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In Proceedings of ICLR, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Johannes von Oswald, Christian Henning. João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. In Proceedings of ICLR, 2020. 2020.
11

---

Under review as a conference paper at ICLR 2021
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank mn language model. Proceedings of ICLR 2018, 2018.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Proceedings of ICLR, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of ICML 2017, pp. 3987-3995, 2017.
A CORPUS EXAMPLES
Figure 2 and present samples from the corpora used for our dataset. As stated in the paper, we can notice a much bigger difference between input class in the case of the multilingual setup, while the differences in the case of the multidomain setup are more subtle and nuanced.
Dataset samples
Mad'arská iNFINITY Coliseum Lan je pokračovánín áspéšnéćo BECUPu, z nêhož si nejeden ná8 tým v minulosti odvezl medaili.
ezech
If Hofmann played the role of paterfamilias, Anais Nin was the bad mother Admiral and De Niro's group. This wasn't 't close.
english
Le Beatle s en est alors emparé pour créer un chef-d chef-d'oeuvre o'ouuvre psychédélique longtemps associé à I' ussaee du LSD.
french
Im ersten Jahr hatten sie schon 278 Anfragen, fast ehemalige Manager Unternehmer wollten mitmachen.
german
Los despidos serán realizados por medio del plan de GM de cese de empleo, por lo que no ofrecerán jubilaciones anticipadas
spanish
Figure 2: Samples from the multilingual dataset
Multidomain dataset samples
Good weather for the crops. Have your sheep been suffering much from the staggers ?
bnc
Have you contributed great deal this year to the butter mountain ?
would like your advice about Rule 143 concerning inadmissibility.
euro
will come up on Thursday If Hofmann played the role of paterfamilias, Anaïs Nin was the bad mother to Admiral and De Niro's 's group. This one wasn't 'tt't close.
My question relates something
news
Otto Prince Bavaria. was chosen as the first King of Greece in 1832. under name Othon.
wiki
His arrival Nafplio, then the Greek capital, was hailed enthusiastically by Makriyannis
Figure 3: Samples from the multi-domain dataset
B FURTHER ANALYSIS
B. POE WEIGHTS BEHAVIOUR
We also inspected the gate values produced by LSTM-gated PoE models observing that the models are indeed not learning class-switching mechanism. We hypothesized that this due the fact that when the experts are still untrained, the LSTM produces some arbitrary but consistent gating values, making those selected modules being the only ones be trained, and thus falling into vicious cycle. As sanity check that supports this hypothesis, we first pre-trained a set of modules while still using our simple gating mechanism. Then, we initialized with these modules network that now used LSTM mixture weights, but training on very short sequences to avoid the effect of catastrophic forgetting affecting the network. In this context, the network learned the appropriate gating expected.
B.2 MULTIDOMAIN MODULE CORRELATION
In comparison with the Multilingual setup, correlations in the MultiDomain case are much weaker. Moreover, they weak even within the same class: When we measure the autocorrelation between
12

---

Under review as a conference paper at ICLR 2021
weights pertaining the last 100 batches with the preceding 100 ones obtain values in the order of 0.65, much lower than for the MultiLingual experiments, where they are in the order of 0.96 (see Figure 4b). This shows that model usage is less consistent per-class, which could be explained by the fact that classes are much more nuanced than in Multil.nngual and their corresponding distributions far more complex. These results also consistent with our experimental observation that the MultiDomain dataset was more amenable to transfer between different classes than the MultiLin- gual, as these classes could be distributed more evenly across the model and could be characterized with multiple mixture weights configurations.
B00kS
News
0.5
Europari
0.0
MWWS
-0.5
1.00
0.13
BNC
wki
1.00 Wiki
News
Europart
BNC
Wike
(a) Heatmap of weights with rows sorted by domain and columns sorted by similarity
(b) Weight correlations
Figure 4: MultiDomain (A=10k) analysis
C TRANSFORMER EXPERIMENTS
We experimented extensively with Transformer models. One difference with respect LSTM mod- els is that Trasnformers, at least in their vanilla versions, are not autoregressive, and thus they cannot transfer information from the past. In standard NLP tasks, they largely overcome this problem by using a large context window on which they can operate effectively. Thus, to afford them similar memory capabilities, we kept buffer of the last b 512 consecutive examples that was continually updated with each incoming mini-bu mini-batch.
(2017) describes learning rate scheduling scheme in which the learning rate is linearly increased until reaching a number of warmup steps, and then annealed from that point on. Considering that in Continual Learning setup the model is not expected to converge, annealing might not be appropriate. Thus, we also experimented with keeping the learning rate flat after that point on. We experimented with both learning rate schedules, plus no scheduling at all. Furthermore, we considered both training with default Adam parameters (a = 10-3, 3 (0.9, 0.999)) or the ones reported by Vaswani et al. (211)) and base learning rates of 1 dmooe: 1e 5e 0.01. Also, we tuned the warmup steps among 400, 2000 and 4000 steps. The best perplexity results we obtained in the Multilingual validation data were 13.2 for À = 10k and 6.02 for À = 100k, whereas in the Multidomain validation data we got 686 average perplexity for À = 10k and 527 for 20k.
Vaswani
D GENERATED OUTPUT
In Figure 5 we present generated samples from different stages of training. These generated exam- ples are produced by sampling one character at time from the models, and using them as input for the next time step. As quantitatively observed in the paper, it adapts much faster to the current input type (French) in comparison with an LSTM, which generates text resembling the language of the previously class even after 10 batches.
13

---

Under review as a conference paper at ICLR 2021
PoE+PW 30
LSTM
for a release was the week Troust
Tvice. (Relátórs had state's annual annual Call Statua plannting
end english
Pglates in George Services are
claimed whet this could get one
more years' physical cost cau polítical but the room of Noxe Common Electrical Taladei Baritef.
lmmnnt Filmarian Roads. cadres et temps disputer Lileana Maan, Institution, le provinces, unbicut
5 batches french
BAG - Runey premium begai maki attempted Jueves Mo., unit encome ergarded next post television genetical dangere tet. For hemous el-Bilanze extranger la fin de I'Etat: "Yens irneu à Show Joban Il vio, les grandes hommes
Definit werde à I équipe pass, libertant Youth Losier Chavez and Jean-Pierro a Vu entre fascal publ
10 batches french
ivc communc services au championnat où qui se sont renfovées de hierre du 23,4er est dit doubles
end french
Figure 5: Generated text at different stages of training
E MODEL SIZES
As it is shown Table 2 the number hidden units varies for most of the models. We vary the hidden size in order to keep similar number of parameters across the models: around 22 million for the multilingual setup and around 600 million for the multidomain one.
MultiLingual
MultiDomain
Model LSTM
Hidden size
#Parameters
Hidden size
sParameters
1300
21.66M
5200
605.2M 2M
Ind. LSTM
550
20.2M
1800
571.2M
PoE/MoE (+PW) 5
550
21.2M
1600
621.8M
PoB/MoE (+PW) 30
200
21.85M
200
635.3M
MoS
500
22M
2620
572M
Table 2: Model sizes
F HYPERPARAMETER SEARCH
Table 3 present the explored hyperparameters for LSTM and PoE. The parameters in bold are the ones chosen for the final models, with the exception of PoE-5 and PoE+PW 5 which are marked with italics.
The meaning of the different hyperparameters for Table 3 is:
nhid: the size of the hidden state of the base L.TTM
dropout: the dropout value used in the base module of the LSTT
learn iter:: how many learning iterations over each batch are done before moving to the next batch
adapt. iter:: it is used in the case of PoE+PW and it shows how many iterations to train the gating weights are done for each learning iteration.
modules: how many modules does the PoE models contain
gating nhid: the size of the hidden state for the LSTM used to calculate the gating weights in the case of PoE
clear gating: is a boolean value which clears the hidden state of the LSTM used for gating weights in the case of PoE
Also, MoS was tuned following the hyperparameters: 1 or learning iterations, learning rate le- 3 5e-4. For the domain setup, considered the combinations: (nsoftmaxes=2, nhid=4750) or (nsoftmaxes=50, nhid=2620). On the other hand, for the multilingual dataset, we considered (nsoftmaxes=2, nhid=12000 100) (nsoftmaxes=75, nhid=500).
14

---

Under review as a conference paper at ICLR 2021
learn
adapt. iter.
gating nhid
clear gating
task
À
model
nhid
dropout
modules
iter.
0.1, 0.2.
lstm
200, 1300
1, 2,
0.4
10k
50, 100
MoF/PoF
200, 550
0.2
2, 5
5, 30
0,
lang.
200
MoE/PoE+PW
200, 330
0,2
2,5
1, 10, 100
5, 30
0.1, 0.2.
Istm
200, 1300
1, 1,2,5
0.4
100k
MoE/PoE
200, 550
0.2
1,2, 5
1
5, 3,30
0.
200
MoE/PoE+PW
200, 350
0,2
1,2,
1. 10, 100
5, 30
0.1, 0.2,
Istm
5200
1,2, 5
10k
50, 100
MoE/PoE
200, 1600
1,2,5
5.
0,
dom.
200
MoF/PoF+PW
200, 1600
0.2
1,2,5
1, 10, 100
5, 30
200, 1300,
0.1,
Istm
1,2,5
5200
20k
50, 100
MoE/PoE
200, 1600
1,2,5
5.
0.1
200
MoE/PoE+PW
200. 1600
0.2
1,2,5
10. 100
5. 30
Table 3: Table with the hyperparameters tested on the models: LSTM, PoE, and PoE+PW. The bold parameters are the ones chosen for I.STM, MoF/PoF-30, MoF/PoE+PW 30 and the italic parameters are the ones chosen for MoF/PoE-5 and MoE/PoE+PW
G STANDARD DEVIATIONS
MultiLingual
MultiDomain
À
10k
À 100k
= 10k
X = 20k
ppl
ppl
@ sw
rec
ppl
ppl@sw
rec
ppl
ppl@ sw
rec
ppl
ppl@ sw
rec
Ind. LSTM
0.42
0.41
0.44
0.12
0.05
0.3
28.5
25.5
0.2
17.1
16.6
0.22
Large LSTM
1.08
1.81
0.28
0.87
2.84
98.1
0.64
MoS
0.5
0.8
0.87
0.15
0.55
1.82
32.4
40.7
0.2
19.9
18.5
0.4
PoE 5
0.23
0.2
0.7
0.14
0.24
2.51
229
332
0.32
18.8
19.6
0.5
PaE.20 30
0.28
0.28
0.72
0.12
0.21
1.5
27.7
14.4
0.22
15.4
15.1
0.48
PaE+PW
0.17
0.33
0.9
0.11
0.48
3.2
26.4
22.1
0.52
15.7
19.6
0.62
PoE+PW
0.21
0.2
0.44
0.1
0.1
0.66
23.7
16.5
0.3
14.3
12.2
0.28
Ensemble
0.249
0.301
0.85
0.12
0.35
2.32
54.5
76.4
0.419
27.6
32.6
0.407
Ensemble 30
0.375
0.525
0.923
0.203
0.549
2.01
35.1
45.9
0.287
24.5
27.8
0.545
MoE 5
0.255
0.274
0.855
0.12
0.426
1.75
64.2
84.4
0.447
42.4
56
0.464
MoE
0.21
0.22
0.1
0.16
2.17
23.2
0.53
17.2
0.45
MoE+PW 5
0.264
0.377
0.806
0.101
0.322
2.36
53.2
71.6
0.36
34.4
40.4
0.445
MoE+PW 30
0.326
0.561
0.753
0.195
0.543
2.24
43.5
43.8
0.315
26.6
40.3
0.74
Table 4: Standard deviation for Average perplexity (ppl), perplexity for 10 batches after a switch (ppl @sw), and recovery time after switch in batches (rec) for both datasets per mean sequence length (A).
15


Reviews:
[{"review_id": "guotFOB741v", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes two benchmarks for continual language modeling: one evaluating character-level multilingual drift between languages which share similar characters and second evaluating word-level drift between English corpora of different domains. The setup is online in the sense of evaluation: they evaluate on the new sentences and then train over them (unlike image datasets), and catastrophic forgetting is hence characterised as having higher error than was in the past when there is a switch between the domains/languages. Hence, the loss functions measuring forgetting quantify the height and length of the rise in error. They compare a mixture of expert baselines with gating by different gating methods on this setup.\n\n\nPrimary Concerns:\n\n1. There are few  sentences and terms that are hard to understand and to me they seem imprecise. Examples would be: \n\n    (1.1) Intro: “human children still manage to acquire multiple languages without being explicitly asked to keep them separated” -- not sure if I buy this as it is known that if children are exposed to situation where there are many languages, they get confused, sometimes many kids find it hard to learn any of them, and it becomes important to give them guiding signal. Do you have any reference to support this hypothesis?\n\n    (1.2) Section 3, second para: “preventing data leakage”: what do you mean by data leakage?\n\n    (1.3) Section 3, third para: hard to follow, notation isn’t clear. And it seems there is a typo in S_i = \\sum_j T_i.\n\n    (1.4) Section 3, fourth para: “for a model to be resilient to forgetting, it must adapt quickly”: this statement is not correct because if a model adapts quickly to a new distribution, the parameter change would lead to forgetting and that’s primary the reason why there are regularization based approaches for continual learning enforcing models to be in the vicinity of old parameters. Too much adaptivity does not ensure less forgetting.\n\n    (1.5) Section 3, loss after switch: what do you mean by a switch? How do you know when a switch happens (task label is not given)? In practice the loss curve is not smooth. How do you identify the switch? Fig 1 (a) is too smooth, does not represent the real loss curve. \n\n2. Regarding experiments, is it not possible to design much simpler methods which work for this problem? If it's known there is expected to be a character/word-sequence distribution shift, I believe it's likely they can be detected easily with traditional n-gram models and style distinguishing attributes typically used for author identification [1,2]. Why isn't it possible to use a baseline which consists of experts for one domain/language where the character-sequence decides which expert to use instead of these weaker gating-based methods? Also, English/czech/german/french seem very distinguishable and share little in common in terms of character sequences [3], hence I am doubtful of the finding that combining these models will improve any single language performance.\n\n3. Why is it not possible to apply traditional continual methods like Experience replay to this setting-- you simply store intelligently selected past sentences in memory (when say error shoots up) and replay using them. There are many other continual learning approaches that potentially could be applied here. Any particular reason for not using them?\n\n[1] Koppel et. al., Computational Methods in Authorship Attribution\n[2] Sapkota et. al., Not All Character N-grams Are Created Equal: A Study in Authorship Attribution\n[3] Gerz et. al., On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling (edited) \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MRKIunxT0M", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#############################################################################################\n\nSummary:\n\nThis paper introduces a dataset and benchmark for language modeling in the online continual learning framework. The key characteristics of this benchmark are: the data are temporally correlated, there are no task identifiers presented to the model (task-free setting), and the evaluation is performed in an online fashion.\n\nThe benchmark consists of a Multilingual character-level dataset and a Multidomain word-level dataset. The authors introduce several metrics and evaluate using several simple baselines of mixtures-of-experts and products-of-experts. \n\n#############################################################################################\n\nPros:\n\n1.\tThe paper is clear and well-written. \n2.\tThe authors provide sufficient details on data collection and modeling\n3.\tThe relevant work section is extensive\n4.\tThe design choices in constructing the dataset are well thought out and make sense given the objective of the paper. In particular, the dataset along with the proposed evaluation metrics captures the three stated objectives of the benchmark. \n5.\tThe authors are upfront about materials left out of the main text. It’s nice when potential questions are anticipated and answered, for example, “why weren’t continual learning SOTA models evaluated?” and “why weren’t transformers considered as baselines?” The authors answer these questions candidly. \n\n#############################################################################################\n\nCons\n\n1.\tThe dataset seems incremental over existing work\n2.\tThe introduced evaluation metrics are described intuitively, but are not analyzed empirically or theoretically \n3.\tThe necessity/value of the introduced dataset is not adequately justified in relation to existing challenges in the continual learning setting. A component of this is showing where existing models fail (and why this dataset will help improve them). \n\n#############################################################################################\n\nRecommendation and explanation\n\nI recommend rejection for the previously outlined reasons. \n\n#############################################################################################\n\nI also have some questions that I hope the author can help address:\n\n1. What is the key innovation over existing work such as d’Autume et al. who also study language models in the continual learning, task-free setting?\n2. What failure of current models does this benchmark address? Note that the answer to this question should also be empirically demonstrated. \n\n#############################################################################################\n\nAdditional feedback\n1. This benchmark could very well be a valuable contribution that fills a hole in the existing body of work, but the paper in its current form does not adequately establish this. The rebuttal should better address how this benchmark fits into existing work by comparing it to existing datasets and more relevant baselines.\n2. The paper as a whole is well written, but I question some of the choices in syntax: terms such as “demarcation” and “desideratum” are spirited but may be better replaced by plainer alternatives.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AF__a-kCNc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\nThis paper proposes a new evaluation framework and gives two available evaluation datasets\nWeakness：\n- the paper needs a major rewrite to improve fluency and to better state motivation and contribution\n- the empirical validation is weak.\n\nReasons for accept:\nThe advantages of this paper are: \n1)\tthis paper proposed a new evaluation benchmark and dataset to promote the related research of online continual learning; \n2)\tthe proposed plastic gate allows it to distribute different distributions among different experts, which has certain effects from the experimental results. \n\nReasons for reject:\nThe shortcomings of this paper are:\n1.\tThis paper is not enough novel and has not contributed enough to continual learning related research; \n2.\tThe core motivation of this paper is not clear enough. The abstract mentioned that \"it is hard to demarcate task boundaries in actual tasks\", and then said that a new benchmark, new metrics, and gating technique are proposed. Stacked statements like this can hardly capture the main problem to be solved.\n3.\tThe advantages of the new metrics are not clear. Because from the experimental results, PPL and PPL@sw have a strong correlation. Therefore, please explain its advantages in detail (including the advantages of this evaluation framework compared with the evaluation framework of related literature, and verify it)\n4.\tThe baseline uses LSTM and does not use CNN, Transformer, .etc, which shows that its generalization is limited.\n5.\tCan you provide the experimental results when λ is other values, and the combination of the number of modules？\n6.\tBecause what you are proposing is a continuous language modeling evaluation framework. Is it possible to evaluate some of the latest online continual learning systems? \nFor example：\n1) Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis\n2) Learning a Unified Classifier Incrementally via Rebalancing\nOr other Task-Free Continual Learning related work. This will have a good evaluation effect on measuring the versatility of your evaluation framework. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1xNlev0BOds", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper’s main contributions are (i) to propose two new benchmarks for online continual learning in the context of language modelling and (ii) evaluate the performance of a number of composition-of-experts-based models on the new datasets using a number of metrics. The multilingual benchmark, derived from an existing multilingual news corpus, consists of sequences of characters where the language is periodically switched, and the MultiDomain benchmark consists of sequences of English words where the corpus is periodically switched. The comparative performances of the various baselines on the two datasets, as well as an analysis of the mixture weights in one of the models during training, are used to provide insights into the qualitative differences between the datasets.\n\nOverall, I am inclined to recommend acceptance for this paper on the margin because it makes a good contribution towards evaluating continual learning models in more real world settings, more specifically in the context of online learning. The datasets proposed are well-suited for purpose for reasons outlined below, and the evaluation using various composition-of-experts models is fairly conducted and followed up with an informative analysis. The key downside of the paper is that no standard continual learning baselines are trained on the proposed datasets; I would be inclined to increase my score if results were shown for 1 or 2 algorithms specifically designed for continual learning with neural networks (as discussed in more detail below).\n\nPositives:\n\t•\tThere is a need to start evaluating continual learning in closer-to-real-life settings; in providing datasets that facilitate evaluation of continual learning models in an online setting without task boundaries, this paper makes a positive contribution in this direction. \n\t•\tThe datasets are simply composed, but seem well suited for evaluating online continual learning because (i) language data is sequential, (ii) by imposing a truncated exponential (and thus memoryless) distribution on the length of subsequences, it is hard for models to cheat in predicting the next task switch, preserving task-agnosticity, and (iii) in both datasets, the subtasks share latent similarities, creating the possibility for forward/backward transfer between them. \n\t•\tThe analysis of the experiments provides interesting insights into the datasets and differences between the baselines. E.g. Figure 1d effectively shows how the weights of one of the Product of Experts models switch after a task change, indicating a degree of specialisation of the modules, and 1e uses the correlations of the mixtures weights used for different subtasks to highlight the latent similarity between pairs of subtasks. \n\t•\tThe paper is clearly written and easy to follow. \n\nMain Concern\n\t•\tLimited set of baselines. While a range of composition-of-experts baselines are used for evaluation, it would have been much better to also include other methods specifically designed for online continual learning, such as those cited in the paper [1, 2] or, though not strictly online, a replay-based method such as CLEAR, which works in the task-agnostic setting. It is claimed in the paper that including state-of-the-art online continual learning methods would have involved “non-trivial adaptations significantly departing from the original models, which would limit any possible conclusions we could draw” as they are designed for image-based datasets. I don’t fully understand the basis of this claim; perhaps the authors could elaborate - as far as I am aware, for example, [1] is not restricted for use on image-based datasets. \n\t•\tSince the subtasks do have discrete boundaries, even though these are not passed to the model during training, it would be possible to evaluate methods that use task boundaries for consolidation on the proposed datasets by either providing knowledge of the boundaries (although this breaks the task-agnosticity) or by using methods that can detect task boundaries - e.g. EWC uses the Forget-Me-Not Process [3]. \n\t•\tOverall, not evaluating the datasets with any standard continual learning baselines is an important weakness. \n\nOther comments\n\t•\tThe proposed method, plastic gates, which performs best amongst the baselines used when combined with product of experts models, seems simple and effective but I am inclined to question how novel it is, since it just amounts to multi-step online gradient descent on the mixture weights. \n\t•\tThe metrics used for evaluating continual learning, loss after switch and recovery time after switch, which are one of the main selling points of the paper are suitable for the datasets provided, but would not be applicable in a setting where either the task boundaries are not known or there are no hard task boundaries to be identified. \n\t•\tTypo Section 2 Paragraph 2: “MNNIST” -> “MNIST”", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews were mixed for this paper. On one hand, some of the reviewers highlighted that the proposed datasets could be useful to researchers. On the other, reviewers found a few important flaws with the current manuscript including missing baselines, issues with the proposed tasks, and possibly inaccurate/imprecise statements.

Our discussion after the author's response focussed on whether the positives aspects of the current paper outweighed some of the perceived weaknesses of the paper. In particular, while some of the initial criticisms from the reviewers were successfully addressed by the authors (including possible imprecisions and to a certain extent motivation), all the reviewers remained convinced that standard continual learning baselines could be adapted to this setting. They also conjectured that these missing baselines might not allow readers to appreciate the strength of the proposed datasets. 

In their response, the authors argued that adapting models would require research. The reviewers are under the impression that it would be useful to test baselines more or less "as-is" even if the authors do not think these baselines will be competitive. For example, in the discussion, a reviewer suggested that "an experience replay baseline could [...] have been implemented" where the replay buffer includes the hidden states of an LSTM. It might also be useful to study baselines that do not strictly obey the proposed setting, again to get a better understanding of the proposed tasks (including how difficult it is).  Overall, having some of these baselines would be one way to better connect the proposed work to the current continual-learning literature. 
Decision: Reject