Improving Language Plasticity via Pretraining with Active Forgetting
Authors: Yihong Chen
Kelly Marchisio
Roberta Raileanu
David Ifeoluwa Adelani
Pontus Stenetorp
Sebastian Riedel
Mikel Artetxe
Venue: NeurIPS 2023 poster (2023)
Keywords: plasticity
continual learning
meta-learning
embeddings
cross-lingual transfer
forgetting
TL;DR: Pretraining a language model with active forgetting imbues it with more plasticity and makes it adapt to new languages faster in low-data scenarios.

Abstract: Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning a new embedding layer for the new language, doing so is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Concretely, by resetting the embedding layer every K updates during pretraining, we encourage the PLM to improve its ability of learning new embeddings within limited number of updates, similar to a meta-learning effect. Experiments with RoBERTa show that models pretrained with our forgetting mechanism not only demonstrate faster convergence during language adaptation, but also outperform standard ones in a low-data regime, particularly for languages that are distant from English. Code will be available at https://github.com/facebookresearch/language-model-plasticity.

Improving Language Plasticity via
Pretraining with Active Forgetting
Yihong Chen T8
Kelly Marchisio 8 Roberta Raileanu
David Ifeoluwa Adelani
Pontus Stenetorp" T
Sebastian Riedel
Mikel Artetxe
UCL Centre for Artificial Intelligence Meta AI *Reka AI
"Cohere AI
(yihong, chen, d.adelani, p. p.stenetorp, s.riedell (as ucl. . ac. uk raileanuêmeta, com
mikel@reka kelly@cohere.co com
Abstract
Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning new embedding layer for the new language, doing is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as simple way of creating PLMs that can quickly adapt new languages. Concretely, by resetting the embedding layer every K updates during pretraining, encourage the PLM to improve its ability of learning embeddings within limited number of updates, similar to meta-lcarning effeet. Experiments with RoBERTa show that models pretrained with forgetting mechanism not only demonstrate faster convergence during language adaptation, but also outperform standard ones in low-data regime, particularly for languages that are distant from English. Code will be available at httphhttss:/ttttiii com/ setebookkeeearrc//laaggge
model plasticity
Introduction
Pretrained language models (PLMs) have been swiftly reshaping the landscape of natural language processing (NLP) by improving upon standardized benchmarks across the board [Radford and Narasimhan 2018 Devlm al. 2019 Liu et 2019 Brown al., 2020 At their core, they acquire knowledge ingesting large datasets and store this knowledge in their parameters during pretraining. Using finetuning or prompting (Brown et al. 2020). such knowledge can then be applied to downstream applications, such as semantic analysis, question answering, and others.
Despite their success, PLMs still have a number of shortcomings [Weidinger al. 202
2022 In
particular, it requires massive data and computation to pretrain them Gururangan et al. 2020 Kaplan
al. 2020, Hernandez al. 2021 Hu et al. 202T Touvron all 2023 Naively retraining new
PLM to accommodate every lingual space shilt would be prohibitively expensive. This makes it
highly relevant research target to create PLMs that can be efficiently adapted to new lingual spaces.
While forgetting in the context of both human and machine learning is often perceived as some- thing negative (for instance catastrophic forgetting (McCloskey and Cohen 1989 Ratclifi 1990 Kirkpatrick et al. 2017)), recent works have shown that for artificial neural networks forgetting
Wee use the term lingual space shift to describe changes in language usage between pretraining and the target downstream application, caused by factors such as language change, time evolution, domain switch. model with high language plasticity would quickly adapt these shifts.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

can also play positive role in increasing their "plasticity", such as improving generalization to 2022 Ig et al. 202T], enabling learning in low-data 202 laha et al. 2021]. or counteracting primacy bias (Nikishin
unseen data [hou et al.] 2022
Chen al.
regimes Allbduulmohsin et al.
et al. 2022 1 Dro et al. 2023]. Given these developments, in this work we ask whether we can draw
upon forgetting as mechanism to improve pretraining and imbue PLMs with similar benefits.
It is well established in the NLP community that models struggle to generalize across languages without substantial intervention [Conneau
et al. 2020, Pfeiffer
2020
2022 Ansell et al. 2022]. which
is especially true for low-resources languages. We thus see this as promising testing ground for forget- ting techniques. Our focus is on the input layer of the PLM, the to- ken embedding layer, as learning it has been shown to be highly ef- fective when adapting between lan- guages [[rrtetxe et al. 2020
Lirual Rosce
Peetran
Now Lingual Space1
Lingual Space
Ligual
Lingust
trandlormer body
hazon
tskan embesdiing lyver prozaiseed in Englat
sakan embocdingg layer adapted now language
protisined
companent
Figure I: Rewiring via relearning token embeddings: where the transformer body (the purple part) is "frozen" and reused for a new language, but the token embeddings are relearned to suit the new language.
Concretely, we introduce an active forgetting mechanism, that resets to- ken embeddings at regular intervals, while leaving all other parameters untouched throughout pretraining. We study whether this forgetting approach creates PLM that can easily rewire (Figure Th to an unseen (possibly distant) language. Intuitively, resetting embeddings forces the transformer body to re-derive reasoning each time instead of relying on memorized short- cuts. Through repetition, the body learns more abstract, high-level reasoning. A model with greater abstraction can easily transfer across languages, since high-level reasoning is more language-agnostin
Our zero-shot evaluations on several cross-lingual transfer benchmarks show that for cases where unlabeled adaptation corpus for the unseen language has few as million tokens (a low-data regime), forgetting PLMs outperforms the baseline by large margins: average gains of +22.2% on XNLI, 3.% on MLQA, and +-6.9% XQuAD. In addition, models pretrained using active forgetting converge faster during language adaptation. Finally, we find that forgetting especially beneficial for languages that are distant from English, such Arabic, Hindi, Thai, and Turkish.
2 Rewire PLMs for New Languages
Using unlabeled data, Artetxe et al. 2020 demonstrates the possibility of rewiring a monolingual PLM for new language; they propose to relearn the embedding layer for the new language while keeping all the other parameters frozen. The underlying assumption that the token embedding layer and the transformer body (the non-token-embeddinn parameters) divide up the responsibility in way that the former handles language-spest general reasoning. Hence, rewiring English PLM for new language boils down to separately adapting the former with unlabeled data in the new language and the latter with English task data. The procedure can be summarized as follows:
lexical meanings, while the latter deals with high-level
1. Pretrain: A transformer- -hased model pretrained on an English corpus. our experiments, choose to pretrain RoBERTa-baas Liu et al. [2019] 12-layer transformer-basee model, on English CC100 [Conneau et al. 2020).
2. Language Adapt: The token embedding layer is finetuned using unlabelled data in the new language, while the transformer body is frozen.
3. Task Adapt: The transformer body finetuned using downstream task data in English, while the token embedding layer is frozen.
4. Assemble: The final model is assembled by taking the adapted token embedding layer from stage 2 and the adapted transformer body from stage 3.

---

QA
Task Adapt
QA
start
forget!
forget!
Assemble
Pretrain
Language Adapt
transtormer bady rendomly inttelleed
transformer body pretrained with MLM
a,
transformer body adapled specilic task
frozen component
taken embedding layer randomly inttaiised
embedding English
tokan embedding layer edepted language
gradient updates
in the pretrain stage, compare standard pretraining with forgetting pretraining. where the token embeddings are actively forgotten at regular interval while the transformer body Iearned as the standard pretraining, Middle: the task adapt stage and the language adapt stage separately adapt the transformer body using English task data and the token embeddings using unlabeled data in the new language. Right: the assemble stage reassemble the adapted body and token embedding layer into usable PLM.
Figure 2: Unsupervised zero-shot cross-lingual transfer. Left:
2.1 On The Difficulty of Rewiring PLMs via Relearning the Token Embeddings
While the above procedure AAtetxe et al. 2020 offers general framework for rewiring mono- lingual PLM with unlabelled data in the new language, it is unclear how efficient such rewiring can be, including both sample efficiency and computational efficiency. To better understand the difficulty rewiring PLMs via relearning the token embeddings, we design an experiment where relearn the token embedding layer using varying amounts of adaptation data. For illustration purpose, we pick English as the pseudo "adaptation language" because the English dataset is large enough bootstrap series of sub-datasets with varying quantity. We create sub-datasets with [1K, 10K, 100K, 1M, 5M, 10M, 100M, IB, 10B] tokens and relearn the English embeddings while keeping the transformer body frozen.
The dashed blue line in Figure summarizes the influence of the adaptation data quantity on the qual- ity of the rewired PLMs (relearned embeddings assembled with the En- glish NLI task body). We that the standard PLMs are easy to rewire if there is enough adapta- tion data. However if the adapta- tion corpus contains fewer than 10 million tokens, the performance of the rewired standard PLMs (the blue dashed line in the figure) drops dras- tically as the adaptation data quan- tity goes down, from near 80 to around 35, a random- -orlosiigg level for NLI tasks. This motivates us to create more rewirable PLMs, i.e. PLMs with more plasticity so that the rewiring process can faster and consume less data.
Influence Adaptation Data Quantity
forget
standard
English XNLI Accuracy
70-
60
50
40
104
109
10a
10
Tokens the Adaptation Corpus
Figure 3: The rewiring performance for standard PLMs (blue dashed line) drops drastically if the adaptation tokens S 10M.

---

3 Pretrain Easy-to-Rewire PLMs via Active Forgetting
Recent works have shown that incorporating forgetting through iterative weights resetting can increase the "plasticity" of neural networks, enabling them to learn from small data and generalize better to unseen data supervised learning (Alabdulmohsin et al. 2021 Taha et al. 2021 Zhou et all 2022]. Building on these efforts, we study if we can bring such forgetting into the pretrain stage so that the resulting PLM would have more plasticity, allowing easier rewiring to new languages.
Our Hypothesis. In effect, when Artetxe et al. 2020 relearned the token embedding layer, the reinitialization of the embeddings can be seen as forgetting applied once at the start of the language adapt stage. However, the PLM (specifically the transformer body) has never encountered forgetting before this stage and may struggle to handle this new situation. Without early exposure to forgetting, the PLM might suffer from slow recovery caused by forgetting before eventually benefiting from it. The learning of a new lexical embedding layer in PLM henceforth consumes lots of data in new languages along with long training horizons as shown in Section 2.l In this paper, ensure swift learning of the new languages with both high sample efficiency and convergence rate, we argue that the PLM must be exposed forgetting during pretraining, allowing itself to maximize the positive impact of forgetting and minimizing the cost of recovery.
Our Method. With this hypothesis in mind, we propose to add an active forgetting mechanism to the pretraining procedure, which resets the token embedding layer periodically as described in Algorithm| ! Concretely, the forgetting mechanism operates by intentionally clearing the weights of the embedding layer, which stores the static representations for all tokens, and reinitializing them to new set of random values every K gradient updates. Since pretraining involves advanced training strategies, like optimizers with states and learning rate schedulers, we also reset them together with the token embedding layer. We refer to language models pretrained with such active forgetting mechanism as forgetting PLMs, in contrast to standard PLMs which are pretrained in standard way. The pretraining loss curve of a forgetting PLM is episodic (Figure D. like in reinforcement learning or meta-learning This episodic learning demonstrates that the active forgetting mechanism can introduce diversity without requiring actual new data. Each forgetting event kind of "branches out" novel environment for the model to explore, as if initiating new episode of learning.
Research Questions. To further examine the proposed forgetting mechanism, we compare forgare ting PLMs and standard PLMs on sample efficiency and convergence speed during language adapt, two key aspects of model plasticity. Our research investigates:
RQ1: Real-world low-resource languages often have scarce data for adapting models. Does
pretraining with active forgetting impart enough plasticity to forgetting PLMs, enabling them learn new languages even with such limited data?
RQ2: Deploying PLMs frequently encounters computational limitations. Endowed with more plasticity, can forgetting PLMs reduce adaptation time for such low-compute scenarios?
RQ3: New languages may be very similar or different from pretraining languages. Does this similarity/differeece impact the relative benefit of forgetting PLMs over standard PLMs?
4 Evaluate Forgetting PLMs for Unsupervised Cross-Lingual Transfer
To evaluate the effectiveness of forgetting PLMs and address RQ1-RQ3, we conduct experiments on several cross-lingnns lonsull transfer benchmarks.
4.1 Experimental Setup
In our work, we closely follow the setup in Artetxe et al. 2020 and Marchisio al. 2022]_ Our pretraining model is RoBERTa-bass a standard 12- 2-layer transformer based language model. We trained language-speciffic sentencepiece tokenizers Kudo and Richardson 2018 with a vocabulary size of 50K over the corresponding data subsets in CCT00. The model was pretrained with the English subset of the CC-100 dataset. The pretraining process consists of 125K updates, with a batch size of 2048. We used a learning rate scheduler with linear decay and an initial learning rate of 7e 4, with

---

Training Curve
Standard Pretraining
Forgetting Pretraining
raining Loss
Training Curve
Step
20k 40k 60k
80k
100k
120k
Figure 4: Pretraining loss curves of forgetting and standard language models. The forgetting mechanism brings an episodic pattern into the loss curve: every embedding forgetting produces loss spike, from which the model learn to recover. Through such repeats of forget-relearn, the model gets used to learn new embeddings from scratch.
Algorithm Active forgetting mechanism. The token embedding layer is reset every K updates.
Input: K, interval between two consecutive forgetting; Phody/emb> current effective number of updates for the body the token embedding layer; Obody/emb, current learning rate for the body the token embedding layer; hody/emb- parameters after the th update for the body or the token embedding layer; Obady bbody/emh- optimizer states after the nh update for the body or the token embedding layer, θ, randomly initialized embedding parameters, each element drawn from (0, 0.02); f. the function that computes the gradients the parameters using the sampled data; g. the function that updates the parameters based the gradients (e.g. one step Adam optimizer) 8, the function that updates learning rate (e.g. one step in the polynomial learning rate scheduler).
Output: the updated parameters and optimizer states p(x) (Pin) Pondy }. O (0(m) O(?) body
emb
I: nemb = mod K
2: Obody ((Abooy)) [adjust learning rate based on r1.
3: Oeemb ((Demb))
4: G(n) [(p(a=1),, .) (compute all gradients]
5: body: body g(G p(n-1) 1) (n-1)
(m)
hody+ body Dody (Ohody, Tt) (update the transformer body
6: if Temn
0 then
7:
p(n)
θ periodical reset token embeddings and the relevant optimizer states)
umb
8:
o (n-1)
Gemb
1)
0
9: end if
10:
p(n)
anh (((()) p(n-1) (n-1)
omh: 2ab Ogmby Teemb) (update the token embeddings)
5

---

Method
vi
sw
65
bg
de
fr
el
ru
Standard
65.8
55.6
68.0
65.5
62.2
63.5
63.1
56.9
Forgetting
62.8
59.5
74.0
71.7
68.5
71.2
70.8
65.8
Relative Gain
- -4.6%
7.00%
+8.8%
+9.5%
+10.1%
+12.1%
+12.2%
+15.6%
Table 1: Accuracy comparison of forgetting and standard PLMs on XNLI (table continues).
Method
zh
hi
tr
ar
th
Avg
em
Standard
53.2
36.8
39.7
38.9
41.2
35.3
53.3
86.1
Forgetting
63.5
45.8
52.9
52.7
59.5
59.7
62.7
85.1
Relative Gain
+19.4%
+244.5%
+33.2%
+33.5%
+44.4%
+ +69.1%
+21.2%
-1.2%
Table 2: Accuracy comparison of forgetting and standard PLMs on XNLI (table continued). On average, forgetting PLMs achieve 21.2% relative gain in accuracy compared to standard PLMs across the languages tested, where averaged relative gain
Exe (lanssgss Relative Gain of #languages
10K warm-up updates. Checkpoints were saved every 500 updates and we always choose the last pretraining checkpoint where possible for optimal performance. For forgetting pretraining, we chose the checkpoint corresponding to the best validation perplexity since the last checkpoint might have token embeddings reset. We set the frequency of forgetting K = 1000 and used a clip-norm of 0.5.
During the language adapt stage, we kept most the finetuned the token embedding layer while keeping the others frozen, as described in Section Note that no forgetting happens during this stage because we want the models learn the new languages as well as possible. In the task adapt stage, both models were finetuned for 10 epochs on the English task data, specifically MultiNLI (Willlams et al. 2018 for the NLI task and SQUAD (Rajpurkaa et al. (2016) for the QA task. After the assemble stage, we evaluate the zero-shot performance of the assembled model on XNIII (Conneau et al. 2018 a cross-lingual III task, along with two cross-lingual QA tasks. We report
the same as for pretraining. We
XQuAD [Artetxe et al. 2020 and MI.QA [Lewis et al. 2020
the NI.I accuracy and QA Fl on the test sets.
Our experiments were implemented using fairseq Out et al. 2019 The pretraining and language adaptation experiments were conducted on 32 Tesla V100 GPUs (each with 32 GB memory) and took approximately 24-36 hours to complete. The time taken for both stages were quite close to each other even though the latter only involved tuning the embeddings. This demonstrates the importance of reducing the computational cost of the language adaptation stage.
focus language adapt in low-data regimes. We simulate low-resources scenarios by limiting the adaptation data for each downstream language to only 5M subword tokens from CC100. This is in contrast with conventional setups, where all the tokens in the corresponding languages in CC100 are used for language adaptation. As Table shows, such setups consume several orders magnitude more data than our 5M-token setup: for instance, the Swahili CC100 subset contains 345M tokens, roughly 69 times larger than our corpus, and the Russian subset contains 34.9B tokens, roughly 6, 980 times larger. Therefore, PLMs that can successfully learn new languages with rich data under traditional setups may struggle do so with our limited 5M-token corpus.
Differing from prior work (Atetxe al. 2020 Marchisio et al. 2022
4.2 Forgetting PLMs Work Better in Low-Data Regimes (RQ1)
Standard PLMs struggle in low-data language adaptation, dropping from 86.1 English NLI accuracy to just 53.3 average accuracy on XNLI with limited 5M token adaptation data. Compared to prior work which uses full data from Wikipedia [[ttttx et al. 2020 or from CC100 [Marchisio et al. 2022 the average accuracy on XNLI drops about 18% (from 66.8/66.3 to 53.3). This indicates standard PLMs are not coping well with the low-data regime. In contrast, forgetting PLMs achieve decent 62.7 average XNLI accuracy, a a +21.2% relative gain over standard PLMs, as shown in Table 2
Forgetting PLMs also outperform standard PLMs MLQA and XQuAD, with average Fl relative gains of +33.8% and +60.9% across languages, as respectively demonstrated in Table 3 and Table Across NLI and QA tasks, forgetting PLMs consistently surpass standard PLMs in low-data regimes.

---

Method
es
vi
de
zh
ar
Avg
en
Standard
49.4
38.3
45.3
34.1
17.7
20.8
34.3
78.9
Forgetting
55.3
45.0
53.4
43.0
28.8
34.7
43.4
78.3
Relative Gain
+
12.0%
+17.6%
+17.8%
+26.2%
+ +62.5%
+67.0%
+
+33.8%
-0.8%
Table 3: Fl-score comparison of forgetting and standard PLMs on MLQA. On average, forgetting
PLMs achieve 33.8% relative gain in Fl compared to standard PLMs across the languages tested, 2. (lagroges) Relative Gain of
where averaged relative gain
# #Tanguages
Method
vi
65
Fu
de
xh
hi
ar
th
tr
Avg
Standard
49.7
57.7
49.4
50.9
48.5
32.4
21.4
22.2
15.4
13.0
36.1
Forgetting
52.9
64.6
56.5
60.9
59.9
43.7
33.3
38.7
38.4
41.4
49.0
Relative Gain
+66.%%
+12.0%
+14.5%
+19.7%
+.23.6%
4-34.0%
+55.8%
+74.2%
+149.7%
+-1888%
+-00.9%
Table 4: Fl-score comparison of forgetting and standard PLMs XQuAD. On average, forgetting PLMs achieve a 60.9% relative gain in Fl compared to standard PLMs across the languages tested, Σx [Innesags Relative Gain
where averaged relative gain
#lanuugges
Why do forgetting PLMs handle the low-data regime better? We hypothesize this is because forgetting PLMs are more robust to different embedding initializations. They encode more universal know wedde in the transformer body. Standard PLMs may encode more "shorteut" knowledge relying on certain embedding initializations. In low data, standard PLMs cannot adjust embeddings towards shortcut routes without access to enough data. Forgetting PLMs do not rely on shortcuts so perform better.
4.3 Forgetting PLMs Learn New Languages with Fewer Parameter Updates (RQ2)
We are also interested in how quickly forgetting PLMs and standard PLMs can learn new languages. Figure 3 summarizes adaptation curves XNLI, MLQA and XQuAD, with each point representing the averaged performance across all languages. In just 5K steps (4% of full adaptation), forgetting PLMS reach 57.8 accuracy XNLI while standard PLMs struggle at random guessing levels of 37 22 Similar trends hold for MLQA and XQuAD. After 5K steps, forgetting PLMs achieve 92% of their full performance XQuAD versus just 53% for standard PLMs (see the last plot in Figure 5p.
Why do forgetting PLMs converge faster? We hypothesize is because periodical embedding resetting forces the body to gradually locate itself on particular manifold, where it can easily cooperate with new embeddings. This makes the body encourage larger embedding updates when adapting to new languages. Active forgetting simulates language switching during pretraining introducing diversity without new data. This allows faster adaptation to real new languages.
4.4 Languages That Are Distant To English Benefit Most From Forgetting PLMs (RQ3)
Up to this point, we have primarily presented the averaged performance. In this section, we delve into a detailed comparison of language-specific performance between forgetting PLMs and standard PLMs on XNLI, MLQA, and XQuAD. To gain deeper insight into which languages benefit the most from the use of forgetting, we present the relative performance changes in Figure 3 for XNLI and MLQA. The results for XQuAD can be found in Figure S in the appendix. Across the spectrum of languages (Table 5 we observe that forgetting provides greater benefits for languages distant to the pretraining language (English) in terms of language family, script and morphology. Specifically, forgetting brings large relative gains for languages such as Arabic, Hindi, Thai, Turkish, and Urdu compared closer languages like German. Script seems important forgetting helps Vietnamese and Swahili less despite their distance from English, likely due to the shared Latin script. Examining adaptation curves within the first 5K steps, forgetting PLMs reach substantially superior performance over standard PLMs for almost all languages except Urdu, while standard PLMs struggle at random guess levels (see Figure / and Appendix D). This demonstrates forgetting PLMs' ability to efficiently adapt to new languages, particularly dissimilar ones, in low-data settings.
PPrecisely, it simulates vocabulary swappings, causing drastic changes to the input of the body.
7

---

XNLI Acturacy ve Adaetat on Stops
XQUAD Adaptation Simps
ferget
standard
standurd
Sondard
20000
62000
150000
13
23820
53200
100003
13
20033
23200
100020
XNU Arcuracy us Adeptstion Steps First SK Steps
MI.AA vs Adeptation Steps (ftrst SK Steps]
XQUAD F1 vs. Adaptarion Steps [First 5K Steps]
leggel
32-
16-
35
1602
20:0
1032
5020
1052
2026
5320
1030
2300
3006
1020
5200
Adierstien Stepe
Adactation Stroe
A.Atstiiton Stace
Figure 5: Adaptation curves XNLI, MIQA,, and XQuAD. Numbers aggregated across languages. The first row contains the full adaptation curves, which comprises 125K adaptation steps. The second row contains the zoom-in versions of curves for the first 5K adaptation steps. Forgetting PLMs converge faster than standard PLMs; for instance, on XQuAD (the last plot), forgetting PLMs reach 92% their final performance within 5K updates, while standard PLMs only reached 53% of their final performance at that point.
Relative Gain Across Languages XNLI
Relative Gain Across
Languages MLQA
Accuracy XNU
MLQA
40-
40
20
Forget
Forget
0
Rolative Gain (%)
50
40-
25
Relative
20
bg fr
2h
hl
th
de
zh
hi
Language
Language
Figure 6: Relative gains of forgetting PLMs over standard PLMs across languages. Forgetting yields substantial relative gains for languages like Arabic, Hindi, Thai, Turkish, and Urdu. However, for languages closely related to English, such German, the relative gains from forgetting modest.
5
Related Work
5.1
Forgetting and its Positive Roles
The common perception of forgetting is that it implies weak memory and loss of acquired knowledge, thus is often regarded as a sign of un-iiteelligence or an undesirable property. In neural networks, catastrophic forgetting (McCloskey and Cohen 1989 Ratchifi 1990 Kirkpatrick et al. 2017 is portrayed as a forgetting phenomenon where neural networks lose the ability to predict old patterns after new inputs alter their weights. Forgetting in this context has negative consequences, knowledge overwrites the old. Plenty of prior research strives to overcome catastrophic forgetting and enable continual learning (Schmidhuber 2013 Kirkpatrick et al. 2017
the new
Lopez- Paz and Ranzato Parisi et al., 2019 Rolnick
2017 Shin et al. 2017 SSchwarz et al. 2018 Mallya and Lazebnik 2018
al.
2019 Beaulieu al. 2020 Veniat al. 2020 Gaya al. 2023
Khetarpal al. 2022
Our work differs from the above ones in that our subject intentional forgetting rather than passive forgetting and its associated negative impact. To put in another way, we seek to understand how

---

torget
38
forget
50
stanclard
standard
60
37
MMLL Accurac
55
XMA Acc. 2P33
26
150
forget
standard
35
80
34
35
32
Adappaaiion Steps
Acaplation Steps
Adaptation
UR
forget
standlard
standard
standard
XOU Accurasy
4.5
XOU Accumiiy
40
XHU Accuracy
42
37
36
1000 2000 3000 4000 Adaptation Stops
5000
1600 2000 3000 4000 5000 Acaptation Stops
1000 2000 3000 4000 5000 Adaptation Steps
Figure 7: Adaptation curves on XNLI within 5K updates for individual languages: Bulgaria, Greek, Spanish, French, Russian, Swahili, Vietnamese and Chinese. For all languages except Urdu, the forgetting PLMs converge faster than the standard PLMs during the language adaptation stage.
forgetting if purposely incorporated an active process during training - might help new learning. Similar positive roles of forgetting have been discussed in the literature. Specifically, Pastötter et al. 20088 demonstrate forgetting enhances the learning of new information by resetting the encoding process and holding the attention at high levels; Levy et al. [2007] show that it helps second
language acquisition by inhibiting the native language; [Brrettt and Zollman| (0099] find it promote the emergence of an optimal language by preventing partial success from reinforce sub-optimal practice. Nørby (2015) further suggests forgetting serves adaptive functions, helping people regulate emotions, acquiring knowledge and staying attuned to the context. More recently Anderson and
Hulbert [021] reviews evidence on active forgetting by prefrontal control and shows how it can adapt the memory to suit either emotional cognitive goals.
5.2 Forgetting Via Partial Neural Weights Reset
In neural networks, forgetting be instantiated in many forms. A simple way is to reset subsets of parameters before the next round of learning. Iterations of such resetting have been shown to benefit generalization with low compute and low data for computer vision tasks [Irankle and Carbin 2019
Alabdulmohsin et al., 2021 Taha et al. 2021 Ramkumar et al. 2023). More recently, Zhou et al.
2022 demonstrate simil lar forgetting strategy helps image classification and language emergence. Closely linked to our method, Chen et al.) [2022) forget node embeddings in order to truncate infinite message-passing among nodes and thereby aid new graph reasoning with nodes. Our work uses similar forgetting mechanism over token emheddings, improving new language reasoning with new tokens. As far we know, we are the first to bring forgetting into pretraining and demonstrate that forgetting pretraining boosts linguistic plasticity. A relevant thread in reinforcement learning (RL)
research studies the plasticity loss phenomenon Lyle et al. 2023 Nikishin et al. 2023 Recent work explores similar forgetting approaches to improve plasticity. Ig et al. [021] periodically reset the current policy by distilling into a reinitialized network throughout training. Intuitively, this releases network capacity storing suboptimal policies and opens up space for the the vet-to-be-discoeer optimal (final) policy. Simpler methods just reset an agent's 's last layers [Nikishin et al. 2022). preventing overfitting to early experiences and primacy bias. Resetting parameters also improves sample efficiency by allowing more updates per environment interaction (D'Oro al. 2023
5.3 Making Pretrained Language Models Multilingual
Pretraining on multilingual data makes PLMS multilingual (Coneeau et al. 2020 but has downsides like needing large multilingual corpus with appropriate mixing, potential interference among lan-

---

guages, and difficulty of covering all languages, Alternatiely, the line of research on cross-lingual transfer makes PLMs multilingual by extending English-only PLMs to other languages. [Atetxe et al.
2020 demonstrate possibility of such extension by relearning the embedding layer with unsupervised data from the new language. Marchisio et al. [022 further increase computational efficiency using a mini-model proxy. Liu et al, 2023a use similar partial reset-reinit approach in vision-language
settings. Approaches based on adapters and sparse finetuning have also been proposed [Peeifeer
et al. 2020 2022 2021 Ansell et al. 2022]. Adapters are bottleneck layers (usually placed after the feedforward layers) that add extra capacity when adapting to a different task or language. Our proposed forgetting mechanism can be applied to adapter- adapter-based methods as we can allow forgetting to happen in the adapter layers. The current choice of forgetting embeddings keeps the architecture intact and incurs additional hyperparameter tuning, allowing to understand the fundamental capability of forgetting pretraining.
Conclusion & Future work
6.1
Conclusions
While forgetting is usually perceived as negative, recent work points out that it can also be beneficial in
certain cases, particularly for quickly learning new tasks, training in non-stationary environments (Igl
et al..a.
2021 Nikishin et al. 2022 D'Oro et al. [023]] and improving sample efficiency [Taha et al.
2021
Zhou et al. 2022]. J. Joining this line of work, our paper demonstrates that forgetting techniques
can improve pretrained language models by imbuing them with more linguistic plasticity. Specifically, our proposed active forgetring mechanism can create PLMs that are easier to rewire for new lingual spaces. Experiments with RoBERTa show that models pretrained via active forgetting can better learn from small amounts of data while also enjoying faster convergence during language adaptation, particularly for languages distant from English.
Going beyond language adaptation, argue that PLMs with more plasticity are promising direction for future research, as they allow easier adaptation to various tasks, domains, languages and can evolve faster as the real world changes. Unlike symbolic methods, such as knowledge graphs. which can easily rewire a fact by modifying the corresponding knowledge triplet, current static PLMs are harder rewire since changing one fact via updating model weights may disrupt multiple other facts without substantial post-hoc intervention. Improving the rewirability via forgetting pretraining thus can be seen as one way of imbuing PLMs with similar benefits as symbolic methods (making the resulted model more controllable i.e. can be modified with tiny cost), complementing the line of post-hoc model editing research [Mitchell al. 2021 2022
6.2 Limitations
Our work uses the simplest forgetting approach directly resetting embeddings to random initializa- tion. Advanced techniques like gradually injecting noise could be explored. We focus on masked language modeling pretraining with language-specific tokenizers. Applying active forgetting to auto-regressive LMs, other pretraining methods (e.g. deberta pretraining (He et all, 2021b )) and various tokenization is promising future work. More broadly, current large language models need more plasticity to expand across tools, tasks, and domains. Our work takes an initial step, showing directly resetting embeddings can significantly improve model plasticity. Further research on more sophisticated forgetting techniques during pretraining could unlock additional gains.
On the theory front, potential connections can be made between forgetting and meta-lcarning Schaul
and Schmidhuber 2010 Thrun and Pratt 2012 Andrychowicz et al.) 2016 Finn all 2017 since both attempt to learn solutions that can quickly adapt themselves to new inputs. Another possible theoretical explanation for why active forgetting works so well might be related to the flatness of the solution in the loss landscape (Alabdulmohsin et al., 2021). Flatter minima tend to enjoy better generalization [Liu et 2023b). Thus, it might be worthwhile study the flatness of the transformer body during the forgetting pretraining.
10

---

Acknowledgments and Disclosure of Funding
We would like thank our reviewers for their valuable suggestions. We are also grateful to those who engaged in interesting discussions during the project, including Pasquale Minervini, Xuanli He, Jiayi Wang. Yuxiang Wu, Hila Gonen, Dieuwke Hupkes, Fabio Petroni, Naila Murray, Alexis Thual, Nicola Cancedda, Yingchen Xu, and Hubert Jacob Banville. Yihong would like to express her thanks to the FAIR-UCL PhD program for generously funding her PhD. David Adelani acknowledges the support of DeepMind Academic Fellowship programme.
References
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constan- tine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 9:1116-1131, 2021.
David Ifeoluwa Adelani. Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen H Muhammad, Peter Nabende, al. Masakhaner 2.0: Africa-centric transfer learning for named entity recognition. In 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 4488 4508. Association for Computational Linguistics (ACL), 2022.
Ibrahim Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitialization generalization in convolutional neural networks. arviv preprint
2021.
Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. Adapting pre- trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336-4349, Gyeongju, Republic Korea, October 2022. International Committee on Computational Linguistics. URL
https: htmp:://acclann htt////aaaaanllooooo/o .rr.or///o/ 2022 col.222..olnn2211 382
Michael Anderson and Justin C. Hulbert. Active forgetting: Adaptation of
memory by prefrontal control. Annual Review of Psychology, 72(1):1-36. 2021.
doi: 10.1 001144660mrree-yyyy0077000000000
URL
paruure--sych--7222112 094114
PMID: 32928060.
Marcin Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances neural information processing systems, 29, 2016.
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1778-1796, 2022.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolin- gual representations. In Proceedings of the 58th Annual Meeting of the Association for Computa- 2020.
lional Linguistics, pages
Jeffrey Barrett and Kevin IS Zollman. The role of forgetting in the evolution and learning of language. Journal of Experimental & Theoretical Artificial Intelligence. 21(4):293-309 2009.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprini arXiv:2002.0055) 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot

---

learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Asso- ciates, Inc., 2020. URL pttpp:: wew. ps cwppaper..p.p......ppp. Files/ ess/paper/
Yihong Chen, Pushkar Mishra, Luca Franceschi, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Refactor gnns: Revisiting factorisation-baseed models from a message-passing perspective. In Advances Neural Information Processing Systems, 2022.
Alexis Conneau, Ruty Rinott, Guillaume Lamplee Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. 10.18653/vI/DI8- 1269. URL http:::::: http://accaaaaooooo.ooooo///////
2018. Association for Computational Linguistics. doi:
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pages 8440-8451, Online, July 2020. Association for Computational Linguistics. doi:
URL
wrg. 2020 aww.mmin.aaa........ 747
Jacob Devlin, Ming-We Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4188 Minncapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1111886//1//11114442 1423. URL https
V/aclantho wowpo/oooyyooo1111111111
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https
/ /openreview..oo wwtpreviewwneee/eeee fett:/orrmmiddoseee 9aBBVJe
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela
Fan, John Ortega, Ricardo Ramos, Annette Rios Gonzales, Ivan Meza-Ruiz Ruiz, al. Americasnli: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 6279-6299, 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta- meta-learing for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https
//openrevies ew /
Jean-Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu. Building a subspace of policies for scalable continual learning. In The Eleventh International Conference on Learning Representations, 2023. URL URL hntp:::////// //ppere///eeeeview
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 8342-8366 2020.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Dehertav3: Improving deberta using electra-style pre-training with disettanddd embedding sharing, 2021a.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021b.
URL htttss://gperreiiiiiii net fott:/ooumm222 - XPZIIotutst
12

---

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu -hen, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning, In International Conference on Learning Representations, 2021. URL httppss//ppeeneveee...ttt forum?
id Qun8iv4qsby
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint orXi::200.088361, 2020.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476. 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barree Barr-insww, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71,2015
Benjamin J Levy, Nathan D McVeigh, Alejandra Marful, and Michael Anderson. Inhibiting your native language: The role of retrieval-induced forgetting during second-language acquisition. Psychological Science, (8(1):29-344 2007.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330. 2020.
Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulić, and Iryna Gurevych. Delving deeper into cross- lingual visual question answering. In Findings of the Association for Computational Linguistics: EACL 2023, pages
2023a.
Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In Proceedings of the 40th International Conference on Machine Learning, 2023b.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11162. 2019. URL httt:://txxxx//// org abs /atps//90...00 11692
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pruceedings of Machine Learning Reseurch, pages 2321190-2221.022 PMLR, 23-29 Jul 2023. URL http:ttt/////rceeiiirro mlr
DIess/v202/ly1e220 html
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018
13

---

Kelly Marchisio, Patrick Lewis. Yihong Chen, and Mikel Artetxe. mini-model adaptation: Efficiently extending pretrained models new languages via aligned shallow training. arXiv preprint 1050:2,212.10003, 2022.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pages 109-166 165. Academic Press, 1989. doi: http:://bb.oooo1110000666600000/
URL https
//www. scwencediiii.ccccooooooooo.cc.. rect. com /ttpr///woo/ooeeee/////rr///////
Eric Mitchell, Charles Lin, Antoine Bossclut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In International Conference on Machine Learning, pages 15817- 15831. PMLR, 2022.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning. pages 16828-16847. PMLR, 2022.
Eygenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In Workshop on Reincarnating Reinforcement Learning ICLR 2023, 2023. URL https: http:::tttpppotoieeeee http:////pereeee...eee net www.o/rummttooommmmmmmm
1d-09AJADE21N
Simon Nørby. Why forget? on the adaptive value of memory loss. Perspectives Psychological Science, 10(5):551-5700 578, 2015. doi: 10.1177/174591161597777 URL httpt::////oooorg
00117777456916155288
PMID: 26385996.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrationss), pages 48-53, 2019.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71. 71, 2019.
Bernhard Pastötter, Karl-Heinz Bäuml, and Simon Hanslmayr. Oscillatory brain activity before and after an internal context change-evidence evidence for reset of encoding processes. Neurolmage, 43(1): 173-181, 2008
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingud Transfer. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, Online, November 2020. Association for Computational Linguistics. doi: 10. 2tt.:6/00//222.mmmlllll... 617. URL https: http://aaaaannnnan.aaaaaaa
2020 www.0000.em11pmmmmmm ma main. 617
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Unks everywhere: Adapting multilingual language models new scripts. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 10186-10203. 2021.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-3492 2022.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.
14

---

Vijaya Raghavan T Ramkumar, Elahe Arani, and Bahram Zonooz. Learn, unlearn and relearn: An online learning paradigm for deep neural networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http:::://peeeevveeeeeee net/ forum?? dN O2MJDST
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Tom Schaul and Jürgen Schmidhuber. Metalearning. Scholarpedia, 5(6):4650, 2010.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Fronfiers in psychology, 4:313, 2013.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference machine learning, pages 4528 4888-453.2200 PMLR, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Ahmed Taha, Abhinav Shrivastava, and Larry S Davis. Knowledge evolution in neural networks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12843-12852, 2021.
Y. Tang, C. Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation with extensible multilingual pretraining and finetuning. ArXiv, abs/2008.00401. 2020. URL http::: www.sticscannthoooooooeoo hotpttcccccco.....ooooorrroosolo 220936592
.rg/Corpus
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Tom Veniat, Ludovic Denoyer, and Mare' Aurelio Ranzato. Efficient continual learning with modular networks and driven priors. arXiv preprint arXiv:20/2.12631, 2020.
Laura Weidinger. John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from language models. CoRR, abs/21 12..4359, 2021. URL https: arw.v..iiiv.///.i/ wwweiiivrrrgaaaa abs/
2112.03354
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Trunsparency, pages 214-229, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Pupers), pages 1112-1122, 2018.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. In International Conference on Learning Representations, 2022.
15


Reviews:
[{"review_id": "2cZ5ojQoi8", "reviewer": "Reviewer_ar7w", "summary": "This paper presents an embedding forgetting mechanism for pre-training, aimed at enhancing robustness in downstream shift embedding fine-tuning. Focusing on the low-resource regime, the study conducts experiments on 10 simulated low-resource languages across three tasks: XNLI, XQUAD, and MLQA.", "questions": "(1) Did you try different hyper-parameters for standard PLM and forgetting PLM when fine-tuning?\n(2) Did you tie the weights of embedding and LM head?\nThanks for authors' detailed rebuttal, which addressed most of my concerns. However, the weaknesses are still existing. I'll keep my score.", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) This paper introduces a simple yet effective method, akin to a form of regularization, for improving the multilingual capabilities of a PLM. It demonstrates impressive performance when adapting to downstream X-tasks.\n(2) The writing is clear, and the illustrative figures are informative.", "weaknesses": "(1) The paper is somewhat limited in scope, as it only applies to low-resource multilingual tasks within the framework of Artetxe et al. [2020].\n(2) To propose a simple yet effective method, I believe solid experiments or rigorous proof are necessary; however, both are missing. In the experimental section, there is a lack of comprehensive ablation studies (e.g., why set low-resource data at 5M? Is this the real scenario in the multilingual setting? Why not use actual low-resource data instead of simulating one, considering that real low-resource data is different from sampled ones? How does the update frequency affect the results?).\n(3) The paper lacks comparisons with other established methods, such as multilingual pre-training, multilingual adapters, and multilingual regularization techniques, among others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. We want to clarify a few misunderstandings and address your concerns as follows.\n### First, we want to revisit and emphasise the scope of this work. \n> (1) The paper is somewhat limited in scope, as it only applies to low-resource multilingual tasks within the framework of Artetxe et al. [2020].  \n\nOur motivation is to improve language models' plasticity. Plasticity of neural networks have been studied in graph learning, computer vision and reinforcement learning[3-10], where forgetting-relearn methods show promise. Our goal is to study plasticity in the context of pretrained language models. We believe this is a emerging research direction and will thrive in the following years. \n\nHowever, translating the `plasticity` concept to the language model setting is not trivial due to the lack of clear experimental setups. We note that, despite the model differences, almost all language models begins with a token embedding layer. As often tied to a specific vocabulary, the token embedding layer limits the plasticity, preventing generalisation to a new vocabulary. \n\nThis observation inspires us to explore the plasticity of language models by manipulating the token embedding layer. Artetxe et al. [2020] draws our attention as it offers a nice experimental framework of **only manipulating the token embedding layer** for adapting between languages. We are not trying to improve SoTA multilingual models but rather testing if pretraining with active forgetting can be promising.\n\nIn this sense, our work is rather well-scoped instead of \"limited in scope\", if contextualised in the line of plasticity research[3-10]. Our choice of Artetxe et al. [2020] as the experimental framework is also well-justified.  We will elaborate on this in our camera-ready version.\n\n\n### Second, we want to address your concerns about our low-data experimental setup. \n> Why not use actual low-resource data instead of simulating one\n>\n> Why set low-resource data at 5M? \n\nWe acknowledge that dealing with real-world low-resources languages can be more challenging than the low-data setup in our paper. We recognize there are rich work in this space: multilingual pretraining[1,12], multilingual adapters[2,13], and multilingual regularization[11]. We have discussed many of them in the related work section. However, the challenge of \"low-resources\" involve **multiple entangled factors**: the quality of the tokeniser, the amount of data, whether the script/language family is distant to the pretraining language(s) etc. \n\n**Simulating allows us to control these factors and isolate the effects of the factor that we are interested in** -- the amount of data in the new language. This factor is essential to our work as our goal is `plasticity` i.e. rewiring model prediction with as little new information as possible. Simulating various amount of data in the new language allows us to compare model plasticity as shown in Fig 3 in our paper. \n\nWe observe forgetting PLMs outperform standard PLMs, when data is between 10K and 5M tokens. Since results on 5M is already decent enough to demonstrate the effectiveness of forgetting and most low-resources languages contain fewer than several million tokens, we chose to report the results on 5M. The curve in Fig 3 shows the results on other data amount.  \n\nIn summary, the choice of simulating a low-data well suits our research goal and allows us to contribute a clean piece of knowledge in the line of plasticity research.  \n\n### Third, we believe our experiments are comprehensive and convincing. \n> the study conducts experiments on 10 simulated low-resource languages\n\nWe evaluated our method on three widely used cross-lingual transfer benchmarks XNLI, XQUAD, and MLQA, encompassing a variety of languages and two different types of tasks, NLI and QA. As shown in Tab 1 in our paper, XNLI contains 14 languages. Therefore we disagree with your comment that we only experiment with 10 languages. This is an unfair comment about our work.\n\n### Other Concerns and Questions\n> How does the update frequency affect the results?\n\nWe assume you mean the forgetting frequency $K$. If so, please see the general response.\n\n**Q1**: For fair comparison, we use the same hyper-parameters following previous work Kelly 2022 and RoBERTa. We will open-source the code.\n\n**Q2**: The embeddings and the weights in LM head are tied as described in RoBERTa paper[14]\n\n------\n\n[1] Conneau, Alexis, et al. \"Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020.\n\n[2] Pfeiffer, Jonas, et al. \"MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.\" EMNLP 2020.\n\n[3] Lyle, Clare, et al. \"Understanding Plasticity in Neural Networks.\" ICML 2023\n\n[4] Zhou, Hattie, et al. Fortuitous forgetting in connectionist networks. ICLR 2022\n\n[5] Chen, Yihong, et al. Refactor gnns: Revisiting factorisation-based models from a message-passing perspective. NeurIPS 2022.\n\n[6] Igl, Maximilian, et al. \"Transient Non-stationarity and Generalisation in Deep Reinforcement Learning.\" ICLR 2020.\n\n[7] Alabdulmohsin,  Ibrahim, et al. The impact of reinitialization on generalization in convolutional neural networks. arXiv:2109.00267, 2021.\n\n[8] Taha, Ahmed, et al. Knowledge evolution in neural networks. CVPR 2021.\n\n[9] D’Oro, Pierluca, et al.  Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022.\n\n[10] Nikishin, Evgenii, et al. The primacy bias in deep reinforcement learning. ICML 2022.\n\n[11] Pfeiffer, Jonas, et al. Unks everywhere: Adapting multilingual language models to new scripts. EMNLP 2021.\n\n[12] Pfeiffer, Jonas, et al. Lifting the curse of multilinguality by pre-training modular transformers. NAACL 2022.\n\n[13] Ansell, Alan, et al. Composable sparse fine-tuning for cross-lingual transfer. ACL 2022.\n\n[14] Liu, Yinhan, et al. Roberta: A robustly optimized bert pretraining approach.\" arXiv:1907.11692 (2019)."}, {"review_id": "QQVVTBAt9B", "reviewer": "Reviewer_TTi1", "summary": "This paper follows the language adaptation procedure of MonoTrans (Artetxe et al., 2020), and proposes a new pre-training method with active forgetting. By resetting the embedding layer every K updates during training, the language model learns to learn the new embedding fast, similar to a meta-learning effect.\n\nThe paper conducted experiments on several languages, and evaluate the adapted models on zero-shot XNLI, MLQA and XQuAD. Unlike the experimental setting of MonoTrans, the paper considers a low-resource pre-training setting where an \"unseen\" language has as few as 5 million tokens for the adaptation step. The results show that the pre-trained models with active forgetting pre-training converge quickly during language adaptation, and outperform the baseline models.", "questions": "- If active forgetting even requires a re-pretraining step, how to adapt currently existing pre-trained models without re-training them?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper studies an important research question, i.e., language adaptation in a low-resource setting. Since the success of language models relies on large-scale pre-training, how to adapt the language models to low-resource languages without large-scale training data is beneficial for low-resource NLP research.\n\nThe proposed active forgetting is simple and effective for the following language adaptation step. Experiments demonstrate that it outperforms baseline models for zero-shot classification and question answering tasks.", "weaknesses": "- I am confused about goals and means:\n\n  - The goal of language adaptation / MonoTrans (Artetxe et al., 2020) is to transfer existing high-resource language models (English in particular) to unseen languages, and the method is to learn language-specific word embeddings, i.e., language adaptation.\n\n  - The goal of this paper seems to improve language adaptation results, rather than the goal behind language adaptation. The key is, transferring the currently existing models rather than pre-training new models. I understand the proposed method can benefit language adaptation. However, it greatly limits the scope, only the pre-trained models with the active forgetting can have this effect, i.e., currently existing pre-trained models are out of scope.\n\n- Experimental setup. As mentioned in L273, the paper claims that multilingual pre-trained language models have issues that they need large corpora. Since many studies have shown that multilingual models have cross-lingual transferability, training multilingual language models in the low-resource setting should be a baseline.\n\n- It would be great to provide the pre-training loss curves, showing how active forgetting affects the pre-training procedure.\n\n- (Minor)  L192, Page7 footnote, missing \".\". Figure 6, \"XNLI Accuracy vs Adaptation Steps\" is too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We would like to address your concerns as follows.\n### First, we would like to clarify our goal, motivation and contributions.\n> The goal of this paper seems to improve language adaptation results, rather than the goal behind language adaptation. The key is, transferring the currently existing models rather than pre-training new models. \n\nWe agree with you that it is important to study various ways of transferring existing pretrained models. And there are already plenty of nice works in this line, e.g. adapters[2,13], regularization[11] and many others. We have discussed them in the related work section. However, the `transferability` of existing standard pretrained models is limited, causing non-trivial efforts to adapt/extend these models to new languages[2,11-13]. \n\nAn alternative way would be improving pretraining technology so that we can make more transferrable pretrained language models (PLMs). This line of research receives little attention despite its importance in alleviating the expensive cost of data and compute in downstream adaptation. As the field will see more pretrained models coming out in the following years, we argue that now is a good timing to consider such new pretraining technology. Because we are running out of high-quality data and language models are getting bigger and bigger. \n\nThis is highly relevant to the plasticity research [3-10], where plasticity is defned as changing model prediction **with as little new information as possible**. In the context of language adaptation, pretrained models with \"plasticity\" should adapt to new languages with as little data in the new language as possible, and therefore reduce the cost of data in downstream adaptation.\n\nTo sum up, our goal is not to reach another SoTA in language adaptation but rather using language adaptation as **a testing bed for understanding plasticity of language models**. We bring together efforts from different communities[3-10] and show that pretraining with active forgetting can be promising for improving language models' plasticity. \n### Second, we want to elaborate on our experimental setup\n> Since many studies have shown that multilingual models have cross-lingual transferability, training multilingual language models in the low-resource setting should be a baseline.\n\nWe agree that multilingual PLMs can be a meaningful baseline for grounding our numbers. We are training a multilingual pretrained baseline. We will update the results here once they are ready in the following days. \n\nBeyond that, we would like to emphasise that our work tackles a different scenario. We aim to have a flexible language model. No matter the pretraining corpus is monolingual or multilingual, this language model should easily generalise itself to  **unseen languages**. This is different from the scenario of multilingual PLMs like XLM-R[1], which requires seeing all the data for all languages from the scratch. Once done with pretraining and there is some new language distant from the pretraining languages you want to support, the multilingual PLMs might still struggle with **zero-shot transfer** as shown in several low-resources language research[15-17]. \n\n### Other Concerns and Questions\n> It would be great to provide the pre-training loss curves, showing how active forgetting affects the pre-training procedure.\n\nPlease see the general response. The active forgetting creates an episodic learning pattern during pretraining, which is often seen in reinforcement learning or meta-learning.\n\n> (Minor) L192, Page7 footnote, missing \".\". Figure 6, \"XNLI Accuracy vs Adaptation Steps\" is too small.\nThank you, we will fix them in the camera-ready version.\n\nQ:\n>  If active forgetting even requires a re-pretraining step, how to adapt currently existing pre-trained models without re-training them?\nThank you, this is indeed a valuable question. \n\nAs discussed in our related work section and Part 1, there are many ways to adapt existing pretrained models to new languages according to prior research [2,11,13]. Our contribution is rather on the pretraining side. We hope our work can inspire research institutions/companies to **improve pretraining technology** and deliver PLMs with more \"transferability\". In this way the downsteam users can adapt them cheaply and without too many tweaking.\n\nOn the other hand, it could be very exciting to take an existing model and make it perform active forgetting. We will leave this for future work since it is beyond the scope of this paper and requires further exploration.\n\n-----\n\n References 1-14 can be found in our response to Reviewer ar7w.\n\n[15] Ebrahimi, Abteen, et al. \"AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages.\" ACL 2022.\n\n[16] Adelani, David Ifeoluwa, et al. \"MasakhaNER: Named Entity Recognition for African Languages.\" TACL 2021.\n\n[17] Adelani, David, et al. \"MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition.\" EMNLP 2022."}, {"review_id": "zoFZM0zUc6", "reviewer": "Reviewer_Divy", "summary": "This work introduces a training technique that leverages actively resetting token embedding to improve zero-shot language transfer. Experiments on RoBERTa show consistent improvement in multiple languages, distant languages in particular.", "questions": "How sensitive is this method to the choice of forgetting frequency?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple and innovative approach.\n2. Consistent improvement across languages and tasks.\n3. Insightful analysis.", "weaknesses": "1. Only experimented on one pretrained model.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n### We appreciate that you recognize the simplicity and effectiveness of our method. We address your comments as follows.\n\n> Only experimented on one pretrained model.\n\nWe would like to extend our experiments to more pretrained models. However, we are limited by the computational resources. One compiling of the entire experimental pipeline require at least 2 pretraining runs, 48 language adaptation runs, 6 task adaptation runs, which takes about 38460 GPU hours (V100 32GB).     \n\nOn the other hand, we chose RoBERTa as it is one of the most widely-used open-sourced pretrained language models. The effectiveness of our method on RoBERTa shows the potential of our method for the transformer family, as our method is not specific to a particular RoBERTa architecture but generalizable to any language model with a token embedding layer. We will open-source our code to facilitate future research on other pretrained models. \n\n\nQ:\n> How sensitive is this method to the choice of forgetting frequency?\n\nPlease refer to the general response. "}, {"review_id": "I3G1fW4Trr", "reviewer": "Reviewer_1FBf", "summary": "The paper proposed an active forgetting mechanism for PLMs pre-training for cross-lingual transfers and adaptations. The authors propose a multi-stage adaptation framework for better cross-lingual transfer/adaption: 1) First, by resetting embedding layers every K update during pre-training for monolingual RoBERTa model; 2) Second, by adapting parameters separately (embeddings vs backbone) for language-specific knowledge and task-specific knowledge (language+task adaptation). The paper shows the proposed mechanism helps faster convergence during cross-lingual transfers and adaptation as well as achieving better downstream task performances. \n", "questions": "* See weaknesses.\n* Have you tried resetting other parameters during pre-training? What about other PLMs? e.g. Does deBERT style training affect the results?\n\n* References:\n\n     1. Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulić, and Iryna Gurevych. Delving Deeper into Cross-lingual Visual Question Answering. Findings of EACL, 2023 → Partial resetting, and re-initialization of parameters for cross-lingual generalization in VL setting.\n\n     2. Vijaya Raghavan T Ramkumar, Elahe Arani, Bahram Zonooz, Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks, TMLR, 2023 → Selective parameters resetting for continual learning (online and few-shot).\n\n\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper shows an interesting empirical finding that resetting partial parameters (in this case embeddings) during pre-training in English leads to better transfers/adaptation in cross-lingual settings.\n* The paper is clear-written and the proposed method is effective empirically for cross-lingual transfers.\n", "weaknesses": "- Although the proposed method achieved good performances in downstream tasks, this work lacks proper ablation experiments on showing the gain is from resetting during pre-training, vs the gain of adapting/resetting parameters separately during the language+task adaptation stage. I.e. the paper should include the results of the following 4 types of experiments:\n    * Standard pre-training + standard adaptation\n    * Forgetting pre-training + standard adaptation\n    * Standard pre-training + language/task adaption\n    * Forgetting pre-training + language/task adaption\n\n- Practically, efficiency (parameters and training) could be an issue with the language adaptation + task adaptation issue. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate that you recognise the effectiveness of our method and find our finding interesting. We would like to address your concerns as follows.\n### First, we want to elaborate on our experimental setup.\n> this work lacks proper ablation experiments on showing the gain is from resetting during pre-training, vs the gain of adapting/resetting parameters separately during the language+task adaptation stage. ...\n\nWe run quick experiments on Arabic and include numbers for standard adaptation here. However, we also want to highlight the difference between standard adaptation and language/task adaptation. \n\nStandard adaptation relies on **labelled** data, which is expensive for a new downstream language. In contrast, the language/task adaptation **does not use any labelled data**. It only uses the unlabelled data from the new language. In our case, we only found $6.7K$ Arabic NLI data. The amount of labelled data is not enough to adapt an English NLI model to Arabic NLI without proper regularization.\n\nOur experimental setup follows [18], where standard-pretraining + language/task adaptation (MonoTrans) is shown to be competitive among a few baselines for zero-shot unsupervised cross-lingual transfer. On top of this finding, our proposed forgetting method can further improve the sample-efficiency of the language/task adaptation, reducing the amount of unsupervised data needed for the new language. This is motivated by a practical scenario where the new languages contain only several thousands of tokens to a few millions of tokens (e.g. the corpus for the new language might contain only 2-3 books). \n\n\n  **Method **                                          | **Supervised Data** | **Unsupervised Data** | **Arabic XNLI ACC** \n------------------------------------------------------|---------------------|-----------------------|---------------------\n **standard pretraining + standard adapt**            | 6.7K                | 0                     | 32.8                \n **standard pretraining + language/task adaptatioon** | 0                   | 5M                    | 41.2                \n **forget pretraining + standard adaptation**         | 6.7K                | 0                     | 34.2                \n **forget pretraining + language/task adaptation**    | 0                   | 5M                    | 59.7                \n\n\n### Second, we would like to address your questions. \n**Q1**: \n> Have you tried resetting other parameters during pre-training? \n\nWe did try resetting the bias terms in the language model head, though it didn't help much in our preliminary experiments. \n\nWe decided to focus on token embedding layer as prior work [18-24] demonstrate that the token embedding layer, which captures most lexical meanings, is crucial for cross-lingual transfer. Moreover, since the token embedding layer is not only the first layer but also the last layer (RoBERTa and many other language models use tied token embeddings), resetting the token embedding layer is equivalent to resetting the last layer. This echos the findings in [4,5,7,8,10], where resetting **later** layers is more effective to model plasticity.\n\nIn the future, it could be interesting to try out resetting other parts and see if they benefit particular tasks. For example, we can leverage some outcome from the line of interpretable LM. Say if we understand the functions of a particular subnetwork, we can consider selectively forgetting them to help relevant downstream tasks, achieving similar effects as parameter-efficient tuning methods[25-26].\n\n**Q2**: \n> What about other PLMs? e.g. Does deBERT style training affect the results?\n\nWe would like to extend our experiments to more pretrained models. deBERT sounds like a good candidate. However, we are currently bottlenecked by the computational resources, similar as many other pretraining research. One compiling of the entire experimental pipeline requires at least 2 pretraining runs, 48 language adaptation runs, 6 task adaptation runs, which takes about 38460 GPU hours (V100 32GB). We are working hard to extend to more models but would require more time and computational resources.\n\nOn the other hand, we chose RoBERTa as it is one of the most widely-used open-sourced pretrained language models. The effectiveness of our method on RoBERTa shows the potential of our method for the transformer family, as our method is not tied to a particular RoBERTa component but generalizable to any language model with a token embedding layer. We will open-source our code to facilitate future research on other pretrained models.\n\n**References**:\nThank you for the references. We will add them to our camera-ready version.\n\n-----\n\nReferences 1-14 can be found in our response to Reviewer ar7w. References 15-17 can be found in our response to Reviewer TTi1\n\n[18] Artetxe, Mikel et al. \"On the Cross-lingual Transferability of Monolingual Representations.\" ACL 2020.\n\n[19] Minixhofer, Benjamin,  et al. \"WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.\" NACCL 2022.\n\n[20] Dobler, Konstantin et al. \"FOCUS: Effective Embedding Initialization for Specializing Pretrained Multilingual Models on a Single Language.\" arXiv preprint arXiv:2305.14481 (2023).\n\n[21] Tran, Ke. \"From English to Foreign Languages: Transferring Pre-trained Language Models.\" (2019).\n\n[22] Jain, Neel, et al. \"How to Do a Vocab Swap? A Study of Embedding Replacement for Pre-trained Transformers.\" (2022).\n\n[23] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer Learning for Low-Resource Neural Machine Translation. EMNLP 2016 \n\n[24] Chao Xing, el al. Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation. NAACL 2015.\n\n[25] Hu, Edward J., et al. \"LoRA: Low-Rank Adaptation of Large Language Models.\" ICLR 2021.\n\n[26] Sourab Mangrulkar, et al. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. 2022"}, {"review_id": "v4qgq2y3Uj", "reviewer": "Reviewer_LV8K", "summary": "This paper proposes active forgetting, a rather straightforward method that resets the embedding layer every K updates during pretraining, to quickly adapt PLMs to new languages. Through experiments on different language pairs with RoBERTa, the authors claim that the proposed method can induce faster convergence and better performance when the languages are distant from English.", "questions": "- In the abstract, the paper claims that it is data and compute inefficient to learn a new embedding layer. But this paper does not address this problem, since a new embedding layer for new languages still has to be learned. Therefore I don't think it is necessary to mention that in the abstract.\n\n- The paper claims the method ensures high sample efficiency. How does the experiment support this?\n\n- Forgetting is “generally” a bad thing for PLMs, but this work counter-intuitively shows that active forgetting can be beneficial for cross-lingual transfer. The authors are encouraged to give more intuition or explore the reasons behind this phenomenon, in a cross-lingual perspective.\n\n- For the experiment w.r.t. RQ3, the authors show that active forgetting is particularly helpful when the new languages are (typological) different from the pretraining language. However, I see also an important possible influencer: the script. The authors are encouraged to also explore if scripts can influence the transfer performance under active forgetting.\n\n- The frequency of active forgetting is set to K=1000. As this is an important hyperparameter, I would encourage the authors to justify their choice.\n\n- I would be very interested to see a plot of loss over updates during language adapting. I would expect the loss to go down in general but fluctuate a lot every time when active forgetting is used.\n\n- It’s natural to think that this active forgetting could be used in other parts of the models instead of merely on embeddings. Therefore it would be very interesting to explore: active forgetting on which part of the models is the most effective?\n", "limitations": "The authors include a section of Limitations in the paper, where they mention that this paper only focuses on the simplest forgetting, which is only applied to the embeddings.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written in general, with a clear motivation and some novelty. \n\n- It is appreciated that the author conduct experiments on many languages from similar languages like German to distant languages like Thai. \n\n- The method itself is counter-intuitive, as it resets the embedding layer periodically which intuitively can be bad for PLMs, but effective to improve the cross-lingual transfer.\n\n- The authors give in-depth analysis and insights into their proposed method, which I believe is interesting to the community. \n", "weaknesses": "Some figures are inconsistent and visually not good. For example, the upper three subfigures in Figure 4 have a visually bad axis scale. Some values are rounded to integers while some are not, as in Figure 6. The authors should carefully redraw and prettify them in the camera-ready version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\n### We appreciate that you recognise the soundness and contribution of our work. We would like to address your comments as follows.\n\n**Figures 4 and 6**: Thank you for pointing out the issues. We will polish the figures in the camera-ready version. \n\n**Q1**:\n> In the abstract, the paper claims that it is data and compute inefficient to learn a new embedding layer. But this paper does not address this problem, since a new embedding layer for new languages still has to be learned. \n\nWe would like to clarify the confusion here. Although the new embedding layer still has to be learned, forgetting PLMs require less compute and data in the new language to reach a good performance: \n- 1) less compute. Fig 4 in our paper shows that forgetting PLMs converge fast within $5K$ adaptation steps. So we can actually stop the adaptation training earlier than the standard methods. For example, on XQUAD, in order to converge to $90\\%$ of their final performances, forgetting PLMs take only ~$5K$ steps while standard PLMs take ~$20K$ steps. This leads to **4x** speedup for the language adaptation run. \n- 2) less adaptation data. We did a few ablations on the adaptation data quantity and studied its impact on learning a new embedding layer. Fig 3 in our paper summarizes the results for learning a new English embedding layer. We can see that, when adaptation data is less than $10M$ (often true for low-resources languages), forgetting PLMs consistently outperform standard PLMs. Thus, to reach the same level of performance in low-resources setting, forgetting PLMs require less adaptation data while relearning the English embeddings (an extreme example would be in Fig 3 in our main paper, to reach 40 on NLI, standard PLMs require ~$100K$ adaptation data while forgetting PLMs only require ~$5K$).  \n\n**Q2**:\n> The paper claims the method ensures high sample efficiency. How does the experiment support this?\n\nThe second point in our response to Q1 addresses this question. More evidence can be found by comparing the perfomance drop of standard PLMs and forgetting PLMs when the adaptation data change from a high-data setting [18,27] to a low-data setting that our work is considering: \n\n| Method                  | Avg adaptation #tokens | Avg XNLI performance |\n| :---------------------- | ---------------------: | -------------------: |\n| Kelly et al 2022 [27]   |                  10.3B |                   72 |\n| Artetxe et al 2022 [18] |                   569M |                 66.7 |\n| Standard                |                     5M |                 53.3 |\n| Forgetting              |                     5M |                 62.7 |\n\nWe can see that, when the adaptation data is reduced from 569M to 5M, forgetting only drops about 6% (from 66.7 to 62.7) while standard drops about 20% (from 66.7 to 53.3). When the adaptation data amount drops, forgetting PLMs still retain a relatively good performance compared to standard PLMs. \n\n\n**Q3**:\n> The authors are encouraged to give more intuition or explore the reasons behind this phenomenon, in a cross-lingual perspective.\n\nOur intuition is that the periodic forgetting of the token embedding layer will force the transformer body learn better high-level abstraction. Because every time forget happens, the body kind of \"re-derive\" these abstraction; by repeating this process again and again, the body finally learns to abstract instead of short-cut memorizing particular embedding values. During cross-lingual transfer, a body with more high-level abstraction can be more easily transferred to new languages, since the high-level abstraction is more language-agnostic.\n\nOur intuition can also be supported by cognitive science literature (see sec 5.1 in our paper) where forgetting is shown beneficial for learning to abstract [27,29] and new languages[28].\n\nWe will add more discussion on this in the camera-ready version. \n\n**Q4**:\n>  For the experiment w.r.t. RQ3, the authors show that active forgetting is particularly helpful when the new languages are (typological) different from the pretraining language. However, I see also an important possible influencer: the script. \n\nThis indeed a very valuable comment. We did observe some impact of the script. For example, although Vietnamese and Swahili are both distant from English, yet forgetting brings modest or no improvements. We guess it might be because they are written in the Latin script, which is the same as the pretaining English. We will add this discussion in the camera-ready version. \n\n**Q5**: Frequency of Forgetting\n\nPlease see general response.\n\n**Q6**: Loss Curves of Forgetting\n\nYes, the forgetting indeed creates a spike and then the model learn to recover to normal loss. Please see general response for details.\n\n**Q7**: \n> active forgetting on which part of the models is the most effective?\n\nWe are also excited about this direction. Our guess is that it will depend on the task. Token embeddings play a big role in cross-lingual transfer, so forgetting on token embeddings is effective. But for other tasks, forgetting on other parts of the model might be more effective. We will explore this in the future.\n\n------\n\nReferences 1-26 can be found in our responses to the other reviewers.\n\n [27] Marchisio, Kelly et al. Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training. ACL 2023 Findings\n\n [28] Levy, Benjamin J et al. Inhibiting your native language: The role of retrieval-induced forgetting during second-language acquisition. Psychological Science, 2007\n\n [29] Simon Nørby. Why forget? on the adaptive value of memory loss. Perspectives on Psychological Science, 2015\n\n [30] Michael C. Anderson and Justin C. Hulbert. Active forgetting: Adaptation of memory by prefrontal control. 2022"}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We would like to address some common questions in this general response. **Figures are attached in the rebuttal pdf.**

### Active Forgetting Creates An Episodic Learning Pattern
Reviewers are curious about the loss curves comparison for standard pretraining and forgetting pretraining. Hence we have included the loss curves for standard pretraining and forgetting pretraining in Rebuttal Figure 1. We can see that the forgetting pretraining creates a very interesting pattern of loss curves. If we zoom in a bit, as shown in Rebuttal Figure 2, episodic learning is happening while we are actually using the same data for pretraining. This is quite different from introducing diversity by including as many languages as possible in the pretraining corpus. It is more like in reinforcement learning or meta-learning.

### Impact of Forgetting Frequency
We would like to elaborate on our choice of forgetting frequency $K$. In our preliminary experiments, we tried $K=100, 1000, 5000$. We find $K=1000$ works well and thus sticks with it. Since we don't want to overtune the hyperparameters, we just use the same $K$ for all the experiments.
We include the loss curves of $K=100$ and $K=5000$ here. We can see that both forgetting too frequently and forgetting too infrequently will hurt the performance. Too frequent forgetting leaves little time for the body to learn something meaningful (the pretraining loss stuck around 11). Too sparse forgetting will make the body hard to adjust to the next forgetting, causing divergence as pretraining goes on.



Meta-review: We note that reviewer ar7w has not acknowledged the author response. However, the author response has largely addressed the reviewers’ concerns. As the authors explain, this paper is not aiming at "improving SoTA multilingual models but rather testing if pretraining with active forgetting can be promising". Indeed this seems like a promising approach. The authors present a novel technical contribution to the community that is experimentally demonstrated to be effective and improve LM plasticity. Two key issues that are raised are the lack of ablation experiments (Reviewer 1FBf) and the need for additional baselines, both of which would give crucial and further insight into the effectiveness of the approach. The authors address these by running / presenting new experiments albeit only limited ones (ie, on Arabic and Thai respectively). The paper would benefit further from more comprehensive experiments in that respect, and, in particular, showing not only detailed results per language, but also aggregated (Avg) results (some of which could go in the Appendix given the space restrictions). The authors are asked to provide these experiments in the camera ready.
Decision: Accept (poster)