SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Up to this point, we have primarily presented the averaged performance. In this section, we delve into a detailed comparison of language-specific performance between forgetting PLMs and standard PLMs on XNLI, MLQA, and XQuAD. To gain a deeper insight into which languages benefit the most from the use of forgetting, we present the relative performance changes in Figure 6 for XNLI and MLQA. The results for XQuAD can be found in Figure 8 in the appendix. Across the spectrum of languages (Table 5), we observe that forgetting provides greater benefits for languages distant to the pretraining language (English) in terms of language family, script and morphology. Specifically, forgetting brings large relative gains for languages such as Arabic, Hindi, Thai, Turkish, and Urdu compared to closer languages like German. Script seems important - forgetting helps Vietnamese and Swahili less despite their distance from English, likely due to the shared Latin script. Examining adaptation curves within the first 5K steps, forgetting PLMs reach substantially superior performance over standard PLMs for almost all languages except Urdu, while standard PLMs struggle at random guess levels (see Figure 7 and Appendix D). This demonstrates forgetting PLMs' ability to efficiently adapt to new languages, particularly dissimilar ones, in low-data settings.
${}^{2}$  Precisely, it simulates vocabulary swappings, causing drastic changes to the input of the body.
Figure 5: Adaptation curves on XNLI, MLQA, and XQuAD. Numbers aggregated across languages. The first row contains the full adaptation curves, which comprises  $125\mathrm{K}$  adaptation steps. The second row contains the zoom-in versions of curves for the first 5K adaptation steps. Forgetting PLMs converge faster than standard PLMs; for instance, on XQuAD (the last plot), forgetting PLMs reach  $92\%$  of their final performance within 5K updates, while standard PLMs only reached  $53\%$  of their final performance at that point.
Figure 6: Relative gains of forgetting PLMs over standard PLMs across languages. Forgetting yields substantial relative gains for languages like Arabic, Hindi, Thai, Turkish, and Urdu. However, for languages closely related to English, such as German, the relative gains from forgetting are modest.
5.1 Forgetting and its Positive Roles
The common perception of forgetting is that it implies weak memory and a loss of acquired knowledge, thus it is often regarded as a sign of un-intelligence or an undesirable property. In neural networks, catastrophic forgetting [McCloskey and Cohen, 1989; Ratcliff, 1990; Kirkpatrick et al., 2017] is portrayed as a forgetting phenomenon where neural networks lose the ability to predict old patterns after new inputs alter their weights. Forgetting in this context has negative consequences, as the new knowledge overwrites the old. Plenty of prior research strives to overcome catastrophic forgetting and enable continual learning [Schmidhuber, 2013; Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017; Shin et al., 2017; Schwarz et al., 2018; Mallya and Lazebnik, 2018; Parisi et al., 2019; Rolnick et al., 2019; Beaulieu et al., 2020; Veniat et al., 2020; Gaya et al., 2023; Khetarpal et al., 2022].
Our work differs from the above ones in that our subject is intentional forgetting rather than passive forgetting and its associated negative impact. To put it in another way, we seek to understand how
Figure 7: Adaptation curves on XNLI within 5K updates for individual languages: Bulgaria, Greek, Spanish, French, Russian, Swahili, Vietnamese and Chinese. For all languages except Urdu, the forgetting PLMs converge faster than the standard PLMs during the language adaptation stage.
forgetting – if purposely incorporated as an active process during training – might help new learning. Similar positive roles of forgetting have been discussed in the literature. Specifically, Pastotter et al. [2008] demonstrate forgetting enhances the learning of new information by resetting the encoding process and holding the attention at high levels; Levy et al. [2007] show that it helps second language acquisition by inhibiting the native language; Barrett and Zollman [2009] find it promote the emergence of an optimal language by preventing partial success from reinforce sub-optimal practice. [Norby [2015] further suggests forgetting serves adaptive functions, helping people regulate emotions, acquiring knowledge and staying attuned to the context. More recently Anderson and Hulbert [2021] reviews evidence on active forgetting by prefrontal control and shows how it can adapt the memory to suit either emotional or cognitive goals.
5.2 Forgetting Via Partial Neural Weights Reset
In neural networks, forgetting can be instantiated in many forms. A simple way is to reset subsets of parameters before the next round of learning. Iterations of such resetting have been shown to benefit generalization with low compute and low data for computer vision tasks [Frankle and Carbin, 2019]. Alabdulmohsin et al., 2021; Taha et al., 2021; Ramkumar et al., 2023. More recently, Zhou et al. [2022] demonstrate a similar forgetting strategy helps image classification and language emergence. Closely linked to our method, Chen et al. [2022] forget node embeddings in order to truncate infinite message-passing among nodes and thereby aid new graph reasoning with new nodes. Our work uses similar forgetting mechanism over token embeddings, improving new language reasoning with new tokens. As far as we know, we are the first to bring forgetting into pretraining and demonstrate that forgetting pretraining boosts linguistic plasticity. A relevant thread in reinforcement learning (RL) research studies the plasticity loss phenomenon [Lyle et al., 2023; Nikishin et al., 2023]. Recent work explores similar forgetting approaches to improve plasticity. [gl et al., 2021] periodically reset the current policy by distilling it into a reinitialized network throughout training. Intuitively, this releases network capacity storing suboptimal policies and opens up space for the yet-to-be-discovered optimal (final) policy. Simpler methods just reset an agent's last layers [Nikishin et al., 2022], preventing overfitting to early experiences and primacy bias. Resetting parameters also improves sample efficiency by allowing more updates per environment interaction [D'Oro et al., 2023].
5.3 Making Pretrained Language Models Multilingual
Pretraining on multilingual data makes PLMs multilingual [Conneau et al., 2020] but has downsides like needing large multilingual corpus with appropriate mixing, potential interference among lan
guages, and difficulty of covering all languages. Alternatively, the line of research on cross-lingual transfer makes PLMs multilingual by extending English-only PLMs to other languages. Artetxe et al. [2020] demonstrate possibility of such extension by relearning the embedding layer with unsupervised data from the new language. Marchisio et al. [2022] further increase computational efficiency using a mini-model proxy. Liu et al. [2023a] use a similar partial reset-reinit approach in vision-language settings. Approaches based on adapters and sparse finetuning have also been proposed [Pfeiffer et al., 2020, 2022, 2021, Ansell et al., 2022]. Adapters are bottleneck layers (usually placed after the feedforward layers) that add extra capacity when adapting to a different task or language. Our proposed forgetting mechanism can be applied to adapter-based methods as we can allow forgetting to happen in the adapter layers. The current choice of forgetting embeddings keeps the architecture intact and incurs no additional hyperparameter tuning, allowing us to understand the fundamental capability of forgetting pretraining.
6 Conclusion & Future work
6.1 Conclusions
While forgetting is usually perceived as negative, recent work points out that it can also be beneficial in certain cases, particularly for quickly learning new tasks, training in non-stationary environments [Igl et al., 2021; Nikishin et al., 2022; D'Oro et al., 2023] and improving sample efficiency [Taha et al., 2021; Zhou et al., 2022]. Joining this line of work, our paper demonstrates that forgetting techniques can improve pretrained language models by imbuing them with more linguistic plasticity. Specifically, our proposed active forgetting mechanism can create PLMs that are easier to rewire for new lingual spaces. Experiments with RoBERTa show that models pretrained via active forgetting can better learn from small amounts of data while also enjoying faster convergence during language adaptation, particularly for languages distant from English.
Going beyond language adaptation, we argue that PLMs with more plasticity are a promising direction for future research, as they allow easier adaptation to various tasks, domains, languages and can evolve faster as the real world changes. Unlike symbolic methods, such as knowledge graphs, which can easily rewire a fact by modifying the corresponding knowledge triplet, current static PLMs are harder to rewire since changing one fact via updating model weights may disrupt multiple other facts without substantial post-hoc intervention. Improving the reirability via forgetting pretraining thus can be seen as one way of imbuing PLMs with similar benefits as symbolic methods (making the resulted model more controllable i.e. can be modified with tiny cost), complementing the line of post-hoc model editing research [Mitchell et al., 2021, 2022].
Our work uses the simplest forgetting approach - directly resetting embeddings to random initialization. Advanced techniques like gradually injecting noise could be explored. We focus on masked language modeling pretraining with language-specific tokenizers. Applying active forgetting to auto-regressive LMs, other pretraining methods (e.g. deberta pretraining [He et al., 2021b], and various tokenization is promising future work. More broadly, current large language models need more plasticity to expand across tools, tasks, and domains. Our work takes an initial step, showing directly resetting embeddings can significantly improve model plasticity. Further research on more sophisticated forgetting techniques during pretraining could unlock additional gains.
On the theory front, potential connections can be made between forgetting and meta-learning [Schaul and Schmidhuber, 2010; Thrun and Pratt, 2012; Andrychowicz et al., 2016; Finn et al., 2017] since both attempt to learn solutions that can quickly adapt themselves to new inputs. Another possible theoretical explanation for why active forgetting works so well might be related to the flatness of the solution in the loss landscape [Alabdulmohsin et al., 2021]. Flatter minima tend to enjoy better generalization [Liu et al., 2023b]. Thus, it might be worthwhile to study the flatness of the transformer body during the forgetting pretraining.
We would like to thank our reviewers for their valuable suggestions. We are also grateful to those who engaged in interesting discussions during the project, including Pasquale Minervini, Xuanli He, Jiayi Wang, Yuxiang Wu, Hila Gonen, Dieuwke Hupkes, Fabio Petroni, Naila Murray, Alexis Thual, Nicola Cancedda, Yingchen Xu, and Hubert Jacob Banville. Yihong would like to express her thanks to the FAIR-UCL PhD program for generously funding her PhD. David Adelani acknowledges the support of DeepMind Academic Fellowship programme.