Fortuitous Forgetting in Connectionist Networks
Authors: Hattie Zhou
Ankit Vani
Hugo Larochelle
Aaron Courville
Venue: ICLR 2022 Poster (2022)
Keywords: Neural Networks
Generalization
Iterative Training
Compositionality
Iterated Learning

Abstract: Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce forget-and-relearn as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.

Published as a conference paper at ICLR 2022
FORTUITOUS FORGETTING IN CONNECTIONIST NETWORKS
Hattie Zhou"
Ankit Vani
Mila, Université de Montréal
Mila, Université de Montréal
Hugo Larochelle
Aaron Courville
Mila, CIFAR Fellow, Google Research, Brain Team
Mila, Université de Montréal, CIFAR Fellow
ABSTRACT
Forgetting is often seen as unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce forget-and-releam trajectories artificial neural networks. In this process, the forgetting step selec- tively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forget- ting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide coherent view on the dynamics of iterative training in neural networks and offer clear path towards performance improvements.
powerful paradigm for shaping the learning
INTRODUCTION
Forgetting is an inescapable component of human memory. occurs naturally neural synapses get removed or altered over time (Wang et al. 22020). and is often thought to be an undesirable char- acteristic of the human mind. Similarly, machine learning, the phenomena of catastrophic forget- ting is often blamed for certain failure modes in the performance of neural networks (McCloskey & Cohen 1989; Ratcliff 1990 French 1999 However, substantial evidence psychology and neuroscience have shown that forgetting and learning have a symbiotic relationship (Bjork & Bjork 2019 Gravitz 2019). A well-known example is the "spacing effect", which refers to the obser- vation that long-ten recall is enhanced by spacing, rather than massing, repeated study sessions. Bjork & Allen (1970) demonstrated that the key to the spacing effect is the decreased accessibility of information in-between sessions.
In this work, we study a general learning paradigm that we refer to as forget-and-relaare and show that forgetting can also benefit learning in artificial neural networks. To generalize to unseen data, we want our models to capture generalizable concepts rather than purely statistical regularities, but these desirable solutions naturally (Geirhos et al.) 2020). Recently, a number of training algorithms have been proposed to im- prove generalization by iteratively refining the learned solution. Knowledge evolution (Taha al., 2021) improves generalization by iteratively reinitializing one part of the network while continu- ously training the other. Iterative magnitude pruning (Frankle Carbin, 2019: Frankle et al. 2019) removes weights through an iterative pruning-retrainin process, and outperforms unpruned mod- els in certain settings. Hoang et al. (2018) iteratively utilize synthetic machine translation corpus through back brck-translations of monolingual data. Furlanello et al. (2018 ; Xie et al. (2020) employ it- erative self distilllllt to outperform successive teachers. Iterated learning Kirby 2001 has been
small subset of the solution space and often more difficult learn
" "Correspondance zhou. hatt e@qmai ccom
"ur paper title is derived from that of French 1999), Catastrophic Forgetting in Connectionist Networks. We distinguish the use of the term "iterated learning", which is specific algorithm, with "iterative train- ing", which refers any training procedure that includes multiple generations of training.

---

Published as a conference paper at ICLR 2022
shown to improve compositionality in emergent languages (Ren et al., 2020 Vani al. 2021 and prevent language drift (Lu et al. 2020). We propose that many existing iterative algorithms are in- stances of more general angee--ad-releamn process, and that their success can be understood by studying the shared underlying mechanism.
Forget-and-relearn is iterative training paradigm which alternates between forgetting stage and a relearning stage. At a high level, we define a forgetting stage as any process that results in a decrease in training accuracy. More specifically, let D = {(x,, re) be the training dataset. use as source of randomness for
Let U represent uniform noise sampled from (0, 1), which stochastic functions. Given a neural architecture, let F be the set of functions computable by a neural network with that architecture, and for any N E F,, let Acc(N) = n Σ (X,) = Yt) be the training accuracy of N. Let Nr be a network trained on D, and let C. := E(Ace [ represent the performance of randomly initialized classifier on D. We say that f (0. + F is a forgetting operation if two conditions hold: (i) P[Ace(](N,, U)) < Acc(N,) | Acc(N,) > C) = 1; (ii) the mutual information [N, U) D) is positive. The first criterion ensures that forgetting will allow for relearning due to a decrease in accuracy. The second criterion captures the idea that forgetting equates a partial removal of information rather than a complete removal. For standard neural networks, it is sufficient to show that training accuracy is lower than the model prior to forgetting, but higher than chance accuracy for the given task.
The Forget-and- -el-eer Hypothesis. Given an appropriate forgetting operation, iterative re- training after forgetting will amplify unforgotten features that are consistently useful under different learning conditions induced by the forgetting step. A forgetting operation that favors the preserva- tion of desirable features can thus be used to steer the model towards those desirable churacteristics.
We show that many existing algorithms which have successfully demonstrated improved general- ization have a forgetting step that disproportionately affects undesirable information for the given task. Our hypothesis stresses the importance of selective forgetting, and shows path towards better algorithms. We illustrate the power of this perspective by showing that we can significantly improve upon existing work in their respective settings through the use of more targeted forgetting opera- tions. Our analysis in Section 5 sheds insight the mechanism through which iterative retraining leads to parameter values with better generalization properties.
2 EXISTING ALGORIHTSS AS INSTANCES OF FORGET- RELEARN
In this section and the rest of the paper, foeus two settings where iterative training algorithms have been proposed to improve performance: image classification and language emergence. We summarize and recast a number of these iterative training algorithms instances of forget-and- relearn. We also relate our notion of forgetting to other ones in the literature in Appendix A5
Image Classification. A number of iterative algorithms for image classification uses a reinitializa- tion and retraining scheme, and these reinitialization steps can be seen as a simple form of forgetting. Iterative magnitude pruning (IMP) (Frankle & Carbin 2019 Frankle et al. 2019) removes weights based on final weight magnitude, and does so iteratively by removing an additional percentage of weights each round. Each round, small magnitude weights are set to zero and frozen, while the remaining weights are rewound to their initial values. Zhou et al. (2019a) show that the masking and weight rewindingg] procedure still retains information from the previous training round by demon- strating that the new initialization achieves better- chancc accuracy prior to retraining. Thus, we can view IMP as forget-and-relean process with additional sparsity constraint. RIFLE (Li et al.) 2020) shows that transfer learning performance can be improved by periodically reinitializing the final layer during the finetuning stage, which brings meaningful updates to the low-level features in earlier layers. We can view this reinitialization step as a forgetting operation that enables relearning.
Knowledge evolution (KE) TTha al. 2021) splits the network's 's weights into a fit-hypothesis and a reset-hypothesis, and trains the network in generations in order to improve generalization perfor- mance. At the start of each generation of training, weights in the reset-hypothesis reinitialized,
In this work, use "rewind" to mean resetting weights back to their original initialization. We use "reset" and "reinitialize" interchangeably indicate new initialization.
2

---

Published as a conference paper at ICLR 2022
while weights in the fit-hypothesis retained. Taha et al. (2021) suggest that KE is similar to dropout where neurons learn to be more independent. However, it is hard to translate the dropout intuition to the case of single subnetwork. Another intuition compares to ResNet, since KE can induce a zero mapping of the reset-hypothesis without reducing capacity. But this zero-mapping always occurs with sufficient training, and it is unclear why it would benefit generalization.
Language Emergence. Several existing algorithms in the language emergence subfield can also be seen as forget-and-relearn algorithms. Emergent communication is often studied in the context of multi-agent referential game called the Lewis game (Lewis) 2008 Foerster al. 2016 Lazaridou et al. 2016). In this game, a sender agent must communicate information about the attributes of a target object receiver agent, who then has to correctly identify the target object. Composi- tionality is a desirable characteristic for the language that agents use, as it would allow for perfect out-of-distribution generalization to unseen combinations of attributes.
Li & Bowling 2219)) hypothesize that compositional languages are easier to teach, and propose an algorithm to exert an "case-of-teaching"" pressure on the sender through periodic resetting of the receiver. We can instead view the receiver resetting as partial forgetting in the multi-agent model. Ren et al. (2020) use iterated learning to improve the compositionality of the emergent language. Their algorithm consists of (i) an interacting phase where agents learn to solve the task jointly, (ii a transmitting phase where a temporary dataset is generated from the utterances of a sender agent, and (iii) an imitation phase where newly-initialized sender is pretrained on temporary dataset of the previous sender's 's utterances, before starting a new interacting phase, and this process iterates. A crucial component during imitation is to limit the number of training iterations, which creates learning bottleneck that prevents the new model from fully copying the old model. This restricted transfer of information can be seen forgetting operation, and showeases an alternative approach to selective forgetting that removes information that is harder to learn.
3 PARTIAL WEIGHT PERTURBATION AS TARGETED FORGETTING
In this section, we show that common forms of partial weight perturbation disproportionately affect information that can hinder generalization in both image classification and language emergence set- tings, and can thus be seen as targeted forgetting operation under the forget-and-releam hypothesis. We provide full details on the perturbation method studied in Appendix A1.1
3.1 IMAGE CLASSIHCATION
What information is undesirable depends on the task at hand. For the issue of overfitting in im- age classification, we want to favor features that are simple, general and beneficial for generaliza- tion (Vallc-Pérez et al.] 2019). Several works have shown that easy examples are associated with locally simpler functions and more generalizable features, while difficutt examples require more ca- pacity to fit and are often memorized (Baldock et al., [222]] Yosinski et al.) 2014 Kalimeris et al.) 2019 Arpit et al. 2017). Thus, as proxy, we define undesirable information here as features asso- ciated with memorized examples. In order to evaluate whether partial weight perturbation selectively forgets undesirable information, we create two types of example splits for our training data: (i) easy or difficult examples, and (ii) true mislabeled examples. At each training iteration, we make copy of the model and perform a forgetting operation on the copy. We then evaluate both models and measure separately the training accuracy the two groups of examples. We perform these experiments using a two-layer MLP trained on MNIST We further extend to a 4-layer convolution model and R8 on CIFAR-10, and ResNet50 on ImageNet in Appendix A1.2
For the definition of easy or difficult examples, we follow prior work (Jiagg et al., 2020 Baldock al. (02T)) and use the output margin as measure of example difficulty. The output margin is defined as the difference between the largest and second-larges logits. We label 10% of training examples with the smallest output margin in the current batch as difficult examples. In the noisy labels experiment, we sample 10% of the training datef] and assign random labels to this group.
*We use an MI.P with hidden layers 300-100-10. 10, trained using SGD with a constant learning rate of 0.1. 'To simplify the task, we restrict to using only 10% of the MNIST training set so that the random label examples can be learned quickly.
3

---

Published as a conference paper at ICLR 2022
40-
20-
60
Tharthy
Tha
200o Aine
Hr-uggll bay
HP Panggr Libeb
Mi-forget Hard
MP Torget Sandom Labels
40-
Ferpet Easy
R2 Forgst Truet Labe
Porget Rancom Labob
20002 33000 3:00e 40320 30002
10100 20002 20000 42000 50000 Troens
10000 20003 30000 40200 20020 10003 20200 33000 40020 51003
Traveng
(a) Easy vs difficult examples.
(b) True vs mislabeled examples.
The left panels in (a) and (b) show training accuracy of the two example groups for two types of weight perturbation. The right panels in (a) and (b) show the accuracy of each example group for the same model with weight perturbation applied. Same colors represent the same for- getting operation, and dashed lines represent the accuracy the difficult mislabeled examples. Results are averaged over 5 runs.
Figure 1:
150-
100
60
Dave Matr
boxrt Mateo
50
Seme Weightt Bert: Comp
N
Same WesggtiZer Comp
Same Wrishe Rwrt Perm
Same Wright 2em Perm
Non hard Namik Camp
Now Rund Zoro Carr
Ves
Parn
Rand Parm
700
A3E
630
30.0
450
660
200
102
602
2:0
400
C60
Trining teadon
Traniing Parmtton
Triining Ieradon
Tranning Baranton
(a) Perturb through reinitialization of masked weights.
(b) Perturb through setting masked weights to O.
Figure 2: The left panels in (a) and (b) show train accuracy of the compositional (Comp) and non- compositional (Perm) language examples; blue lines here are hidden behind orange lines. The right panels in (a) and (b) show the accuracy of each group for the same model with weight perturba- tion applied. Same colors represent the same forgetting operation, and dashed lines represent the accuracy on the non-compositional examples. Results are averaged over 5 runs.
We consider KE-style perturbation to be the reinitialization random subset of the weights, and IMP-style perturbation to be the zeroing of small magnitude weights and rewinding of the remaining weights. More details are provided in Table AT Figure [ shows that both types of weight pertur- bation adversely impact difficult mislabeled example groups more severely than the easy or true labeled examples, even when both example groups reach 100% train accuracy. Related to our ob- servations, Hooker et al. (2019) show that pruned models disproportionately affect tails of the data distribution. There are also theoretical works tying robustness-tor ity, flat minima. and generalization (Zhou et al. 2019b
causstees-tto-pramtttrr-ne parameter-noiss to compressibil-
Arora et 2018 Foret al. 202I).
3.2
COMPOSITIONAL LANGUAGE EMERGENCE
In the version of the Lewis game we study, the objective to identify objects with two attributes: SHAPE and COLOR. The sender receives a one-hot representation of the SHAPE and COLOR attributes of an object and produces a discrete message. The receiver takes this message as input and needs to classify the correct object from set of candidate objects. The two agents are typically parameter- ized as LSTMs (Hochreiter Schmidhuber) 1997) and trained simultaneously using REINFORCE
Williams 1992 or Gumbel-sofimax (Jang et al. 2017 Maddison et al.) 2017). A compositional language would have a consistent and independent attribute symbol mapping for all input objects. This is desirable for out-of-distribution generalization, since unseen combinations of attributes will still have a meaningful language output. Thus, the undesirable information in this task corresponds non-compositional language. We define two candidate languages as designed by Li & Bowling
4

---

Published as a conference paper at ICLR 2022
2019), and use them to provide possible targets to the model. One language is compositional, where each attribute (4 shapes and 8 colors) is represented by a distinct and consistent symbol-positiin. The other language is non-composiiional, which is formed by randomly permuting the mappings between messages and objects from the compositional language. In this case, each object still has a unique message, but the individual symhols do not correspond to consistent meaning for shape or color. An example of a compositional and permuted language is provided in Table A2 and A3
Since the sender controls the language output, we simplify the Lewis game here and train only the sender model to fit a given language through supervised learning. The target language is formed by mixing the two pre-specified languages. For the 32 objects in the dataset, half of them are mapped to the compositional language and half to the composiiiion language. Similar to the experiments in Section B.I we make a copy of the model at each training iteration, and perform the forgetring operation on the copy. We evaluate both models and measure the accuracy of the two example groups separately. For the weight perturbations, we consider either mask based on initial weight magnitude random mask. For both mask criteria, we either reinitialize the masked weights or set them to O. More details can be found in Table A4 As shown in Figure 2 we find that all forms of weight perturbation under study lead more forgetting of the non-compositional language.
4 TARGETED FORGETTING IMPROVES PERFORMANCE
According to the forget-and-relear hypothesis, the forgetting step should strike a balance between enabling relearning in the next generation and retaining desirable information from the previous generation. We show in this section that through targeted forgetting of undesirable information and preserving of desirable information, we can significantly improve upon existing algorithms.
4.1 REDUCING OVERFITTING IN IMAGE CLASSIFICATION
We build upon the KE algorithm since it is widely applicable and leverages a large number of iter- ative training generations. Following Taha et al. (2021), we study tasks that have a small number of training examples per class, where the network is prone to overfitting, and adopt the same exper- imental framework unless otherwise specified. Detailed hyperparameters, training procedure, and dataset descriptions can be found in Appendix AA21 A2.2
As described by Taha et al. (2021), in KE the mask criteria M is determined prior to training and generated randomly for weight-level splitting (WELS), or based on the location of the kernel for kernel-level splitting (KELS). Given a split-rate 8r, s,% of weights (for WELS) kernels (for KELS) in each layer / has a corresponding M' value of 1. For a neural network parameterized by O, we define the fit-hypothesis each generation, the weights in the fit-hypothesis are kept the same, and the weights in the reset- hypothesis are reinitialized. Then, the network is trained from this new initialization for e epochs, where e is the same for each generation.
M O and reset-hypothesis as (1 M) O. At the start of
Can we perform the forgetting step in way that more explicitly targets difficult examples in order to improve generalization performance? Baldock et al.) (202T) introduce prediction depth as measure of example difficulty in a given model. Prediction depth refers the layer after which an example's activations can generate the same predictions using a KNN probe as the model's final predictions. They show that increasing prediction depth corresponds to increasing example difficulty. They study this correspondence using various notions of example difficulty, including output margin, adversarial input margin, and speed learning for each data point.
Based on the observations in Baldock et al. 20211), we hypothesize that by reinitializing the later layers of the neural network, we can remove information associated with difficult examples more precisely than the mask criteria used in KE. Thus, we propose a new forgetting procedure called later-layer forreeetnnn (LLF). Given a layer threshold L, we define the LLF mask criterion for each layer/ l as:
MLr {
ifl i
We note that the concept of difficult examples is used only as a proxy for information in the network that is more complex, brittle, and specialized. We do not suggest that these examples should not be
5

---

Published as a conference paper at ICLR 2022
learned. In fact, memorization of examples in the tails of the data distribution are often important for generalization (Feldman & Zhang 2020), which we also achieve in the relearning stage.
Following Taha et al. (2021), we perform LLF on top of non-iterative loss objectives designed to duce overfitting. We report performance with label-smoothing (SMTH) (Müller et al. 2019 Szegedy et al., 2016)) with = 0.1, and the CS-KD regularizer (Yun et al. 2020) with 7 and Adl. 3. Additionally, we add long baseline which trains the non-iterative method for the same total number of epochs as the corresponding iterative methods. We present results for ResNet18 (He et al. 2016) in Taaleeaane and DenseNet16 69 (Huang et al. 2017) in TTT We find that LLF substantially out- performs both the long baseline and the KE models across all datasets for both model architectures.
Taha et al. (2021) introduced KE to specifically work in the low data regime. LLF as a direct extension of KE expected to work in the same regime. For completeness, we also evaluate LLF (Le & Yang 2015), and WideResNet (Zagoruyko & Komodakis, 2016) and DenseNet-BC on CIFAR-TO T0 and CIFAR- 100 (Krizhevsky et al. (2009) in Appendix We see a significant gain on Tmay-lma but not on CIFAR-IO or CIFAR-I00.
using ResNet50 on
Analysis of Related Work. Surprisingly, we find that when compared to the long baselines with equal computation cost, KE actually underperforms in terms of generalization performance, which contrary to the claims in Taha et al. (2021). We can understand this low performance by looking at the first few training epochs in each generation in Figure Ba We see that after the first generation, the training accuracy remains near 100% after the forgetting operation. This lack of forgetting suggests that KE would not benefit from retraining under the forget-and-releam hypothesis. We also observe that KE experiences optimization difficulties post forgetting, as evidenced by decrease in training accuracy in the first epochs of each generation. This could explain why it under-performs the long baseline. Although KE can enable reduction in inference cost by using only the slim fithhypothesis, our analysis reveals that this benefit comes at the cost of lower performance.
Concurrent work (Alabdulmohsin et al. 202T studies various types of iterative reinitialization ap- proaches and proposes a layer- wise reimitialization scheme, which proceeds from bottom to top and reinitializing one fewer layer each generation. refer to this method as LW and provide a compar- ison to their method in Table 1 We find that LLF outperforms LW across all datasets we consider. More discussion on this comparison can be found in Appendix A2.4
Analysis of LLE In order to view the success of LLF and other similar iterative algorithms in terms of the selective forgetting of undesirable information, we should also observe that selective forgetting of desirable information should lead to worse results. We can test this by reversing the LLF procedure: instead of reinitializing the later layers, we can reinitialize the earlier layers. The standard LLF for Flower reinitializes all layers starting from block 4 in ResNet18, which amounts to reinitializing 75% of parameters. The reverse versions starting from block 4 and block 3 reinitializes 25% and 6% of weights respectively. We see in Figure 3b that the reverse experiments indeed perform worse than both L... and the long baseline.
Additionally, by looking how prediction depth changes with the long baseline compared LLF training in AA we can observe that LLF pushes more examples to be classified in the earlier layers. This further suggests that forget-and-releary with LLF encourages difficult examples to be relearned using simpler and more general features in the early layers.
4.2 INCREASING COMPOSITIONALITY IN EMERGENT LANGUAGES
In order to promote compositionality in the emergent language, Li & Bowling (0019) propose to periodically reinitialize the receiver during training. They motivate this by showing that it is faster for a new receiver to learn a compositional language, and hypothesize that forcing the sender to teach the language to new receivers will encourage the language to be more compositional. As is common in this line of work (Brighton & Kirby] 2006 Lazaridou et al. 2018 Li & Bowling 2019 Ren et al. 2020), compositionality of the language is measured using topographic similarity (p), which is defined as the correlation between distances of pairs of objects in the attribute space and the corresponding messages in the message space. Li & Bowling (2019) show that p. measured using Hamming distances for objects and messages, is higher with the ease-of-tert of-tecching approach than training without resetting the receiver.
6

---

Published as a conference paper at ICLR 2022
Table 1: Comparing KE and Later Layer Forgetting for ResNet18. Results are mean accuracy and standard error over 3 runs and reported for hyperparameter settings with best validation performance. KE experiments use KELS split with split rate of 0.8. LLF uses L {10. 14), corresponding to block 3 and 4 in ResNet18. N3, N8, NIO indicate the additional number of training generations on top of the baseline model. LLF consistently outperforms all other methods.
Method
Flower
CUB
Aircraft
MIT
Dog
Smth (NI)
51.02 10.09
58.92 10.24
57.16 10.91
56.04
10.39
63.64
10.16
Smth long (N3)
59.51 10.17
66.03
10.13
62.55
10.25
59.53
1:060
65.39
1:055
Smth + KE (N3)
57.95 10.65
63.49
10.39
60.56
10.36
58.78
1:054
64.23
1:005
Smth LLF (N3) (Ours)
63.52 +0.13
70.76
+0.24
68.88
#0.11
63.28
40.69
67.54
#0.12
Smth long (N10)
66.89 +0.23
70.50
+0.13
65.29
+0.51
61.29
+0.49
66.19
#0.03
Smth KE (N10)
63.25 #0.17
66.51
+0.07
63.32 +0.30
59.58
+0.62
63.86
+0
Smth + L.IF (N10) (Ours)
70.87 +0.41
72.47
40.31
70.82 +0.50
64.40
40058
68.51
+..39
Smth + LW (N8)
68.43 +0.27
70.87
#0.15
69. 10
#0.27
61.67
+0.32
66.97
#0.21
Smth LLF (N8) (Ours)
69.48 +0.21
72.30
+0.28
70.37
+0.49
63.58
10.16
68.45
+0.25
CS-KD (NI)
57.57 #0.61
66.61
+0.02
65.18 #0.68
58.61
+0.25
66.48
#0.12
CS-KD long (N3)
64.44 10.62
69.50
10.24
65.31
10.67
57.16
10.34
66.43
10.24
CS-KD + KE (N3)
63.48 +1.30
68.76
#0.57
67.16
+0.23
58.88
+0.03
67.05
#0.31
CS-KD LLF (N3) (Ours)
67.20 +0.51
72.58
+0.02
+0.21 71.65
62.41
+0.45
68.77
20.24
CS-KD long (N10)
68.68 10.28
69.59
10040
64.58
1.007
56. 12 1.043
64.96
10.17
CS-KD KE (N10)
67.29 #0.74
69.54
+0.60
68.70
+0.33
57.61 +0.91
67.11
#0.11
CS-KD + LLF (N10) (Ours)
74.68 10.19
73.51
+0.35
72.01
+0.23
62.89 +0.59
69.20
10.12
CS-KD + LW (N8)
73.72 #0.74
71.81
+0.21
+0.34 71.95
70.82
59. 18 +0.41
68.09
20.24
CS-KID LLF (N8) (Ours)
73.48
#0.31
73.47
+0.35
40.23
62.26
40.47
69.24
+0.20
We challenge this Cease-of-teaching interpretation in & Bowling (2019) and offer alternative explanation. Instead of viewing the resetting of the receiver as inducing an ease-of-teaching pressure on the sender, we can view form of forgetting is sub-optimal, since we care about the quality of the messages, which is con- trolled by the sender. This motivates a balanced forgetting approach for the Lewis game, whereby both sender and receiver can forget non compossiiiii aspects of their shared language. Therefore, we propose an explicit forgetting mechanism that removes the asymmetry from the & Bowling (2019) setup. We do this by partially reinitializing the weights of both the sender and the receiver and refer to this method as partial balanced forgetting (PBF). We use the "same weight reinit" per- turbation method from Section 3.2 with 90% of weights masked. Further training details are found in Appendix A3 We show in Figure that this method, which does not exert an "case-of-teaching" pressure, significantly outperforms both no resetting and resetting only the receiver in terms of p.
asymmetric form of forgetting in the two-agent system. This
5 UNDERSTANDING THE FORGET-AND-RELEANE DYNAMIC
In Section we discussed the importance of targeted forgetting in the forget-and-relaam dynamic. In this section, we study the mechanism through which iterative retraining shapes learning.
5.1 IMAGE CLASSIFICATION
We consider two hypotheses for why iterative retraining with LLF helps generalization. We note that these hypotheses are not contradictory and can both be true.
1. Later layers improve during iterative retraining with LLF. Due to co-adaptation during training, the later layers may be learning before the more stable early layer features are fully developed (notably in early epochs of the first generation), which might lead to more overfitting in the later
7

---

Published as a conference paper at ICLR 2022
100
Traiing Accursy
80
Text accrra
70
70-
KE
65-
early
LLF
atdl ayes
60-
liee layers
20
meverse reint: block
standard LLP -Doccs
55-
30-
reverse reint: block3
50-
2005
605
1.0
20
Training Iteration
Training Generaiion
Staining Generation
(a) Train accuracy at initialization and in the first epachs of training for 5 generations.
(b) Test accuracy of resetting early layers vs. later layers
(c) Test accuracy for different
freeze layer settings
(c) show the min, max, and average of 3 runs. (a) shows that KE induces no forgetting after the first 2 generations. (boss that resetting early layers significantly under-performs LLF. shows that freezing early layers prevents any improvement subsequent training, while freezing later layers still leads to significant improvements.
Figure 3: Analysis experiments performed on Flower dataset with ResNet18. (b)
layers in some settings. Reinitializing the later layers might give them opportunity to be relearned by leveraging fully developed features from the earlier layers.
2. Early layers improve during iterative retraining with LLF. Since the early layers are continuously refined through iterative training, the features which are consistently useful under new learning conditions may be amplified. These features may be beneficial to generalization.
To test hypothesis we freeze the early layer after the initial training generation and reinitialize the later layers for retraining in subsequent generations. If the later layers improve result of learning on top of well-developed early layer features, we should expect performance improvement during relearning even when early layers are frozen. As shown in Segureeejundd "freeze early layers", we observe no significant performance improvement in this setting, suggesting that simply relearning later layers has little effect on generalization performance.
To test hypothesis 2, we design reverse experiment where the later layers (except for the output FC layer) reinitialized and then frozen at each subsequent relearning generation. The early lay- ers are trained continuously in each generation. As shown in Figure 3c under "freeze later layers", find that even with random frozen layer in the middle, iterative retraining of the early layers still improves performance significantly. To test the interpretation that early layers improve through learning under different conditions, we perform the same experiment before, except we keep the later layers frozen at the same initialization in each generation. This reduces the amount of vari- ability seen during iterative retraining. As shown in Figure 3c under "freeze fixed later layers", we find the performance of this to be much worse than the version with different reinitialization each generation, demonstrating the importance of having variable conditions for relearning. These anal- yses suggest that the main function of iterative retraining is to distill features from the unforgotten knowledge that are consistently useful under different learning conditions.
und
5.2 COMPOSITIONALITY
The analysis in Section 5.1 supports the hypothesis that features that are consistently learned across generations are strengthened in the iterative retraining process. this hypothesis explains why itera- tive algorithms improve compositionality in the Lewis game, it would predict that (i) compositional mappings are consistently learned across generations, (ii) the compositional mapping is strengthened during repeated retraining, and (iii) increases during the retraining stage.
For iterated learning, Ren et al. (2020) show that by limiting the number training iterations the imitation phase, the new sender can learn language with better p than the previous one, arguing that the compositional components of the language are learned first. The success iterated learning has primarily been attributed to this increase in compositionality during the imitation phase, which follows the explanation in the cognitive science thcories from which iterated learning derives (Kirby) 2001). However, when viewed as instance of forget-and-relean,, hypothesis predicts that the
8

---

Published as a conference paper at ICLR 2022
2.
Topeoraent aww.aty 60
2.a
0.0-
taroted Leamingg
Aeoadar (in r91s
inseraction
Feneet dee gh441
receier
Baruder
Tu Nareton
Tuinig Koradon
(a) Iterated learning.
(b) Iterated learning, zoomed the first 40000 steps.
(c) Ease-of-tcaching.
Figure 4: Topographic similarity (p) in the Lewis game with different forms of forgetting. presents the across all phases of iterated learning: (b) zooms into the first 40000 steps to illustrate that p improves in stages for every generation: first during reinitialization and imitation (forget), but it is generally during interaction that the sender starts outperforming the previous generation's 's (relearn). (c) plots p the end of each generation for the case-of-teaching setting.
(a)
improved performance in iterated learning is primarily driven by the interaction (retraining) phase. The retraining phase reinforces features that are consistently learned, and we conjecture that these correspond to the compositional mappings. We show in Figure [bb] that the interaction phase is indeed primarily responsible for the improvement in p. By looking at p during both the imitation and interaction phases using optimal hyperparameters under the setting in Ren al. (2000), we see that increases in occur during the interaction phase, while the optimal imitation phase does not even need to result in a higher p than the end of the previous generation.
To show that the compositional components of the language repeatedly learned and strengthened during retraining, we visualize the learned mappings from each input dimension to message dimen- sion in Appendix A4 We see that the same compositional mappings are retained across generations while the con-compositional aspects fluctuate and decrease in magnitude. We also observe that the probability of compositional mappings increase with increasing number of generations, illustrating the strengthening effects of iterative retraining.
6 CONCLUSION
We introduce forget-and-reeeam as general framework to unify number of seemingly disparate iterative algorithms in the literature. Our work reveals several insights into what makes iterative training successful paradigm. show that various forms of weight perturbation, commonly found iterative algorithms, disproportionately forget undesirable information. We hypothesize that this key factor to the success of these algorithms, and support this conjecture by showing that how well we can selectively forget undesirable information corresponds to the performance of the resulting algorithms. Although we focus discussion the forgetting of undesirable information, our analysis in Section (5) implies that this is in service of preserving desirable information. We demonstrate that the relearning stage amplifies the unforgotten information, and distills from it features that are consistently useful under different initial conditions induced by the forgetting step.
This work has implications in understanding and designing new iterative algorithms. places em- phasis on what is forgolten rather than what is learned, and demonstrates that designing new methods for targeted information removal can be a fruitful area of research. It can often be easier to define and suppress unwanted behavior than to delineate good behavior, and we illustrate two ways of do- ing so. Our results suggest that there exists Goldilocks zone of forgetting. Forget too little and the network could easily retrain to the same basin; forget too much and the network would fail to accu- mulate progress over multiple relearning rounds. Where this zone lies raises interesting questions in our understanding of neural network training that can be explored in future work. Finally, the idea that features that are consistently useful under different learning conditions is reminiscent of the line of work in invariant predictions (Arjovsky et al.) 2019 Mitrovic et al. 2021 Zhang et al. 2021 Parascandolo et al. 202I). Understanding this dynamic in the iterative regime may point the way to creating algorithms that capture the same benefit without the expensive iterative training process.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like thank Kevin Guo, Jason Yosinski, Sara Hooker, Evgenii Nikishin, and Ibrahim Alabdulmohsin for helpful discussions and feedback on this work. The authors would also like acknowledge Ahmed Taha, Fushan Li, and Yi Ren for open sourcing their code, which this work heavily leverages. This work was financially supported by CIFAR, Hitachi, and Samsung.
REPRODUCIBILIIY
We make our code available
htrp forgetting
REFERENCES
Ibrahim Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitialization on generalization in convolutional neural networks. arXiv preprint arXiv:2109.00261 2021.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 2020.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization.
arXiv preprint arXiv: orXiv:997.002893 2019.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang, Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pp- 254-263. PMLR, 2018
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 233-242 PMLR, 06-11 Aug 2017.
Robert JN Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. arXiv preprint arXiv:2106 09647, 2021.
Jeffrey Barrett and Kevin Zollman. The role of forgetting in the evolution and learning of language.
J. Exp. Theor. Artif. Intell., 21:293-309, 12 2009.
Robert A. Bjork and Ted W. Allen. The spacing effect: Consolidation or differential encoding?
Journal of Verbal Learning and Verbal Behavior, (5):567-572, 1970.
Robert A. Bjork and Elizabeth L. Bjork. Forgetting the friend of learning: implications for teaching and self-regulated learning. Advances in Physiology Education, 2019.
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life, 1222)229-242, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition,
pp. 248-255. Icee, 2009.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint arXiv:2008.03700 2020.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in neural information processing systems, pp. 2137-2145, 2016.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference Learning Represen- tations, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2022
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 2019.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 1999.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616.
PMLR, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Lauren Gravitz. The importance forgetting, Nature, 571(Uuly:S12-SI4, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Cong Duy Vu Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back- translation for neural machine translation. In NMT@ACL, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint arXiv:1911.052444 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Yiding Jiang. Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2020.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity. Ad- vances in Neural Information Processing Systems, 32:3496-3506, 2019.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXhv:1412.6980, 2014.
Simon Kirby. Spontaneous evolution of linguistic structure-an an iterated learning model of the emer- gence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 5(2):102- 110, 2001.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A al. Overcom-
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of 2009.
features from tiny images.
11

---

Published as a conference paper at ICLR 2022
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arxiv:1612.07182, 2016.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of lin- guistic communication from referential games with symbolic and pixel input. arXiv preprint arXiv:/804.03984, 2018.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
David Lewis. Convention: philosophical study. John Wiley & Sons, 2008.
Fushan Li and Michael Bowling. Ease-of- -eeachingg and language structure from emergent commu- nication. arXiv preprint arXiv: 024003, 2019.
Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, and Dejing Dou. Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer. Interna- tional Conference on Machine Learning, pp. 6010-6019, PMLR, 2020.
Iyya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In International Conference on Machine Learning. pp. 6437-6447. PMLR, 2020.
Xiaolong Ma. Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Ming- hai Qin, Sijia Liu, Zhangyang Wang, et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Processing Systems, 34, 2021.
Chris J. Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relax- ation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5157 2013.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Jovana Mitrovic, Brian McWillias, Jacob Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In International Conference Learn- ing Representations, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv: /wt.:/996600222. 2019.
Maria-Elew Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. doi: 10. 109//CVV1P.20008.4
722-729, 2008.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Interna- tional Conference on Machine Learning, pp. 5142211-5555 5142-5151. PMLR, 2019.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 413 420, 2009. doi: 10.1 10.1109/CVP..2009 5206537.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2)2285, 1990.
12

---

Published as a conference paper at ICLR 2022
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional lan- guages emerge in a neural iterated learning model. International Conference on Learning Representations, 2020.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557. PMLR, 2018.
Ravid Shwartz-Ziv Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprini arXiv: /00i::700..00 1t///7030..0000 00810, 2017.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gordon Wil- son. Does knowledge distillation really work? arXiv preprint urXiv:2206.05944 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Ahmed Taha, Abhinav Shrivastava, and Larry Davis. Knowledge evolution neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12843-12852, 2021.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Under- standing and improving knowledge distillation. arXiv preprint arXiv:2002.03532, 2020.
Guillermo Valle-Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the crrameter-function map biased towards simple functions. In International Conference on Learning Representations, 2019.
Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville. Iterated learning for emergent systematicity in VQA. In International Conference on Learming Representations, 2021.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD 20rds-200-2011 Dataset. Technical Report CNS-TR-2011-00) California Institute of Technology. 2011.
Chao Wang, Huimin Yue, Zhechun Hu, Yuwen Shen, Jiao Ma, Jie Li, Xiao-Dong Wang, Liang Wang, Binggui Sun, Peng Shi, Lang Wang, and Yan Gu. Microglia mediate forgetting via complemenn--eepeddent synaptic elimination. Science, 2020.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-22 226, 1992.
Qizhe Xie, Minh- Thnh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Annik Yalnizyan- Malnizyan-Carson Carson and Blake A. Richards. Forgetting enhances episodic control with struc- tured memories. bioRxiv, 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13876-13885, 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
1615::1605.07146 2016.
13

---

Published as a conference paper at ICLR 2022
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to out-of-distributio generalization? arXiv preprint arXiv:2106.02890, 2021.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a.
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. Non- Non-vacuous gen- eralization bounds at the imagenet scale: a PAC-bayesian compression approach. In International Conference on Learning Representations, 2019b.
14

---

grey pink purple Published as a conference paper at ICLR 2022
A
APPENDIX
A1 PARTIAL WEIGHT PERTURBATION AS TARGETED FORGETTING
A1.1 DETAILS WEIGHT PERTURBATION METHODS
We can categorize this class of forgetting method in terms of both mask criteria and a mask action. The mask criteria is a function during the forgetting step by producing a binary mask with the same dimensions as the parameters of the neural network. The mask action Rd > R2 determines how the replacement values of the weights are computed. The mask-1 action 81 operates on weights with mask value 1, and mask-O action so operates on weights with mask value O. For neural network parameterized by θ forgetting through partial weight perturbation can then be defined by the update rule
R + {0, 1)² which determines which weights will be changed
+ - m O ( ( + (1 m o so(
(2)
Table A1: Types of weight perturbations for image classification experiments. Random mask criteria means that masks are generated randomly, and IMP-style mask criteria equals to 1 for corresponding weights with large final magnitudes. Mask-1 I Action refers to the operation on weights with corre- sponding mask value of 1, and Mask-O Action refers to the operation weights with corresponding mask value of
Method
Mask Criteria
Mask-1 Action
Mask-0 Action
KE-style Random Reinit
Random
Identity Rewind
Reinitialize Set to 0
IMP-style Weight Rewind
final
Table A2: A example of a compositional language adopted from Li & Bowling 2019
black
blue
green
grey pink purple
red
yellow
circle be lc
re
pc
de
square bs
gs
rs
pS
us da
ys
star ba
la
ga
pa
triangle bt
gt
rt pt
ut
yt
Table A3: A example of non-compositional language adopted from Li & Bowling 2019
black
blue
green
red
yellow
circle ys
gc
lc ra
bc ut
ua
TS
square
ga
yc
de
pt
rc
dt
star yt
Is
ba
bs
bt
It uc
triangle
pc
ps
da
ds
us
ya
gt
A1.2 EXTENSION TO CIFAR- AND IMAGENET
We extend our analyses of partial weight perturbation to CIFAR-10 (Krizhevsky et al. 2009) on layer convolutional model (Conv4) and ResNet18 model, and to ImageNet (Deng et al.) 2009) on ResNet50. All CIFAR- 10 models are evaluated using a 50% reset rate. We find reliable trends for both KE-style and IMP-style forgetting for all datasets.
For CIFAR- 10, we follow the exact same procedure as the MNIST experiments. Following the setup in Frankle & Carbin 2019), Conv4 model is trained using the Adam optimizer, and ResNet18
15

---

Published as a conference paper at ICLR 2022
Table A4: Types of weight perturbations for compositionality experiments. Random mask criteria means that masks are generated randomly, and weight mask criteria equals to for corresponding weights with large final magnitudes. Frequency indicates how often masks are resampled. Mask-1 Action refers to the operation weights with corresponding mask value of 1, and Mask-0 Action refers to the operation on weights with corresponding mask value of O.
Method
Mask Criteria
Frequency
Mask-1 Action
Mask-0 Action
Random Reinit
Random
Per Iter
Identity
Reinitialize
Random Zero
Random
Per Iter
Identity
Set to 0
Same Weight Reinit
knit
Once
Identity
Reinitialize
Same Weight Zero
- snstl
Once
Identity
Set to 0
30-
Tharing
Tharnyy
02-
40-
HP Forgit tept) Ensy
HP Forgit tepll) Hare
22
HP- Ferget tept) Eacy
ui Forget tepl: lhrd
2 2
pet sorged tepo) tosy
path soreet tept) tosy
pith Furget tep0) Hare
pigh Forget tepo: Harrs
20000 40020 52000 80c00 200000 bnriien
5
20000 ADDDE 53000 10020100010
20000 eoo0s 63000 50OGS 200000
0
20000 40002 63200 20020120020
Traiiin
Borsoon
Tradiig
(a) Conv4 on 10
(b) ResNet18 on CIFAR-10
Figure A1: The left panels in (a) and (b) show training accuracy of easy and hard example groups for different types of weight perturbation. The right panels in (a) and (b) show the accuracy of each example group for the same model with weight perturbation applied. Same colors represent the same forgetting operation, and dashed lines represent the accuracy on the hard examples. Results are averaged over 5 runs. Lowess smoothing is applied for visual clarity.
trained using SGD. We train on 10% of training data for fast convergence. Results on easy and hard examples are shown in Figure Al We find that KE-style forgetting results in consistent and significant difference between easy and hard examples. However, the IMP-style forgetting operation renders the fost-forgetting accuracy close to chance for CIFAR-10 models, and there is only a small difference between easy and hard example groups. the lottery ticket literature, has long been observed that "late resetting" is needed to get performant sparse models as datasets and models get larger (Frankle et al., 2019). In late resetting, we would rewind kept weights to the value a few epochs after training, rather than to the initial untrained values. We perform variant of IMP- style forgetting where weights are rewound to epoch 3. We find that IMP-style forgetting with late resetting significantly increases accuracy for both groups, and much more for the easy group than the hard group. We also show the same trend with true and random labeled example groups in Figure A2 These results demonstrate that IMP with late resetting achieves more targeted forgetting of undesirable information, which can be seen through the larger gaps between example groups These observations suggest another possible reason for the increased performance when using late resetting: IMP with late resetting better leverages the benefits of forget-and-relean
For ImageNet experiments, given computational constraints, we use a pretrained ResNet50 model on ImageNet and separate examples into easy and hard groups based on output margin. We only use training examples that are correctly classified as to not disadvantage the difficult examples. This means that both group has 100% accuracy fre-forgetting. We then perform various forgetting oper- ations on top of the pretrained model and measure the accuracy of the two groups post-forgetting. For KE-style forgetting, we randomly reinitialize 20% of weights on the same pretrained model and aggregate results over 5 samples. For IMP-style forgetting, we set varying percentages of small weights to zero proxy given or intermediate weights. Results are shown in Table A5 and we find that hard examples are more severely affected in all cases.
using a pretrained model and do not have access to initial
16

---

Published as a conference paper at ICLR 2022
130-
102-
72-
22
Traktrg Accuracy
Trating Acuurace
20-
KE Fogiit Tran Lacels
LE Forgrt Trie Label:
KE Forglt Ransen Labst
KF Forgit Fansem Labeic
M4- torge. dard1 Trus Lelads
MP rurg, har21 Trus Leted.
PP farget (nod) Random lazen
PP fargen tep2) Ramdom labe
narge dend) Laber
rarge tepil Latets
40ze sc000
CO0DT 100020
(a) Conv4 on CIFAR-10
(b) ResNet18 on CIFAR-10
Figure A2: The left panels in (a) and (b) show training accuracy of true and mislabeled example groups for different types of weight perturbation. The right panels in (a) and [b]]sw the accuracy of each example group for the same model with weight perturbation applied. Same colors represent the same forgetting operation, and dashed lines represent the accuracy on the mislabeled examples. Results are averaged over 5 runs. Lowess smoothing is applied for visual clarity.
an)
Table A5: Weight perturbations on ResNet50 for ImageNet. Using a pretrained model, we filter out the ImageNet training examples that are wrongly classified by the pretrained model. Hard examples are the 10% of remaining training examples with the largest output margin, and easy examples are the remaining training examples. We measure the accuracy of the two groups after performing various forgetting operations on the pretrained models. Random reinit values are the mean and standard error over samples.
Method
Easy Example Accuracy
Hard Example Accuracy
Random Reinit 20% of Weights
45.6% + 0.7%
17.9% + 0.3%
Set Small Weights to Zero (50%)
89.8%
47.3%
Set Small Weights to Zero (40%)
97.9%
65.5%
Set Small Weights to Zero (30%)
99.7%
80.1 1
Set Small Weights to Zero (20%)
100%
90.3%
A2 TARGETED FORGETTING IMPROVES PERFORMANCE
A ARCHITECTURES AND TRAINING DETAILS
All networks are trained with stochastic gradient descent (SGD) with momentum of 0.9 and weight decay of 10 We also use a cosine learning rate schedule (Loshchilov & Hutter 2017). Taha eet al.) (2021) use an initial learning rate of 0.256, but we find that smaller learning rate than what used in Taha al. (202T) to be beneficial for certain datasets, thus we consider a learning rate in f0.1, 0.256 for all experiments and report the setting with the better validation performance. Models are trained with a batch size of 32 for 200 epochs each generation. No early stopping is used.
For the layer threshold I., we determine the value through hyperparameter tuning on a validation set. In the settings we consider, we find that it is sufficient to sweep through the starting layer or middle layer of model blocks, beginning from the second block. This allows us to search for within a restricted set of 6 of fewer values, making it practical even in deep models. All ResNet18 models use a default layer threshold of 10 (start of block 3), with exception of the Flower dataset which uses threshold of 14 (start of block 4) on the SMTH experiments. All DenseNetl 69 models use a default layer threshold of 40 (start of denseblock 3), except for the Aireraft dataset which uses a threshold of 68 (middle of denseblock 3).
17

---

Published as a conference paper at ICLR 2022
Table A6: Summary of the five datasets used in Section 4.1 adopted from
Taha et al. 2021
Classes
Train Size
Val Size
Test Size
Total Size
Flower Nilsback & Zisserman
2008
102
1020
1020
6149
8189
CUB Wah et al. (001T)
200
5994
N/A
5794
11788
Aircralt Maj ct al. (0133)
100
3334
3333
3333
10000
MIT67 Quatton Torralba
2009
67
5360
N/A
1340
6700
Stanford-Dogs Khosla et
2011
120
12000
N/A
8580
20580
Later Layer Forgetting Tiny-ImageNet. Results are test accuracy averaged over 3 runs and reported for hyperparameter settings with best validation performance. N3 and N10 indicate the additional number of training generations top of the baseline model. The long baseline scales the step LR schedule based on the additional epochs.
Table A7:
Method
Tiny-ImageNet
Smth (N1)
54.37
Smth long (N3)
51.16
Smth LLF (N3) (Ours)
56.12
Smth long (N10)
49.27
Smth + L.FF (N10) (Ours)
56.92
A2.2 DATASET SUMMARY
Taha al. 2021
different resizing of images for different datasets, and the specifie resizing parameters are not specified in the paper. For consistency, we resize all datasets to (256, 256). We validation split to tune hyperparameters and report results the test when available.
A2.3 EXTENSION TO LARGER DATASETS AND MODELS
In this section, evaluate LLF larger datasets and models than what used in the comparison with KE. We train ResNet50 on Tiny-ImageNet (Le & Yang) 2013), which consists of 200 classes with 500 training examples each. We adopt the training setup from Ma al. (2021) and reset layers starting from the third block (L = 23) during LLF, As illustrated in Table A7 we find LLF outperform the baselines Tiny-ImageNet.
Furthermore, we also adopt two highly competitive baselines for CIFAR-10 and CIFAR-100 (Krihhvssyy al., 2009) in the literature: WidcResNee-28-11 (Zagoruyko & Komodakis, 2016) and DC (A = 12) (Huang et al., 2017). CIFAR-I0 contains 10 classes with 5000 training examples each, while CIFAR-100 contains 100 classes with 500 examples each. For WedeResNet- 28-10, 10, we reset from the second block (I. 10), and for DenseNet-BC. we reset only the fully- connected output layer (L = 99) during LLF. We report our results in Table A8 In these settings, we do not see any improvements from LLF over our baselines.
For these dataset extensions, we follow the same simple heuristic that was used for the five small datasets in the main text. Namely, take existing training setups and hyperparameters from the literature as is and treat it as one generation, and search for over a restricted set values (6 or fewer). Better results may be obtained through finer hyperparameters tuning.
A2.4 COMPARISON TO LW
Concurrent work (Alabdulmohsin et al. 2021 studies various types of iterative reinitialization ap- proaches and finds that reinitializing entire layers produced the best performance. They propose a layer-wise reinitialization scheme, proceeding from bottom to top and reinitializing one fewer layer each generation. In addition, Alladuummohsin et al. (202T) performs weight rescaling and activa- tion normalization for each layer that is not reinitialized. We refer to this method as LW and add
18

---

Published as a conference paper at ICLR 2022
Table A8: Later Layer Forgetting on CIFAR-I0 and CIFAR-100 using WideRes.Net-28-1 (WRN) and DenseNet-BC (DN). Results are test accuracy averaged over 3 runs and reported for hyperpa- rameter settings with best validation performance. We consider three additional training generations on top of the baseline model. The long baseline scales the step LR schedule based on the additional epochs.
Method
CIFAR-10
CIFAR-I00
WRN Smth (NI)
96.09
81.20
WRN Smth long (N3)
96.32
81.29
WRN Smth + LLF (N3) (Ours)
95,91
80.95
DN Smth (NI)
95.21
77.12
DN Smth long (N3)
95.68
78.12
DN Smth + LLF (N3) (Ours)
95.52
78.13
comparison to their method in Table [ We find that LLF outperforms LW across all datasets consider. However, we note that L.LF introduces an additional hyperparameter for the layer thresh- old, which LW does not require. Additionally, we find that under our experimental settings, several choices proposed in LW actually harms performance. We remove the addition of weight rescaling and normalization, and do not reinitialize batchnorm parameters, in order to achieve the best per- formance under the LW schedule. The reinitialize the first few layers. Doing so in LW hinders their progress the first few genera- tions of training, and also leads to poorer generalization performance even after completing the full sequential schedule.
hypothesis suggests that it is unnecessary
109
389
330
Sedine 5o 506
Kaninn 1450
Lnserboce
Layeriddox
(a) KNN probe for KE
(b) KNN probe for Long Baseline.
(c) KNN probe for LLF.
Figure A3: KNN probe train accuracy for various ResNet18 models trained on Flower. The layers shown are the end of each block and after the softmax operation. We K=3 to compute accuracy using the activations from each layer. The different curves in each panel show that overall prediction depth (as approximated by average KNN accuracy across layers) decreases with more training. LLF in (c) shows the most significant change, with the KNN probe performing near perfect at layer 6, which is earlier than the other two methods.
A3 DETAILS ON PARTIAL BALANCED FORGETTING
Our experimental settings follows Li & Bowling 2019). We use one-layer LSTMs with a hidden size of 100 to parameterize both the sender and receiver agents. We train using batch size of 100, and use the Adam optimizer (Kingma & Ba 2014) with a learning rate of 0.001. The language contains vocabulary size of 8 and a message length of 2. We use 4 SHAPP and 8 COLOR attributes, and the object set consist of 32 combinations these attributes.
The agents are trained with REINFORCE and entropy regularization using the following objectives:
Egs s [R(i, Ves log S [)] + 25 Vgs 5mm5(mm [t)
gs
19

---

Published as a conference paper at ICLR 2022
Table A9: Comparing KE and Later Layer Forgetting for ResNet18. Results are accuracy averaged over 3 runs and reported for hyperparameter settings with best validation performance. KE experiments use KELS split with split rate of 0.8. LLF uses L {10, 14), corresponding to block 3 and 4 in ResNet18. N3, N8, NIO indicate the additional number of training generations on top of the baseline model. LLF consistently outperforms other methods.
Method
Flower
CUB
Aircraft
MIT
Dog
Smth (NI)
51.02
58.92
57.16
56.04
63.64
Smth long (N3)
59.51
66.03
62.55
59.53
65.39
Smth + KE (N3)
57.95
63.49
60.56
58.78
64.23
Smth + LLF (N3) (Ours)
63.52
70.76
68.88
63.28
67.54
Smth long (N10)
66.89
70.50
65.29
61.29
66.19
Smth + KE (N10)
63.25
66.51
63.32
59.58
63.86
Smth + LLFF (N10) (Ours)
70.87
72.47
70.82
64.40
68.51
Smth + LW (N8)
62.84
70.58
67.36
61.24
66.58
Smth + LW NoNorm NoRescale (N8)
65.97
71.15
67.97
63.13
66.96
Smth + LW(-BN (-BN) (N8)
63.67
70.10
66.79
60.10
Smth + LW (-BN) NoNorm NoRescale (N8)
68.43
70.87
69.10
61.67
66.97
Smth + LLF (N8) (Ours)
69.48
72.30
70.37
63.58
68.45
CS-KD (N1)
57.57
66.61
65.18
58.61
66.48
CS-KD long (N3)
64,44
69.50
65.23
57.16
66.43
CS-KD + KE (N3)
63.48
68.76
67.16
58.88
67.05
CS-KD + LLF (N3) (Ours)
67.20
72.58
71.65
62.41
68.77
CS-KD long (N10)
68.68
69.59
64.58
56.12
64.96
CS-KD + KE (N10)
67.29
69.54
68.70
57.61
67.11
CS-KI) LLF (N10) (Ours)
74.68
73.51
72.01
62.89
69.20
CS-KD + LW (N8)
65.80
70,94
66.37
57.96
66.74
CS-KD + LW NoNorm NoRescale (N8)
69.96
70.91
69.53
60.22
67.47
CS-KD + LW(-BN) (N8)
69.54
68.87
CS-KD + LW(-BN) NoNorm NoRescale (N8)
73.72
71.81
70.82
59.18
68.09
CS-KD + LLF (N8) (Ours)
73.48
73.47
71.95
62.26
69.24
V V = E (R(f, Van log R( m, c)] + R VanH|x m, c)
We use 2 = 0.1 and A 0.1 in all our experiments, which differs slightly from & Bowling 2019) that use X 0.05, however observe that the difference is small between these settings. We train the "no reset" and "reset receiver" baselines for 6000 iterations per generation for 50 gener- ations, following the hyperparameters in Bowling 2019). We train PBF method with "same weight reinit" for 3000 iterations per generation for 100 generations for best performance. However we note that even using the same 50 generation hyperparameter, PBF significantly outperforms its baselines.
A4 EMERGENT LANGUAGE VISUALIZATIONS
In this section, we aim to visualize the languages learned by the sender Lewis game in the set- tings Ren et al. (020) and & Bowling (2019). Despite all the runs having near-perfect training accuracy the degree compositionality exhibited by the emergent languages varies significantly depending the use of forget-and-retrail
evident in Figure 4 In both these settings, each object has two attributes that can take values in A = (I,...,71) and Ag = (l...., ...,,T2) respectively, and (.....5y)
each message has two tokens that can take values in Mi (1...., $1) 81) and M2 = ( 82) respectively. Let us represent represent m!!) if message token has the value j. and 0 otherwise. When r 72 81 82
d; as attribute has the value j. and 0 otherwise. Similarly, we
20

---

Published as a conference paper at ICLR 2022
Table A10: Comparing KE and Later Layer Forgetting for DenseNet169. Results are test accuracy averaged over 3 runs and reported for hyperparameter settings with best validation performance. KE experiments use WELS split with a split rate of 0.7. LLF uses L {40, 68). N3, N8, N10 indicate the additional number of training generations on top of the baseline model. LLF consistently outperforms other methods.
Method
Flower
CUB
Aircraft
MIT
Dog
Smth (N1)
45.87
61.59
58.06
57.21
66.46
Smth long (N3)
59.39
70.87
67.55
62.29
68.82
Smth + KE (N3)
58.35
68.85
65.63
60.35
68.64
Smth + LLF (N3) (Ours)
62.31
71.97
70,53
64.60
70.19
Smth long (N10)
67.47
71.98
70,69
60.72
67.48
Smth KE (N10)
65.19
70.20
67.47
60.77
68.62
Smth + L.FF (N10) (Ours)
70.09
73.12
74.31
61.69
69.93
CS-KD (NI)
52.95
64.28
64.87
57.61
66.90
CS-KD long (N3)
60.03
63.80
67.68
57.21
67.01
CS-KD + KE (N3)
63.33
66.78
68.47
58.81
68.49
CS-KD + LLF (N3) (Ours)
65.43
73.54
72.22
61.74
70.61
CS-KD long (N10)
63.84
63.58
67.22
55,49
64.69
CS-KD + KE (N10)
69.24
66.34
69.51
58.33
67.73
CS-KD + LLF (N10) (Ours)
72.23
74.20
73.30
63.18
71.07
(i.e. the total number of input attribute values is equal to the total number of message token values), a compositional language with maximum topographic similarity (p) would have each input attribute activation dg mapped to a unique message token activation my
(1)
Thus, can represent the inputs and the messages both concatenations of two one-hot (r))
tors a = (1) .......... a a2
(1) ......."" (=²)
mm())) respec- and m = {m] ..... m mm) ...., m2
(#2)
tively. In Figures A8 we illustrate the mappings from input dimensions to message di- mensions m. Visualized values closer to indicate a more consistent mapping across the en- tire dataset, indicating compositional mapping. Concretely, for each active input dimension as we vary Agt across all its values and collect the corresponding message token probabilities ..... m, (+2) produced by the sender via two softmax operations. Then, the mean of these message probability vectors E,yz [m]] across Ayt. is used as the row cor-
i = (mè, (1) ....., nū (e1), m2 (1)
responding to input dimension a For perfectly compositional mapping, the visualized matrix
resembles permutation matrix.
For iterated learning in Figure A4 and PBF in Figure A6 we see the emergence of composi- tional mapping. The compositional component is retained across generations whereas the non- compositional components are disproportioonatey affected in the forgetting step and are more likely to get remapped. Finally, during the retraining phases, we see the compositional mappings get strengthened.
A5 CONNECTIONS TO OTHER TYPES OF FORGETTING
In the continual learning literature, catastrophic forgetting (McCloskey & Cohen
1989
Ratcliff
1990) refers to the decrease in performance on older tasks due to the learning new tasks. There is
large body of literature focused avoiding catastrophic forgetting (Ratclifi 1990: Robins 1995
French 1999 Kirkpatrick et al. 2017 Serra et al., 2018). In this work, we view forgetting in a positive light, to say that it aides learning. It is conceivable that catastrophic forgetting
be leveraged to help learning with a well-chosen secondary task in a forget-and and-releam process. is worth noting that the continual learning problem is in some sense the opposite of what consider
21

---

Published as a conference paper at ICLR 2022
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Input
Input
Input
Input
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Message dim (a)
Message dim
Message dim
Message dim
(b)
(c)
(d)
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
Input
Input
Input
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Message dim (e)
Message dim
Message dim
Message dim
(f)
(g)
(h)
Figure A4: Mappings from imput dimension to message dimension learned by the sender using iterated learning in the setting Ren al. (2020), logged after every generation for the first 32200 training iterations across all phases. A compositional language appears as a permutation matrix, where each input dimension maps unique message dimension.
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
Input
Input
Input
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Message dim (a)
Message dim
Message dim
Message dim
(b)
(c)
(d)
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
Input
Input
Input
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Message dim
Message dim
Message dim
Message dim
(e)
())
(g)
(h)
Figure A5: Mappings from input dimension to message dimension learned by sender without any form of rertean in the setting of Ren et al. (2020), logged uniformly for the first 33000 training iterations. A compositional language would have appeared as a permutation matrix, where each input dimension maps to unique message dimension.
22

---

Published as a conference paper at ICLR 2022
Imput ciim
0.75
Imput citn
0.75
0.75
0.75
Input
Input
0.25
0.25
0.25
0.25
Messaoe dim (a)
Message dim
Message dim
Messaoe dim
(b)
(c)
(d)
1.00
1.00
1.00
1.00
Inpur dim
Input dim
Impur dim
Impur dim
0.30
0.30
0.30
0.30
0.25
0.25
0.25
0.25
Messade dim (c)
Messade dim
Messade dim
Messade dim
(f)
(h)
Figure A6: Mappings from input dimension to message dimension learned by the sender using Partial Balanced Forgetting (PBF) in the setting of & Bowling (019), logged after every two generations for the first 48000 training iterations. more compositional language has unique non- overlapping message dimension activations for different input dimensions.
0.75
0.75
0.75
0.75
Input
imput
imput
Imput
0.25
0.25
0.25
0.25
Messade dim (a)
Message dim
Message dim (c)
Message dim
(b)
(d)
1.00
1.00
1.00
1.00
Input 6/m
Input dmm
Imput dim
Imput d.m
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
Messade dim (e)
Messade dim
Message dim
Message dim
(f)
(h)
Figure A7: Mappings from input dimension to message dimension learned by the sender by reset- ting the receiver periodically following the setting of Li & Bowling (2019), logged after every generation for the first 48000 training iterations. A more compositional language has unique non- overlapping message dimension activations for different input dimensions.
23

---

Published as a conference paper at ICLR 2022
1.00
1.00
1.00
1.00
Input dim.
0.75
Input dim
0.75
Input dmm
0.75
Input dm
0.75
0.25
0.25
0.25
0.25
Message dim (a)
Message dim
Message dim
Messaoe cim
(b)
(c)
(d)
1.00
1.00
1.00
1.00
Input dim
Input dim
Input dim
0.30
0.50
0.50
Inpat
0.50
Messaoe dim
Messaoe dim
Messaoe dim
Messaoe dim
(h)
Figure A8: Mappings from input dimension to message dimension learned by the sender without any form of forget-and-reerain in the setting [i & Bowling (2019). logged uniformly for the first 48000 training iterations. A more compositional language has unique non-overlapping message dimension activations for different input dimensions.
in forget-and-relearn. Continual learning seeks learn different tasks without forgetting old ones, while we explicitly assume that the tasks are available for iterative relearning.
Shwartz-Ziv & Tishby (2017) studies the progression of mutual information between each layer and the input and target variables throughout the training of deep neural networks. They reveal that the converged values lie close the information bottleneck theoretical bound, which implies a decrease in the mutual information with the input. In sense, the information bottleneck leads the "forgetting" of unnecessary information. It would be interesting explore whether there are fundamental similarities between the information loss caused by SGD training dynamics and the effect resulting from forget-and-relean
Barrett & Zollman 2009) studies forgetting in the Lewis game from the perspective of suboptimal equilibria. They show that using learning rule that remembers the entire past (i.e. Herrnstein reinforcement learning), the model converges to worse equilibrium than learning rules that discard past experience. Yalnizyan-Carse the reinforcement learning formulation. These works further point the possibility of designing forgetting mechanisms that are built in to the learning process itself.
Richards (202I) also studies the benefits of forgetting in
Although claim many iterative training algorithms as instances of forget-and-relenn there also many that we do not, even though they may be related. Algorithms like born-again networks (Furlanello et al. 2018) and self-training with noisy students (Xie et al., (020) rely iterative self-distillation, and do not have distinct forgetting and relearning stages. Instead, knowledge is passed to the next generation through pseudo-labels. A number of papers have tried to understand why students trained through self-distillation often generalize better than their teachers (Phuong &
Lampert 2019 Tang al. 2020 Allen-Zhu & Li, 2020) Stanton al. 202T); perhaps viewing
these methods through the lens of forgetting undesirable information offers an alternative path.
24


Reviews:
[{"review_id": "kFYTOf8PwU", "reviewer": "Reviewer_SWHW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a curious new perspective on forgetting that can have significant potential impact on the deep learning research in general. Since forget-and-relearn is a general framework, it can improve a broad range of deep learning applications: not only low-resource image classification and language emergence, but a myriad of others including more practical vision tasks, nlp tasks, speech processing, RL applications and untold others. However, there are significant issues in the current version of the paper that prevent me from recommending acceptance.\n\nThe first and main issue is that claiming forget-and-relearn as a general framework **needs more experimental support**. For each major conclusion (difficult examples, compositionality), it is important to check that it holds across many setups with different dataset sizes, training objectives and model architectures. However, this paper only evaluates them on two relatively niche problems. In contrast, related works such as Taha et al. (2021) make a much weaker claims, concerning only with small datasets in computer vision, and yet they explore a more diverse set of tasks including ImageNet classification.\n\nMy second (and less significant) concern is that authors do not compare how forget-and-relearn relates to prior art that also connects forgetting and improved generalization, albeit using different terms. One potentially relevant work (R Schwartz-Ziv and N. Tishby, 2017) studies training neural networks from the information perspective and finds that there is a distinct phase where neural network improves generalization by reducing the amount of information about input features in its intermediate representations. Another is Furlanello et al. (2018) where authors propose born-again networks - a form of knowledge distillation where multiple copies of the same network are repeatedly re-trained from scratch to improve the training performance. However, unlike forget-and-relearn where model retains some of its weights, born-again networks retains the useful information from the previous versions using knowledge distillation.\n\nMy final and least important issue is the presentation. While the paper is generally well-written, it has a rather unconventional structure with mixing theoretical and experimental sections. While there deviating from the conventional structure *can* sometimes be justified, it makes the paper more difficult to follow. In the case of this specific paper, I do not understand the motivation for changing the structure, especially since many related papers (Taha et al. 2021, Alabdulmohsin et al 2021) managed without it.\n\nReferences:\n- Taha, A., Shrivastava, A., & Davis, L.S. (2021). Knowledge Evolution in Neural Networks. ArXiv, abs/2103.05152.\n- Alabdulmohsin, I.M., Maennel, H., & Keysers, D. (2021). The Impact of Reinitialization on Generalization in Convolutional Neural Networks. ArXiv, abs/2109.00267.\n- Shwartz-Ziv, R., & Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. ArXiv, abs/1703.00810.\n- Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., & Anandkumar, A. (2018). Born Again Neural Networks. ICML.\n\n\n", "summary_of_the_paper": "The paper attempts to unify several prior observations on how partial forgetting during neural network training can affect the final training results. Authors propose forget-and-relearn, a general framework that explains how partial (but not total) one or several times midway through training can improve the final result by mostly forgetting \"undesirable\" information. To test their hypothesis, authors run experiments in image classification and language emergence, where they partially \"forget\" learned parameters during training using either later-layer forgetting (LLF) or partial balanced forgetting (PBF), and measure how forgetting affects overall model performance and some specific behaviors. Authors show that a simple perturbations such as \"forgetting\" layers later in the network can improve performance by disproportionally forgetting difficult and mislabeled examples. In the language emergence task, authors show how forget-and-relearn improves compositionality of the learned language by disproportionally forgetting the non-compositional features.", "summary_of_the_review": "The paper opens a new and potentially immensely useful perspective on training deep neural networks. However, making claims this strong requires proportionally strong and versatile supoort of experiments and/or theoretical results. Therefore, while this paper has promise, I find its current form too raw and unpolished for publication in a major venue such as ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TkaAcCUQw45", "reviewer": "Reviewer_UMJe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Update after rebuttal\n* CF was just an example, main point is to provide a better judgment for readers, hence it is necessary to go beyond accuracy and report various metrics. An important point to highlight, given current work is highly empirical, is to show why this method works rather than showing it works.\n* Authors should report other metrics such as Precision/Recall, AUC.\n* Authors should avoid using any strong claims/words without having any theoretical evidence.\n* I agree work is interesting and offers a unique direction for training DNNs, but more work is needed to back a few claims presented in this work.\n* Visualizing filters at later layers will further strengthen this work.\n* Good job in modifying the work and providing additional results.\n* Overall authors have addressed major concerns and confusion.\n\n# Strength\n* Easy to understand the paper, well written\n* Good sets of experiments and sufficient information provided to replicate the results.\n\n# Weakness\n* Strong claims and experiments fail to justify those.\n* Comparison with human or brain should be avoided, in fact, recent findings suggest that forgetting is also an essential part, and citations to such works are essential to motivate this research (https://www.nature.com/articles/d41586-019-02211-5).\n* Accuracy is not an efficient metric to analyze forgetting in neural networks.\nSuppose the hypothesis shows that forgetting and relearning is essential. In that case, the author should conduct experiments showing backward and forward transfers and analyze how much information you lose in the forgetting phase. Show fisher information matrix before, after forgetting, and after relearning, and answer if the same weights are active during all three stages. \n\n* I am also not convinced of the way experiments are conducted; why only tune the last few layers? This is a kind of transfer learning; you keep representation intact and tune on newer tasks or prune on the current task, allowing the least active neurons to change weights to generalize to a given task. It is pruning with sparsity added to obtain low-layer features.  Adding sparsity always helps; even in continual learning, it has shown advantages, which current work fails to cite [3-4]. So I won’t call this forgetting which is misleading, but another form of sparse pruning approach should be called reinitialize and pruning. \n\n* One can achieve sparsity by even using regularization [1], or prune weights using activation functions such as neural activity sparsity [2] or k-winner take all (kWTA)[5]; authors should at least compare with these approaches and show how efficient and computationally efficient their pruning strategy is. Hence I am not convinced by the results, given that important baselines are missing.\n\n* Co-adaption will always lead to overfitting on later layers; this is not true. It may be true in some scenarios, but not for all. I would avoid such claims without experiments or keep language-neutral, saying other effects can also be seen for various models.\n\n* Large scale experiments should be performed with natural images; how does this affect hold for cifar and imagenet. MNIST, there are very few difficult examples; even those examples share most of the representation with other not-so-difficult examples. Another approach would be augmenting data and creating examples that are difficult to understand for neural networks and showing how this pruning-based approach would work and scale. \n\n* In terms of comparison with LW, you should also report performance on their benchmarks. Additionally, the hyperparameter introduced in their work is essential, and performance can vary if you do not carefully tune it. Did you tune it? If not, reporting numbers on their benchmark would be handy. I would like to know the computational cost introduced for all phases (training, forget and learn) and memory footprint (given you copy the network).\n\n\n\n* [1] https://pubmed.ncbi.nlm.nih.gov/25987366/\n* [2] https://arxiv.org/abs/2104.06153\n* [3] https://arxiv.org/abs/1801.01423\n* [4] https://arxiv.org/abs/1905.10696\n* [5] https://proceedings.neurips.cc/paper/2015/file/5129a5ddcd0dcd755232baa04c231698-Paper.pdf", "summary_of_the_paper": "# Interesting Idea, but needs major revision and additional results \nIn this work, the authors introduced forgot and relearn framework to unify disparate existing iterative algorithms. They draw several insights and show that iterative training can achieve stable performance and generalize better on various tasks. They propose various hypotheses showing how forgetting undesired information can help calculate the importance of features important for downstream tasks. These back these claims by conducting experiments on various benchmarks. ", "summary_of_the_review": "# Additional experiments and visualizations is necessary \n* Besides the points stated above, I am also interested in seeing the statistical significance of the result. Can the author report a standard error for all experiments for k trials (at least k >2)? Please show hidden representation with later layers during the before phase, forget/reinitialize phase and relearn/pruning phase. \n* Additionally, the effect of sparsity on model performance, how to ensure signals or later features are sparse enough for the model to generalize better. Having theory to back these claims is ideal, given current work is heavily experimental, I would encourage authors to add more insight and results to back their claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fx5piVVGqD", "reviewer": "Reviewer_MVhg", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is beautifully written and lets the reader smoothly build an understanding of recent DL regularization methods over the proposed forget-and-relearn foundation. \nThe scheme of a clearly stated hypothesis at the beginning that undergoes falsification attacks throughout the paper is well-realized and should set a trend. \nThe paper is technically sound, from the definition of the forgetting operation to the setup of the various experiments and their presentation. The math discourse flows.\nAuthors show great confidence in recent literature. All the paper is disseminated with well-discussed references that are valuable to the reader also beyond this specific work. \n\nA relevant contribution of this work is the Later-Layer-Forgetting NN training method that substantially outperforms baselines and competitors in image classification. The results of the experiments in the entirely different scenario of language emergence give further evidence to the generality of the LLF approach.\nNevertheless, I believe that the core contribution of this work remains the forget-and-relearn interpretation of existing (and future) DL methods, which shows a remarkable explanatory potential - demonstrated by the straightforward derivation of the LLF method itself - and could positively animate the ML and DL debate as other proposed interpretations did in the past, such as the lottery ticket hypothesis.\n\nI hope authors persist in this line of research in the future. I would undoubtedly follow new works in this direction. In this regard, I believe authors can integrate their conclusion with a richer discussion of future directions. I think there are many:\n- The accuracy gains persist with larger datasets such as Imagenet? How many forget and relearn cycles we should perform? Can we simply wait for convergence also for big datasets?\n- Can the forget-and-relearn paradigm help in refining our understanding of the continual learning problem? How should we use LLF in a continual learning scenario?\n- What is the impact of LLF in deep reinforcement learning? For instance, the iterative training procedure of AlphaGo Zero that imposes to learn under a new condition at each generation could especially benefit from forgetting-and-relearn methods such as LLF.\n- How LLF should be applied to other learning problems and architectures (e.g., transformers)?\n- Do we register further gains in accuracy by gradually forgetting fewer layers in LLF?", "summary_of_the_paper": "The authors propose forgetting as a key concept to understand the success of recent DL regularization methods such as Iterative Magnitude pruning. The authors articulate their view in the *forget-and-relearn* hypothesis and extensively test it through the paper. They leverage this understanding to shape a new simple forget-and-relearn regularization scheme that effectively reduces overfitting in image classification and promotes compositionality in language emergence problems.", "summary_of_the_review": "This paper offers a new hypothesis to explain the success of several recent DL methods. This hypothesis is well supported by experiments and offers a straightforward method to reduce overfitting, effective in image classification and language emergence problems. I believe it could positively animate the ML and DL debate as other proposed interpretations did in the past, such as the lottery ticket hypothesis. In my opinion, this article should be highlighted at the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7H4X4O7EW7O", "reviewer": "Reviewer_hRdL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe work presents an alternative hypothesis to the idea of forgetting behavior in models. Indeed, it is an interesting proposal. Forgetting to leave room to better capture existing features and relearning the more difficult samples is a curious perspective. \n\nStrengths:\n\n* The ideas in this work are novel to the best of my knowledge. \n* The work is clear, well written and enjoyable to read.\n* The experiments nicely demonstrate the paradigm \n\nWeaknesses:\n\n* The datasets in the paper are not referenced. Please, include references, and include in the text a description of them, like sizes, etc.\n* The work is shown for two tasks in their limited setups. This poses a question of whether the hypothesis can be proven and is general. The work does not prove the hypothesis, which jeopardizes the generality of the method. Unfortunately, a formal proof is not included.\n* Still, if we assume the hypothesis to be generally true, how can the reader understand to apply the paradigm to any other problem?\n* A few experimental details are not discussed. For example, how is L chosen to be within the specific values in the big networks? A better way would be to show the performance as function of L. \n\nQuestions to the authors:\n- The improvements to iterative algorithms obtained under “forget-to-relearn” were handled based on properties from the model and its learning. How can we know what features, layers, etc. should be reset in the general case? If we don’t know a-priori where or how the tail of features is learned, how can we improve training further? I would appreciate a general discussion from the authors added to the paper on this matter.\n- In language, it has been proposed that a network could be pre-trained for shorter sequences and later on with longer ones (e.g., [1]). Could you discuss how the forget-relearn paradigm relates to such cases where the network is pre-trained and then extended to train on additional weights or inputs?\n- Recent research in generalization of neural networks, suggests that overparameterization, first overfits and then a second dip allows for better generalization of models [2]. Are these two processes complementary or contradictory? Why? \n\n[1] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018.\n[2] Belkin et al., Reconciling modern machine-learning practice and the classical bias–variance trade-off, 2019.\n", "summary_of_the_paper": "The paper explores the forgetting effect as a positive effect for learning. The authors suggest that a “forgetting to relearn” hypothesis is the underlying mechanism. This mechanism removes unvaluable or weak features at forget time, while models relearn and reinforce the remaining features. Then the hypothesis is used to explain effects on iterative training algorithms and further suggest some improvements. The hypothesis is empirically evaluated by improving iterative algorithms in two cases: image classification, and language emergence via the Lewis game. In image classification, the authors propose later-layer forgetting (LLF) based on the idea that easier samples are decided in the first layers of a convolutional neural network, while difficult examples are learned in deeper layers. LLF reinitializes only the weights in deeper layers. The results over 5 datasets (Flower, CUB, Aircraft, MIT, Dog) show improvements over long training and other restarting methods, as well as with ResNet and DenseNet as models.\nIn language, the manuscript tests the Lewis game with a mixture of compositional and non-compositional language as the easy and difficult samples, respectively. Also, in this case, the authors come up with Partial Balanced Forgetting (PBF) which partially reinitializes both the sender and the receiver in the game, showing better results than previous hypothesized ideas about this game. Last, they test for the learning aspects during retraining under the forget and relearn paradigm, such as the strengthen of the model during retraining. \n", "summary_of_the_review": "Despite the main idea being counterintuitive at first glance, the topic becomes clearer and more interesting with the reading and the experiments, nicely explaining previous works. Therefore, I believe that the ideas have merit, however there is no theoretical proof that proves the hypothesis. There are some impracticalities of this work, that I would appreciate if the authors can address. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the paper, it introduces a forget-and-relearn framework to the iterative learning algorithm. It provides serval new insight that forgetting could be favorable to learning and validates the insights via image classification and language tasks. The idea is novel and inspiring. Although there are some debates on the experiment and the generality of the proposed method, I think authors answered those questions decently and many researchers would be interested in this direction.
Decision: Accept (Poster)