Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: # Interesting Idea, but needs major revision and additional results 
In this work, the authors introduced forgot and relearn framework to unify disparate existing iterative algorithms. They draw several insights and show that iterative training can achieve stable performance and generalize better on various tasks. They propose various hypotheses showing how forgetting undesired information can help calculate the importance of features important for downstream tasks. These back these claims by conducting experiments on various benchmarks. 
main_review: # Update after rebuttal
* CF was just an example, main point is to provide a better judgment for readers, hence it is necessary to go beyond accuracy and report various metrics. An important point to highlight, given current work is highly empirical, is to show why this method works rather than showing it works.
* Authors should report other metrics such as Precision/Recall, AUC.
* Authors should avoid using any strong claims/words without having any theoretical evidence.
* I agree work is interesting and offers a unique direction for training DNNs, but more work is needed to back a few claims presented in this work.
* Visualizing filters at later layers will further strengthen this work.
* Good job in modifying the work and providing additional results.
* Overall authors have addressed major concerns and confusion.

# Strength
* Easy to understand the paper, well written
* Good sets of experiments and sufficient information provided to replicate the results.

# Weakness
* Strong claims and experiments fail to justify those.
* Comparison with human or brain should be avoided, in fact, recent findings suggest that forgetting is also an essential part, and citations to such works are essential to motivate this research (https://www.nature.com/articles/d41586-019-02211-5).
* Accuracy is not an efficient metric to analyze forgetting in neural networks.
Suppose the hypothesis shows that forgetting and relearning is essential. In that case, the author should conduct experiments showing backward and forward transfers and analyze how much information you lose in the forgetting phase. Show fisher information matrix before, after forgetting, and after relearning, and answer if the same weights are active during all three stages. 

* I am also not convinced of the way experiments are conducted; why only tune the last few layers? This is a kind of transfer learning; you keep representation intact and tune on newer tasks or prune on the current task, allowing the least active neurons to change weights to generalize to a given task. It is pruning with sparsity added to obtain low-layer features.  Adding sparsity always helps; even in continual learning, it has shown advantages, which current work fails to cite [3-4]. So I won’t call this forgetting which is misleading, but another form of sparse pruning approach should be called reinitialize and pruning. 

* One can achieve sparsity by even using regularization [1], or prune weights using activation functions such as neural activity sparsity [2] or k-winner take all (kWTA)[5]; authors should at least compare with these approaches and show how efficient and computationally efficient their pruning strategy is. Hence I am not convinced by the results, given that important baselines are missing.

* Co-adaption will always lead to overfitting on later layers; this is not true. It may be true in some scenarios, but not for all. I would avoid such claims without experiments or keep language-neutral, saying other effects can also be seen for various models.

* Large scale experiments should be performed with natural images; how does this affect hold for cifar and imagenet. MNIST, there are very few difficult examples; even those examples share most of the representation with other not-so-difficult examples. Another approach would be augmenting data and creating examples that are difficult to understand for neural networks and showing how this pruning-based approach would work and scale. 

* In terms of comparison with LW, you should also report performance on their benchmarks. Additionally, the hyperparameter introduced in their work is essential, and performance can vary if you do not carefully tune it. Did you tune it? If not, reporting numbers on their benchmark would be handy. I would like to know the computational cost introduced for all phases (training, forget and learn) and memory footprint (given you copy the network).



* [1] https://pubmed.ncbi.nlm.nih.gov/25987366/
* [2] https://arxiv.org/abs/2104.06153
* [3] https://arxiv.org/abs/1801.01423
* [4] https://arxiv.org/abs/1905.10696
* [5] https://proceedings.neurips.cc/paper/2015/file/5129a5ddcd0dcd755232baa04c231698-Paper.pdf
summary_of_the_review: # Additional experiments and visualizations is necessary 
* Besides the points stated above, I am also interested in seeing the statistical significance of the result. Can the author report a standard error for all experiments for k trials (at least k >2)? Please show hidden representation with later layers during the before phase, forget/reinitialize phase and relearn/pruning phase. 
* Additionally, the effect of sparsity on model performance, how to ensure signals or later features are sparse enough for the model to generalize better. Having theory to back these claims is ideal, given current work is heavily experimental, I would encourage authors to add more insight and results to back their claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper attempts to unify several prior observations on how partial forgetting during neural network training can affect the final training results. Authors propose forget-and-relearn, a general framework that explains how partial (but not total) one or several times midway through training can improve the final result by mostly forgetting "undesirable" information. To test their hypothesis, authors run experiments in image classification and language emergence, where they partially "forget" learned parameters during training using either later-layer forgetting (LLF) or partial balanced forgetting (PBF), and measure how forgetting affects overall model performance and some specific behaviors. Authors show that a simple perturbations such as "forgetting" layers later in the network can improve performance by disproportionally forgetting difficult and mislabeled examples. In the language emergence task, authors show how forget-and-relearn improves compositionality of the learned language by disproportionally forgetting the non-compositional features.
main_review: This paper proposes a curious new perspective on forgetting that can have significant potential impact on the deep learning research in general. Since forget-and-relearn is a general framework, it can improve a broad range of deep learning applications: not only low-resource image classification and language emergence, but a myriad of others including more practical vision tasks, nlp tasks, speech processing, RL applications and untold others. However, there are significant issues in the current version of the paper that prevent me from recommending acceptance.

The first and main issue is that claiming forget-and-relearn as a general framework **needs more experimental support**. For each major conclusion (difficult examples, compositionality), it is important to check that it holds across many setups with different dataset sizes, training objectives and model architectures. However, this paper only evaluates them on two relatively niche problems. In contrast, related works such as Taha et al. (2021) make a much weaker claims, concerning only with small datasets in computer vision, and yet they explore a more diverse set of tasks including ImageNet classification.

My second (and less significant) concern is that authors do not compare how forget-and-relearn relates to prior art that also connects forgetting and improved generalization, albeit using different terms. One potentially relevant work (R Schwartz-Ziv and N. Tishby, 2017) studies training neural networks from the information perspective and finds that there is a distinct phase where neural network improves generalization by reducing the amount of information about input features in its intermediate representations. Another is Furlanello et al. (2018) where authors propose born-again networks - a form of knowledge distillation where multiple copies of the same network are repeatedly re-trained from scratch to improve the training performance. However, unlike forget-and-relearn where model retains some of its weights, born-again networks retains the useful information from the previous versions using knowledge distillation.

My final and least important issue is the presentation. While the paper is generally well-written, it has a rather unconventional structure with mixing theoretical and experimental sections. While there deviating from the conventional structure *can* sometimes be justified, it makes the paper more difficult to follow. In the case of this specific paper, I do not understand the motivation for changing the structure, especially since many related papers (Taha et al. 2021, Alabdulmohsin et al 2021) managed without it.

References:
- Taha, A., Shrivastava, A., & Davis, L.S. (2021). Knowledge Evolution in Neural Networks. ArXiv, abs/2103.05152.
- Alabdulmohsin, I.M., Maennel, H., & Keysers, D. (2021). The Impact of Reinitialization on Generalization in Convolutional Neural Networks. ArXiv, abs/2109.00267.
- Shwartz-Ziv, R., & Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. ArXiv, abs/1703.00810.
- Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., & Anandkumar, A. (2018). Born Again Neural Networks. ICML.



summary_of_the_review: The paper opens a new and potentially immensely useful perspective on training deep neural networks. However, making claims this strong requires proportionally strong and versatile supoort of experiments and/or theoretical results. Therefore, while this paper has promise, I find its current form too raw and unpolished for publication in a major venue such as ICLR.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors propose forgetting as a key concept to understand the success of recent DL regularization methods such as Iterative Magnitude pruning. The authors articulate their view in the *forget-and-relearn* hypothesis and extensively test it through the paper. They leverage this understanding to shape a new simple forget-and-relearn regularization scheme that effectively reduces overfitting in image classification and promotes compositionality in language emergence problems.
main_review: The paper is beautifully written and lets the reader smoothly build an understanding of recent DL regularization methods over the proposed forget-and-relearn foundation. 
The scheme of a clearly stated hypothesis at the beginning that undergoes falsification attacks throughout the paper is well-realized and should set a trend. 
The paper is technically sound, from the definition of the forgetting operation to the setup of the various experiments and their presentation. The math discourse flows.
Authors show great confidence in recent literature. All the paper is disseminated with well-discussed references that are valuable to the reader also beyond this specific work. 

A relevant contribution of this work is the Later-Layer-Forgetting NN training method that substantially outperforms baselines and competitors in image classification. The results of the experiments in the entirely different scenario of language emergence give further evidence to the generality of the LLF approach.
Nevertheless, I believe that the core contribution of this work remains the forget-and-relearn interpretation of existing (and future) DL methods, which shows a remarkable explanatory potential - demonstrated by the straightforward derivation of the LLF method itself - and could positively animate the ML and DL debate as other proposed interpretations did in the past, such as the lottery ticket hypothesis.

I hope authors persist in this line of research in the future. I would undoubtedly follow new works in this direction. In this regard, I believe authors can integrate their conclusion with a richer discussion of future directions. I think there are many:
- The accuracy gains persist with larger datasets such as Imagenet? How many forget and relearn cycles we should perform? Can we simply wait for convergence also for big datasets?
- Can the forget-and-relearn paradigm help in refining our understanding of the continual learning problem? How should we use LLF in a continual learning scenario?
- What is the impact of LLF in deep reinforcement learning? For instance, the iterative training procedure of AlphaGo Zero that imposes to learn under a new condition at each generation could especially benefit from forgetting-and-relearn methods such as LLF.
- How LLF should be applied to other learning problems and architectures (e.g., transformers)?
- Do we register further gains in accuracy by gradually forgetting fewer layers in LLF?
summary_of_the_review: This paper offers a new hypothesis to explain the success of several recent DL methods. This hypothesis is well supported by experiments and offers a straightforward method to reduce overfitting, effective in image classification and language emergence problems. I believe it could positively animate the ML and DL debate as other proposed interpretations did in the past, such as the lottery ticket hypothesis. In my opinion, this article should be highlighted at the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper explores the forgetting effect as a positive effect for learning. The authors suggest that a “forgetting to relearn” hypothesis is the underlying mechanism. This mechanism removes unvaluable or weak features at forget time, while models relearn and reinforce the remaining features. Then the hypothesis is used to explain effects on iterative training algorithms and further suggest some improvements. The hypothesis is empirically evaluated by improving iterative algorithms in two cases: image classification, and language emergence via the Lewis game. In image classification, the authors propose later-layer forgetting (LLF) based on the idea that easier samples are decided in the first layers of a convolutional neural network, while difficult examples are learned in deeper layers. LLF reinitializes only the weights in deeper layers. The results over 5 datasets (Flower, CUB, Aircraft, MIT, Dog) show improvements over long training and other restarting methods, as well as with ResNet and DenseNet as models.
In language, the manuscript tests the Lewis game with a mixture of compositional and non-compositional language as the easy and difficult samples, respectively. Also, in this case, the authors come up with Partial Balanced Forgetting (PBF) which partially reinitializes both the sender and the receiver in the game, showing better results than previous hypothesized ideas about this game. Last, they test for the learning aspects during retraining under the forget and relearn paradigm, such as the strengthen of the model during retraining. 

main_review: 
The work presents an alternative hypothesis to the idea of forgetting behavior in models. Indeed, it is an interesting proposal. Forgetting to leave room to better capture existing features and relearning the more difficult samples is a curious perspective. 

Strengths:

* The ideas in this work are novel to the best of my knowledge. 
* The work is clear, well written and enjoyable to read.
* The experiments nicely demonstrate the paradigm 

Weaknesses:

* The datasets in the paper are not referenced. Please, include references, and include in the text a description of them, like sizes, etc.
* The work is shown for two tasks in their limited setups. This poses a question of whether the hypothesis can be proven and is general. The work does not prove the hypothesis, which jeopardizes the generality of the method. Unfortunately, a formal proof is not included.
* Still, if we assume the hypothesis to be generally true, how can the reader understand to apply the paradigm to any other problem?
* A few experimental details are not discussed. For example, how is L chosen to be within the specific values in the big networks? A better way would be to show the performance as function of L. 

Questions to the authors:
- The improvements to iterative algorithms obtained under “forget-to-relearn” were handled based on properties from the model and its learning. How can we know what features, layers, etc. should be reset in the general case? If we don’t know a-priori where or how the tail of features is learned, how can we improve training further? I would appreciate a general discussion from the authors added to the paper on this matter.
- In language, it has been proposed that a network could be pre-trained for shorter sequences and later on with longer ones (e.g., [1]). Could you discuss how the forget-relearn paradigm relates to such cases where the network is pre-trained and then extended to train on additional weights or inputs?
- Recent research in generalization of neural networks, suggests that overparameterization, first overfits and then a second dip allows for better generalization of models [2]. Are these two processes complementary or contradictory? Why? 

[1] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018.
[2] Belkin et al., Reconciling modern machine-learning practice and the classical bias–variance trade-off, 2019.

summary_of_the_review: Despite the main idea being counterintuitive at first glance, the topic becomes clearer and more interesting with the reading and the experiments, nicely explaining previous works. Therefore, I believe that the ideas have merit, however there is no theoretical proof that proves the hypothesis. There are some impracticalities of this work, that I would appreciate if the authors can address.
review_title: null

Meta-review: In the paper, it introduces a forget-and-relearn framework to the iterative learning algorithm. It provides serval new insight that forgetting could be favorable to learning and validates the insights via image classification and language tasks. The idea is novel and inspiring. Although there are some debates on the experiment and the generality of the proposed method, I think authors answered those questions decently and many researchers would be interested in this direction.
Decision: Accept (Poster)