Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper addresses the problem of catastrophic forgetting during sequential task learning with recurrent neural networks, a notoriously difficult continual learning setting. 

The main theoretical contribution is a formalism, Natural Continual Learning (NCL), which generalizes several related continual learning approaches in the regularization-based family, including several recent SOTA techniques. This formalism applies to the probabilistic learning setting using neural networks, so it is not restricted to supervised learning or recurrent models.

Experimental evidence is provided to support the claim that NCL outperforms previous approaches.

Finally, interesting analogies are made to models of biological learning, which are also supported by accompanying experiments; similar analysis tools are applied to practical instantiations of the NCL framework in controlled experimental settings.

main_review: # Originality
## Context and Related work
The paper build on a well developed line of work in continual learning research, namely the regularization-based family of approaches. Such methods rely on relatively "coarse" approximations of second-order quantities, and hence have to come up with practical workaround which mitigate the issues arising from errors in such approximation and their systematic interactions throughout learning.

***

# Quality

## Technical Soundness
* I believe the technical framework proposed for NCL provides clarity and unifies several related approaches.
* However, this it is not itself novel, since all such approaches start from a Bayesian perspective and diverge in practical implementations of diverse approximations of similar quantities.
* While I believe there is merit to a unified approach, the primary limitations of this family of approaches are efficient and effective approximations; I believe the conceptual goals of this family of continual learning approaches were made clear in the literature for some time now, even as early as the EWC paper.
 

## Assumptions and Limitations
* While the text correctly identifies limitations of related approaches, it does not characterize the significance of the difference between related work and practical implementations of the proposed method. Could you please rectify this or point me to the relevant paragraphs?


## Completeness of Evaluation
* I am not convinced that the presented experiments sufficiently support the claims of superiority to related methods or indeed of relevance to the wider set of continual learning approaches.
* Could you please compare like-for-like with existing approaches on standard settings beyond MNIST classification? While RNNs are of particular importance, it is not clear that the tasks considered are themselves sufficient to demonstrate the superiority of proposed approach, at least not in my view. This is because we already have many solutions to toy continual learning problems; in my view, sufficient progress means solving continual learning in more realistic problem settings.

***

# Significance

## What's unique about this work?
* I appreciate the in-depth study of continual learning with RNNs presented in this paper, the detailed experiments and carefully controlled problem setups considered. This is a valuable paper! 
* However, I do not believe it presents enough evidence for the proposed practical algorithms to support claims of superior performance to related or alternative methods. Please let me know whether there are other reasons why no other continual learning approach could be successfully applied to this niche and please argue for the niche itself rather than claiming general applicability. If you mean general applicability in practice, I believe much more evidence is required. If you mean general applicability in principle, please quantify exactly what "general" means and provide an exhaustive list limitations, of both conceptual and practical nature.


# Summary
* This paper is interesting to a number of researchers working on regularization-based continual learning solutions and perhaps modelers of biological learning systems with recurrent models. However, I cannot argue for wider applicability and relevance, which is reflected in my score.
* Furthermore, while progress is being made and well reported in this paper, the significance of this progress to date since to be comparable to that of previously published papers, since the capabilities of continual learning algorithms proposed in this paper are virtually identical to previous work. I could not identify a strong argument for superior capabilities, but I am willing to be convinced otherwise.
* While the perspective is interesting and the analyses revealing, I am not convinced that they contribute enough to recommend acceptance.


limitations_and_societal_impact: The discussion of impact is not particular to this paper, and does apply to the entire field of machine learning, but is well thought out, despite not being specific to this paper.

I would suggest including a discussion of the impact of the work itself and its context rather than making too general remarks.

As it is the case with most papers and discouraged by prevailing reviewing practices, the paper expends little effort to characterize the limitations of the proposed practical algorithm, the diminished relevance of experiments presented within to setups of interest, to related approaches or the wider subfield. Truisms and 'catchphrases of the day' are offered in place of a honest discussion of what current continual learning methods have to offer today, and what is left for future work. I know this is difficult, but please make an effort along these lines!
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel regularization based continual learning approach, NCL, which is mainly used for recurrent neural network (RNN). Since NCL is based on Bayesian continual learning, NCL utilizes the approximate posterior of previous tasks as a prior for current task, and the Laplace approximation is used to compute the approximate posterior. Through simple toy analyses on convex and non-convex loss landscapes for NCL and other baseline methods, authors show that previous methods have a problem on the trajectories of convergence when learning a new task. Based on the results on simple toy analyses, authors proposed a novel trust region based regularization method that modulate the gradient by considering the curvature of prior distribution via its precision matrix. The proposed technique is highly similar to projection based methods, but its origin is quite different from previous methods. Experiment results show that NCL outperforms other baselines in stimulus-response and MNIST tasks. 
main_review: Pros

- This paper proposed a novel trust-region based regularization methods that can outperform other baselines. The methods modifying the gradients used to update the parameters are quite novel and impressive, and the authors well analyzed the similarity between NCL and other projection based methods in related work.

- By showing simple toy analyses with other baselines, it is easy to see the problems of previous methods and exact trajectories when learning a new task. 

Cons

- It would be great for carrying out some large-scale experiments on language or time-series dataset. Based on the experiment results that considered, it is hard to decide NCL is effective on general RNN architecture.

- I think the overall performance of NCL could be highly sensitive to the hyper parameter $r$ in Eq.(7). Since the $\gamma$ in Eq.(8) depends on the $r$, I think it is not easy to choose the proper value of $r$,


Question

- The mathematical form of updating weights in NCL is highly similar to gradient projection based methods except the term $(\theta - \mu_{k-1})$ in Eq.(8). Does the excellence of NCL over gradient projection based methods is from considering Hessian in Laplace approximation or the term $(\theta - \mu_{k-1})$ which tries to reduce the deviation from $\mu_{k-1}$?


Minor comments

- In my humble opinion, in line 146, there is a typo: involves "maximizing" the posterior --> involves "minimizing" the posterior.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a method called Natural Continual Learning (NCL). NCL unifies the concepts of weight regularization and gradient projection into non-interfering subspaces. Gradient projection is implemented as a trust region optimization using the Fisher information matrix. The authors also introduce a Kronecker-factor approximation to improve the scalability properties of their method. The comparison against other continual learning approaches is conducted using recurrent neural network (RNN) architectures. 
main_review: I found the paper to be original. 

The paper is mostly clearly written (see Minor Issues later on) and well structured. 

A major advantage of the paper is that it is quite easy to follow the authors' motivation and thought process that led to the development of the proposed method.

The paper's significance is reduced by the fact that the experimental evaluation was somewhat unconventional and it was conducted only using RNNs. I consider this a significant downside, given that the method is not designed exclusively with RNNs in mind. The paper would be considerably strengthened were the authors to show strong results in more standard CL benchmarks, and when using more challenging datasets than MNIST.

Specifically for subsection 3.2, I found the conclusion reached (see lines 264-266) very interesting and I think it should be validated further using other datasets.

Minor Issues:
- Lines 59-60: The authors write "or network size decreases." At first, I thought that the authors meant a decreasing network size over time, but I think they just mean a limited network capacity relative to the learning task at hand. If the latter is correct, you should probably rephrase.
- In the problem statement paragraph, the authors should probably emphasize that each task has its own distinct data distribution.
- I would also recommend, instead of using the words task and dataset interchangeably (e.g., in subsection 2.1), to instead use only the word task as is customary in CL papers.
- For the vector Δ, introduced in line 147, I would use a bold lowercase character to emphasize the fact that it is indeed a vector and not a matrix.
- Line 334: I think "is not" should be replaced with "are not."
- I recommend using the same vertical axis for Figures 3A and 3B so that they are directly comparable.

Questions:
- Why did you choose to focus only on RNNs instead of experimenting with multiple types of architectures?
- In the non-convex case of the toy problem, why did you set the Task 1 loss to be convex?
- What is the motivation for expanding the number of tasks in the Stroke MNIST benchmark, and why did you make the specific choices detailed in lines 273-278?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a model for continual learning. Specifically, it intends to merge parameter regularization methods and gradient projection methods. The regularization is achieved by taking a bayesian perspective over the distribution over parameters, that is approximated by a Laplace distribution in a variational fashion, with online updates. The gradient projection is addressed by trust-region optimization, resulting in an update rule that uses the precision matrix to precondition the gradient of the current task objective, plus a term that keeps the parameters close to the previous mean. Experiments are carried out for sequential problems on a stimulus-response dataset and on stroke MNIST.

main_review: The paper is overall clear and well written, and the approach seems technically sound. All the mathematical notation is correct and the authors provide intuition for the most important equations, which helps readibility.

In terms of technical contribution, the novelty of this work is somewhat limited. As acknowledged by the authors, the bayesian perspective is very common in continual learning and the variational approximations involved here are quite popular. The use of the Fisher information matrix in lieu of the hessian is also standard. To my understanding, the overall model is closely related to Online-EWC [1], where however in the overall update the precision matrix weight the gradient of the current task objective rather than the regularization term. Also, it is not very clear to me the benefit of this scheme, besides getting rid of the need to tune an hyperparameter for the regularization loss.

My main concerns lie however in the experimental section. Specifically:
  - it is peculiar that the authors limit the application of their model to RNN problems only. Indeed, there is nothing specific in the formulation that suggests that the model would behave especially well for sequential problems. Given the fact that those are not so popular in continual learning, I think the paper would benefit from the comparison on standard experiments such as the ones derived from MNIST (split, rotated, permuted) or CIFAR-10. 
  - the datasets tested seem to be quite simple, and more complex learning problems may be included to make the conclusions more convincing. 
  - Some other standard models such as EWC and its online version should be added to the comparisons, given they are very popular and close in spirit to the model proposed here.
  - for the main comparisons, the authors report the loss (not clear whether it is on training or test sets) as a main metric. Other metrics would be more suitable, such as accuracy or forgetting measures.

---
Overall, I think the paper is slighly below the standard for NeurIPS. The approach presented here is clear and solid, but not fundamentally novel to my understanding. However, as mentioned above, the main weaknesses lie specifically in the experimental section. I think with a revision of the experiments this submission could meet the standard for acceptance. 

References:
[1] Schwarz, Jonathan, et al. "Progress & compress: A scalable framework for continual learning." International Conference on Machine Learning. PMLR, 2018.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce an extension to Bayesian continual learning whereby updates in successive tasks are constrained to be close to a “trust region” of the parameters found in preceding tasks. Exploiting the Bayesian nature of the model, the trust region in question can be defined in terms of the prior precision matrix. The authors show that the method, when applied to RNNs, is effective against baselines in a) a suite of neuroscience related tasks, and b) stroke MNIST datasets.
main_review: The paper offers an elegant contribution to the continual learning literature, and I can find very little to criticise, so this review is pretty short. The formal presentation is rigorous and clear and (as far as I was able to check) appears sound. Figure 1 presents the intuition behind the work very clearly. The evaluation is thorough. The method’s relationship to the rest of the literature is well described (although I’m not sufficiently familiar with this particular corner of the continual learning field to vouch that nothing is missing). The paper is logically organised and well written, and I find myself in the unusual position of not having found a single typo. Overall I’m happy to recommend acceptance.

I have one small recommendation. It wasn’t obvious to me why the authors chose an RNN model (and only an RNN model) to evaluate their method. Relatedly, it would have been interesting to see the method applied to other standard (non dynamic) continual learning benchmarks. I assume that these are the sorts of tasks where previous methods don’t do so well. (Apologies if the authors said that somewhere, but I can’t find it looking back over the paper.) Perhaps some intuition about why the method does well in this particular setting could also be given.
limitations_and_societal_impact: The paper is at the level of basic research, and - in my opinion - is far enough from deployment not to have to societal impact explicitly
review_title: null

Meta-review: The paper proposes a framework that unifies regularization based techniques in CL and projection techniques. As other reviewers pointed out (e.g. v6dD) this is maybe one of the better formally motivated and better studied direction for addressing CL. And hence at a high level the final result might not seem surprising (e.g. that regularizion methods can be seen as a trust region method, and then there is a deep connection between projection methods and these regularization methods). 

However I want to stress that the final algorithm and its derivation is by no means trivial. And I think the improvement (at least in the theoretical understanding) of these methods is of great values for those focusing on such techniques. So from my perspective, in terms of novelty and significance I think the manuscript has provided enough of both. And while there might be some scalability worries, I think there is a significant subset of the CL community that will find this result very useful. 

The main weakness of the work is the empirical exploration. While I believe the experiments are done carefully, and provide evidence for the efficacy of the algorithm (plus I welcome the new experiments on feedforward models), the scale of the empirical section is a bit weaker compared to an average paper on this topic. However, particularly considering the new results mentioned by the authors that should be integrated in the paper, I think they might just be sufficient to reach the requirement of the conference. 


Decision: Accept (Poster)