Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work highlights the shortcomings of the backpropagation algorithm for continual learning applications and proposed continual backprop that used a generate-and-test method to continually inject random features alongside SGD which enables better learning on non-stationary data streams. This approach is evaluated with the semi-stationary scenario of Bit-Flipping and continual supervised learning with permuted-MNIST as well as on non-stationary RL problems.
main_review: * The shortcomings of SGD have motivated many works on its adaptation for continual learning and several adaptations such as Orthogonal Gradient Descent [1], Stable-SGD [2] and alternative local learning approaches [3,4] have been proposed. A comparison with these methods would help with better assessing the proposed approach.

* Other benchmarks such as split-CIFAR100, split-IMAGENET, CoRe50 are frequently used to benchmark the continual supervised learning approaches. It would be fair to expand the comparison to a subset of these datasets.

* Classification accuracy might not be the best metrics to evaluate the continual leaning potential. Other metrics such as the backward and forward transfer are better suited [5].

[1] Farajtabar, Mehrdad, et al. "Orthogonal gradient descent for continual learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.
[2] Mirzadeh, Seyed Iman, Mehrdad Farajtabar, and Hassan Ghasemzadeh. "Dropout as an implicit gating mechanism for continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.
[3] Lindsey, Jack, and Ashok Litwin-Kumar. "Learning to learn with feedback and local plasticity." arXiv preprint arXiv:2006.09549 (2020).
[4] Madireddy, Sandeep et al. “Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning”, arXiv preprint arXiv:2007.08159 (2021)
[5] Mai, Zheda,  et al. "Online Continual Learning in Image Classification: An Empirical Survey." arXiv preprint arXiv:2101.10423 (2021).

summary_of_the_review: The approach looks promising, but is limited in terms of the experiments and the comparison with existing approaches that makes it difficult to assess the true potential of this approach
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper demonstrates and proposes a solution for a new problem in continual learning which is the inverse of catastrophic forgetting. Compared to prior work, they study problems where the data distribution changes much more rapidly. They demonstrate that backpropagation based optimization loses its ability to adapt when tracking these rapidly changing continual learning problems.  They show a degradation in performance over time on permuated MNIST, non-stationary RL problems, and the bit-flipping problem. They propose a solution to this problem by reinitializing some portion of the weights of every layer. They propose a utility function to choose which layers to reinitialize based on a combination of adaptation-utility and contribution-utility. They demonstrate that utilizing this method, they can achieve better performance that does not degrade over time and that it works in more cases than l2 weight decay.
main_review: Strengths:
- This paper proposes and tackles a novel important problem in continual learning that has not been observed in prior work.
- They demonstrate on both proposed solution is able to perform significantly better on this problem and does not degrade in performance over time compared to the vanilla back propagation on feed-forward networks on CV, RL, and bit-flipping problems
- They show empirical results on a variety of different Feed-forward networks with different hyperparameters and activation functions. They compare against 

Weakness:
- The paper does not compare the tradeoff of their problem and method with the prior work on catastrophic forgetting. It is possible that the problems and their solutions are opposites of each other and would be very useful to study.
- The paper would benefit from more directly exploring their hypothesis of "decaying plasticity" of neural networks by exploring the change in weights, gradients, and loss landscape over time.
- This paper only demonstrated their algorithm on Feed-forward networks. Since the paper is largely empirical in demonstrating this new problem, it would be significantly stronger if it could demonstrate this problem on Deep convolutional models, and attention networks. The model sizes and datasets are also relatively smaller. Does this problem continue with overparameterized models or other regularization techniques.

Questions/Suggesiton:
It may be useful to position the work compared to "Neural Rejuvenation" [1] since the reinitialization method is somewhat similar.

Are the errors shown test errors or training errors? Could this be a problem of overfitting and the proposed method is acting as regularization?

Have you experimented with even lower or higher learning rates? The accuracy for the lowest in the current graphs seems to still converge quite quickly.

Could we compare to a baseline of fully reinitializing the model when the data distribution changes to understand the upper-bound on the expected accuracy?

[1] Qiao, Siyuan, et al. "Neural rejuvenation: Improving deep network training by enhancing computational resource utilization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
summary_of_the_review: This paper proposes and tackles a very novel problem in continual learning that would have great significance to the field and is important to explore. They are able to demonstrate a somewhat novel effective algorithm that significantly improves performance, and does not degrade over time. Unfortunately, this paper is largely empirical and does not have sufficiently diverse experiments to characterize this problem. Given this, I currently recommend weak rejection for this paper. While it explores a large variety of activation functions and hyperparameters, all the networks are relatively small Feed-forward networks. They compare several methods and demonstrate that regularization is somewhat effective in some cases, but does not fully explore why regularization works or other methods. It currently falls short of fully explaining the root cause of the problem.

=================================================================
Post Rebuttal:
I would again like to thank the authors for including the additional experiments on model size and dataset change speed. I believe this paper is borderline, and am updating my review to marginally above the acceptance threshold. The additional experiments improved understanding and I believe this is a very useful direction to explore. I also agree with reviewer gcKs and Evj3, that this should be compared to existing continual learning methods and papers. The other datasets should also be useful to experiment with since they should still demonstrate at least some loss in Neural plasticity with 20 stationaries according to the other experiments. I also agree with reviewer kzUn on the depth of understanding this paper provides. This paper highlights a very useful novel direction to explore continual learning problems and a reasonable solution, but would drastically be improved if it provided much further depth.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses a modification of the BP algorithm, the Continual BP (CBP), that founds on the intuition that  standard BP cannot work in online learning scenarios due to weights departing from the initial condition of being small and random, after some training experience. The paper contributes with an empirical analysis of the phenomenon as well an algorithm to address the issue. 
main_review: STRENGTH

The topic of learning online, in continual, incremental or any other online setting, is certainly of high and growing interest in the community. Works trying to tackle the problem in its fundamental aspects, starting from the weight adjustment algorithms are very much interesting for the community. In this respect, this paper certainly takes an interesting and somewhat original view into the difficulties of learning online.

Additionally, the empirical analysis covering both supervised learning and reinforcement learning scenarios is certainly a strong aspect of the paper, at least for what pertains breadth of application (though, as discussed in the following, it comes at the cost of depth of the results).

WEAKNESSES

The paper has some substantial issues, ranging from solidity of the claims, scholarly value, novelty as well as empirical validation of the paper ideas.

In terms of claims, I strongly disagree with the sentence in the concluding paragraph of the introduction about “…we contribute to the understanding of why BP and its variants fail in non-stationary problems”.  The paper does not help understanding the root causes of the problem. It  provides empirical results showing that a certain amount of randomness reintroduced dynamically into the model helps in keeping the model able to learn. This is far from explaining why this happens. For instance, I would have expected to see at least experiments to rule-out the fact that the inability to learn is due to gradient vanish (this can be easily plotted by monitoring the gradient magnitude). I had the feeling that the departure from the small weight initialization might cause reduced learning due to larger weights bringing the neurons into the saturated parts of their activation functions. My first intuition was to think about regularizing on the norm of the weights, which is in fact providing effective results in the empirical analysis. Apart from the permuted MNIST case, the L2 regularization seems to be providing nearly the same advantage as the proposed CBP algorithm. 

In terms of scholarly value, the paper is often confusing with respect to the community it is referring to: it is somehow linking the paper both to the continual learning community as well as to the community of learning in non-stationary environments. The paper needs to make a clear choice as the underlying assumptions (e.g. forgetting Vs remembering), the related literature and the related benchmarks differ substantially. Currently, the paper is in between the two worlds, and unsatisfactory in the way it addresses the point above with respect to both. For instance, in the continual learning community, one might wonder on what are the relationships between the proposed method and other weight-constraining/regularization approaches such as EWC and LWF.

In terms of novelty, the solution proposed in the paper builds heavily on existing approaches, mostly on generate-and-test extended on non-major aspects, such as support for multi-layer perceptrons and non-LTU activations. Hence, I do not see much original content in the proposed CBP algorithm, whereas there might be some truly original insights coming from the intuition of the root causes of BP inability in learning online (but as said, these are underdeveloped in the paper).

The empirical analysis, as anticipated, could make good use of experiments showing the behaviour of the gradient magnitude across learning iterations. Current experiments, although commendable in the fact that they cover both supervised and reinforcement learning, do not provide substantial proof of CBP being better than existing weight regularization techniques.

summary_of_the_review: The topic of the paper is of potential interest for the community and the empirical analysis sufficiently broad. On the negative end, the depth of technical discussion and of the empirical analysis does not support the claims. Originality of the contributed CBP algorithm is minor.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of fast adaptation in a non-stationary online continual learning(CL) setting. It argues that keeping weight randomnization is important to fast adaptation in CL. However, current CL methods only performs weight randomization in the beginning of the algorithm; the weights loss randomness overtime, leading to degraded model performance. The paper presents a continual weight reinitialization algorithm to overcome the issue. In particular, it proposes to evaluate the utility of each hidden unit -- including importance to the current task and adaptation capability. Then selects a set of hidden units with low score and resets their incoming and outgoing weights. The authors conduct experiments to evaluate the performance of the proposed method.


main_review: Strong Points

* The paper takes one of the most import issues in continual learning: non-stationary online CL setting. For me, the problem itself is real and practical.

* The proposed approach is reasonable and addresses the problem of weight randomnization in CL setting.

* Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty, the method has been well motivated by pointing out the limitations in SOTA methods.

Weak Points

* The proposed generate-and-test algorithm is built upon previous work by Mahmood and Sutton 2013. So the novelty is limited.

* Although the paper focuses on fast adaptation to new tasks in CL. It is important to evaluate how the proposed method deals with catastrophic forgetting. I would like to see experimental results on the catastrophic forgetting evaluation.

* "Continual backprop" in the title and paper does not reflect the attributes of the proposed algorithm. I would suggest changing "Continual backprop" to "Continual reinitialization" as a more appropriate summary of the paper and the algorithm.

* "feature" in the paper is used to refer to a hidden unit in a network (See page 5). However, "feature" commonly refers to a vector extracted from an embedding layer in a network. I would suggest that the authors change "feature" to "hidden unit" in order to avoid misunderstanding.

* Fig. 9, in semi-stationary reacher setting, PPO+L2 performs better than the proposed continual PPO. It's worth discussion the reasons for the results.


summary_of_the_review: Overall, I vote for marginally accepting. I like the idea of continual reinitialization and handling it by the proposed generate-and-test method. My major concern is about the limited novelty of the paper -- its build upon previous work by Mahmood and Sutton 2013, and some misleading terms such as "continual backprop" and "feature".
Hopefully the authors can address my concern in the rebuttal period.

[After rebuttal]:
After reading the other reviewer's comments and authors' feedback, I would like to lower my score due to the same concerns as other reviewers as mentioned.
review_title: null

Meta-review: The paper studies an important newly identified problem in continual learning of rapid adaptation, and proposes the use of a generate-and-test method to continually inject random features alongside SGD, enabling better learning on non-stationary data streams.
Unfortunately the paper remained borderline in the discussions. While reviewers liked the overall research direction and contributions, they also agreed the paper in current form still would benefit from deeper insights into the proposed method, stronger empirical evidence.
Experiments cover broad applications including RL, but don't seem to give very clear advantages over other weight regularization schemes, and other metrics of quality could be added. We appreciate the authors have added additional experiments testing it both for the two important regimes of under- and over-parameterized networks, though those can be expanded.
We are sorry that this good paper remained narrowly below the bar in this case, and hope the detailed feedback helps to strengthen the paper for a future occasion.
Decision: Reject