Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work addresses multi-task learning where task boundaries are unknown. The approach is to construct a dynamic decision tree with nodes made up of small networks. Nodes are merged and promoted in the tree based on learned error bounds on value function estimates. Inference through the tree works by selecting nodes with the highest value prediction. It is an interesting approach for modular learning, even within the same environment.

The paper is clearly written. The approach is clearly explained. The empirical evaluation is thorough, with good control of the baselines.

Node promotion is done so that a new task can take a new path through the tree, avoiding catastrophic forgetting. This is done by comparing the value function estimates of a child node to its parents. A large enough difference in value function estimate triggers a node promotion. An important assumption here is that different tasks differ substantially in long term returns. This may not be the case in general. Tasks can differ in various ways apart from the total reward. Sufficient discussion is not provided on this assumption and how it affects the results.

At the end of page 5, it is claimed that sequential MNIST is “significantly more realistic”. Why is this the case? 

Figure 4 is a bit hard to understand. What is the “reward” here? What is the action space for this task?

In table 1, why is the SANE accuracy for digit “0” not 100%? All other methods are 100% and this is the only class so far unless I’m misinterpreting the table.

While SANE manages to avoid catastrophic forgetting compared to the baselines, the classification accuracy for each digit does settle back down to less than 50% in most cases. Why is this the case?

How effective is the strategy of detecting task change using value function bounds in the mini world environment? The rewards do not seem to be very different. I would be interested in seeing some examples of transitions that trigger a node promotion.

The results in this environment are not so convincing. Is it perhaps because of the values being similar across tasks?

Some more discussion of how this work differs from other approaches that combine deep learning and decision tree could be included.

In section 2.3 “Training the Critics”, it is stated that the critic is trying to minimize the value v(s). Why is this done?

Future work could include using distribution RL for detecting outliers and node promotion.

This is an exciting direction of work and a difficult problem without known task boundaries. The approach is sound with decent empirical evaluations. The paper could benefit from some clarity in the explanation of the results and the assumptions made in the method. Some more in-depth analysis of the trees contracted by this method and their dynamic behavior through learning would shed some more light on some of the results.


review_title: An interesting approach for a difficult multi-task problem

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes SANE -- an architecture and a training algorithm for continual learning. The SANE model consists of a tree where each node can act as an RL agent and where nodes act according to the dispatching mechanism based on their reward prediction. This allows to activate and update only those agents that are specialized in the current task and thus may prevent catastrophic forgetting caused by updating the whole model.

Novelty:
Different parts of the method did have appearance in the prior literature. For example, the general idea to organize the model into a hierarhical structure with a certain kind of dispatching can be found in hierarchical softmax [1] with a more modern variant [2]. More recently, [3] proposed a similar model with a very close motivation to address catastrophic forgetting. In my opinion, authors should make a more extensive literature review and more clearly justify novelty of the method.

Clarity:
After reading the paper several times, I did not get a clear enough picture of the proposed model operates. Perhaps, a more formal algorithmic description would help. 

Significance:
The empirical evaluation only consists of relatively simple experiments. MNIST, for example, is hardly representative of a real-world continual learning task. Authors do not compare SANE to string baselines that are specifically designed to prevent catastrophic forgetting, even to EWC which is cited. Even though, EWC does rely on task boundaries information, it would at the very least define the gap that exists to the methods that are agnoistic to this kind of priveledged information. 
The lack of a thorough experimental study makes it difficult to argue for high significance of this work to the community.

Other comments:
 I think what authors mean by "discounted reward" and $r_{disc}$ is usually denoted as "discounted return", this should be clarified.

References:
[1] A scalable hierarchical distributed language model. Andriy Mnih and Geoffrey Hinton. 2008.
[2] Self-organized Hierarchical Softmax. Yikang Shen, Shawn Tan, Chrisopher Pal, Aaron Courville. 2017
[3] Gated Linear Networks. Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Agnieszka Grabska-Barwinska, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, Marcus Hutter. 2019
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Edit

Updated score from 4 to 5

## Summary

* The paper proposes a task agnostic CL method called SANE. It uses a hierarchical, modular tree-like architecture where nodes are neural networks (policy + critic + replay buffer). 
* The architecture is quite interesting. Alongside regular parameter update operations, it also supports updates to the tree-structure.
* Each state (that a learning agent encounters) activates a new path in the tree (thus reducing negative interference) while still enabling knowledge sharing.
* Experiments on MNIST and Mini-Grid to show the usefulness of SANE over the baselines.

## Objective

* Addressing the problem of continual learning in the context of reinforcement learning.

## Strong Points

* The proposed architecture is quite intriguing. 
* It has several practical desiderata like:
  * Updates on the tree-structure. More nodes can be added (on the fly) while keeping the overall complexity (of training/inference) in check.
  * Works in the task-agnostic setting, which is more general and practical than the task-aware setting.
* The writing (while missing some key details, see "Areas for Improvement") generally flows well and conveys high-level ideas.
* It is quite interesting (and motivating) that SANE retains knowledge of the previous tasks as it continues training on the new tasks. I think of the tree-structure as a way of inducing a prior for the learning agent. The tree-update operations are along the direction of learning a useful inductive bias (without hardcoding the structure). SANE could be a good step in the direction of learning priors/inductive biases from data alone.

## Areas for Improvement

* While the paper describes the high-level idea well, it is very vague in several design choices/implementation details. The lack of detail makes it hard to evaluate the usefulness of the approach. Some examples:
  * The paragraph on merging nodes is very vague. What does it mean to find the closest nodes in the policy space? Do you compute the L2 norm between the predictions of the policies of all possible inputs? How are policies combined by doing a "weighted sum of usage count"? What does "retraining once" mean. How do you decide when to stop training the merged node.

* Setup: The paper mentions that they are operating in the continual RL setup. The MNIST task is a one-step RL problem (or a one step decision-making problem). The MiniGrid setup is more exciting, but the paper considers only three tasks (difficult to evaluate the approach's utility when the number of tasks increases). It does not show convincing results on even those (more at a later point). 

* Unconvincing results: 

Let's start with the MiniGrid results. The paper mentions that "The Empty8x8 graph is the most insightful: CLEAR has almost perfect recall of this environment while 2-Room is training, but catastrophically forgets it during the training of Unlock.". Let us see the performance of CLEAR on Unlock -- Figure 5 (2nd row, 3rd column, MiniGrid Unlock). We see that around 1.5M steps, the CLEAR baseline reaches about 90% performance and starts to saturate. SANE is performing approximately 20% at this point, and even after training for 2.25M steps, SANE reaches around 80% success. My argument is, CLEAR starts overfitting after 1.5M steps while SANE is not overfitting even after 2.25M steps. Now, look at Figure 5 (2nd Row, 1st Column MiniGrid Empty). Around 1.5M steps, CLEAR is performing close to the SANE model.  I could argue that there is no need to train CLEAR (and other baselines) for additional 750K steps just because the SANE model needs to be updated. Updating for so many extra steps is also causing the CLEAR baseline to overfit to the Unlock task. Please note that I am not criticizing the decision to train SANE for 2.25M steps. I am criticizing the choice of training CLEAR much longer. 

Regarding the MNIST results, I have the same worry. We can see that the baselines reach close to 100% very quickly by zooming into the plots, while the SANE model takes much longer. Repeating myself,  I am not criticizing the decision to train SANE much longer. I am criticizing the choice of training the other baselines much longer, thus forcing them to overfit to the current task.

* Choice of baselines: Apart from the criticism of the experiment results, I found the baselines' choice to be quite weak as well. While the authors rightly point out that task-agnostic learning is a much harder setup, they consider very few baselines (actually only one baseline, since Impala is not for CL). Regarding CLEAR and CLEARx40, I do not think CLEARx40 is a fair baseline. Using much more parameters makes the model more likely to overfit, given that the mode is always trained for a fixed number of steps, irrespective of when it starts to overfit. Some more reasonable baselines would have been:

@InProceedings{Aljundi_2019_CVPR,
author = {Aljundi, Rahaf and Kelchtermans, Klaas and Tuytelaars, Tinne},
title = {Task-Free Continual Learning},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}

@incollection{NIPS2019_9357,
title = {Online Continual Learning with Maximal Interfered Retrieval},
author = {Aljundi, Rahaf and Belilovsky, Eugene and Tuytelaars, Tinne and Charlin, Laurent and Caccia, Massimo and Lin, Min and Page-Caccia, Lucas},
booktitle = {Advances in Neural Information Processing Systems 32},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {11849--11860},
year = {2019},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/9357-online-continual-learning-with-maximal-interfered-retrieval.pdf}
}


@inproceedings{
Lee2020A,
title={A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning},
author={Soochan Lee and Junsoo Ha and Dongsu Zhang and Gunhee Kim},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=SJxSOJStPr}
}

* Limited related work: Apart from some articles mentioned in the previous point, some recent works look at learning ensembles of "self-activating" policies.

https://icml.cc/Conferences/2020/ScheduleMultitrack?event=6293

@inproceedings{
Goyal2020Reinforcement,
title={Reinforcement Learning with Competitive  Ensembles of Information-Constrained Primitives},
author={Anirudh Goyal and Shagun Sodhani and Jonathan Binas and Xue Bin Peng and Sergey Levine and Yoshua Bengio},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=ryxgJTEYDr}
}




* Additional Results: It is difficult to understand what the nodes are learning. The appendix has one example, which is not sufficient to understand what the nodes are learning. It is also not clear why would the learning model try to activate different paths. This relates to the problem of learning options in RL and the commonly observed problem where only one option is active during most of the training. At the minimum, it will be useful to see some results/plots describing how the tree evolves during training and how the frequency of use of different nodes changes.

* Ablations: The paper introduces several hyper-parameters but does not consider any sort of ablations with them. This makes it difficult to understand what hyper-parameter values are important and which are set arbitrarily.

## Some other questions

Please note that these are some general questions, and I did not consider any of these design choices as "wrong" or "against the paper."

* Why did the paper use L2 norm and not KL penalty (section 2.3)

* The paper mentions that they are operating in a task-agnostic setting. While that is true, they are still training/evaluating the model in the more limited sequential (one-task-at-a-time) setting. Is there a reason to not work in the more general any-task-at-any-time setting?

* Could the authors also include a discussion of the training time for the baselines vs. SANE. My understanding is, training SANE takes a lot more time/steps.

* Could the authors add some description about how rewards are computed for the MNIST setup.
review_title: Interesting Idea, limited execution

final_score: rating: 5: Marginally below acceptance threshold
review_content: Post-rebuttal update:

I thank the authors for their responses to my queries and for updating the paper. I think the paper has improved, the main reason being that the replay-ratio parameter sweeps for CLEAR in the MNIST experiments show that SANE does improve the stability-plasticity tradeoff, in particular the runs with a 0.97 replay ratio (only shown in the rebuttal pdf but I think should make it into the appendix of the paper), which show that the memory of CLEAR cannot be improved to match SANE's without severely affecting its ability to learn new tasks. The only other set of experiments, the sequence of three minigrid tasks, however, do not show an overall improvement over CLEAR, and so overall the experimental evaluation is still weak. It is hard to judge how good the method is without extending the range of baselines (I appreciate the addition of the and task settings. As a result, I am only increasing my score from a 4 to a 5, but I do think SANE is an interesting method and I encourage the authors to keep working at it to prove its effectiveness.

---------------------------
Original review:

This paper presents SANE (Self-Activating Neural Ensembles) a new dynamic architecture method for mitigating catastrophic forgetting in neural networks in the context of reinforcement learning that is task-agnostic. The architecture consists of a tree where each node comprises its own value critic, policy and replay buffer. Action selection at each time step begins by starting at the root of the three and activating the child that predicts the most optimistic value, and repeating this procedure from the activated child until a leaf node is reached, whose policy is used to act with. Training has two components: (i) the parameters in activated nodes are updated using REINFORCE, and (ii) structural updates are made to the tree whereby children that are sufficiently different to their parents are promoted up the tree and given their own children, and children that are similar to each other are merged in order to limit growth of the tree. The method is empirically compared to two baselines, IMPALA and IMPALA with CLEAR (an experience-replay based continual learning method), on two sequences of tasks, sequential MNIST and a sequence of three egocentric gridworld tasks (MiniGrid). On sequential MNIST, SANE displays less forgetting but less adaptivity than the baselines, and on the gridworld tasks the results are mixed.

SANE is an interesting and inventive method but I do not think it is ready for publication yet because (i) the experiments do not sufficiently support the claim that SANE improves catastrophic forgetting over the (limited set of) baselines, and (ii) SANE has a lot of components and hyperparameters but it is not clear which aspects are important for it to work. I believe the paper could be much improved by (i) demonstrating that the baselines have been adequately tuned (and perhaps increasing the set of methods compared to, e.g. with another dynamic architecture method), (ii) including a number of ablation studies or other experiments that better elucidate the inner workings of SANE and justify architectural choices.

Positives:
	•	SANE incorporates a number of interesting mechanisms that might intuitively be useful for mitigating catastrophic forgetting. For example, by only updating activated nodes, modularity is preserved during training in a way that can protect forgetting in other parts of the network. Also, the structural updates provide an interesting way of expanding and contracting the architecture as necessary under fixed resource constraints - an important challenge for continual learning. 
	•	The problem that SANE tackles, i.e. task-agnostic continual learning, is an important one for which there are not a large number of approaches in the literature. 
	•	An effort is made to equalise the number of parameters used by the baseline agents and SANE in the experiments for a fair comparison, and experiments are repeated with several random seeds. 

Concerns:
	•	SANE is claimed to improve on the baselines of IMPALA and CLEAR on the axis of mitigating catastrophic forgetting, but the experiments do not demonstrate this clearly. 
	◦	In the sequential MNIST experiments, SANE retains some performance on earlier tasks as training progress but the performance of both IMPALA and CLEAR seems to drop almost instantaneously after the task is switched. On the other hand, both IMPALA and CLEAR learn the new task extremely quickly compared to SANE, which usually does not even reach the same maximum level of performance as the baselines. This suggests that SANE shifts the tradeoff of adaptivity vs. remembering towards remembering, but doesn’t show that the tradeoff is improved vs the baselines. In the CLEAR paper, there is an important hyperparameter that mediates this tradeoff, which is the proportion of replay experiences vs. new experiences used for training - this parameter should be tuned here to be able to show that CLEAR is indeed inferior to SANE on this task. (Judging from the code it seems that a 50/50 split is used, is this correct?) 
	◦	In the sequence of three MiniGrid tasks, even though the performance of the CLEAR agent on the first task eventually drops below that of the SANE agent, the SANE agent initially deteriorates a lot faster. For the second task there seems to be no significant difference between agents. Additionally, the SANE agent is much slower to learn on the third task, never reaching the performance of the CLEAR agent, again demonstrating that the SANE agent is trading off adaptivity for slightly better memory. It’s not evident that SANE is improving the tradeoff vs. CLEAR without tuning the latter’s hyperparameters. 
	•	It is difficult to discern which elements of SANE are important for its performance in order to justify some of the architectural choices. For example: 
	◦	The critics are trained to estimate both the value and the variance of the value, and, in the inference stage, the node with the highest upper bound of estimated reward is chosen. How important is this UCB-like policy for the performance of the agent, and the value of the hyperparameter alpha that scales the standard deviation term? Would SANE work just as well if just the value estimate was used to choose the next node? Could this lead to a wrong solution if applied in a highly stochastic environment? 
	◦	The policies of the nodes in SANE use REINFORCE but IMPALA and CLEAR use VMPO; is there a reason for this difference? 
	◦	In the structural updates, for node promotion the value functions of a child and its parent are compared, but for node merging, the policies are compared - is there a rationale behind these choices?  
	◦	In the MiniGrid experiments, SANE is run with either 10 or 15 top level nodes, resulting in a relatively big difference in performance for the 1st and 3rd tasks, giving some indication of the sensitivity of this hyperparameter. Why were 20 top-level nodes chosen for the MNIST experiments and was the performance sensitive to this choice? How about other hyperparams such as number of child nodes and the number of times a node has to have been used before it can be promoted? 

Other comments:
	•	The number of baselines used for comparison is limited, especially given that one of them (IMPALA) is not designed for continual learning. It would at least be good to cite other task-agnostic methods such as [1,2]. 
	•	It is slightly odd to frame sequential MNIST as an RL problem, when it is a supervised learning task. 
	•	A visualisation is given in the appendix showing an example of a node that seems to represent a policy that always moves the agent forward one step in the gridworld. Presumably this is shown to indicate the specialisation of modules in the tree; it would be very interesting to see more evidence of specialisation/diversity across modules since this is one of the motivating factors for the architecture and also to see how the tree structurally adapts at task switches - are there more promotions when the data distribution changes?  

[1] Aljundi, Rahaf, Klaas Kelchtermans, and Tinne Tuytelaars. "Task-free continual learning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[2] Zeno, Chen, et al. "Task agnostic continual learning using online variational bayes." arXiv preprint arXiv:1803.10123 (2018).
review_title: Interesting idea but lacking in empirical validation and justification of architectural choices

Meta-review: The reviewers have arrived at the consensus that this is a paper with an interesting idea, both novel and well-explained, but not quite backed up with sufficient empirical evidence. Like them, I think there is a lot of potential in modular methods for continual learning, and I know these are challenging advances to demonstrate. So I encourage you to persist, iterate and submit a stronger version of this paper in the future!
Decision: Reject