Self-Activating Neural Ensembles for Continual Reinforcement Learning
Authors: Sam Powers
Abhinav Gupta
Venue:  (2021)
Keywords: continual reinforcement learning
lifelong learning
deep reinforcement learning

Abstract: The ability for an agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of generally intelligent agents. Most methods devised to address this problem depend heavily on well-defined task boundaries which simplify the problem considerably. Our task-agnostic method, Self-Activating Neural Ensembles (SANE), uses a hierarchical modular architecture designed to avoid catastrophic forgetting without making any such assumptions. At each timestep a path through the SANE tree is activated; during training only activated nodes are updated, ensuring that unused nodes do not undergo catastrophic forgetting. Additionally, new nodes are created as needed, allowing the system to leverage and retain old skills while growing and learning new ones. We demonstrate our approach on MNIST and a set of grid world environments, demonstrating that SANE does not undergo catastrophic forgetting where existing methods do.

Under review as a conference paper at ICLR 2021
SELF-ACTIVATING NEURAL ENSEMBLES FOR CON- TINUAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The ability of agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of gen- erally intelligent agents. Most methods devised to address this problem depend heavily on well-defined task boundaries which simplify the problem considerably, Our aask-agnostic method. Self-Activainng Neural Ensembles (SANE), uses hi- erarchical modular architecture designed to avoid catastrophic forgetting without making any such assumptions. At each timestep a path through the SANE tree is activated; during training only activated nodes are updated, ensuring that unused nodes do not undergo catastrophic forgetting. Additionally, new nodes are created needed, allowing the system to leverage and retain old skills while growing and learning new ones. We demonstrate our approach on MNIST and a set of grid world environments, demonstrating that SANE does not undergo catastrophic for- getting where existing methods do.
INTRODUCTION
Lifelong learning is of critical importance for the field of robotics; an agent that interacts with the world should be able to continuously learn from it, acting intelligently in a wide variety of situations. In marked contrast to this ideal, most standard deep reinforcement learning methods are centered around a single task. First, task is defined, then a policy is learned to maximize the rewards the agent receives in that setting. If the task is changed, a completely new model learned, throwing away the previous model and previous interactions. Task specification therefore plays central role in current end-to-end deep reinforcement learning frameworks. But task-driven learning scalable?
In contrast, humans do not require concrete task boundaries to be able to effectively learn separate tasks - instead, we perform continual (lifelong) learning. The same model is used to learn new skills, leveraging the lessons of previous skills to learn more efficiently, without forgetting old be- haviors. However, when placed into continual learning settings, current deep reinforcement learning approaches do neither: the transfer properties of these systems are negligible and they suffer from catastrophic forgetting (McCloskey & Cohen 1989: French 2006).
The core issue of catastrophic forgetting is that neural network trained on task starts forget what it knows when trained on second task, and this issue only becomes exacerbated as more tasks are added. The problem ultimately stems from training one network end-to-end sequentially; the shared nature of the weights and the backpropagation used to update them mean that later tasks overwrite earlier ones (McCloskey & Cohen, 1989; Rateliff 1990).
To handle this, past approaches have attempted wide variety of ideas: from task- regulariza- tion (Kirkpatrick et al.) (017), learning different sub-modules for differrnnt tasks (Rusu et al.) 2016), to dual-system slow/fast learners inspired by the human hippocampus (Schwarz et al. 2018). The core problem of continual learning, which none of these methods address, is that the agent needs to autonomously determine how and when to adapt to changing environments, as it is infeasi- ble for human to indefinitely provide an agent with aask-boundary supervision. Specifically, these approaches rely the notion of tasks to identify when to spawn new sub-modules, when to freeze weights, when to save parameters, etc. Leaning on task boundaries is unscalable and side-steps the core problem.
There are a few existing task- -fee methods. Some address the problem by utilizing fixed subdivisions of the input space (Aljundi al. 2018 Veness et al. 2019) which we believe limits their flexibility.

---

Under review as a conference paper at ICLR 2021
Recent experience- experience-based approaches such as Rolnick et al. (2018) forgo explicit task IDs, but to do so they must maintain large replay buffer, which is unscalable with an ever-increasing number of tasks. To our knowledge only one other continual learning method, Neural Dirichlet Process Mixture Models (Lee et al. 2020), adaptively creates new clusters, and we show in our experiments that SANE outperforms it.
Our method approaches the problem head-on by dynamically adapting to changing environments. Our task-agnostic hierarchical method, Self-Activating Neural Ensembles (SANE), depicted in Fig- ure I is the core of the method. Every node in the tree is a separate module, none task-specific. At each timestep a single path through the tree is activated to determine the action take. Only acti- vated nodes are updated, leaving unused modules unchanged and therefore immune to catastrophic forgetting. Critically, our tree is dynamic; new nodes are created when existing nodes are found to be insufficient. This allows for the creation of new paths through the tree for novel scenarios, preventing destructive parameter updates to the other nodes.
SANE provides the following desirable properties for continual reinforcement learning: (a) It mit- igates catastrophic forgetting by only updating relevant modules; (b) Because of its task-agnostic nature, unlike previous approaches, it does not require explicit supervision with task IDs; (c) It achieves these targets with bounded resources and computation. In our results, we demonstrate the ability of SANE to learn and retain MNIST digits when presented fully sequentially, challenging task our baselines struggle with. We also demonstrate SANE on a series of three grid world tasks, showing that SANE works in the reinforcement learning setting.
MODEL
Node
ERCI
FERT
Critic
LBIT
B11T
LRRI
CDRT
BRLI
(Palicy)
Figure 1:
The overall tree structure of the SANE system, here with layers of nodes. Each node contains actor and critic.
The structure of our Self-Activating Neural Ensembles (SANE) system is tree, shown in Figure [ Each node in the tree contains both an actor and a critic, described in 2.1 At each timestep we activate a path through the tree from the root leaf, each layer selecting the node whose critic estimates the highest activation score (described in 2.2p. We then sample an action from that leaf node's policy to execute in the environment. During training, the activated nodes are the only ones updated: all other nodes are unchanged. This enables behaviors selected for by other paths through the tree to remain unchanged.
The tree structure is critical to the success of the system; a node's 's children can be seen as partitioning the input space that activates the node, specializing their policies these subspaces. When they differ sufficiently from their parent in performance, they are promoted to a higher level of the tree and obtain children of their own, further partitioning the space. Nodes are constantly being created, promoted to higher levels, or merged as the situation demands (described in Section 2.4).
One critical question to ask is why use a dynamic structure and not just huge static tree with only parameter updates? In the case of the static tree, if path through the network is beneficial for one task, it is likely to be chosen as a starting point for the subsequent task, and updated in-place for this new setting. Examples of this can be found in Jacobs et al. (1991); Shazeer et al. (2017). where specialized losses are necessary to distribute activation across in the case of a stationary distribution. With SANE, the change of
Static vs. Dynamic Trees:
the ensemble of experts, environment would be detected and a new path created, thus minimizing disruption of the previously beneficial behaviors.
2

---

Under review as a conference paper at ICLR 2021
Inference
Replay Buffer Update
State
Action
(b Tascel
Activates
Sends Replay
[%y Fa)
Entry
Roplay Registration
Figure 2: On the left is an example of inference, which results in the generation of actions execute in the environment. On the right is the associated adding of replay entries to the nodes that activated 0, after the collection phase.
2.1 NODE CONTENTS
Every node contains the following: a policy P:, a critic C,, a replay buffer containing list of (state discounted reward rdisc), and a count of the number of times this node has been used (usage count).
The policy p: presented here is vector of n logits, where is the number of actions available to the agent. The critic C, a neural net mapping a state 8 to two scalars: the value estimate of the discounted reward received if the current node is activated, (s)s), and estimate of the absolute error:
un (s(s) ~ |ddisc (+)]
With these we can define an optimistic estimate (upper confidence bound) and pessimistic estimate (lower confidence bound) for what reward a node can achieve:
vUCR, us))) + t (s)))
(8) a
(3)
a syper-parametee representing how wide margin around the expected value we allow, VUCB is analogous to an upper confidence bound in multi-armed bandit literature: training proceeds and v; becomes more accurate, u, will decrease, providing natural transition from exploration to exploitation for each node it learns. This may however act too optimistically in highly stochastic environments, an area of future work.
Each node's 's replay buffer contains the history of experiences per node and is used to train the node's 's critic. The usage count is the number of times node has been activated and is used in the merging of nodes. These described more in sections 2.3 and 2.4
2.2 INFERENCE
For each observed state 8, from the environment, activate path through the tree from the root to leaf node, whose policy is used to determine the next action to take in the environment, a, The environment then responds with reward r; and a new observation 8 (See Figure
We find the path recursively: from an activated node, the next node to activate is selected from its children by taking the node with the highest optimistic upper bound of reward. This process continues until leaf node is reached. After the set of activated nodes is determined, an action is selected stochastically from the activated leaf's policy, and executed the environment. New nodes are not created mid-episode to keep the tree idempotent, which allows us to readily run environments in parallel.
During inference, three things happen at each timestep to support updating the tree. All active nodes:
are registered receive the replay entry for that timestep after the episode completes.
compare their critic's estimate to their parent's. s. If they differ sufficiently (Section 2.4). the child will be promoted during the next tree structure update.
increment their usage count.

---

Under review as a conference paper at ICLR 2021
Promotion
Merging
Goned Into
Merged Into
Parented Under
Figure 3:
The two parts of updating the structure of the tree: promoting nodes to a higher level, via cloning, and the merging of nodes together when the maximum size has been exceeded.
2.3 PARAMETER UPDATES
SANE training updates consist of two steps: parameter updates (replay buffer, policy, and critic) and tree structure updates (creating new nodes, merging nodes, etc.). After T steps of data collection have occurred, we trigger both parameter update and a tree structure update.
Replay Buffer Update: Each node's replay buffer contains the history of the times that node was activated, and is used to train the node's critic. Each entry in the replay buffer contains an input state s and discounted reward received Taises which also used in the training of the policies. The discounted reward is computed in the standard fashion, in Espeholt et al. (2018).
When the addition of new entries results in the replay buffer exceeding max length, instead of removing the oldest entries, random entries removed instead. This is to mitigate each critic from developing a recency bias.
Policy Update: The policy for each node is trained on-line using REINFORCE (Williams 1992) with Monte Carlo returns.
The critic for each node is trained off-line using the replay buffer. More specifically, we split a node's 's replay buffer into batches each of size k. The critic has two functions estimate: u(s) (the expected value) and u(s) (the uncertainty 1.2 loss for both reward and error prediction. Note that this is the slowest step in the training of the SANE tree. Since the nodes all use separate critics, this step can be effectively parallelized. We only train a critic when its node was activated at least once since the last time nodes were trained.
Training the Critics:
defined in equation Tp. We use
2.4 STRUCTURE UPDATES
One of the critical properties of SANE is that the structure updates dynamically. Promoting nodes enables existing behaviors to stay unchanged, and merging nodes allows us to keep memory and computation bounded, both depicted in Figure The term promotion is used refer to cloning a node and placing it in the tree one level from the original. Merging is the process of combining nodes and necessary cap the computational and memory costs that scale with tree size.
Drift is the term for when an environment is non-stationary, e.g. when the reward distribution or the state transition distribution are changing over time. Often where drift occurs, catastrophic forgetting follows because networks update to the new setting, forgetting the old.
Node Promotion:
To recognize drift, non-leaf nodes the time of creation have their critic cloned and frozen, creating static critic called the anchor. We compare the prediction of a node's critic to the prediction its parent's ' anchor makes. We say that sufficient change has occurred when the reward bounds predicted by node's i critic do not include the value predicted by its parent's anchor, which functions bascline. In such a scenario, the node should be promoted up a level of the tree, because we wish to capture the new situation being observed, and minimize disruption of previously learned behavior.
static
Concretely, we say that sufficient change has occurred when either of the following inequalities hold, for given input state s, node and parent's 's anchor VUCB, (s)) vs(s), ULCB.. > (s). In order to avoid an extraneous number of promotions for outliers, Icaf node must have been used more than kn times since it created or last promoted.
4

---

Under review as a conference paper at ICLR 2021
An implicit assumption here is that the change in reward received is sufficient for distinguishing relevant changes in setting. We believe this to be a general assumption; if, as the task changes, a node's behavior results in the same received reward (and exploration does not expose better path), then the change was not relevant for determining the optimal policy, and can be effectively ignored.
To prevent unlimited memory consumption and increasingly long activation times, we limit the number of children each node can have. The limit is defined per level of the tree: level k has limit lg. We start by finding the two entries in the set that are closest in policy space. Conceptually, this is so we can find general patterns in the state space that elicit the same optimal policy. To do this we convert the policies from logits to probabilities, then find the two policies that have the smallest L2 distance. L2 was selected because it is symmetric and can be computed batch-wise rapidly. The usage counts, policies, and replay buffers are combined by doing a weighted average of the nodes' contents by usage count, explained in more detail in section A.2 The critic for the new node is trained using the merged replay buffer, after being initialized from the critic of the selected node with higher usage.
Merging Nodes:
Non-leaf nodes additionally update the anchor by combining the nodes' anchors' replay buffers and retraining once. Merging nodes is the exception to the general rule of never updating the anchors.
The first node in the tree initialized as child of the root and has random critic and a uniform policy. That node is cloned twice, which become the first node's children. This continues until the tree has reached the pre-specified maximum height. From now on, the height of the tree remains fixed. To maintain this, whenever node is promoted, that node is recursively cloned to generate children until the maximum height is re-achieved.
Initialization:
3 EXPERIMENTS
We present types of experiments that demonstrate the behavior of SANE. The first trains on the digits of MNIST sequentially, to show recall under complete domain shift. The second learns series of grid world environments, to show that the method works in a reinforcement learning setting. Both never re-train on a task they' 'ee seen before; this is to ensure there are concrete points of domain shift. The environments are simple enough that this not ensured by randomness alone.
In these experiments we only consider two-layer SANE trees, with their sizes (Téu, f1) indicating respectively the maximum number of nodes at the first layer, and the maximum number of children each of those may have. A [20, 4] tree therefore has 20 top layer nodes and 80 second layer nodes, for a total of 100 nodes. The experiments are run over 4 seeds unless otherwise noted, and the error bars represent the standard error of the mean.
In both cases we use IMPALA (Espeholt et al. 2018 as the non-continual learning baseline, to demonstrate the effects of catastrophic forgetting, and CLEAR (Rolnick et al.) 2018) as a continual learning baseline. CLEAR is a variant of IMPALA that adds an extensive replay buffer to maintain experiences from old tasks. In particular, we selected CLEAR because it is one of the few other methods that does not require task IDs as input. We chose for CLEAR the same number of entries in its replay buffer as the upper bound of the number SANE uses, across all nodes: 409,600 frames. All network and implementation details are given in the Appendix. For the MNIST experiment we additionally use Neural Dirichlet Process Mixture Model baseline (Lee) et al. 2020), described further below.
Baselines:
CLEAR has an implicit trade-off between how easily it learns new task, and how much it recalls old ones, as defined by the ratio of entries in each training batch that are selected from the replay buffer, versus newly acquired entries. Additionally, if we naively run CLEAR for the same number of timesteps as SANE, we might end up overfitting, We thus additionally consider two types of modification to CLEAR: changing the replay ratio and early stopping. In the latter case we continue to evaluate the network for the rest of the allotted time, to maintain alignment.
We conduct two experiments: MNIST and MiniGrid. MNIST allows us to test network behavior in the presence of huge domain shifts as well as a relatively large number of tasks, to highlight how approaches like CLEAR struggle in these scenarios. MiniGrid allows us to demonstrate how SANE avoids forgetting in the reinforcement learning setting. Note that while previous efforts have used Atari, we do not use Atari for a couple of reasons: particularly ill-suited for continual learning per the desiderata given in (Farquhar & Gall 2018), for
Environments/Datascts
(a) this setting is
5

---

Under review as a conference paper at ICLR 2021
AND
MM
PIIITT
MABST:
PRIST
3.0
Figure 4: Sequential MNIST results: Each graph visualizes all networks the given digit. The x- axis is consistent across graphs. Training on newly introduced digit occurs in the shaded regions, and evaluation test recall happens everywhere else. SANE is the only method to catastrophi- cally forget. Digit 9 can be found in the Appendix.
instance there is little cross-task resemblance, task IDs can be readily inferred (b) the required compute high (a bottleneck new model development).
3.1 MNIST EXPERIMENTS
MNIST (LeCun al. 2010) is a dataset of the digits 0-9, each represented by grayscale 28x28 image. Demonstrating continual learning with MNIST has an extensive history. Permuted MNIST Kirkpatrick et al. 2017 Schwarz et al. 2018) Goodfellow et al., 2015) trains the full set of digits concurrently, adding significant pexel-level shifts to change the appearance between tasks. Split MNIST (Farquhar & Gal 2018) splits the 10 digits into 5 sequential tasks of 2 digits each. Both have assumptions we can now relax; in the former case, the assumption of balanced classes, and in the latter the existence of negative examples for each class being trained. We therefore introduce Sequential MNIST, an alternative task in which digits are trained completely sequentially: first the Os, then the 1s, etc.
To convert this classification task to an RL task, we define the action to be the selection of class (i.e. the action space is of size 10), and the reward is defined be if the correct class is selected, otherwise. We train for 300,000 frames per digit. No normalization is done other than scaling pixel intensities between [0, 1].
For this experiment we additionally run Neural Dirichlet Process Mixture Model (Lee et al. 2020) as a baseline, using the code provided by the authors. This is a classification-omi method, so for this baseline we do not do the conversion to an RL task described above. We use the default MNIST configuration provided in their code. Due time constraints NDPM is shown with only one run, though preliminary experiments indicated consistent results. Additionally, we experimented with different configurations of CLEAR by tuning the replay ratio parameter and by carly stopping (see Appendix A.7 and selected replay fraction of 0.875 for our CLEAR baseline.
The results are presented in Figure The networks are trained on the given digit during the timesteps indicated by the shaded region. In the non-shaded regions of the plots, the networks evaluated on that digit alone, to analyze recall. As this figure visualizes recall and not generalization, these evaluation steps done on the MNIST Train dataset.
Additionally we test the methods by evaluating them on the Test dataset for the digits cumulative to the most recent digit seen during training. These results are presented in Table [] "Mem Last" is

---

Under review as a conference paper at ICLR 2021
Method
0
0-1
0-2
0-3
0-4
0-5
0-6
0-7
0-8
0-9
NDPM
100.0
65.0
41.0
34.0
32.0
15.0
11.0
10.0
13.0
15.0
IMPALA
100.0
53.2
32.1
24.6
19.0
15.1
14.3
13.1
10.6
10.0
CLEAR (r=0.88)
100.0
94.9
54.0
39.7
36.5
19.3
20.2
22.5
11.8
15.4
SANE
97.5
82.2
75.0
57.3
47.1
50.6
29.9
26.8
28.7
33.0
Mem Last
100
50
33.3
25
20
16.7
14.3
12.5
11.1
10
Table 1: The scores of the methods on the MNIST Test dataset, on all digits trained so far, e.g. column "0-3" occurs after training digit 3 and tests cumulative recall on all four digits to that point. "Mem Last" is the score achieved if the answer given is always the last number seen.
provided as baseline for what the expected results would be the network always identifies the last-seen digit.
Only SANE consistently recalls digits after their training period, avoiding experiencing catastrophic forgetting. This setting is particularly challenging for typical methods, because in each training regime there is always only one correct action to take, and singular desired answer. It is worth noting that SANE did not perform flawlessly; there are a few digits (5, 8, 9) that SANE learns inconsistently in the allotted number of frames.
easy to overfit to providing the
There are a few specific behaviors to note: (a) CLEAR's s added replay buffer does provide benefit for recall of earlier digits, though the benefit tapers off as more digits are observed. (b) SANE's curves are reliably cyclic: when SANE begins to do well on new digit, its performance decreases. However, when the next new digit is presented, the performance on prior digits increases again. Our hypothesis is that this is due to overfitting the negative examples when training on the next digit, leading to some reverse transfer. (c) SANE'S performance in the Test condition plateaus lower than the other methods; this is due to the application of a tanh to the logits of the policy to improve stability, meaning even in Test every action has chance of selection, described further in section A.2 (d) NDPM learns each digit less well than the previous, with only digit O actually being mastered, and even that digit still experiences more forgetting than SANE.
prior digits
the current digit, which is mitigated by
3.2 MINIGRID EXPERIMENTS
g
Hirigrid: Room
Minirid: Unlock
Minigri:: Key Comridor
-INPRIA CLEAS midE JANTT degs
ca
1.80
Figure 5: Results for the MiniGrid experiments, the order which they were trained. Training occurs in the shaded region, and the rest of the graph depicts recall while the other environments are trained.
MiniGrid (Chevalier-Boisver et al. 2018) a lightweight grid world library designed to procedu- rally generate environments. We train for 750,000 frames the 2 Room environment, followed by 1.5 million frames on the Unlock environment, and finally 1.5 million frames on the Key-Corridor environment, depieted in Figure 5 Each is randomized at the completion of each episode. The ob- servation is an egocentric representation of the environment, capturing state information about the 7x7 square in front of the agent.
The training results are presented in Figure5] As with MNIST, the shaded region indicates when that environment was being trained, while the unshaded parts are evaluation steps that environment,

---

Under review as a conference paper at ICLR 2021
to visualize recall. We experimented with different configurations of CLEAR (see Appendix A.7 and for MiniGrid selected early stopping, though a replay ratio of 0.75 performed comparably.
SANE's performance is comparable to that of CLEAR with early stopping, indicating that SANE can successfully mitigate catastrophic forgetting in RL environments, even improving on performance in places. CLEAR without early stopping performs worse than both, indicating that it is likely overfitting. The fact that SANE trains extensively on the Unlock environment after plateauing (for approximately an additional million timesteps), may indicate that SANE is less susceptible to overfitting, potentially another benefit of the modularity of our architecture. This benefit is not to be understated; without inherent resilience to overfitting, agents that are continuously acting in the world must have some automatic mechanism to detect it. a feature which most CL methods do not presently have, with the exception of (Aljundi et al., 2018). IMPALA exhibits catastrophic forgetting as expected, even though it struggles with learning the second two environments.
ANALYSIS OF SANE
Num Nodes Promoted: Ninigris
Num Nodes Promoted: MNGST
Dumbier
-2 temm
-Dat
-D684
-Dut2
Dor
Figure 6: The nodes promoted by SANE while rumning the MiniGrid (left) and MNIST (right) experiments. For clarity they have been smoothed by averaging over window of size 20.
Creation of new nodes: Figure 6 shows when SANE decided to promote nodes during one run of each experiment. For MiniGrid, it steadily increases per task, whereas for MNIST the pattern is less distinct. The former behavior is expected; low as negative examples are provided to the current best nodes, then high as nodes improve over their parents. may remain high because the tree starts to run out of capacity around 300k steps, and promotion+merging nodes is less sample efficient than promotion alone. The behavior of MNIST is less clear though, and future work will investigate it more closely. We provide some further intuition for the behavior of the SANE network by visualizing a few states that activate the same node in Figure 7 located in the Appendix.
Parameter and architecture analysis: Unfortunately SANE is relatively slow (about day to train Minigrid, versus an hour for CLEAR) and considerably and set of ablations has been out of reach. Informal experimentation has shown little sensitivity to most parameters, with the primary exceptions being the size of the nodes and the number of epochs they are trained for, as well as in e-greedy. The former two are particularly inconvenient because they are highly tied to the runtime of the method.
intensive, so complete grid search
5 RELATED WORK
The goal of continual learning, or lifelong learning, is to create an agent that can learn to accomplish a wide variety of tasks sequentially (Lesort et al., 2019). To accomplish this a system must bal- ance stability (the extent to which existing knowledge is retained) and plasticity (how readily new knowledge is acquired) (Parisi et al. 2018).
The former in particular has posed substantial problem for neural networks, which undergo catas- trophic forgetting, tending to completely forget old tasks when presented with new ones, often quite abruptly (French) 2006). Attempts to correct for this tend to fall into these broad categories, which discuss below: dynamic networks, dual memory, regularization, rehearsal, sparse-coding, and ensembles (Parisi et al., 2018 Kemker et al. 2017).
Dynamic architectures and dual memory: One example of dynamic architectures, where sources are added as as-needed, is CHILD (Ring 1997]. which determines when to create new units by analyzing average errors observed in network weights. Another dynamic architecture method is Progressive Networks (Rusu et al.) (0066), where new neural networks (" ("columns") are added as new tasks are encountered, with lateral connections created to the networks of previously learned tasks.
8

---

Under review as a conference paper at ICLR 2021
Progress and Compress (Schwarz et al. 2018) adapts this into duallmmmoon method by maintain- ing two columns and regularly distilling the active column into the knowledge-brrs column.
Regularization: Elastic Weight Consolidation (Kirkpatrick et al. 2017) and Synaptic Intelligence (Zenke et al. 2017) are examples of regularization. Both automatically learn an importance mea- sure for the individual neuron weights for particular tasks, such that new tasks minimally influence weights that were important for prior tasks. The method presented by |Aljundi et al. (2018) is a task- free regularization method that uses features of the loss as is updated: when a plateau is detected after a peak, the parameter importance weights are updated.
One rehearsal method is CLEAR (Rolnick al., 2018), which maintains replay buffer over old tasks, using reservoir sampling and a behavioral cloning loss to ensure the network con- tinues to respond appropriately
Rehearsal:
the inputs of old tasks, giving this method some regularization as well. CLEAR is built on top of IMPALA (Espeholt et al., (018), an asynchronous non-continual learning actor-critic method. Another rehearsal method is Maximally Interfered Retrieval (Aljundi) et al., 2019), which trains the entries from its buffer that a proposed update (using recently ob- served data) most negatively impact. MIR also introduces a variant using generative model. Both of these do not require task IDs.
Sparse coding and ensembles: Fixed Expansion Layer Networks (Coop et al. 2013) (FEL) is an example of both sparse-coding and ensemble method, though not one demonstrated in RL envi- ronments; it introduces a layer that is sparsely activated, such that the error signal does not propagate through all weights, and creates an ensemble of these to improve results. PathNet (Fernando et al.) 2017). which may also be considered an ensemble method (Kemker et al., (017), uses an evolu- tionary strategy to select a path through the network. The usage of specific paths instead of the whole network is a similarity PathNet shares with SANE. However, the method still requires task boundaries determine when to freeze paths, a key difference from SANE.
Task-boundary free: In addition to CLEAR, MIR, and Task-Free Continual Learning, there are few other explicitly task- bok-boondyry free works. Goyal et al. (2019) also present a distributed net- work of primitives that independently determine the score used to activate them. However, their primitives are constrained in terms of what part of the input space they see, and new primitives are not created over time. Gated Linear Networks (Veness et al. 2019) introduces hierarchical backpropagation-free method that gates neuron selection by associating inputs with predefined con- text vectors. Continual Neural Dirichlet Process Mixture Models Lee et al.) (2020) leverages the ability of standard DPMMs to handle clustering with an arbitrary number of clusters, increasing as necessary when new data is not well-explained by existing cluster. Finally, (enoo et al., 2019) approach task-free continual classification by estimating a Bayesian posterior online.
Additionally, some of the components of SANE have occurred in non-CL literature. For instance, Mnih & Hinton (2009) and (hhen et al. (2017) utilize a method that selects a leaf node by making choices at each internal node of tree, though tree creation is by clustering not by drift detection.
Ghosh et al. (2017) presents Divide and Conquer, an idea conceptually similar to SANE; it partitions the input space and learns an ensemble of specialized policies, though they cluster with K-Means while SANE forms partitions automatically.
CONCLUSION
The core challenge behind continual learning is knowing how and when to adapt to novel situations without overwriting previous behavior. Most existing methods side-step the problem by allowing the task boundaries to be specified, giving a clear signal for when to add parameters, freeze them, reinitialize them, etc. Our method, Self-Activating Neural Ensembles, needs such crutch. It automatically detects and adapts to drift in the environment by adding new nodes to the SANE tree. Additionally, since only the necessary set of nodes is activated at each timestep, nodes that did not participate are not updated with the new interaction data and therefore are not susceptible to catastrophic forgetting. Finally, our method automatically merges nodes as necessary, ensuring the resources consumed remain bounded.
Future Work: Improving the speed of training has been a bottleneck, and is one direction for future work. Other directions include experimenting with more layers, sharing parameters between nodes, having a common pre-encoder akin to the visual cortex, actively mining negative examples from other nodes, adapting to continuous and stochastic domains, leveraging distributional models, and many others.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. CoRR, abs/1812.03396. 2018. URL httlhttp:/aaae wwaexiii..rr..o..oo abs/1812. 03596
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. CoRR, abs/1908.044 04842, 2019.
wttpooggg//bba//g 04742
Maxime Chevalier- Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: httt:::://itthuee www hwwpmmccbbyyygeeee www.mmmmmmiiiieemmmi... 2018.
Robert Coop, Aaron Mishtal, and Itamar Arel. Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting. Neural Networks and Learning Systems, IEEE Transactions on, 24:1623-1634 10 2013. doi: 10.1 111111/TNNSSS201332222
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted lctor- architectures. CoRR, abs/I 0/5/1802.0161. 2018. URL httt:://aaxiiit/ http:/aaxivv..g/aaaaa..a 010.16101001611
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning, arXiv preprint 1805.0090 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017. URL httt:::/axxiv..gg/bbb////////
08734
Robert French. Catastrophic Forgetting in Connectionist Networks, volume 3. 01 2006. ISBN 9780470018868 doi: 0000044700086000000000
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and-
conquer reinforcement learning. CoRR, abs/1711.098744, 2017. URL http:://axxi..oort/
09874
lan J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting gradient-based neural networks, 2015.
Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng. Sergey Levine, and Yoshua Ben-
gio. Reinforcement learning with competitive ensembles of information-coonstaineed primitives.
CoRR, abs/1906. 10667, 2019.
Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3:78-88, 02 1991. doi: 10.1162/ncco.1991...1.999
Ronald Kemker, Angelina Abitino, Mare McClure, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. CoRR, abs/1708.00072. 2017. URL http:///rxiv org/
abs/ 708. 02072
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Brabskk-Barwinska. Demis Hass- abis, Claudia Clopath. Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forget- ting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3520 2017. ISSN -427-8424. doi: 10.1073/pnas.160 161 1611835114. content/114/13:0
URL https::/wwttt pnas org/
Heinrich Küttlerr Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Dis- tributed RL. arXiv preprint arX1v:1910.03552, 2019. URL lhttps://githhne com/
Cicebookrssearchfooocchho
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http:///am..eccc..oo//e////die// 2010.
10

---

Under review as a conference paper at ICLR 2021
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task- fase-free continual learning. CoRR, abs/2001.00689, 2020. URL http::
org/abs 20cla/a//........ 2001. 00689
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Na- talia Díaz Rodríguez. Continual learning for robotics. CoRR, 0001197700012,. 2019. URL
http://arxiv.org// wwt.:axwww....gaba////oa//// 00182
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation Advances in Research and Theory, 24(C):009-16,, 1989. ISSN 0079-7421. doi: 10. 10.10///0779- 7421 (08)600535366-3
Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 21, pp. 1081-1088. Curran Asso- ciates, Inc., 2009. URL URL /ttps///ppsss neeeeeeeediinnnnnnnsssssse ccc/ppper/208//1lle 14555002550cc55aa55522------
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Con- tinual lifelong learning with neural networks: review. CoRR, abs/1802.07569, 2018. URL
Roger Rateliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97 2:285-308, 1990.
Mark Ring. Child: A first step towards continual learning. Machine Learning, 07 1997. doi: 10.1023/A:10073312357.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Greg Wayne. Experience replay for continual learning. CoRR, abs/181 1. 11688161116,,,,,1111111111112211 2018. URI.
1861.11682
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, Abs/1606.04671, 2016. URLhhtt:://are /tattt:://ariiv org. http:/ab/lll..... 04671
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka BarbiikkkaB Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning. In International Conference on Machine Learning, pp. 35588-4333 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The CoRR, Abs/1701.00638, 2017. URL httthh/p/// wttp///axvvvv/vaa//ii//eea/aaava org/ abs 1701 06538
gated mixture-oun mfxture-of-expert experts layer.
Yikang Shen, Shawn Tan, Christopher Joseph Pal, and Aaron C. Courville, Sel-orgganizzed cal softmax. CoRR,
2017. /t//:/tte///arrxx..rrr/a///////p
Joel Veness, Tor Lattimore, Avishkar Bhoopchand, David Budden, Christopher Mattern, Agnieszka Grabska-Band Barkinkka Peter Toth, Simon Schmitt, and Mareus Hutter. Gated linear networks. CoRR, abs/1910.01526, 2019. httt::////aaxx wwwp:/gggg/bbbbbsee .1126.156111
R. J. Williams. Simple statistical gradient algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Improved multitask learning through synaptic intelligence. CoRR, abs/1703.04200 2017. URL lntp:://aaxxvvoog//aa////////// abs/////bb//033/./ 04200
Chen Zeno Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes, 2019.
11

---

P or in other words the ratio of the Under review as a conference paper at ICLR 2021
A
APPENDIX
Code is provided in the supplementary.
A.1 NODE VISUALIZATIONS
Here see four instances of states the same node activated. These examples were simply the last four usages we recorded where this node was activated, not cherry picked from a larger set. In all cases
that the ideal next action is to move forward, and indeed that is what this node's policy encodes for.
Figure 7: An example of four instances where the node triggered. When this node fires, the action taken is for the agent to move forward.
Additionally, in the supplementary there are two videos, minigrid.tree.mpp tree.mp4 and mnist tree.mp4, which demonstrate how the network grows over time. Nodes are created at the bottom of the video pink, and darken to blue as they get used, according to the same scaling as usage counts are scaled, see below.
A.2 SANE IMPLEMENTATION DETAILS
There number of details left out of the main SANE explanation, streamline the explanation down to the most significant concepts. These details are explained here, roughly in decreasing order of significance.
The entries in the replay buffer are importance-weighter when being used to train the critic. In particular they are scaled by the simple ratio: (eur(a) probability of the selected action using the node's current policy to that if the node's 's policy at the time the action was originally taken.
When merging nodes, only the first 3/4 of the set of nodes is eligible to he merged. This is under the premise that the most recent nodes were likely newly created, and we're giving them a little longer to be used before we collapse it into another node.
During training SANE operates in an epsilon- epsilon-greedy manner, selecting random action some fraction of the time, to encourage exploration.
All nodes' policies are scaled as (tanh(P), meaning each logit falls in the range (-3, 3). This is for few reasons. (1) This stabilizes the REINFORCE gradient applied to the policy. (2) Even a node is exceptionally confident in an action, when it encounters a situation it must still have some non-zero chance of trying another action (related to epsilon-greedy). (3) Raising the probability of wirriooully-disdvvanaggeee action to a range where gets selected frequently should not take large number of examples.
We maintain sets usage counts: (1) The total number of times the node has been activated across its entire ancestry (i.e. including the count of the node it was cloned from), roughly representing how many unique replay entries the critic has been exposed to. During merge this count becomes the average of the two nodes'. This is the usage used in weighting policies and replay buffers during merge. (2) The usage count since this node was created last cloned. Used for determining whether the node is eligible be cloned again. (3) The usage count since the last time the node was trained, only used to determine whether train the node's critic during the update step.
12

---

Under review as a conference paper at ICLR 2021
Usage count (I) from the above point is not used directly in determining node's weight; instead the weight is given by max(1, tanh wisage_count////1 have not been highly tuned, and work surprisingly well regardless of task. Ensuring minimum of means that under-used nodes are never completely ignored during merge.
* 100). These constants
To adjust for the fact that rewards are not consistently scaled across tasks, we maintain maz reward constant, set to the absolute value of an observed reward that exceeds the current mor_ mor_reward. This value is multiplied by .999 every timestep. All rewards are scaled as 10 * rewardd max reward, to scale their values between [-10, 10).
The critic network's 's weights are initialized according to normal distribution with mean 0 and standard deviation 0.001/in. faanffesuusss In other words, the values are very close
To adjust for the fact that the very first node's anchor is initialized randomly, and thus not effective baseline, have edge-case way to initiate the promotion of node: when it is used more than 10,000 times. (Usage count 2 above.)
We take leaf out of IMPALA's book and, since critics are updated asychronously, every node has a duplicate critic that is what gets trained, and then its weights are copied into the shared critic that is used during data collection. This does mean critie weights change mid-data collection, but not mid-training.
Every training step we update the non-leaf nodes' policies to be the weighted average of their children's. These policies are used for two purposes: (1) if the node cloned, its policy will end up the policy of leaf, and thus used, and (2) the policies used to determine which nodes to merge, so a reasonably meaningful one is beneficial. These aspects are primarily important in trees with more than two layers.
Periodically (every 100 updates) we send to each non-leaf node batch of entries from each of its children. This is in case one of the children has been getting activated to the exclusion of the others, ensuring the parent still represents its other children.
When bootstrapping the estimated value for the last entry in the replay buffer to compute the discounted rewards, found helpful multiply that value by 0.9. This is so if the estimate is wrong, it doesn't 't perpetuate itself via the discounted rewards entered into the replay buffer.
A.3 SANE EXPERIMENT DETAILS
Many of SANE's constants are common between experiments:
Description
Value
Policy learning rate
lc-4
Critic learning rate
le-3
Replay buffer size
4096
Min usages for promotion km
1500
MiniGrid constants:
Description
Value
Num parallel envs
Timesteps between updates (T)
1000
Error scale a
0.5
Merge ratio
Critic train epochs
Replay batch size (k)
4096
Reward discount
.99
Epsilon
0.02
MNIST constants:
13

---

Under review as a conference paper at ICLR 2021
Description
Value
Num parallel envs
Timesteps between updates (7)
250
Error scale
1.0
Merge ratio
3/4
Critic train epochs
Replay batch size (k)
1024
Reward discount
0
Epsilon
0.04
MNIST gets divided into more batches per training cycle, MNIST's discount factor must be be- cause the images presented to the agent have no correlation to each other in time, and MNIST gets a higher exploration rate.
A.4 BASELINE. IMPLEMENTATION DETAILS
For IMPALA used the Torchbeast implementation (Küttler et al. 2019). We built CLEAR top of this base. We used the same constants for both baselines, except IMPALA used an entropy cost of 0.0001.
Code available cont i nual r1 torchbeast's s default values available in httpe///aaa////llllo//oc////e//a Listed here are the ones we changed:
MiniGrid constants
Description
Value
Unroll length (k)
20
Entropy cost
0.005
Learning rate
0.006
Num learner threads
Discounting
0.99
MNIST constants
Description
Value
Unroll length (k)
20
Entropy cost
0.01
Learning rate
0.001
Num learner threads
Discounting
A.5
MINIGRID REMAINING RESULT
MNIST: 9
CLEAS c.000
Figure 8: The result training on digit 9, annexed here for want of space.
14

---

Under review as a conference paper at ICLR 2021
A.6 NETWORK SPECIFICATIONS
To simplify things, we adopt the convention of describing 2d convolutional layers with Convi(e w, 8), indicating there are output channels, window size w w, and stride s. Simi- larly, denote max pooling over a window size of with MozPool(w)). Finally, we denote a linear layer with output channels with (cn(c).
In all cases the non-linearity used is ReLU, and for simplicity remains implied.
A.6.1 BASE NETWORKS
For MiniGrid the base encoder, shared by SANE, CLEAR, and IMPALA is
Conv(16, 2,1) > MarPool(2) + Cono (32,2 - Conv(64,,,11) + Flatten + Lin(32)
For MNIST the base encoder is:
Conv(24,5,1) - MaxxPoo((2) Cona(48,5,) 5) - MaxPool > Flatten -> Lin(32)
(5)
For CLEAR and IMPALA, the output of these encoders concatenated with a one-hot of the last action and the last reward, and projected using linear layer to the final output size: the total number of actions (10 for MNIST, for MiniGrid).
For SANE, the encoder is followed by
encoder + Lim(128) - Lin(128) + Lin(2)
(6)
Where the the 2 output values are the expected value e and the uncertainty u for that node. This gives total of about 75,000 parameters for MNIST and around 35,000 for MiniGrid.
A.6.2 INCREASED PARAMETER NETWORKS
A number of iterations were tried for these larger networks, particularly with emphasis maintain- ing certain bottleneck layer dimensions, though we do not claim have spanned the whole space of options. These are what were used in the paper:
For MiniGrid, giving around 1.2M parameters:
Conv(1024,2 2, 1) Conw(1024,1)) 1, Max MaxPool(2)) - Conv(32,2.)) Conw(64, + Flatten - Lin(32)
For MNIST, giving around 5.5M parameters:
Conu(24, 5, 1) C20x8 1,1) ~ Conv(2048, 1,1) - Conw(24, 1,1) > Max (ool()
Conv(48, 5, 1) - Conv (1024, 1, 1) Conv 1024, 1, 1) Conv 48, 1, 1) Max MaxPooll (2)
Flatten + (in
(8)
15

---

Under review as a conference paper at ICLR 2021
A.7
CLEAR REPLAY FRACTION COMPARISON
A.7.1 MNIST
WONDS
MENDO
WW
MM
Figure 9: Comparison of CLEAR methods observe the trade-off between speed of learning task and recall of old tasks, in MNIST. The number represents the fraction of the training batch (of size that from the replay buffer. SANE also included for comparison.
In Figure we present comparison of CLEAR runs, varying the fraction of each training batch that is sampled from the replay buffer. can see that the higher ratios do indeed both learn more slowly (for instance digit 2), and remember previous tasks longer. Higher ratios did not reliably learn the digits, so they were omitted from the analysis. From this we see that r=0.88 does the best, so that is what we use in the comparison in Figure 4 We additionally ran early stopping (stopping after exceeding a threshold 0.97), but it exhibited no recall on past experiments, so we omitted reduce clutter in the graph.
A.7.2 MINIGRID
Minigrid:
Jniock
-OLEAR
-0BA rub
OLEAS mi2 Jeaffy st6t
1.80
2.50
1.54
2.54
Figure 10: Comparison of CLEAR methods to observe the trade-off between speed of learning new task and recall of old tasks, in Minigrid. The number represents the fraction of the training batch (of size 8) that is from the replay buffer. SANE also included for comparison.
16

---

Under review as a conference paper at ICLR 2021
In Figure 10 see comparison for Minigrid, with the same method used for MNIST. Early stopping is done when the value exceeds (0.78, 0.95, 0,92) respectively, selected by observing prior runs. Note replay ratio of 0.88 performs poorly in the second task and does not succeed at all at the third task, allowing for the superior recall of the first task. Farly stopping and replay ratio of 0.75 perform comparably;
select early stopping for our comparison due to better asymptotic performance in 2 Room. Note that, by chance, every experiment type (including SANE) had one fail to learn task 3 within the allotted time, and was thus re-run.
17


Reviews:
[{"review_id": "XTd9W7h-RaH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit\n\nUpdated score from 4 to 5\n\n## Summary\n\n* The paper proposes a task agnostic CL method called SANE. It uses a hierarchical, modular tree-like architecture where nodes are neural networks (policy + critic + replay buffer). \n* The architecture is quite interesting. Alongside regular parameter update operations, it also supports updates to the tree-structure.\n* Each state (that a learning agent encounters) activates a new path in the tree (thus reducing negative interference) while still enabling knowledge sharing.\n* Experiments on MNIST and Mini-Grid to show the usefulness of SANE over the baselines.\n\n## Objective\n\n* Addressing the problem of continual learning in the context of reinforcement learning.\n\n## Strong Points\n\n* The proposed architecture is quite intriguing. \n* It has several practical desiderata like:\n  * Updates on the tree-structure. More nodes can be added (on the fly) while keeping the overall complexity (of training/inference) in check.\n  * Works in the task-agnostic setting, which is more general and practical than the task-aware setting.\n* The writing (while missing some key details, see \"Areas for Improvement\") generally flows well and conveys high-level ideas.\n* It is quite interesting (and motivating) that SANE retains knowledge of the previous tasks as it continues training on the new tasks. I think of the tree-structure as a way of inducing a prior for the learning agent. The tree-update operations are along the direction of learning a useful inductive bias (without hardcoding the structure). SANE could be a good step in the direction of learning priors/inductive biases from data alone.\n\n## Areas for Improvement\n\n* While the paper describes the high-level idea well, it is very vague in several design choices/implementation details. The lack of detail makes it hard to evaluate the usefulness of the approach. Some examples:\n  * The paragraph on merging nodes is very vague. What does it mean to find the closest nodes in the policy space? Do you compute the L2 norm between the predictions of the policies of all possible inputs? How are policies combined by doing a \"weighted sum of usage count\"? What does \"retraining once\" mean. How do you decide when to stop training the merged node.\n\n* Setup: The paper mentions that they are operating in the continual RL setup. The MNIST task is a one-step RL problem (or a one step decision-making problem). The MiniGrid setup is more exciting, but the paper considers only three tasks (difficult to evaluate the approach's utility when the number of tasks increases). It does not show convincing results on even those (more at a later point). \n\n* Unconvincing results: \n\nLet's start with the MiniGrid results. The paper mentions that \"The Empty8x8 graph is the most insightful: CLEAR has almost perfect recall of this environment while 2-Room is training, but catastrophically forgets it during the training of Unlock.\". Let us see the performance of CLEAR on Unlock -- Figure 5 (2nd row, 3rd column, MiniGrid Unlock). We see that around 1.5M steps, the CLEAR baseline reaches about 90% performance and starts to saturate. SANE is performing approximately 20% at this point, and even after training for 2.25M steps, SANE reaches around 80% success. My argument is, CLEAR starts overfitting after 1.5M steps while SANE is not overfitting even after 2.25M steps. Now, look at Figure 5 (2nd Row, 1st Column MiniGrid Empty). Around 1.5M steps, CLEAR is performing close to the SANE model.  I could argue that there is no need to train CLEAR (and other baselines) for additional 750K steps just because the SANE model needs to be updated. Updating for so many extra steps is also causing the CLEAR baseline to overfit to the Unlock task. Please note that I am not criticizing the decision to train SANE for 2.25M steps. I am criticizing the choice of training CLEAR much longer. \n\nRegarding the MNIST results, I have the same worry. We can see that the baselines reach close to 100% very quickly by zooming into the plots, while the SANE model takes much longer. Repeating myself,  I am not criticizing the decision to train SANE much longer. I am criticizing the choice of training the other baselines much longer, thus forcing them to overfit to the current task.\n\n* Choice of baselines: Apart from the criticism of the experiment results, I found the baselines' choice to be quite weak as well. While the authors rightly point out that task-agnostic learning is a much harder setup, they consider very few baselines (actually only one baseline, since Impala is not for CL). Regarding CLEAR and CLEARx40, I do not think CLEARx40 is a fair baseline. Using much more parameters makes the model more likely to overfit, given that the mode is always trained for a fixed number of steps, irrespective of when it starts to overfit. Some more reasonable baselines would have been:\n\n@InProceedings{Aljundi_2019_CVPR,\nauthor = {Aljundi, Rahaf and Kelchtermans, Klaas and Tuytelaars, Tinne},\ntitle = {Task-Free Continual Learning},\nbooktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\nmonth = {June},\nyear = {2019}\n}\n\n@incollection{NIPS2019_9357,\ntitle = {Online Continual Learning with Maximal Interfered Retrieval},\nauthor = {Aljundi, Rahaf and Belilovsky, Eugene and Tuytelaars, Tinne and Charlin, Laurent and Caccia, Massimo and Lin, Min and Page-Caccia, Lucas},\nbooktitle = {Advances in Neural Information Processing Systems 32},\neditor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\\textquotesingle Alch\\'{e}-Buc and E. Fox and R. Garnett},\npages = {11849--11860},\nyear = {2019},\npublisher = {Curran Associates, Inc.},\nurl = {http://papers.nips.cc/paper/9357-online-continual-learning-with-maximal-interfered-retrieval.pdf}\n}\n\n\n@inproceedings{\nLee2020A,\ntitle={A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning},\nauthor={Soochan Lee and Junsoo Ha and Dongsu Zhang and Gunhee Kim},\nbooktitle={International Conference on Learning Representations},\nyear={2020},\nurl={https://openreview.net/forum?id=SJxSOJStPr}\n}\n\n* Limited related work: Apart from some articles mentioned in the previous point, some recent works look at learning ensembles of \"self-activating\" policies.\n\nhttps://icml.cc/Conferences/2020/ScheduleMultitrack?event=6293\n\n@inproceedings{\nGoyal2020Reinforcement,\ntitle={Reinforcement Learning with Competitive  Ensembles of Information-Constrained Primitives},\nauthor={Anirudh Goyal and Shagun Sodhani and Jonathan Binas and Xue Bin Peng and Sergey Levine and Yoshua Bengio},\nbooktitle={International Conference on Learning Representations},\nyear={2020},\nurl={https://openreview.net/forum?id=ryxgJTEYDr}\n}\n\n\n\n\n* Additional Results: It is difficult to understand what the nodes are learning. The appendix has one example, which is not sufficient to understand what the nodes are learning. It is also not clear why would the learning model try to activate different paths. This relates to the problem of learning options in RL and the commonly observed problem where only one option is active during most of the training. At the minimum, it will be useful to see some results/plots describing how the tree evolves during training and how the frequency of use of different nodes changes.\n\n* Ablations: The paper introduces several hyper-parameters but does not consider any sort of ablations with them. This makes it difficult to understand what hyper-parameter values are important and which are set arbitrarily.\n\n## Some other questions\n\nPlease note that these are some general questions, and I did not consider any of these design choices as \"wrong\" or \"against the paper.\"\n\n* Why did the paper use L2 norm and not KL penalty (section 2.3)\n\n* The paper mentions that they are operating in a task-agnostic setting. While that is true, they are still training/evaluating the model in the more limited sequential (one-task-at-a-time) setting. Is there a reason to not work in the more general any-task-at-any-time setting?\n\n* Could the authors also include a discussion of the training time for the baselines vs. SANE. My understanding is, training SANE takes a lot more time/steps.\n\n* Could the authors add some description about how rewards are computed for the MNIST setup.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X5wEinoJ5aa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes SANE -- an architecture and a training algorithm for continual learning. The SANE model consists of a tree where each node can act as an RL agent and where nodes act according to the dispatching mechanism based on their reward prediction. This allows to activate and update only those agents that are specialized in the current task and thus may prevent catastrophic forgetting caused by updating the whole model.\n\nNovelty:\nDifferent parts of the method did have appearance in the prior literature. For example, the general idea to organize the model into a hierarhical structure with a certain kind of dispatching can be found in hierarchical softmax [1] with a more modern variant [2]. More recently, [3] proposed a similar model with a very close motivation to address catastrophic forgetting. In my opinion, authors should make a more extensive literature review and more clearly justify novelty of the method.\n\nClarity:\nAfter reading the paper several times, I did not get a clear enough picture of the proposed model operates. Perhaps, a more formal algorithmic description would help. \n\nSignificance:\nThe empirical evaluation only consists of relatively simple experiments. MNIST, for example, is hardly representative of a real-world continual learning task. Authors do not compare SANE to string baselines that are specifically designed to prevent catastrophic forgetting, even to EWC which is cited. Even though, EWC does rely on task boundaries information, it would at the very least define the gap that exists to the methods that are agnoistic to this kind of priveledged information. \nThe lack of a thorough experimental study makes it difficult to argue for high significance of this work to the community.\n\nOther comments:\n I think what authors mean by \"discounted reward\" and $r_{disc}$ is usually denoted as \"discounted return\", this should be clarified.\n\nReferences:\n[1] A scalable hierarchical distributed language model. Andriy Mnih and Geoffrey Hinton. 2008.\n[2] Self-organized Hierarchical Softmax. Yikang Shen, Shawn Tan, Chrisopher Pal, Aaron Courville. 2017\n[3] Gated Linear Networks. Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Agnieszka Grabska-Barwinska, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, Marcus Hutter. 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "TSEW0lnhBvl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal update:\n\nI thank the authors for their responses to my queries and for updating the paper. I think the paper has improved, the main reason being that the replay-ratio parameter sweeps for CLEAR in the MNIST experiments show that SANE does improve the stability-plasticity tradeoff, in particular the runs with a 0.97 replay ratio (only shown in the rebuttal pdf but I think should make it into the appendix of the paper), which show that the memory of CLEAR cannot be improved to match SANE's without severely affecting its ability to learn new tasks. The only other set of experiments, the sequence of three minigrid tasks, however, do not show an overall improvement over CLEAR, and so overall the experimental evaluation is still weak. It is hard to judge how good the method is without extending the range of baselines (I appreciate the addition of the and task settings. As a result, I am only increasing my score from a 4 to a 5, but I do think SANE is an interesting method and I encourage the authors to keep working at it to prove its effectiveness.\n\n---------------------------\nOriginal review:\n\nThis paper presents SANE (Self-Activating Neural Ensembles) a new dynamic architecture method for mitigating catastrophic forgetting in neural networks in the context of reinforcement learning that is task-agnostic. The architecture consists of a tree where each node comprises its own value critic, policy and replay buffer. Action selection at each time step begins by starting at the root of the three and activating the child that predicts the most optimistic value, and repeating this procedure from the activated child until a leaf node is reached, whose policy is used to act with. Training has two components: (i) the parameters in activated nodes are updated using REINFORCE, and (ii) structural updates are made to the tree whereby children that are sufficiently different to their parents are promoted up the tree and given their own children, and children that are similar to each other are merged in order to limit growth of the tree. The method is empirically compared to two baselines, IMPALA and IMPALA with CLEAR (an experience-replay based continual learning method), on two sequences of tasks, sequential MNIST and a sequence of three egocentric gridworld tasks (MiniGrid). On sequential MNIST, SANE displays less forgetting but less adaptivity than the baselines, and on the gridworld tasks the results are mixed.\n\nSANE is an interesting and inventive method but I do not think it is ready for publication yet because (i) the experiments do not sufficiently support the claim that SANE improves catastrophic forgetting over the (limited set of) baselines, and (ii) SANE has a lot of components and hyperparameters but it is not clear which aspects are important for it to work. I believe the paper could be much improved by (i) demonstrating that the baselines have been adequately tuned (and perhaps increasing the set of methods compared to, e.g. with another dynamic architecture method), (ii) including a number of ablation studies or other experiments that better elucidate the inner workings of SANE and justify architectural choices.\n\nPositives:\n\t•\tSANE incorporates a number of interesting mechanisms that might intuitively be useful for mitigating catastrophic forgetting. For example, by only updating activated nodes, modularity is preserved during training in a way that can protect forgetting in other parts of the network. Also, the structural updates provide an interesting way of expanding and contracting the architecture as necessary under fixed resource constraints - an important challenge for continual learning. \n\t•\tThe problem that SANE tackles, i.e. task-agnostic continual learning, is an important one for which there are not a large number of approaches in the literature. \n\t•\tAn effort is made to equalise the number of parameters used by the baseline agents and SANE in the experiments for a fair comparison, and experiments are repeated with several random seeds. \n\nConcerns:\n\t•\tSANE is claimed to improve on the baselines of IMPALA and CLEAR on the axis of mitigating catastrophic forgetting, but the experiments do not demonstrate this clearly. \n\t◦\tIn the sequential MNIST experiments, SANE retains some performance on earlier tasks as training progress but the performance of both IMPALA and CLEAR seems to drop almost instantaneously after the task is switched. On the other hand, both IMPALA and CLEAR learn the new task extremely quickly compared to SANE, which usually does not even reach the same maximum level of performance as the baselines. This suggests that SANE shifts the tradeoff of adaptivity vs. remembering towards remembering, but doesn’t show that the tradeoff is improved vs the baselines. In the CLEAR paper, there is an important hyperparameter that mediates this tradeoff, which is the proportion of replay experiences vs. new experiences used for training - this parameter should be tuned here to be able to show that CLEAR is indeed inferior to SANE on this task. (Judging from the code it seems that a 50/50 split is used, is this correct?) \n\t◦\tIn the sequence of three MiniGrid tasks, even though the performance of the CLEAR agent on the first task eventually drops below that of the SANE agent, the SANE agent initially deteriorates a lot faster. For the second task there seems to be no significant difference between agents. Additionally, the SANE agent is much slower to learn on the third task, never reaching the performance of the CLEAR agent, again demonstrating that the SANE agent is trading off adaptivity for slightly better memory. It’s not evident that SANE is improving the tradeoff vs. CLEAR without tuning the latter’s hyperparameters. \n\t•\tIt is difficult to discern which elements of SANE are important for its performance in order to justify some of the architectural choices. For example: \n\t◦\tThe critics are trained to estimate both the value and the variance of the value, and, in the inference stage, the node with the highest upper bound of estimated reward is chosen. How important is this UCB-like policy for the performance of the agent, and the value of the hyperparameter alpha that scales the standard deviation term? Would SANE work just as well if just the value estimate was used to choose the next node? Could this lead to a wrong solution if applied in a highly stochastic environment? \n\t◦\tThe policies of the nodes in SANE use REINFORCE but IMPALA and CLEAR use VMPO; is there a reason for this difference? \n\t◦\tIn the structural updates, for node promotion the value functions of a child and its parent are compared, but for node merging, the policies are compared - is there a rationale behind these choices?  \n\t◦\tIn the MiniGrid experiments, SANE is run with either 10 or 15 top level nodes, resulting in a relatively big difference in performance for the 1st and 3rd tasks, giving some indication of the sensitivity of this hyperparameter. Why were 20 top-level nodes chosen for the MNIST experiments and was the performance sensitive to this choice? How about other hyperparams such as number of child nodes and the number of times a node has to have been used before it can be promoted? \n\nOther comments:\n\t•\tThe number of baselines used for comparison is limited, especially given that one of them (IMPALA) is not designed for continual learning. It would at least be good to cite other task-agnostic methods such as [1,2]. \n\t•\tIt is slightly odd to frame sequential MNIST as an RL problem, when it is a supervised learning task. \n\t•\tA visualisation is given in the appendix showing an example of a node that seems to represent a policy that always moves the agent forward one step in the gridworld. Presumably this is shown to indicate the specialisation of modules in the tree; it would be very interesting to see more evidence of specialisation/diversity across modules since this is one of the motivating factors for the architecture and also to see how the tree structurally adapts at task switches - are there more promotions when the data distribution changes?  \n\n[1] Aljundi, Rahaf, Klaas Kelchtermans, and Tinne Tuytelaars. \"Task-free continual learning.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n[2] Zeno, Chen, et al. \"Task agnostic continual learning using online variational bayes.\" arXiv preprint arXiv:1803.10123 (2018).", "extra_scores": {}, "extra_text": {}}, {"review_id": "MinW53KvF_e", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work addresses multi-task learning where task boundaries are unknown. The approach is to construct a dynamic decision tree with nodes made up of small networks. Nodes are merged and promoted in the tree based on learned error bounds on value function estimates. Inference through the tree works by selecting nodes with the highest value prediction. It is an interesting approach for modular learning, even within the same environment.\n\nThe paper is clearly written. The approach is clearly explained. The empirical evaluation is thorough, with good control of the baselines.\n\nNode promotion is done so that a new task can take a new path through the tree, avoiding catastrophic forgetting. This is done by comparing the value function estimates of a child node to its parents. A large enough difference in value function estimate triggers a node promotion. An important assumption here is that different tasks differ substantially in long term returns. This may not be the case in general. Tasks can differ in various ways apart from the total reward. Sufficient discussion is not provided on this assumption and how it affects the results.\n\nAt the end of page 5, it is claimed that sequential MNIST is “significantly more realistic”. Why is this the case? \n\nFigure 4 is a bit hard to understand. What is the “reward” here? What is the action space for this task?\n\nIn table 1, why is the SANE accuracy for digit “0” not 100%? All other methods are 100% and this is the only class so far unless I’m misinterpreting the table.\n\nWhile SANE manages to avoid catastrophic forgetting compared to the baselines, the classification accuracy for each digit does settle back down to less than 50% in most cases. Why is this the case?\n\nHow effective is the strategy of detecting task change using value function bounds in the mini world environment? The rewards do not seem to be very different. I would be interested in seeing some examples of transitions that trigger a node promotion.\n\nThe results in this environment are not so convincing. Is it perhaps because of the values being similar across tasks?\n\nSome more discussion of how this work differs from other approaches that combine deep learning and decision tree could be included.\n\nIn section 2.3 “Training the Critics”, it is stated that the critic is trying to minimize the value v(s). Why is this done?\n\nFuture work could include using distribution RL for detecting outliers and node promotion.\n\nThis is an exciting direction of work and a difficult problem without known task boundaries. The approach is sound with decent empirical evaluations. The paper could benefit from some clarity in the explanation of the results and the assumptions made in the method. Some more in-depth analysis of the trees contracted by this method and their dynamic behavior through learning would shed some more light on some of the results.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have arrived at the consensus that this is a paper with an interesting idea, both novel and well-explained, but not quite backed up with sufficient empirical evidence. Like them, I think there is a lot of potential in modular methods for continual learning, and I know these are challenging advances to demonstrate. So I encourage you to persist, iterate and submit a stronger version of this paper in the future!
Decision: Reject