Self-Activating Neural Ensembles for Continual Reinforcement Learning

1 INTRODUCTION . Lifelong learning is of critical importance for the field of robotics ; an agent that interacts with the world should be able to continuously learn from it , acting intelligently in a wide variety of situations . In marked contrast to this ideal , most standard deep reinforcement learning methods are centered around a single task . First , a task is defined , then a policy is learned to maximize the rewards the agent receives in that setting . If the task is changed , a completely new model is learned , throwing away the previous model and previous interactions . Task specification therefore plays a central role in current end-to-end deep reinforcement learning frameworks . But is task-driven learning scalable ? In contrast , humans do not require concrete task boundaries to be able to effectively learn separate tasks – instead , we perform continual ( lifelong ) learning . The same model is used to learn new skills , leveraging the lessons of previous skills to learn more efficiently , without forgetting old behaviors . However , when placed into continual learning settings , current deep reinforcement learning approaches do neither : the transfer properties of these systems are negligible and they suffer from catastrophic forgetting ( McCloskey & Cohen , 1989 ; French , 2006 ) . The core issue of catastrophic forgetting is that a neural network trained on one task starts to forget what it knows when trained on a second task , and this issue only becomes exacerbated as more tasks are added . The problem ultimately stems from training one network end-to-end sequentially ; the shared nature of the weights and the backpropagation used to update them mean that later tasks overwrite earlier ones ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) . To handle this , past approaches have attempted a wide variety of ideas : from task-based regularization ( Kirkpatrick et al. , 2017 ) , to learning different sub-modules for different tasks ( Rusu et al. , 2016 ) , to dual-system slow/fast learners inspired by the human hippocampus ( Schwarz et al. , 2018 ) . The core problem of continual learning , which none of these methods address , is that the agent needs to autonomously determine how and when to adapt to changing environments , as it is infeasible for a human to indefinitely provide an agent with task-boundary supervision . Specifically , these approaches rely on the notion of tasks to identify when to spawn new sub-modules , when to freeze weights , when to save parameters , etc . Leaning on task boundaries is unscalable and side-steps the core problem . There are a few existing task-free methods . Some address the problem by utilizing fixed subdivisions of the input space ( Aljundi et al. , 2018 ; Veness et al. , 2019 ) , which we believe limits their flexibility . Recent experience-based approaches such as Rolnick et al . ( 2018 ) forgo explicit task IDs , but to do so they must maintain a large replay buffer , which is unscalable with an ever-increasing number of tasks . To our knowledge only one other continual learning method , Neural Dirichlet Process Mixture Models ( Lee et al. , 2020 ) , adaptively creates new clusters , and we show in our experiments that SANE outperforms it . Our method approaches the problem head-on by dynamically adapting to changing environments . Our task-agnostic hierarchical method , Self-Activating Neural Ensembles ( SANE ) , depicted in Figure 1 , is the core of the method . Every node in the tree is a separate module , none task-specific . At each timestep a single path through the tree is activated to determine the action to take . Only activated nodes are updated , leaving unused modules unchanged and therefore immune to catastrophic forgetting . Critically , our tree is dynamic ; new nodes are created when existing nodes are found to be insufficient . This allows for the creation of new paths through the tree for novel scenarios , preventing destructive parameter updates to the other nodes . SANE provides the following desirable properties for continual reinforcement learning : ( a ) It mitigates catastrophic forgetting by only updating relevant modules ; ( b ) Because of its task-agnostic nature , unlike previous approaches , it does not require explicit supervision with task IDs ; ( c ) It achieves these targets with bounded resources and computation . In our results , we demonstrate the ability of SANE to learn and retain MNIST digits when presented fully sequentially , a challenging task our baselines struggle with . We also demonstrate SANE on a series of three grid world tasks , showing that SANE works in the reinforcement learning setting . 2 MODEL . The structure of our Self-Activating Neural Ensembles ( SANE ) system is a tree , as shown in Figure 1 . Each node in the tree contains both an actor and a critic , described in 2.1 . At each timestep we activate a path through the tree from the root to a leaf , at each layer selecting the node whose critic estimates the highest activation score ( described in 2.2 ) . We then sample an action from that leaf node ’ s policy to execute in the environment . During training , the activated nodes are the only ones updated ; all other nodes are unchanged . This enables behaviors selected for by other paths through the tree to remain unchanged . The tree structure is critical to the success of the system ; a node ’ s children can be seen as partitioning the input space that activates the node , specializing their policies to these subspaces . When they differ sufficiently from their parent in performance , they are promoted to a higher level of the tree and obtain children of their own , further partitioning the space . Nodes are constantly being created , promoted to higher levels , or merged as the situation demands ( described in Section 2.4 ) . Static vs . Dynamic Trees : One critical question to ask is why use a dynamic structure and not just a huge static tree with only parameter updates ? In the case of the static tree , if a path through the network is beneficial for one task , it is likely to be chosen as a starting point for the subsequent task , and updated in-place for this new setting . Examples of this can be found in Jacobs et al . ( 1991 ) ; Shazeer et al . ( 2017 ) , where specialized losses are necessary to distribute activation across the ensemble of experts , even in the case of a stationary distribution . With SANE , the change of environment would be detected and a new path created , thus minimizing disruption of the previously beneficial behaviors . 2.1 NODE CONTENTS . Every node i contains the following : a policy pi , a critic Ci , a replay buffer containing a list of ( state s , discounted reward rdisc ) , and a count of the number of times this node has been used ( usage count ) . The policy pi presented here is a vector of n logits , where n is the number of actions available to the agent . The critic Ci is a neural net mapping a state s to two scalars : the value estimate of the discounted reward received if the current node is activated , vi ( s ) , and an estimate of the absolute error : ui ( s ) ≈ |rdisc − vi ( s ) | ( 1 ) With these we can define an optimistic estimate ( upper confidence bound ) and a pessimistic estimate ( lower confidence bound ) for what reward a node can achieve : vUCB , i ( s ) = vi ( s ) + α ∗ ui ( s ) ( 2 ) vLCB , i ( s ) = vi ( s ) − α ∗ ui ( s ) ( 3 ) α is a hyper-parameter representing how wide a margin around the expected value we allow . vUCB is analogous to an upper confidence bound in multi-armed bandit literature : as training proceeds and vi becomes more accurate , ui will decrease , providing a natural transition from exploration to exploitation for each node as it learns . This may however act too optimistically in highly stochastic environments , an area of future work . Each node ’ s replay buffer contains the history of experiences per node and is used to train the node ’ s critic . The usage count is the number of times a node has been activated and is used in the merging of nodes . These are described more in sections 2.3 and 2.4 . 2.2 INFERENCE . For each observed state st from the environment , we activate a path through the tree from the root to a leaf node , whose policy is used to determine the next action to take in the environment , at . The environment then responds with a reward rt and a new observation st+1 ( See Figure 2 ) . We find the path recursively : from an activated node , the next node to activate is selected from its children by taking the node with the highest optimistic upper bound of reward . This process continues until a leaf node is reached . After the set of activated nodes is determined , an action is selected stochastically from the activated leaf ’ s policy , and executed in the environment . New nodes are not created mid-episode to keep the tree idempotent , which allows us to readily run environments in parallel . During inference , three things happen at each timestep to support updating the tree . All active nodes : • are registered to receive the replay entry for that timestep after the episode completes . • compare their critic ’ s estimate to their parent ’ s . If they differ sufficiently ( Section 2.4 ) , the child will be promoted during the next tree structure update . • increment their usage count . 2.3 PARAMETER UPDATES . SANE training updates consist of two steps : parameter updates ( replay buffer , policy , and critic ) and tree structure updates ( creating new nodes , merging nodes , etc. ) . After T steps of data collection have occurred , we trigger both a parameter update and a tree structure update . Replay Buffer Update : Each node ’ s replay buffer contains the history of the times that node was activated , and is used to train the node ’ s critic . Each entry in the replay buffer contains an input state s and a discounted reward received rdisc , which is also used in the training of the policies . The discounted reward is computed in the standard fashion , as in Espeholt et al . ( 2018 ) . When the addition of new entries results in the replay buffer exceeding its max length , instead of removing the oldest entries , random entries are removed instead . This is to mitigate each critic from developing a recency bias . Policy Update : The policy for each node is trained on-line using REINFORCE ( Williams , 1992 ) with Monte Carlo returns . Training the Critics : The critic for each node is trained off-line using the replay buffer . More specifically , we split a node ’ s replay buffer into batches each of size k. The critic has two functions to estimate : v ( s ) ( the expected value ) and u ( s ) ( the uncertainty as defined in equation 1 ) . We use L2 loss for both reward and error prediction . Note that this is the slowest step in the training of the SANE tree . Since the nodes all use separate critics , this step can be effectively parallelized . We only train a critic when its node was activated at least once since the last time nodes were trained .

Summary: This work addresses multi-task learning where task boundaries are unknown. The approach is to construct a dynamic decision tree with nodes made up of small networks. Nodes are merged and promoted in the tree based on learned error bounds on value function estimates. Inference through the tree works by selecting nodes with the highest value prediction. It is an interesting approach for modular learning, even within the same environment.