Modular Lifelong Reinforcement Learning via Neural Composition
Authors: Jorge A Mendez
Harm van Seijen
ERIC EATON
Venue: ICLR 2022 Poster (2022)
Keywords: lifelong learning
continual learning
reinforcement learning
composition
modularity
compositionality

Abstract: Humans commonly solve complex problems by decomposing them into easier subproblems and then combining the subproblem solutions. This type of compositional reasoning permits reuse of the subproblem solutions when tackling future tasks that share part of the underlying compositional structure. In a continual or lifelong reinforcement learning (RL) setting, this ability to decompose knowledge into reusable components would enable agents to quickly learn new RL tasks by leveraging accumulated compositional structures. We explore a particular form of composition based on neural modules and present a set of RL problems that intuitively admit compositional solutions. Empirically, we demonstrate that neural composition indeed captures the underlying structure of this space of problems. We further propose a compositional lifelong RL method that leverages accumulated neural components to accelerate the learning of future tasks while retaining performance on previous tasks via off-line RL over replayed experiences.

Published as a conference paper at ICLR 2022
MODULAR LIFELONG REINEORCEMENT LEARNING VIA NEURAL COMPOSITION
Jorge A. Mendez", Harm van Seijen', and Eric Eaton'
'Department of Computer and Information Science University of Pennsylvania fmendezme, beaton]@seass upenn. edu
Miccoooott Research
harm. vansei www@microsoft.ccommm
ABSTRACT
Humans commonly solve complex problems by decomposing them into easier subproblems and then combining the subproblem solutions. This type of compo- sitional reasoning permits reuse of the subproblem solutions when tackling future tasks that share part of the underlying compositional structure. In a continual or lifelong reinforcement learning (RL) setting, this ability decompose knowledge into reusable components would enable agents to quickly learn new RL tasks by leveraging accumulated compositional structures. We explore particular form of composition based on neural modules and present a set of RL problems that in- tuitively admit compositional solutions. Empirically, we demonstrate that neural composition indeed captures the underlying structure of this space of problems. We further propose a compositional lifelong RL. method that leverages accumu- lated neural components to accelerate the learning of future tasks while retaining performance on previous tasks via off-line RL over replayed experiences.
INTRODUCTION
Reinforcement learning (RL) has achieved impressive success at complex tasks, from mastering the game of Go (Silver et al., 2016; 2017) to robotics (Gu et al., 2017; Andrychowicz et al., 2020). However, this success has been mostly limited to solving a single problem given enormous amounts of experience. In contrast, humans learn to solve myriad of tasks over their lifetimes, becoming better at solving them and faster at learning them over time. This ability to handle diverse problems stems from our capacity to accumulate, reuse, and recombine perceptual and motor abilities in dif- ferent manners to handle novel circumstances. this work, we seek to endow artificial agents with a similar capability to solve RI. tasks using functiona compositionality of their knowledge.
In lifelong RL, the agent faces a sequence of tasks and must strive to transfer knowledge future tasks and avoid forgetting how to solve earlier tasks. We formulate the novel problem of lifelong RL of functionally compositional tasks, where tasks can be solved by recombining modules of knowl- edge in various ways. While temporal compositionality has long been studied in RL, such in the options framework, the type of functional compositionality we study here has not been explored in depth, especially not in the more realistic lifelong learning setting. Functional compositionality involves a decomposition into subproblems, where the outputs of one subproblem become inputs to others. This moves beyond standard temporal composition functional compositions of layered perceptual and action modules, akin to programming where functions are used in combination to solve different problems. For example, a typical robotic manipulation solution interprets perceptual inputs via a sensor module, devises a path for the robot using high-level planner, and translates this path into motor controls with robot driver. Each of these modules can be used in other com- binations to handle a variety of problems. We present new method for continually training deep modular architectures, which enables efficient learning of the underlying compositional structures.
The modular lifelong RL agent will encounter a sequence of compositional tasks, and must strive to solve them as quickly as possible. Our proposed solution separates the learning process into three stages. First, the learner discovers how to combine its existing modules to solve the current task to the best of its abilities by interacting with the environment with various module combinations. Next, the agent accumulates additional information about the current task via standard RL training with
Part of this work was carried out while the author was at Microsoft Research.

---

Published as a conference paper at ICLR 2022
the optimal module combination. Finally, the learner incorporates any newly discovered knowledge from the current task into existing modules, making them more suitable for future learning. We demonstrate that this separation enables faster training and avoids forgetting, even though the agent is not allowed to revisit earlier tasks for further experience. Our main contributions include:
1. We formally define the lifelong compositional RL problem as a compositional problem graph, encompassing both zero-shot generalization and fast adaptation to new compositional tasks.
2. We propose two compositional evaluation domains: a discrete 2-D domain and a realistic robotic manipulation suite, both of which exhibit compositionality at multiple hierarchical levels.
3. We create the first lifelong RI. algorithm for functionally compositional structures and show empirically that it learns meaningful compositional structures in our evaluation domains. While our evaluations focus on explicitly compositional RL tasks, the concept of functional composition broadly applicable and could be used in future solutions to general lifelong RL.
4. We propose to use batch RL techniques for avoiding catastrophic forgetting in lifelong setting and show that this approach is superior to existing lifelong RL. methods.
RELATED WORK
Lifelong or continual learning Most work in lifelong learning has focused on the supervised set- ting, and in particular, on avoiding catastrophic forgetting. This has typically been accomplished by imposing data-driven regularization schemes that discourage parameters from deviating far from earlier tasks' solutions (Zenke et al., 2017; Li & Hoiem. 2017; Ritter et al., 2018), or by replaying real (Lopez-Paz & Ranzato, 2017: Nguyen et al., 2018: Chaudhry et al., 2019; Aljundi et al., 2019) or hallucinated (Achille et al., 2018; Rao et al., 2019; van de Ven et al., 2020) data from earlier tasks during the training of future tasks. Other methods have instead aimed at solving the problem of model saturation by increasing the model capacity (Yoon et al., 2018; Li et al., 2019: Rajasegaran et al., 2019). A few works have addressed lifelong RL by following the regularization (Kirkpatrick et al., 2017) or replay (Isele & Cosgun, 2018; Rolnick et al., 2019) paradigms from the supervised setting, exacerbating the stability-plasticity tension. Others have instead proposed multi-stage pro- cesses whereby the agent first transfers existing knowledge to the current task and later incorporates newly obtained knowledge into shared repository (Schwarz et al., 2018; Mendez et al., 2020). We follow the latter strategy for exploiting and subsequently improving accumulated modules over time.
Compositional supervised learning While deep nets in principle enable learning arbitrarily com- plex task relations, monolithic agents struggle to find such complex relations given limited data. Compositional multi-task learning (MTL.) methods use explicitly modular deep architectures to cap- ture compositional structures that arise in real problems. Such approaches either require the compo- sitional structure to be provided (Andreas et al., 2016; Hudson & Manning, 2018) or automatically discover the structure in a hard (Rosenbaum et al., 2018; Alet et al., 2018; Chang et al., 2019) or soft (Kirsch et al., 2018; Meyerson & Mikkulainen, 2018) manner. A handful of approaches have been proposed that operate in the lifelong setting, under the assumption that each component can be fully learned by training on a single task and then reused for other tasks (Reed & de Freitas, 2016; Fernando et al., 2017; Valkov et al., 2018). Unfortunately, this is infeasible if the agent has access to little data per task. Recent work proposed a framework for lifelong supervised compositional learning (Mendez & Eaton, 2021), similar in high-level structure to our proposed method for RL.
Typical evaluations of compositional learning use standard benchmarks such as ImageNet or CIFAR- 100. While this enables fair performance comparisons, it fails to give insight about the ability find meaningful compositional structures. Some notable exceptions exist for evaluating compositional generalization in supervised learning (Bahdanau et al., 2018; Lake & Baroni, 2018; Sinha et al., 2020). We extend the ideas of compositional generalization to RI., and introduce the separation of zero-shot compositional generalization and fast adaptation, which is particularly relevant in RI..
Compositional RL A handful of works have considered functionally compositional RL, either assuming full knowledge of the correct compositional structure of the tasks (Devin et al., 2017) or automatically learning the structure simultaneously with the modules themselves (Goyal et al., 2021; Mittal et al., 2020; Yang et al., 2020). We propose a method that can handle both of these settings. Two main aspects distinguish our work from existing approaches: 1) our method works in a lifelong setting, where tasks arrive sequentially and the agent is not allowed to revisit previous tasks, and 2) evaluate models on tasks that explicitly compositional at multiple hierarchical levels.
2

---

Published as a conference paper at ICLR 2022
A closely related problem that has long been studied in RL is that of temporally extended actions, or options, for hierarchical RL (Sutton et al., 1999; Bacon et al., 2017). Crucially, the problem we consider here differs in that the functional composition occurs at every time step, instead of the tem- poral chaining considered in the options literature. These two orthogonal dimensions capture real settings in which composition could improve the RI. process. Appendix A contains deeper discus- sion of these connections. Other forms of hierarchical RL have learned state abstractions (Dayan & Hinton, 1993; Dietterich, 2000; Vezhnevets et al., 2017; Abel al., 2018). While these are also related, they have mainly focused on a two-layer abstraction, where the policy passes the output of a state abstraction module as input to an action module. Instead, we consider arbitrarily many layers of abstraction that help the learning of both state and action representations. One additional form of composition RL uses full task policies as the components, and combines policies together when solving new task (e.g., by summing their value functions; Todorov, 2009; Barreto et al., 2018; Haarnoja et al., 2018; Van Nickerk ct al.. 2019; Nangue Tasse et al., 2020). Instead, we separate each policy itself into components, such that these components combine to form full policies.
Batch RL RL from fixed data introduces a shift between the data distribution and the learned policy, which strains the capabilities of standard methods (Levine et al., 2020). Recent techniques constrain the departure from the data distribution (Fujimoto et al., 2019b; Laroche et al., 2019; Kumar et al., 2019). This issue closely connects lifelong RL: obtaining backward transfer requires modifying the policy for earlier tasks without extra experience. We exploit this connection by storing a portion of the data collected on each task and replaying it for batch RL after future tasks are trained.
3 THE PROBLEM OF LIFELONG FUNCTIONAL COMPOSITION IN RL
Complex RL problems can often be divided into easier subproblems. Recent years have taught us that, given unlimited experience, artificial agents can tackle such complex problems without any sense of their compositional structures (Silver et al., 2016; Gu et al., 2017). However, discovering the underlying subproblems and learning to solve those would enable learning with substantially less experience, especially when faced with numerous tasks that share common structure. While our work shares this premise with hierarchical RI. efforts, those works do not consider the formulation we present here, where functional composition occurs at multiple hierarchical levels of abstraction.
Formally. an RL problem is given by a Markov decision process (MDP) 2 = (S, A. R. T. T,7), where is the set of states, A the set of actions, T S + S is the probability distribution P(s' a) of transitioning to state s' upon executing action a in state 8, R: S x A + R is the reward function measuring the goodness of a given state-action pair, and y is the discount factor that reduces the importance of rewards obtained far in the future. The agent's behavior is dictated by (possibly stochastic) policy æ : S - AA that selects the action to take at each state. The goal of the agent is to find the policy x² E II that maximizes the discounted long-term returns E|Ei > RR(ss a:)].
An RL problem Z is a composition of subproblems F1, F2.... if its optimal policy 'T" can be constructed by combining solutions to those subproblems: x² (s) = m, m2 ...., where each my E M X² + Y/s is the solution to the corresponding F. We first consider these subproblems from an intuitive perspective, and later define them precisely. In RL, each subproblem could involve pure sensing, pure acting, or a combination of both. For instance, in our earlier robotic manipulation example, the task can be decomposed into recognizing objects (sensing), detecting the target object and devising plan to reach it (combined), and actuating the robot to grasp the object (acting).
In lifelong compositional RL, the agent will be faced with sequence of MDPs 2(1) g())..., (Tmm over its lifetime. We assume that all MDPs are compositions of different subsets from k shared subproblems F = {Fi...., }. The goal of the lifelong learner is to find the set of solutions
to these subproblems as set of modules M {m {mp....,mpp m. such that learning to solve new problem reduces to finding how to combine these
modules optimally. Each module can be viewed
S(1)
as processing stage in hierarchical process- ing pipeline, or equivalently as functions in a pro- gram, and the goal of the agent is to find the cor- rect module to execute at each stage and the in- stantiation of that module (i.e., its parameters). We formalize the compositional RL problem as a graph g = (V,E) (e.g., Figure 1) whose nodes
112
F1
X22g
F2
171p
S(4)
Figure 1: Compositional RI. problem graph.
3

---

Published as a conference paper at ICLR 2022
are the subproblem solutions along with the state and action spaces: V FUŠUÄ, where $ = unique((S()) ..... ((Tm)))) and À = unique((A()) ...... (Tma))) are the unique state and action spaces. Each subproblem F; corresponds to latent representational space Vi. generated by the corresponding module m, X; + Di. For example, an object detection module could map an image to a segmentation mask, to be used by subsequent modules to make action decisions. Similarly, the SS and A()ss can serve as representation spaces (A2, ))).
Problem Z is then specified as a pair of state and action nodes (())) in the graph, and the goal is to find a path between those nodes corresponding to a policy x: =(()* that maximizes R(O,, More generally, the graph formalism allows for recurrent computations via walks with cycles, and parallel computations via concurrent multipaths; an extended definition of multiwalks trivially captures both settings. In this work, consider the path formulation, and restrict the number of edges in the graph by organizing the modules into layers, as explained in Section 5.1. This formalism is related to that of Chang et al. (2019) for the supervised compositional setting, but is adapted to RL. From this definition, we outline two concrete problem settings, based on the amount of information available to the agent. Both settings show how learning could benefit from compositional solutions.
Zero-shot generalization with full information In some scenarios, the agent may have access to a task descriptor that encodes how the current task relates to others terms of their compositional structures. This descriptor might be sufficient to combine modules into a solution (i.e.. zero-shot generalization), provided that the agent has learned to map the descriptors into solution structure. The descriptor could take different forms, such as multi-hot encoding of the various components, natural language, or highlighting the target objects in the input image. Our experiments study multi- hot descriptors as means to provide the compositional structure. Formally, we assume that the descriptor is given as an external input E T, and that there exists some function & : T x .M + II that can map this input and an optimal set of modules M into an optimal policy for the current task s(t, M) = (0) This would enable the agent to achieve compositional generalization: the ability to solve a new task entirely by reusing components learned from previous tasks.
Fast adaptation with restricted information In other scenarios, the agent is not given the luxury of information about the compositional structure. This is common in RL, where the only supervisory signal is typically the reward. In this case, the agent would be incapable of zero-shot transfer. Instead, we measure generalization as its ability to learn from limited experience a specifc function s(t)_ .M + IT that combines existing modules into the optimal policy for the current task s!9) (M) = (+)) Intuitively, if the agent has accumulated useful set of modules M, then we would expect it to be capable of quickly discovering how to combine and adapt them to solve new tasks.
4 EXAMPLE COMPOSITIONAL GENERALIZATION TASKS FOR RL
We now define two domains that intuitively exhibit the properties outlined in Section 3. More details, including the observation and action spaces and the reward, are provided in Appendix B.
Discrete 2-D world The agent's goal is to reach specific target in an environment populated with static objects that have different effects. The tasks are compositional three hierarchical levels: Agent dynamics: We artificially create four different dynamics, each corresponding a permuta- tion of the actions (e.g., the turn left action moves the agent forward, turn right rotates left). For each task, the effect of the agent's ' actions determined by the chosen dynamics. Static objects: We introduce a chain of static objects in each task, with a single gap cell. Wa 1 is block the agent's movement; the agent must open a door in the gap cell to move to the other side. F Floor cells have an object indicator, but have effect on the agent. Food gives small positive reward if picked up. Finally, 1 lava gives a small negative reward and terminates the episode. Target objects: There are four colors of targets; each task involves reaching a specific color.
There are 64 tasks of the form "reach the COLOR target with dynamics N interacting with OBJECT." If the agent has learned to "reach the red target with dynamics 0" and "reach the green target with dynamics 1", then it should be able to "reach the red target with dynamics 1" by recombining its knowledge. Tasks were simulated on gym-min grid (Chevalier- Sthevalier-Boisvertt et al., 2018).
These 2-D tasks capture the notion of functional composition we study in this work, and prove to be notoriously difficult for existing lifelong RL methods. This demonstrates both the difficulty of the problem of knowledge composition and the plausibility of neural composition solution. To
4

---

Published as a conference paper at ICLR 2022
show the real-world applicabilitty of the problem, propose second domain of different robot arms performing a variety of manipulation tasks that vary in three hierarchical levels.
Robot manipulation We consider four popular commercial robotic arms with seven degrees of freedom (7-DoF): Rethink Robotics' Sawyer, KUKA's IIWA, Kinova's Gen3, and Franka's s Panda.
Robot dynamics: These arms have varying kinematic configurations, joint limits, and torque lim- its, requiring specialized policies. All dynamics are simulated in robosuite t (Zhu et al., 2020). Objects: The robot must grasp and lift a can, a milk carton, a cereal box. or a loaf of bread. The varying sizes and shapes imply that common strategy
manipulate all these objects.
Obstacles: The workspace the robot must act in may be free (i.c., no obstacle), blocked by a wall the robot needs circumvent, or limited by door frame that the robot must traverse.
Each task is one of the 48 combinations of the above elements, just like in the 2-D case. Intuitively, the agent has learned to manipulate the milk carton with the IIWA arm and the cereal box with the Panda arm, then it could recombine knowledge to manipulate the milk carton with the Panda arm.
5 PROPOSED APPROACH FOR MODULAR LIFELONG RL
We now describe our framework for learning modular policies for compositional RL tasks. At high level, the agent constructs a different neural net policy for every task by selecting from a set of available modules. The modules themselves are used to accelerate the learning of each new task and are then automatically improved by the agent with new knowledge from this latest task.
5.1 NEURAL MODULAR POLICY ARCHITECTURE
We propose to handle modular RL problems via neural composition. Modular architectures have been used in the supervised setting to handle compositional problems (Andreas et al., 2016; Rosen- baum et al., 2018). In RI., few recent works have considered similar architectures, but without a substantial effort to study their applicability to truly compositional problems (Goyal et al., 2021; Yang et al., 2020). One notable exception proposed a specialized modular architecture handle multi-task, multi-robot problems (Devin et al., 2017). We take inspiration from this latter architec- ture design our own modular architecture for more general compositional RL.
Following the assumptions of Section 3, each neural module m, in our architecture is in charge of solving one specific subproblem F; (e.g., finding an object's grasping point in the robot tasks), such that there is one-to-one and onto mapping from subproblems to modules. All tasks that require solving F; will share m,- To construct the network for a task, modules are chained sequence, thereby replicating the graph structure depicted in Figure with neural modules.
typical modular architecture considers a pure chaining structure, in which the complete input is passed through sequence of modules. Each module is required to not only process the information needed to solve its subproblem (e.g., the obstacle in the robot examples), but also to pass through information required by subsequent modules. Additionally, the chaining structure induces brittle dependencies among the modules, such that changes to the first module have cascading effects. While in MTI. it is viable to learn such complex modules, in the lifelong setting the modules must generalize to unseen combinations with other modules after training on just few tasks in sequence. One solution is to let each module m, only receive information needed solve its subproblem F; such that it need only output the solution to F. Our architecture assumes that the state can factor into module-specific components, such that each subproblem F; requires only access subset of the state components and passes only the relevant subset to each module. For example, in the robotics domain, robot modules only receive as input the state components related the robot state. Equivalently, each element of the state vector is treated as a variable and only the variables necessary for solving each subproblem F; are fed into m,- This process requires only high-level information about the semantics of the state representation, similar to the architecture of Devin et al. (2017).
At each depth d in our modular net, the agent has access to ka modules to choose from. Each module is small neural network that takes as input the module-specific state component along with the output of the module at the previous depth d- 1. Note that this differs from gating networks in that the modular structure is fixed for each individual task instead of modulated by the state or input. The number of modular layers dmax is the number of subproblems that must be solved (c.g., dmax 3 for (1) grasping an object and (2) avoiding an obstacle with (3) robot arm).
5

---

Published as a conference paper at ICLR 2022
5.2 SEQUENTIAL MODULE LEARNING
Algorithm 1 Lifelong Compositional RL
T 0 loop
Our method for lifelong learning of modular policies accelerates the learning of new tasks by leveraging existing modules, while simulta- neously preventing the forgetting of knowledge stored in those modules. We achieve this by separating the learning into two main stages: an online stage in which the agent discovers which modules to use and explores the environment by modifying a copy of those modules, and an off-line stage in which the original modules are updated with data from the new task. The ratio- nale is that, in the early stages of training, there are two conflicting goals: 1) flexibly adapting the module parameters the current task to ex- plore how to solve it and 2) keeping the module parameters as stable as possible to retain per- formance on earlier tasks. Our method (Algo- rithm 1) consists of the following stages.
t getTask(); T+T+1
if T // Initialize modules
steps <- 0; 8d 4- 7 Vd
else // Find module combination
8, steps + discreteSearchh) bckModules + (lone
7(1, 00 + modullareett(M
// Online exploration
while steps < onlineSteps
8, a, 8' + rollouts( iterSteps)
bufferl//.puah(e.....8
x x0,, Q - RLStepRS a, 10,,,',,00000
steps t steps iterSteps
// Off-line comp. improvement
if T < numModules
M t bckModules
for = I,..., offlineSteps
Initialization Finding a good set of initial modules is major challenge. In order to achieve lifelong transfer to future tasks, we must start from a sufficiently strong and diverse set of modules. For this, we train each new task on disjoint sets of neural modules until all mod- ules have been initialized. Intuitively, this cor-
for = l,..., se....skeeaaaks
8, a, r' 4 bbfffer[/.samplee
π (f) + arg ming NLL(T((),
a' + arg ( (
targetQ + + Q (s',a')
Q(t) + arg min (2 targetQ)
responds to the assumption that the first tasks
presented to the agent made up of disjoint sets of components or subproblems, though we show empirically that this assumption is not necessary in practice.
Online training of new tasks In most RL tasks, during the initial stages of training, the agent struggles to find relevant knowledge about the task. Lifelong RL methods typically ignore this, and incorporate new (likely incorrect) information into the shared knowledge all throughout training. Instead, we keep shared modules fixed during this online training phase, subdivided into two stages:
Module selection We consider two versions of our method. In case, the choice of modules is given to the agent. In the other case, the agent must discover which modules are relevant for the current task. Since a diverse set of modules has already been initialized, this stage uses exhaustive search of the possible module combinations in terms of the reward they yield when combined.
Exploration Once the set of modules to use for the current task has been selected, the agent might be able to perform reasonably well on the task. However, especially on the earliest tasks, it is unlikely that modules that have never been combined will work together perfectly. Therefore, the agent might still need to explore the current task. In order to avoid catastrophic damage to existing neural components and at the same time enable full flexibility for exploring the current task, we execute this exploration via standard RI. training on copy of the shared modules.
The rationale for executing selection and exploration separately is that we want the selected modules to be updated as little as possible in the next and final stage. If we were to instead jointly explore via module adaptation and search over module configurations, we are likely to find solution that makes drastic modifications to the selected modules. If instead we restrict module selection to the fixed modules, then this stage is more likely to find solution that requires little module adaptation.
Off-line incorporation of new knowledge via batch RL While the exploration stage enables learning about the current task, this is typically insufficient for training a lifelong learner, since 1) we would need to store all copies of the modules order to perform well on new tasks, and 2) the modules obtained from initialization are often suboptimal, limiting their potential for future transfer. For this reason, once the agent has been given enough experience on the current task, newly discovered knowledge is incorporated into the original shared modules. It is crucial that this accommodation step does not discard knowledge from earlier tasks, which is not only necessary for solving those earlier tasks (thereby avoiding catastrophic forgetting) but possibly also for future

---

Published as a conference paper at ICLR 2022
tasks (i.e., forward transfer). Drawing from the lifelong learning literature, we use experience replay over the previous tasks to ensure knowledge retention. However, while experience replay has been tremendously successful in the supervised setting, its success in RL has been limited to very short sequences of tasks (Isele & Cosgun, 2018; Rolnick et al., 2019). In part, this limited success stems from the fact that typical RL methods fail in training from fully off-line data, since the mismatch between the off-line data distribution and the data distribution imposed by the updated policy tends to cause degrading performance. For this reason, we propose a novel method for experience replay based on recent batch RL techniques, designed precisely to avoid this issue. Concretely, at this stage the learner uses batch RL over the replayed data from all previous tasks as well as the current task, keeping the selection over components fixed and modifying only the shared modules. Note that this is a general solution that applies beyond compositional algorithms to any lifelong method that includes a stage of incorporating knowledge into shared repository after online exploration. Ap- pendix E validates that this drastically improves the performance of one non-componte con-compositional method.
We use proximal policy optimization (PPO) for online training and butch-comstrained Q learning (BCQ) for batch RL. BCQ simultaneously trains an actor т to mimic the behavior in the data and a critic Q to maximize the values of actions that are likely under the actor (Fujimoto et al., 2019b; In the lifelong setting, this latter constraint ensures that the Q-values are not eeereestmmtte for states and actions that are not observed during the online stage. Note that other batch RL methods could be used instead. Additional implementation details are included in Appendix C.
6 EXPERIMENTAL EVALUATION
Our first evaluation sought to understand the compositional properties of the tasks from Section 4 and the models learned on those tasks, yielding that learned components can be combined in novel ways to solve unseen tasks. This motivated our second evaluation, which explored the benefits of discovering these components in a lifelong setting, showing that our method accelerates the learning of future tasks and avoids forgetting. Evaluation details and ablative tests are in Appendices D and E, and source code is available at: github. com/ cwm/Lifelontee
6.1 ZERO- TRANSFER TO UNSEEN DISCRETE 2-D TASK COMBINATIONS VIA MTL
To assess the compositionality of the tasks we constructed, we provided the agent with fixed graph structure following the formalism of Section 3. This way, the agent knows a priori which modules to use for each task and need only learn the module parameters. The architecture uses four modules of each of three types, one for each task component (static object, target object, and agent dynamics). Each task policy contains one module of each type, chained as static object -> target object -> agent. Modules are convolutional nets whose inputs are the module-specift states and the outputs of the previous modules. Agent modules output both the action and the Q-values. We trained the agent in a multi-task fashion using PPO, collecting data from all tasks at each training step and computing the average gradient across tasks to update the parameters. Since each task uses a single module of each type, gradient updates only affect the relevant modules each task.
We trained the agent on various sets of discrete 2-D tasks and compared it against two baselines: training separate single- task (STL) agent on each task, and training a single
MTL Non-modular
STL
Non-madular
0.8
0.75
50
monolithic network across all tasks in the same multi- task fashion. To ensure a fair comparison, the mono- lithic MTL network re- ceived as input a multi- hot encoding of the compo- nents that constituted each task. Figure 2 (left) shows that our method is substan- tially faster than the base- lines by sharing relevant in- formation across tasks.
avg recurn
0.6
0.50
30
0.4
20
0.25
10
0.00
0.25
0.50
0.75
1.00
10
20
30
40
50
60
# 1M steps per task
tasks
Figure 2: Returns of STL (on 64 tasks) and MTL (on various tasks,
per the colorbar) on 2-D tasks. The modular architecture captures the relations across tasks, accelerating learning. Training on more tasks improves results (left). Generalization of pre-trained modules to unseen combinations vs. the number of training tasks (right). Modules can be combined in novel ways to achieve high performance without training. Shaded regions and error bars show std. errors across 6 seeds.
7

---

Published as a conference paper at ICLR 2022
These results suggest that our modular architecture accurately captures the relations across tasks in the form of modules. To verify this, we evaluated the performance of the agent on tasks it had not encountered during training. To construct the network for each task, we again provided the agent with the hard-coded graph structure, but kept the parameters of the modules fixed after MTT. training. Figure 2 (right) shows the high zero-shot performance our method achieves, revealing that the modules can be combined in novel ways to solve unseen tasks without any additional training.
6.2 LIFELONG DISCOVERY OF MODULES ON DISCRETE 2-D TASKS
The results of Figure 2 encourage us to envision a lifelong learning agent improving modules over a sequence of tasks and reusing those modules learn new tasks much more quickly. In this sec- tion, we study the ability of Algorithm I to achieve this kind of lifelong composition. We consider two instances of our approach: one in which the agent is given the hard-coded graph structures (Comp.+Struct. and one in which the agent is given no information about how modules are shared and must therefore discover these relations autonomously via discrete search Comm.+Seercc).. Once the structure is selected for one particular task, data for that task is collected via PPO training, start- ing from the parameters of the selected modules. Finally, this collected data is used for incorporating knowledge about the current task into the selected modules via batch RL., using data from the current task and all tasks that reuse any of those modules to avoid forgetting. To match our assumptions, unless otherwise stated the initial tasks presented the agent contain disjoint sets of components.
We compared against the following baselines: STL: P& C (Schwarz et al., 2018), a similar method that keeps shared parameters fixed in a first stage, and pushes new knowledge into the shared pa- rameters in second stage, but uses a monolithic network, making it much harder to find solution that works for all tasks; Online EWC (Schwarz al., 2018), which continually trains all tasks into a monolithic network with a quadratic penalty for deviating from earlier solutions; and CLEAR (Rol- nick et al., 2019), which uses replay over previous tasks' trajectories with importance sampling and behavior cloning, but trains a standard monolithic network in single stage. Lifelong baselines were provided with the ground-truth multi-hot indicator of the task components as part of the input.
Figure 3 (left) shows the average learning curves of the lifelong agents trained on all 64 possible 2-D tasks. Even though tasks are trained sequentially, we show the averaged curves to study the ability to accelerate learning: the fact that the curves for our compositional methods are above STL shows that they achieve forward transfer. Note that this acceleration occurs despite our methods using an order of magnitude fewer trainable parameters than STL (86, 350 vs. 1, 080, 320). Additionally, both our methods improve the modules over time, as demonstrated by the trend of increasing zero-shot performance as more tasks are seen (Figure 3, center). P&C also learns faster than STL, but we will see, P&C catastrophically forgets how to solve earlier tasks. Other lifelong baselines perform much worse than STL, since they are designed to keep the solutions to later tasks close to those of earlier tasks, which fails in compositional settings where optimal task policies vary drastically.
Comp.+Struce
PSC
EWC
Zero-shot
Off-ine
SearthNG
STL
CLEAR
Online
Final
0.6
0.75
avg return
return
0.4
0.5
0.50
uso) 1earch Comp -Starc?
0.25
avg
0,2
Comp.4 Search
0.0
Comp-+ +
Search-NC Comp.
Struct
STL
PSC
CLEAR
EWC
0.0
0.5
1.0
20
40
60
# 1M steps per task
seen tasks
Figure 3: Avg. returns of STL and lifelong agents 64 compositional 2-D discrete tasks. Our compositional methods accelerate the training w.r.t. STL, demonstrating forward transfer (left). As compositional methods train on more tasks, they improve modules, achieving higher zero-shot per- formance when combined in novel ways (center). P&C also achieves forward transfer. but it forgets how to solve earlier tasks, while our compositional methods retain Prrformance-Comee Struct. even achieves backward transfer (right). CSapc+ performs better than baselines that receive multi-hot task descriptors. "Zero-shot" for Searh+ (shaded) after discrete search, which does require data. Oin-line and Final performances for STL are omitted for clarity, since it is does not perform lifelong training. Shaded regions and error bars represent std. errors across seeds.
8

---

Published as a conference paper at ICLR 2022
In these 2-D tasks, lifelong learners should completely avoid forgetting, since there exist models (compositional and monolithic) that can solve all possible tasks (see Figure 2). Figure 3 (right) shows the average performance each task progresses through various stages: the beginning (Zero- shot) and end (Online) of online training, the consolidation of knowledge into the shared parameters (for Comp. and P&C only, Off-line), and the evaluation after all tasks had been trained (Final). Our methods are the only that achieve forward transfer without suffering from any forgetting. Moreover, Comp. -Struct. achieves backward transfer: improving the earlier tasks' performance after training on future tasks, as indicated by the increase in performance from the Off- Off-line to Final bars.
To study the flexibility of our method, we evaluated it on random sequence of tasks, without forcing the initial tasks to be composed of distinct components. Figure 3 shows that this lack of curriculum (Comp.+Search-. does not hinder the forward or backward transfer of our approach. In addition, Appendix E demonstrates that our method can learn well with different numbers of modules.
6.3 LIFELONG DISCOVERY OF MODULES ON REALISTIC ROBOTIC MANIPULATION TASKS
Having validated that our approach achieves forward transfer without forgetting on the 2-D tasks, we carried an equivalent evaluation the more complex and realistic robotic manipulation suite. The architecture was similarly constructed by chaining one module for each type of obstacle, object, and robot arm, and each such module was a multi-layer perceptron. We compare against P&C, the best lifelong baseline in the 2-D tasks, and STL. We empirically found that PPO would output overconfident actions in this continuous-action setting (i.e., high-magnitude outputs unaffected by the stochastic action sampling) when initialized from the existing modules directly, which limited our agent's ability to learn proficient policies. Therefore, we applied some modifications to the online PPO training mechanism which permitted successful training of all tasks at the cost of often inhibiting zero-shot transfer (see Appendix D). Figure (left) shows the learning curves of both our compositional methods. All lifelong agents learned noticeably faster than the base STL agent, and compositional methods were
fastest, despite using an order of magnitude fewer trainable
+Search Comp.+5truc
STL
Zero shot
Off-line
P&C
Online
Final
parameters than STL (165, 970
0.8
0.75
vs. 1,040, 304). Figure 4
avg success
0.6
0.50
(right) also shows that the off- line stage led to a decrease in performance. However, like in the 2-D domain, training on subsequent tasks led to back- ward transfer, partially recover- ing the performance of the ear- lier tasks as future tasks were learned. P&C was incapable of retaining knowledge of past tasks, leading to massive catas- trophic forgetting.
0.25
Daca used for diemete
0.00
5TL
P&C
Comp. Search
Camp. Struct
2 1M steps per task
Figure 4: Avg. success of STL and lifelong agents on 48 compo- sitional robot manipulation tasks. Compositional methods again achieve forward transfer (left). The off-line stage causes a drop in performance, but further training the modules on future tasks achieves backward transfer and partially recovers the lost perfor- mance (right). Shaded/error bars represent std. errors over 3 seeds.
7 CONCLUSIONS AND LIMITATIONS
We formulated the problem of lifelong compositional RL, presenting a graph formalism along with intuitive compositional domains. We developed an algorithm that learns neural compositional mod- els in lifelong setting, and demonstrated that is capable of leveraging accumulated components to more quickly Iearn new tasks without forgetting earlier tasks and while enabling backward transfer. As a core component, we use batch RL as a mechanism to avoid forgetting and show that this is a strong choice for avoiding forgetting more broadly in methods with multi-stage training processes.
One limitation of our work is the scalability with respect to the number of modules, requiring to attempt all possible combinations for the discrete search. While experiments showed that this is feasible even on relatively long sequences of 64 tasks, specialized heuristics to reduce the search space would be needed if the number of combinations were much larger.

---

Published as a conference paper at ICLR 2022
8 REPRODUCIBILITY STATEMENT
We have taken a number of steps to guarantee the reproducibility of our results. First, we pro- vide detailed textual descriptions of the compositional environments used for our evaluations in Appendix B and full details of our algorithmic implementation in Appendix C. Next, we de- scribe precisely the experimental setting of our evaluation, including model architectures, base- lines, hyper-parametery and computing resources in Appendix D. In addition to this, we publicly released the source code needed to reproduce our results at: github. cwwmciim///..... fe fenong Mendez20222oduu arLi i fel ongRI.
ACKNOWLEDGMENTS
We would like to thank Marcel Hussing for valuable discussions about robotics experiments. We also thank David Kent and the anonymous reviewers for their comprehensive and useful feedback. The research presented in this paper was partially supported by the DARPA Lifelong Learning Machines program under grant FA8750-188 17-20-18---117, the DARPA SAIL-ON program under contract PRO01120C0044 the DARPA ShELL program under agreement HR001 12190133, and the Army Research Office under MURI grant W91
REFERENCES
David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for life- long reinforcement learning. In Proceedings of the 35th International Conference on Machine Learming (ICML-18), pp. 10-19, 2018.
Alessandro Achille, Tom Eccles, Loic Matthey, Chris Burgess, Nicholas Watters, Alexander Lerch- ner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. In Advances in Neural Information Processing Systems 3/ (NearlPS-18), pp. 9873- 9883, 2018
Ferran Alet, Tomas Lozano-Perez, and Leslie P Kaelbling. Modular meta-learning. In Proceedings of the 2nd Conference on Robot Learning (CoRI.-19), pp. 856-868, 2018
Rahaf Aljundi, Min Lin, Baptiste Goujand, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances Neural Information Processing Systems 32 (NeurIPS- 19), 2019.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR- 16). pp. 39 499448.20 2016.
OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng. and Wojeiech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-21 20, 2020.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), pp. 1726-1734, 2017.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What required and can it learned? In 6th International Conference Learning Representations (ICLR-18), 2018.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Proceedings of the 35th International Conference on Machine Learning (ICML-18), pp. 501-510, 2018.
Michael Chang. Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically compos- ing representation transformations as a means for generalization. In 7th International Conference Learning Representations (ICI.R-19) 2019.
10

---

Published as a conference paper at ICLR 2022
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations (ICLR-19), 2019.
Maxime Chevalier- Chevalier-Boisvertt Lucas Willems, and Suman Pal. Minimalistic gridworld environment for OpenAl Gym. https: /ttts://gt /igghtu......... ccommmxx maximmmeeeeee wwwa//ime/////ammmmm seym-minigriae 2018.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in Neural Infor- mation P Processing Systems 6 (NeurIPS-93), pp. 271-278, 1993.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In Proceedings of the 2017 IEEE International Conference Robotics and Automation (ICRA-217), pp. 2169-2176, 2017.
Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decom- position. Journal of Artificial Intelligence Research (JAIR), 13:227-303, 2000.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution channels gradient descent in super neural networks. vrXiv:/701.0SS0 2017.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. Deep Reinforcement Learning Workshop, NeurIPS-19, 19,, 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning (ICML- 19), pp. 2052-2062, 2019b.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In 9th International Conference Learning Representations (ICLR-21), 2021.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA-17), pp. 3389-3396, 2017.
Tuomas Haarojj, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine Composable deep reinforcement learning for robotic manipulation. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA-18), pp. 6244-6251, 2018.
Drew Arad Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. 6th International Conference on Learning Representations (ICLR-18) 2018
David Isele and Akansel Cosgun, Selective experience replay for lifelong learning. In Proceedings of the Seiody-Second AAAI Conference on Artificial Intelligence (AAAI-18), pp. 3302-3309, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 1 33)1:3355213526 2017.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. In Advances in Neural Information Processing Systems 31 (NeurIPS-18), pp. 2408- 2418, 2018
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q- learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems 32 (NeurIPS- -19), 2019.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills recurrent networks. In Proceedings of the 35th International Conference of Machine Learning (ICMI.-18), pp. 2873-2882, 2018

---

Published as a conference paper at ICLR 2022
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. Proceedings of the 36th International Conference on Machine Learning (ICML-19), pp. 3652-3661, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. AXXh:22005001443 2020.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the 36th International Conference on Machine Learning (ICML-19), pp. 3925-3934, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(12);:935--2447 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30 (NeurIPS-17), pp- 6467-64766 2017.
Jorge Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. In Advances in Neural Information Processing Systems 33 (NeurIPS-20), pp. 14398-14409, 2020.
Jorge A Mendez and Eric Eaton. Lifelong learning of compositional Conference on Learning Representations (ICLR-21), 2021.
structures. structttres 9th International
Elliot Meyerson and Risto Mikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. In 6th International Conference on Learning Representations (ICLR- 18), 2018.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. Proceedings of the 37th International Conference on Machine Learning, pp. 6972-6986, 2020.
Geraud Nangue Tasse, Steven James, and Benjamin Rosman. A Boolean task algebra for rein- forcement learning. In Advances in Neural Information Processing Systems 33 (NeurIPS-20), pp- 9497-9507, 2020.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Tumnr. Variational continual learn- ing. In 6th International Conference on Learning Representations (ICLR-18), 2018.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Ran- dom path selection for incremental learning. Advances in Neural Information Processing Sys- tems 32 (NeurIPS-19), pp. 12669-12679, 2019.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems 32 (NeurIPS-19) pp. 7647-7657, 2019.
Scott Reed and Nando de Freitas. Neural programmen--interpreeee.. In 4th International Conference on Learning Representations (ICLR-16), 2016.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured Laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems 31 (NeurIPS-18), pp.
2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experi- replay for continual learning. In Advances in Neural Information Processing Systems 32 (NeurIPS-9,) 2019.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In 6th International Conference on Learning Representations (ICLR-18) 2018.
12

---

Published as a conference paper at ICLR 2022
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinst Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning. In Proceedings of the 35th International Conference Machine Learning (ICML- 18), pp. 4528-4537, 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):4844488 4894489944999 2016.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of Go without human knowledge. Nature, (35407766))34----1 2017.
Koustuv Sinha, Shagun Sodhani, Joelle Pineau, and William L Hamilton. Evaluating logical gener- alization in graph neural networks. arXiv:2003.06566 2020.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 112((12):181- 211, 1999.
Emanuel Todorov. Compositionality of optimal control laws. Advances in Neural Information Processing Systems 22 (NeurIPS-09), pp. 1856-1864, 2009.
Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning program synthesis. In Advances in Neural Information Processing Systems 31 (NerrIPS-18), pp. 8687-8698, 2018.
Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):1-14, 2020.
Benjamin Van Nickerk, Steven James, Adam Earle, and Benjamin Rosman. Composing value func- tions in reinforcement learning. In Proceedings of the 36th International Conference Machine Learning (ICML-19), pp. 6401-6409, 2019.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In Proceedings of the 345h International Conference on Machine Learning (ICML-17), pp. 3540- 3549, 2017.
Ruihan Yang, Huazhe Xu, YI WU, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In Advances in Neural Information Processing Systems 33 (NeurIPS-20), pp. 4767 466774477,22 2020.
JaeHong Yoon, Jeongtae Lee, Eunho Yang, and Sung Ju Hwang. Lifelong learning with dynamically expandable network. In 6th International Conference on Learning Representations (ICLR-18), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning (ICML-17), pp. 3987- 3995, 2017.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martin robosuite: A modular simu- lation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. /2293, 2020.
13

---

Published as a conference paper at ICLR 2022
Appendices to
Modular Lifelong Reinforcement Learning via Neural Composition
A CONNECTION BETWEEN FUNCTIONALLY COMPOSITIONAL RL AND HIERARCHICAL RL
In Section 2 of the main paper, we briefly discuss the connection between our proposed functionally compositional RL problem and the popular hierarchical RL setting. Given the close connections between the two, in this section we expand upon that discussion.
The primary conceptual difference between hierarchical RL and our proposed functionally compo- sitional RL problem is that hierarchical RL considers composition of sequences of actions in time, whereas we consider composition of functions that, when combined, form a full policy. In particular, for a given compositional task, all functions that make up its modular policy are utilized at every time step to determine the action to take (given the current state).
Going back to example of robot programming from Section I in the main paper, modules in our compositional RL formulation might correspond to sensor processing drivers, path planners, or robot motor drivers. In programming, at every time step, the sensory input is passed through modules in some pre- pre-prrgrmmeed sequential order, which finally outputs the motor torques to actuate the robot. Similarly, in compositional RL, the state observation is passed through the different modules, used in combination to execute the agent's actions.
Hierarchical RL takes complementary approach. Instead, each "module" (e.g.. option) is a self- contained policy that receives as input the state observation and outputs an action. Each of these options operates in the environment, for example reach particular sub-goal state. Upon termi- nation of an option, the agent selects different option to execute, starting from the state reached by the previous option. In contrast, our compositional RL framework assumes that single policy is used to solve complete task.
An integrated approach is possible that decomposes the problem along both a functional axis and temporal axis. This would enable selecting different functionally modular policy at different stages of solving task, simplifying the amount of information that each module should encode. Consequently, the framework we propose could be used to learn individual options, which would then composed sequentially. This would enable options to be made up of functionally modular components, simplifying the form of the options themselves and enabling reuse across options. Research in this direction could drastically improve the data efficiency of RL approaches.
B COMPOSITIONAL ENVIRONMENT DETAILS
In this section, we describe in more detail the environments we propose in Section in the main paper for evaluating functional composition in lifelong RL.
B. DISCRETE 2-D WORLD
Each task in the discrete 2-D world consists of an 8 x 8 grid of cells populated with a variety of objects, and is built upon commmminngriie Below, we describe the core elements of our proposed environment and how they vary according the task components.
Observation space The learner receives partially observable view of the 7 x 7 window in front of it, organized as a w x e image-like tensor, where h w 7 are the height and width of the agent's 's field of view, and c 7 is the number of channels. Each channel corresponds to one of: wall, floor, food, lava, door, target, and agent. The first four channels are binary images with ones at locations populated with the relevant objects. The door channel contains zero for any cell without door or with an open door, and a one for any cell with a closed door. The target channel has all-zeros except for locations in which a t target is located, which are populated with an integer indicator of the color between one and four. Finally, the agent channel has all-zeros except for the location of the agent, which is populated with an integer indicator of
14

---

Published as a conference paper at ICLR 2022
Go green target interacting Go to blue target interacting with food with agent with lava with agent
red target interacting to purple target interacting with wall with agent
with floor with agent
Figure B.I:
Visualization of various instantiations of our compositional discrete 2-D tasks. The highlighted area represents the agent's field of view.
the agent's 's orientation (right, bottom, left, up) between and four. The agent can observe past all objects except walls and closed doors, which occlude any objects beyond them.
Action space Every time step, the agent curn.right, move forward, pick.ob ject, drop.ob object, and open_door These dis- crete actions deterministie, that they always have the intended outcome. However, artifi- cially create distinct dynamics by permuting the ordering of the actions in the following four orders; for readability, we have grayed out actions whose effect stays constant across permutations:
execute one of six actions: turn left,
0. turn left, turn right, move forward, pick object, drop object, open door turn left, turn rig right, open door, pi object, drop ob move forward
2. turn_left, move.forwaree. turn_ri ght, pi pkoojjec,,, drop_ob ject, open_door
3. turn_left, wwve open_door, pi pick_object,
turn_right
Reward function We primarily rely on the original sparse reward function provided by gym sym-minigrid which gives zero at every time step except at the end of a successful episode. The terminal reward value is computed as 1 0.9( H), where the time step which the target is reached and H is the horizon of the environment. In tasks where food is present, the agent gets an additional reward of 0.05 for every piece of food picks up with the pick ob jejet action. In contrast, the agent receives penalty of -0.05 if it steps on lava object.
The 8 8 grid surrounded by a wall. The initial state of every episode is set by randomly sampling the locations of all objects in the scene. First, horizontal location is picked in range [2, w - 2]. where the static object is placed. All cells i, such that i = are populated with the task's static object, except for one individual cell at random vertical location y: i = x, y. Cell æ, y is left empty in all tasks except those whose static object is wall, which case closed door is placed. The agent is placed randomly in some location not occupied by the static object, and facing randomly in any the four possible directions. Finally, one target object of each of the four possible colors is randomly placed in any remaining free spaces in the environment.
Initial conditions
Episode termination For all tasks, the episode terminates upon reaching the correct target, or after H 64 time steps, whichever happens first. The episode immediately terminates if the agent steps on 1 ava object.
Figure B.1 shows example tasks created by sampling one component of each type.
B.2 ROBOTIC MANIPULATION TASKS
Each task in the robotic manipulation domain consists of single robot arm continuous state- action space, with single object and (optionally) obstacle. The dynamics of the task are simu- lated on robosu te, and all robots use general-purpose gripper by Rethink Roboties with two parallel fingers. We now provide details of underlying MDP of tasks within this domain, and how it varies according the task components.
Observation space Each time step, the agent is given rich observation that describes all elements in the task. The robot arm state is described by 32-dimensional vector, concatenating the sine and cosine of the joint positions, the joint velocities, the end-effector position, the end-effector orienta- tion unit quaternions, and the gripper fingers' positions and velocities. The target object's ''s state is described by 14-dimensional vector that concatenates the position and orientation of the object in
15

---

Published as a conference paper at ICLR 2022
Figure B.2: Visualization of various instantiations of our compositional robotic tasks.
global coordinates, and the position and orientation of the object relative to the end-effector. The ob- stacle is similarly described with its position and orientation in global and end-effector coordinates. The goal towards which the target is to be lifted is again described with its position and orientation in both coordinate frames, as well the relative position of the target object with respect the goal. Note that many of these elements are redundant and principle unnecessary for solving the task at hand. However, found that this combination of observations leads tasks that much more easily learned by the STL agent.
The agent's actions continuous-valued eight-dimensional vectors, indicating the change in each of the seven joint positions and the gripper. For reference, this corresponds to the JOINT_POS ITION controller in robosuite.
Action space
Reward function We design dense rewards for our environments similar to those used in rebosuite. a high level, the agent receives an increasingly large reward for approaching, grasping, and lifting the object. Additionally, in tasks involving the wa wall obstacle, we found it nec- lift the object past the wa Concretely, in tasks with no wall, the reward is given by the following piece-wise function:
essary to provide agent with additional reward to encourage
0.1 (anh(5d))
if not grasping
0.25 + 0.5(1 - t2nh()5h))
if grasping and not success if success
and in tasks with wa 1 1, the reward is given by:
0.05(1 - tanh 5d...)
if not past wall
0.05 + 0.05(1 tanh (5d)
if past wall and not grasping
0.25 0.5(1 tanh (25h))
if grasping and not success if success
where df is the distance from the gripper to the center of the object, h is the vertical distance from the object to the target height truncated and d the x, z distance from the gripper the wall.
Initial conditions The robot arm is placed at the center of the left edge of a flat table of width 0.39 and depth d 0.49. The obstacle placed one quarter of the way from left right in the table, and at the center. The object location is sampled uniformly at random in the right half of the table, so that the robot must always surpass the obstacle before reaching the object.
Episode termination
The episode terminates after reaching the horizon of H = 500 time steps.
Figure B.2 shows example tasks created by sampling one component of each type.
C ADDITIONAL ALGORITHMIS DETAILS
Our lifelong compositional RL agent faces tasks in sequence, and is always provided with (at least) a task index to indicate the current task being trained. Additionally, our Comp.+Struct. variant is also informed of the graph structure that relates the various tasks.
At a high level, our method uses discrete search during the module selection stage, then trains a copy of the module parameters online via PPO in the exploration stage, and subsequently uses small
16

---

Published as a conference paper at ICLR 2022
data set of the current and past tasks to train the actual module parameters via BCQ. Implementation details of each of these steps are provided below.
Online training of new tasks: Module selection Upon encountering new task, the agent selects the best modules to solve the task without any modifications to the module parameters. This is done to ensure that the choice of modules requires minimal modifications to those parameters, is needed to retain knowledge of the earliest tasks. In our experiments, we perform a discrete search over all possible combinations of parameters, which requires the least amount of additional assumptions. We roll out each of the resulting policies for ten episodes and pick the module combination that yields the highest average return across the ten episodes.
Online training of new tasks: Exploration It is unlikely that the chosen modules will imme- diately solve the task without modifications, especially for tasks encountered early in the agent's lifetime, when it has not yet learned fully general modules. In the supervised setting, the agent given data set that is representative of the data distribution of the current task, which enables the agent to incorporate knowledge into the modules directly using the provided data after module selection (Mendez Eaton, 2021). However, in RL there is no given data set, and the agent must instead explore the environment to collect data that represents near-optimal behavior. To do this, we execute PPO training. leveraging the selected modules to initialize the policy, but without mod- ifying the actual shared module parameters. BCQ in the next step requires training action value function Q, so for compatibility we modify PPO to compute the state value function (V) from the Q function that actually train. In the discrete setting, we assume deterministic actor and compute the maximum value from the computed Q values: V = max, (ss, a). In the continuous setting, instead sample n 10 actions and compute the average value across all of them obtain an approximation of V.
Off-line incorporation of new knowledge via batch RL One popular strategy in the supervised setting to incorporate new knowledge into shared parameters is to train parameters with a mix of new and replayed data. In RL, this type of off-line training via standard techniques often leads to degrading performance, due to the mismatch between the original data distribution and the data distribution imposed by the updated policy. Therefore, we use BCQ as a training mechanism that constrains changes to this distribution.
Discrete: In the discrete action setting. the algorithm trains an actor and a critic. The actor is trained to imitate the behavior distribution in the data, so we use the same network as used during PPO training. The critic is trained to compute the Q values, constrained to re- gions of the data distribution with high probability mass. Concretely, instead of computing ass + max, Q(s', a') as dictated by the standard Bellman equation, BCQ computes Q(s', ,a), where 7 is a threshold below which actions are considered
Q = maxa"=m(sa"") too unlikely under the data distribution. In our experiments, after this stage we evaluate the policy by rolling out actions via Boltzmann sampling of Q.
Continuous: The original continuous-action BCQ is substantially more complex, since the actor should mimic a general distribution over the continuous-valued actions, for which BCQ uses a variational autoencoder to represent the arbitrary distribution. However, in our case, we assume that a Gaussian policy can represent the data distribution, since the data was itself generated by Gaussian actor. Therefore, like in the discrete case, we train the PPO actor π to imitate the data distribution. This choice also enables us to drop the perturbation model trained in BCQ to train a separate actor. BCQ trains two separate Q functions manner similar to clipped double Q-learning. The target value is computed by
r ++ max min (=(1))
25(s', + (1 max
Q5(8' (-
a'>x(3)
which again estimate by sampling = 10 actions from the actor Subsequent evaluations in our experiments are executed by sampling actions from the actor x.
D EXPERIMENTAL SETTING
In this section, we provide additional details about our experimental setting, including a description of our baselines and the selected hyper-parametto Note that we our own implementations for all baselines, since there is no soun-sourced code available for them. Source code to reproduce our results can be found at: g hub. com/Li fe ang-MM./Meneeeee z 222dddul arLi fe ongRI.
17

---

Published as a conference paper at ICLR 2022
D.1 EVALUATION ON 2-D TASKS
sand or dgean
Model architecture To construct our architecture for the discrete 2-D tasks, we assume that the underly- ing graph structure first processes the static object, then the target object, and finally the agent dynamics. In- tuitively, the agent's actions require all information about target and static objects, and the agent module should be closest to the output since it di- rectly actuates the agent. The plan for reaching target object requires
Corcatwate
Terge mocile
modale
Figure D.3: Modular architecture for discrete 2-D tasks. information about how to interact with the static object. Interacting with the static object instead could be done without information about the target object.
Static object modules consume as input the five channels corresponding to static objects, and pass those through one convolutional block of c 8 channels, kernel size k ReLU activation, and max pooling of kernel size k = 2, and another block of c = 16 channels, kernel size k = 2, and ReLU activation. Subsequent modules, which require incorporating the outputs of the previous modules, include pre-processing network that transforms the module-specine state into a represen- tation compatible with the previous module's output, and then concatenate the two representations. The concatenated representation is then passed through a post-processing network to compute representation of the overall state up to that point. Target object modules pre-process the target object channel with a block with the same architecture as static object modules, concatenate the pre- processed state with the static object module's ' output, and pass this through a single convolutional block of 32 channels, kernel size k 2, and ReI.U activation. Similarly, agent modules pass the agent channel through a pre-processing net with the same architecture of the target object module (minus the concatenation with the static object module output), concatenate the pre-prooe pre-processed state with the output of the target object module, and pass this through separate multi-layer perceptrons for the actor and the critic with a single hidden layer of n 64 hidden units and tanh activation. The learner then constructs separate architecture for each task by combining one module of each type. The architecture shown in Figure D.3.
Baselines
We used the following baselines to validate performance.
STL trains a separate model for each task via PPO. Each such model follows the architecture described above, but uses a single module of each type. We chose use this architecture instead of standard architecture with the input passed entirely into the first module to ensure a fair comparison, yielded substantially better results in our initial evaluations.
P&C uses similar architecture to that of STL for its knowledge base (KB) and active columns, with additional lateral connections from the KB to the active column. To enable P&C to dis- tinguish among tasks, we add a multi-hot indicator of the task components as inputs to the first fully-connected layer. In the original P&C algorithm, the progress phase trains the active column online and the compress phase trains the KB column online as well, but imposing an EWC penalty on the shared parameters. We slightly modify this to make P&C closer to our algorithm: the com- press phase instead uses replayed data from the current task distill knowledge from the active column into the KB. By doing this only over the latest portion of the data, this closely matches the P&C formulation that generates data online with the distribution imposed by the (fixed) active policy. This also enables P&C to leverage the entirety of the online interactions for the progress phase, instead of trading off which portion to use for progressing and which portion to use for compressing. We use PPO to train the active column parameters during the progress phase.
EWC uses the same architecture as STL, but again with a multi-hot indicator to discern the dif- ferent tasks. Throughout its training. EWC modifies all parameters via PPO with an additional quadratic penalty that encourages parameters to stay close to the solutions of the earlier tasks.
CLEAR trains the same architecture training process. In order to prevent forgetting, for every sample collected online and used to compute the PPO loss, 17 samples are replayed from previous tasks to compute the custom CLEAR loss, which balances an importance sampling policy gradient objective (V-trace) and an imitation objective. We ensure replayed samples evenly split across all previously seen tasks.
EWC, and also modifies all parameters throughout the
18

---

Published as a conference paper at ICLR 2022
Table D.I: Summary of optimized hyper-parameters used by our method and the baselines.
Algorithm
Hyper-parameter
Discrete 2-D world
Robotic manipulation
# env. steps
IM
3.6.
# env. steps / update
4,096
8,000
learning rate minibatch size
le-3
1e-3
256
8,000
PPO
epochs per update
4
80
À (GAE)
0.95
0.97
Y (MDP)
0.99
0.995
entropy coefficient
0.5
Gaussian policy variance
fixed
# parameters
0.080,332 320
1, 0,040, 304
# modules per depth
4
# rollouts module comb.
10 100, 000
# replay samples/ / task
100, 000
Compositional
# BCQ epochs
10
100
# parameters
86, 350
165, 970 TO
X
10
7 (EWC)
P&C
# replay samples / task
100, 000
100, 000
# distillation epochs
100
# parameters
70, 282
104, 384
À
10, 000
EWC
7 (EWC)
# parameters
18, 928
CLEAR
"
# parameters
18, 928
via grid-search over the learning rate (from {1e-6, 3c-6, le-5, 3.-5, 1e-4, 3c-4, 1e-3, 3c-3, 1e-2, 3c-2)) and the number of environment interactions per training step (from {256, 512, 1024, 2048, 4096, 8192)). Since the static object affects the difficulty of the task, we sampled one task with each static object (wall, floor, food, and lava) for each hyper-parameter combination, and computed the average performance as the value to optimize. We reused the obtained PPO hyper- seeer--rrmmterss for all lifelong agents. For lifelong agents, tuned their main ppprr-prrameter by training P&C and EWC, we searched for the regularization À in {1e-3. 3, 1e 2, 1e 1e0, lel, 1e2, 1e3, 1e4). For CLEAR, searched for the replay ratio 1j in {0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2). Table D.1 summarizes the obtained hyper-parameters.
Hyper-parameters We tuned the STL
five tasks over five random seeds. For
Computing resources Our discrete 2-D experiments machines, each with two GeForce GTX 1080 Ti GPUs, eight-core Intel Core i7-7700K CPUs, each machine, our compositional
carried out two small development
and 64GB of RAM. When running two parallel experiments method took an average of approximately one day of wall-clock time to train on all 64 tasks, includ- ing a full evaluation of all previously seen tasks after training on each task.
Concatenate
D.2 EVALUATION ON ROBOT MANIPULATION
Concatenate
Model architecture Following the success- ful results on the discrete 2-D domain, which demonstrated that our chosen architecture cor- rectly captured the underlying structure of the tasks, we manually designed a graph structure for the robotics domain that closely matches that of the discrete 2-D setting. In particular, we assume that the obstacle is processed first, the object next, and the robot last. The obstacle module passes the obstacle state through a single hid- den layer of 32 hidden units with tanh activation. The object module pre- rrepprcessees the object state with another tanh layer of n 32 nodes, concatenates the pre- proceeeeeeee state with the output of the obstacle module, and passes this through another tanh layer with 32 units. Finally, the
State
Robot
module
module
Figure D.4: Modular architecture robot tasks.
19

---

Published as a conference paper at ICLR 2022
robot module takes the robot and goal states as input, processes those with two hidden tanh layers of n = 64 hidden units each, concatenates this with the output of the object module, and passes the output through a linear output layer. Following standard practice for continuous control with PPO, we use completely separate networks for the actor and the critic (instead of sharing the early layers). However, we do enforce that the graph structure is the same for both networks. Moreover, for the critic, the action is fed as an input to the robot module along with the robot and goal states. The architecture shown in Figure D.4.
We did no formal hyper-parameter tuning for this setting. Instead, we created the tasks ensuring that STL performed well. Once the tasks were fixed, we perturbed key PPO hyper- parameters (e.g., the learning rate) and verified that the changes led to decreased performance. We maintained the PPO hyper-parameters fixed for lifelong training. We did tune the regularization coefficient/ À for P&C by training on five tasks over three random seeds, varying À in (1e-3, le- 2, le-1, 1e0, le1, le2, 1e3, le4). Table D.1 summarizes the syper-parametert used for our robotic manipulation experiments.
Plyper-parmmete
Modifications to the exploration stage We found empirically that the lifelong compositional agent would often suffer from premature convergence in robotics experiments, ceasing to explore the space early on in the training. Consequently, we incorporated three modifications to the base to downscale the output layers of the policy and critic net- works by a factor of 0.01 whenever the initial policy achieved no success; this ensured that, if the policy was not close solving the task, the agent would not be following highly (and incorrectly) specialized policy, while still leveraging the compositional representations at lower layers. The sec- ond modification was apply tanh activation the last layer of the policy network to limit the actions
PPO algorithm. The first change
magnitude of the outputs, which ensured that stochastic sampling was effective-othrree would be clipped by the robot simulator and the agent would act deterministically regardless of the variance of the policy. The final adaptation was to fix the variance of the policy to a constant value of n² = 1 (log(o) = 0) for the seven joint actions and o² 1/e (log(a) = -055) for the gripper action, since we found that entropy regularization was leading the agent to maximize the variance of irrelevant joint actions while making relevant joints nearly deterministic.
Computing resources Experiments robotic manipulation tasks were more computationally in- tensive, due to the cost of physics simulation. We ran these evaluations on an internal cluster, requiring 40 cores and 80GB of RAM per experiment (without GPUs). Our compositional method required approximately two days of wall-clock time to train on all 48 tasks, again including a full evaluation of all past tasks after completing the training of each new task.
E ADDITIONAL EXPERIMENTAL RESULTS
One natural question that arises when evaluating methods with various algorithmic and architectural building blocks is, which of these constituent parts are crucial for the obtained performance? In this section, we empirically validate our design choices.
We first verify that the modules required to solve the tasks we design diverse. Results from Section 6.1 in the main paper show that the discrete 2-D tasks are truly compositional: if the modules are learned well, they can be recombined and reused to solve unseen tasks. However, it is possible that some of these components are essentially the same. For example, perhaps the module for learning to reach the green target could be replaced with the module to reach the red target. This would severely limit the usefulness of our evaluations. As sanity check, we evaluate the effect of using the incorrect module for evaluation on task, separated by type of module. Figure E.5- reveals that using the incorrect static object modules leads to a small (but noticeable) drop in performance, while using the incorrect target object or agent module leads to a drastic drop to nearly random performance. This validates that the modules differ substantially.
We continue by analyzing our architecture choice. A more natural choice of architecture, which we considered early in our development cycle, is a simple module chaining, where the input is passed entirely through first module, whose output is passed to the next module, and so on. This is in con- trast to architecture, where the input is decomposed into task components and passed separately to distinct modules. We repeated the experiment of Section 6.1 in the main paper with a purely chained architecture and show the results in Figure E.5-b. We found that this chained architecture
20

---

Published as a conference paper at ICLR 2022
0.8
0.0
0.75
0.75
avg retur
0.6
0.50
0.4
0.25
0.2
0.2
0.25
0.00
STL
CLEAR+ Comp.
REC+8BL
0.0
50
0.0
0.5
1.0
Comp
Strect
Replay
1M steps per task
No chargg
Thrpet Apant
MTL
MTL Chain
CLEiB-Corp
Zem- thzt
on Rral
Stalk
STL
PSc NcclBr
On Br
(a) Module diversity
(b) Module generalization
(c) Ablation curves
(d) Ablation barcharts
Figure E.5: Ablative analyses on discrete 2-D tasks. (a) Performance of modular MTL agent when constructing the policy with incorrect modules, demonstrating that modules are specialized to solve their assigned subproblem. (b) Generalization to unseen combinations with our proposed modu- lar architecture (MTL) and with a standard chained modular architecture (MTL Chain), showing that modules require far less data to generalize if they trained on decomposed state representa- tions. (c-d) Performance of our modular architecture trained via CLEAR is much lower than with our method, and our proposed batch RL mechanism avoid forgetting drastically improves the performance of P&C. Shaded regions and error bars denote standard errors over six random seeds.
cannot generalize nearly as quickly as our modified architecture. This is intuitively reasonable. Consider for example the first module, which is in charge of static object detection. In the chained architecture, this module is further in charge of passing information to subsequent modules about all remaining task components, whereas our architecture need only focus each module on the relevant component, without distractor features from other task components. One alternative view of the same problem is that, to achieve zero-shot generalization, the output of all modules at one depth needs to be compatible with all modules at the next depth. This requires that the output spaces of all modules be compatible. One way to encourage this compatibility is to restrict the inputs to the modules to only the relevant task information, as we do.
To test how our method per- forms if the architecture uses more or fewer modules of each type than task compo- nents, we repeated the ex- periments on the 2-D do- main with varying numbers of neural components, using completely random task se- quences (i.c., no curriculum). Figure E.6 shows that our method is remarkably insen- sitive to the number of mod- ules, performing well with range of choices.
modules modules
modules 6 modules
Zero-shot
Orf-lina
Online
0.8
0.75
avg retum
0.50
0.4
0.25
0.2
0.00
3 modules
modulest
5 modules
modules
STL
0.0
0.5
1.0
1M steps per task
Figure E.6: Avg. returns of Comp. Search-NC with varying num- ber of modules on 2-D discrete tasks. The number of modules has only minor effects on the overall performance of our approach. Shaded regions and error bars represent std. errors across 6 seeds. *4 modules is the original (correct) value from Section 6.2.
So far, we have shown that modular architectures enable improved performance in the compositional tasks we consider. Since all the lifelong basclines we consider use monolithic architectures, one could think that the improved performance of our method might come solely from the use of a better architecture. To verify that this is not the case, trained modular architecture with CLEAR. As shown in Figure E.5-c, while the performance CLEAR indeed improves, it falls substantially short of matching the performance of our approach. Since CLEAR uses an objective that closely mimics batch RL, but does so online during training of new tasks, this highlights that the advantage of our method comes primarily from the separation of the learning process into multiple stages.
Another major contribution of our work is the use of batch RL as a means for avoiding catastrophic forgetting. To analyze the effect of this choice, we train a version of P&C that replaces EWC in its compress stage with batch RI. Notably, shown in Figure E.5-d, we found that this almost entirely suppressed the effect of forgetting. Moreover, this led to even-better forward transfer of P&C. This result stresses the fact that avoiding forgetting is not only required for retaining performance of earlier tasks, but also for accumulating knowledge that better transfers to future tasks.
21

---

Published as a conference paper at ICLR 2022
Comp.+
P&C+BBL Replay
Zero-shot
Off-line
STL
Online
We repeated the batch RL abla- tive test on our robotic manip- ulation tasks, yielding that the
0.75
succes
0.6
0.50
training P&C is accelerataed beyond that of STL and batch RL avoids forgetting, However, in this harder setting, the mono- lithic structure is insufficient to express policies for all tasks and therefore Off-line and Final performance are substantially degraded (see Figure F.7).
0.4
0.25
0.2
0.00
Comp + Struct
STL
P&C+BRL Replay
0.0
1M steps per task
Figure E.7: P&C batch RL avoids forgetting, but cannot fully
incorporate new knowledge due to the monolithic structure.
22


Reviews:
[{"review_id": "z249oWnOlla", "reviewer": "Reviewer_77pf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found this to be an interesting paper. The idea of functional compositionality is rather intriguing, and its application to RL and continual learning is original. The authors have motivated why a modular policy should improve continual learning on tasks with compositional structure, and they have validated their approach with a series of compositional tasks, showing that their approach outperforms several alternative approaches to continual learning on a variety of metrics. In particular, they show empirically that their approach demonstrates better forward transfer (with both better zero-shot performance and faster learning on new tasks) and that their approach not only avoids catastrophic forgetting, but that it can actually demonstrate a degree of backward transfer (with training on new tasks improving performance on old tasks).\n\nMethodologically, the primary limitation of their approach is that the computational graph linking modules together needs to be specified in advance. For each graph node, a set of possible modules is defined in advance (although the module parameters still need to be learned), and the agent must determine which of the possibilities to insert at each node. Moreover, as the authors have pointed out, the complexity of finding the right combination of modules can grow combinatorially with the number of possible modules, making it difficult to scale this to a larger number of modules. Another limitation is that the overall framework must be divided into distinct training phases, with an initialisation phase that must give the modules some initial training before they can be deployed in the computational graph. A final limitation is that while the framework shows good performance on compositional gridworld tasks, the empirical performance advantage seem weaker on higher dimensional robotic manipulation tasks. However, in spite of these limitations, the approach is novel, and these are issues that further development of this approach would need to address.\n\nFor me, the primary weakness with this paper is that the exposition of functional compositionality in Section 3 could be clearer, and improving this would help the paper reach a wider audience. The framework and how it was applied to RL only became clearer after reading the subsequent sections, particularly Section 5.1 and the algorithmic details in the Supplement. Providing concrete examples may help ground what was otherwise a rather abstract discussion. For instance, it was initially unclear what the module were doing (as I now understand it, they are just stages in a multi-stage hierarchical processing ), and defining them as solutions to abstract subproblems and as functions that map undefined inputs **X** to undefined outputs **Y** did not help. Part of the issue was that I initially understood the modules in set M as being insertable at any node in the computational graph, which they are not. Moreover, it was not clear in this section that the computational graph is pre-defined, as I initially thought the agent could flexibly compose the modules together to form any graph, and part of the problem it needed to solve was to find the correct graph. Relatedly, Algorithm 1 includes many variables and functions whose meaning did not become clear until a second reading of the paper -- for instance, how the discreteSearch() worked, how the modules are composed together, and how the algorithm relates to the graph in Fig 1. Again, I had initially understood the discreteSearch() to be discovering the graph in addition to the modules that should go in the graph node.\n\nThere is one part of the algorithm that remains unclear to me. Towards the end of paragraph 1 in Section 6.2, the authors mention batch RL as using \"data from the current task and all tasks that reuse those modules to avoid forgetting\". By this statement, I read two possibilities: (a) the algorithm considers all tasks that share some (but not all) modules with the current task or (b) the algorithm considers all tasks that use the exact same set of modules as the current task. It'd be helpful if the authors could clarify what is meant here. Additionally, if (a) is meant, then how are the gradients propagated so that catastrophic interference does not occur in the modules that are not shared with the current task. And if (b) is meant, then how are these tasks considered “different” from the current task given that they are using the exact same policy network.\n\nThe authors also claim the use of batch RL to overcome catastrophic forgetting as one of their contributions. Yet it is unclear to me how this is novel. Rehearsal methods have previously been used to overcome catastrophic forgetting in lifelong learning, and in the RL context specifically, Rolnick et al (2019) specifically used replay techniques to overcome forgetting.\n\nFinally, it may be worth noting that besides functional compositionality and temporal hRL compositionality, there has also been work, at least in the cognitive sciences, in getting agents to learn compositional internal models of tasks for the purposes of model-based planning (see e.g. Franklin & Frank, 2018 - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006116). Indeed, the authors' compositional gridworld task is reminiscent of the compositional task in this paper, which is in turn based off of the compositional \"grid-sailing\" task in Fermin et al, 2010 (https://www.tandfonline.com/doi/full/10.1080/00222895.2010.526467).", "summary_of_the_paper": "This paper introduces and explores the use of *functional compositionality* in lifelong reinforcement learning (RL). In contrast to hierarchical reinforcement, which explores temporal compositionality (the chaining of options or subpolicies across time), functional compositionality involves assembling a novel, overall function (in the case of RL, the policy function) by composing together neural modules that perform multi-stage processing of the policy's inputs. Each module can be regarded as a small function that takes some abstract input X and maps it to an abstract output Y -- the initial set of modules will take the state space as the input X, while the final set will take the actions as the output Y. Modules can perform their computations in parallel, with their outputs concatenated together afterwards, or they can be chained together, with the output of one set of modules serving as the input of the next set. In this setup, the continual learning problem consists of two phases: first, the correct set of modules for the task must be selected and composed into the policy; then, the parameters of those neural network modules must be updated from the data generated by the agent. Selecting the incorrect set of modules will lead to poor transfer and catastrophic forgetting (as the parameters of the incorrect neural network modules get overwritten). This modularity can help improve robustness by reducing dependencies between modules -- this is achieved both through parallelism and through the need of downstream modules to generalise to novel combinations of upstream modules. The authors validate their approaches on a set of tasks with compositional structure, which a functionally compositional policy would be able to exploit, as it could substitute out modules according to the components of the current task. They compare their framework against alternative continual learning baselines and show that their framework is able to (i) avoid catastrophic forgetting; (ii) demonstrate better zero-shot transfer to new tasks; and (iii) even exhibit better backward transfer on training tasks.", "summary_of_the_review": "Overall, I would recommend this paper for acceptance. The application of functional compositionality to RL policies is quite original, and the approach has been validated empirically. But the paper would benefit significantly from greater clarity in its exposition of functional compositionality and in better defining certain variables and functions in Algorithm 1.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "sf2yNlyqTz", "reviewer": "Reviewer_HvzU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I very much agree with the authors' goals: lifelong reinforcement learning is an extremely interesting and underexplored area. I also agree that compositional modularity is an extremely promising direction that merits further study by itself. Unfortunately, the paper suffers from several problems in its current version. However, don not think that these problems are insurmountable: the authors should be able to submit an updated version addressing my main issues in time.\n\n#### Strengths\n- The topic and chosen solution are each highly important research directions with enormous room for future research.\n- The paper is extremely well written. The language is fluent and the overall structure of the paper is solid.\n- The algorithm introduced is a well-engineered solution to the outlined problem\n\n#### Weaknesses\n- (This is my main point of concern. If it is addressed to my satisfaction, I will increase my score to weak accept.) While all explanations and descriptions throughout the paper are easily understood, they severely lack in detail that is required to put many statements into context. And while most of this detail can be found in the appendix, I cannot recommend a paper where the appendix is essential to understanding core parts of the main paper for acceptance.\n- It is always a delicate balance to jointly introduce a new domain and a new approach, as this can quickly create the impression that a paper co-developed an approach and a domain the approach is bound to excel at, diminishing the appeal of both. Unfortunately, this is the case for the current paper: the particular input permutations (changing the color of the goal) and action-space permutations are an extremely good fit to the algorithm and it's targeted modular architecture. One example is that the number of permutations is aligned with the number of modules.\n- Consequently, I find the domain a little too simple. Each tasks will decompose perfectly in representation and action space, even for the robotics domain. This is an unrealistic assumption. An additional experiment on something with a less well-defined structure would have been very interesting. This would have allowed to also investigate other interesting properties, like learning more complex rules that could e.g. involve questions of recursivity or similar.\n- Throughout the paper, it is implied that composition is a sequential process. This is a useful simplification as it allows routing-like stacking of neural layers. However, I would strongly argue that it only a simplification: true modular compositionality should allow for arbitrary connections (imagine e.g., two perception skills that are required for logical inference, or recursive rules).\n\n#### Detailed remarks\n- p3, l8: I don't understand what \"a two-layer abstraction\" should mean\n- p4, top: Mapping the policy onto a graph of sub-skills seems off, as a graph does not allow for parallelisms that may be required for more complex tasks\n- section 5: here is where the bulk of details are missing:\n  - section 5.1: \"each neural module mi in our architecture is in charge of solving one specific subproblem F\" what does that mean? that there are as many modules as subproblems?\n  - section 5.1: \"A better solution is for each module to receive as input only the information relevant to it, such that its output need not characterize any additional information.\" I have no idea what this means\n  - section 5.1: \"our architecture assumes that the state is composed of module-specific components\" too vague\n  - section 5.1: \"At each depth d in our modular net\" what is dmax in general? How is dmax determined?\n  - section 5.2: \"while simultaneously preventing the forgetting of knowledge stored in those modules. We achieve this by separating the learning into multiple stages.\" way too little detail. I do not understand this either\n  - section 5.2: \"For this, we train each new task on disjoint sets of neural modules until all modules have been initialized.\" does this mean that the subtasks are also disjoint?\n  - section 5.2: \"incorporate new (likely incorrect) information\" the information may be superfluous or distracting, but it's definitely not \"incorrect\"\n  - section 5.2: \"Since a diverse set of modules has already been initialized, we do this via discrete optimization of the reward with respect to the possible combinations of modules.\" what does this mean? what is discrete optimization?\n  - section 5.2: \"However, while experience replay has been tremendously successful in the supervised setting, its success in RL has been limited to very short sequences of tasks.\" please elaborate\n  - section 5.2: A module is copied before it is updated to prevent catastrophic forgetting. I did not find anywhere how the updates are later re-introduced into the architecture.\n- section 6: I argue that at least a rough explanation of the architecture is essential when introducing results. I want to know: what are the modules? how many are there? how were they chosen? how many parameters do the different approaches have (the existing pairwise comparisons are insufficient). I especially miss an explanation of how the different modules are partitioned and why they were designed that way?", "summary_of_the_paper": "The paper introduces an algorithm for lifelong reinforcement learning using functional neural composition. The algorithm first maps a new problem onto a composition of previously acquired modules; then, the agent trains/finetunes the selected module combination on the new task; and finally, the agent incorporates this newly acquired information into the existing modules. This algorithm is then evaluated on two domains, one multi-task lifelong gridworld domain and a multi-task lifelong robotics domain. The algorithm produces superior results to several other lifelong learning approaches.", "summary_of_the_review": "A well chosen topic, with a highly interesting solution.\n\nStrengths:\n- good research direction\n- paper is extremely well written\n- the core of the paper, the introduced algorithm, is a well-engineered solution to the problem\n\nWeaknesses:\n- all details relevant to contextualize both the approach and the results are in the appendix\n- the paper jointly introduces a domain and an approach, opening it to the critic that the experiments are hand-crafted towards the approach\n- the experiments seem too simple\n- I disagree with some assumptions behind compositionality", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mp1Fh65D9YO", "reviewer": "Reviewer_wUbR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality, Clarity, Soundness, Correctness**\n\nThe paper is well written, the problem is well formulated and introduced, the experiments are clear, and the proposed approach is simple but sound. The benefits of the modular architecture over the non-modular architecture are clear though perhaps less pronounced than expected. Experiments are repeated multiple times to produce results of statistical significance, and the appendix gives all the hyper-parameters and architecture details to reproduce the experiments. I appreciate that the paper clearly points out the extra amounts of data that go into the module-search procedure, which helps with comparison against the other methods. To the best of my knowledge, the claims made in the paper are supported by the empirical results shown. \n\nMy main criticism is that the paper stops short of tackling the most important problems in lifelong compositional RL, which are: (i) designing the architecture when the correct number of modules is unknown (and the depth, i.e. the number of layers of modules), (ii) training when the correct sequence of modules is unknown and brute-force search is too costly, (iii) training without disjoint initial training tasks. I think each of these problems is hard, and fully solving only one of them would merit at least another publication. But I am afraid that the current paper implements the “baselines” that one would like to see in any attempt to solve said problems, but makes no attempt to do the latter. At least some solution attempts could be taken e.g. from the supervised literature, e.g. Chang 2019 who train a module-selection policy via RL. Another strong simplifying assumption is that the state can be separated (a priori) into relevant sub-parts for each module such that only relevant information is fed to each module - this is perhaps the fourth (iv) important and difficult problem in training compositional architectures.\n\n\n**Improvements**\n\n1) The main improvement that I would like to see is an attempt to train the architecture without knowledge of the correct module-sequence and without disjoint initial training tasks (the latter might turn out to be not possible, which would also be an interesting finding). This attempt would probably consist of several sub-experiments including ablations and controls (e.g. training with too many modules, etc.). I understand that this is mostly beyond of what’s possible during the rebuttal phase, yet I think this would be by far the biggest improvement of the current manuscript.\n\n2) While the plots in Fig 2 (left), Fig 3 (left) and Fig 4 (left) nicely visually summarize the performance, they do not provide information about “how long” it takes for the modular MTL to “overtake” STL. Does MTL need to see many tasks where it gets gradually faster and faster to reach high performance, or is there an initial phase of little improvement followed by a “take-off” where all subsequent tasks are learned rapidly (does that occur after each module has been trained once)? Showing the mean only hides all of these dynamics and it would be nice to see this in (a series?) of appropriate plots. It would also help answer questions like: is modular MTL worth the overhead if it’s only 32 or 16 tasks instead of 64, etc.\n\n3) Figure 2 (right) - the performance gains of modular over monolithic MTL are smaller than intuitively expected. One reason could be that the monolithic MTL successfully makes use of the correct module-sequence which is fed to the monolithic architecture as well. To test for this it would be nice to see performance of both modular and monolithic MTL when feeding in random information for the module-sequence to use. Prediction: modular MTL without search should catastrophically deteriorate monolithic MTL should deteriorate if it uses this information but should remain the same if the information is not used.\n\n**Comments**\n\n4) Another control experiment: “knocking out” individual modules (e.g. reinitializing one module randomly) for the modular MTL with search should lead to deterioration of the corresponding tasks and only these tasks. I have no strong reasons to believe otherwise, yet it would be an interesting control experiment to see whether the modules have really fully specialized, or whether they become responsible for multiple task variations when training with search. \n", "summary_of_the_paper": "**Update after authors' response:** After reading the other reviews, the updated manuscript and the authors response I think the authors have made several small improvements based on the reviewers' suggestions. Typically I would now be leaning towards a score of 7. However, this year's conference only allows for either a 6 or an 8. To me personally the paper is still slightly closer to 6 than 8, but to make a clear statement in favor of acceptance and facilitate the reviewers' discussion I have raised my score to an 8. I will clearly state my opinion (and hesitation to strongly endorse acceptance of the paper) during the reviewers' discussion.\n\n**Summary**: The paper proposes a novel method for lifelong compositional RL. The latter is formally introduced in the paper as lifelong RL (learning to solve a continuous stream of tasks without re-visiting previously seen tasks) with families of tasks that have a known compositional structure. The paper proposes to use a modular neural architecture (consisting of layers of modules) to capture the compositional nature of the tasks to learn. Through an appropriate module-selection process the (topologically disjoint) modules become functionally disjoint. This is shown to facilitate forward-transfer (faster learning of novel task variations that follow the compositional structure). The paper investigates two module-selection mechanisms: a fixed mechanism where the correct sequence of modules for each task is known, and a brute-force combinatorial search over all module combination to find the highest scoring combination. Once modules are selected, the selected module-parameters are trained to increase current task-performance. This is then consolidated with previous task experience (in an off-line RL fashion) via batch-RL to avoid catastrophic forgetting. Experiments are shown on a set of 2D grid-world tasks, and a number of simulated robot arms with different sensory setups, and the method performs well against a number of baseline methods.\n\n**Main contributions**\n1) Definition of compositional families of RL tasks. As the paper correctly points out, a number of previous work on compositional architectures has used variations of off-the-shelf (lifelong) learning tasks, which are not explicitly compositional and a beneficial composition of modules is often not known intuitively (often not even the number of modules is intuitively clear). The two task-families introduced overcome these issues and are a nice contribution towards more meaningful experimentation. The gridworld task is conceptually easy and does not have very complicated perceptual problem or high-dimensional continuous action-spaces; the robotic-tasks on the other hand is closer to real-world applications (and the corresponding complexities). Significance: I think the tasks are well chosen and might become standard-tasks for lifelong compositional RL.\n\n2) Proposal of a compositional architecture and training procedure. Given that the compositional structure of the tasks is precisely known, it is relatively straightforward to propose an exactly matching architecture. The main innovation is a training procedure to train such an architecture. Unfortunately the paper proposes two fairly straightforward solutions that need strong simplifying assumptions: one, the correct sequence of modules is known and applied accordingly, or two, brute-force search over all module combinations (which becomes exponentially costly with increasing numbers of modules, and requires training-data, and relies on disjoint sets of tasks initially until all modules have been trained at least on one task). Significance: the two module-selection mechanisms serve as important controls and lead to promising results. But they are also fairly straightforward conceptually, and rely on often unrealistically strong assumptions. What I would have liked to see for more impactful results is an attempt to train the architecture from scratch without brute-force search and disjoint tasks initially - one possible attempt would be to use RL to train a module selection policy, similarly to Chang 2019, which is used in the paper as a main inspiration for defining compositional RL tasks.\n\n3) Definition of lifelong compositional RL and using techniques from offline RL to avoid catastrophic forgetting. Significance: these are sensible choices, and thus valid contributions, but both have been conceptually proposed before in slightly different settings (Chang 2019 defines compositional supervised problems and using offline RL to avoid catastrophic forgetting and even have backwards-transfer is also not very far-fetched).\n", "summary_of_the_review": "The paper aims at addressing a very timely and important, and notoriously difficult problem: lifelong RL with task-families that have compositional structure. In principle, learning about this compositional structure should greatly facilitate learning-speed for new task-variations (forward-transfer). There are (at least) four main problems to solve: (i) how many modules are needed and what is the required computational depth (i.e. maximum number of sequential module calls), (ii) how are modules selected for a particular training instance (when this is not clear a-priori and brute-force search across module-combinations is too costly), (iii) how are modules trained initially after random initialization (how do they acquire specialization), (iv) how can it be ensured that modules only process relevant information (and thus become invariant to irrelevant variation). The fifth problem, avoiding catastrophic forgetting, is actually solved in the paper by borrowing a technique from batch RL. All of these problems are hard and active areas of research. Admittedly the answer to (iii) might be a curriculum, similar to what's proposed in the paper.\n\nThe current paper makes strong simplifying assumptions to reduce the severity of each problem. While this is a sensible starting point to establish baseline results, what I’m missing from the current manuscript is an attempt to tackle (some of) these problems. I am not saying this is trivial, and I would consider it a significant addition to the paper. Without such an addition the paper shows promising and interesting results for situations where these problems do not appear (which is often unrealistic) or for situations where these problems could be solved through some means in the future. The paper is well written, and clear, results are good, and I have no complaints regarding the correctness of results and claims. However, given the strong simplifying assumptions the results are not very surprising, which ultimately limits the impact and significance of the paper. I currently see no reason to reject the paper, but also have a hard time assigning a high score. I think with a bit of extra work (admittedly beyond what’s possible in the rebuttal) the paper could become a landmark piece of work in lifelong compositional RL.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "atcniIAcy3g", "reviewer": "Reviewer_52Uy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper describes a method that is sensible and seems to work well.  It is a creative combination of existing ideas.  \n\nMy most favorite (and, also, in a way, least) feature of this paper is the strategy of instantiating the modular structures for all the tasks (with appropriate sharing and then performing end-to-end experience replay on the whole ensemble simultaneously!  It's a very nice way to get each of the modules to understand its shared \"job\" in the overall task ensemble.   But it also seems unsatisfying that we have to remember and re-use all the data.\n\nCan you explain a bit more about what the various colors mean in Figure 2 left?  I guess you're making a plot for 3 (or so) different numbers of tasks (e.g., 20, 40, 60)?\n\nAlso, the shaded error regions seem surprisingly small.  Is there really so little variance?  What sources of error does a seed control?  The tasks it is trained on?  Initial weights? Is there any randomization in the domain?\n\nLegends for sparse dotted lines are really hard to view.  It would probably be better just to use solid lines of different colors.\n\nThe phenomenon of backward transfer is nice, but why is it surprising?  It seems almost inevitable given your training setup.\n\nMinor: \"are the only that achieve\"\n\n\"Therefore, upon training a new task, we down-scaled the actions and Q-values.\"  This seems like a very blunt instrument for addressing the problem with PPO.  Isn't there something that would let you retain the actual module weight values and therefore some hope of \"zero-shot\" module recombination?  \n\n\"...it was still able to learn slightly faster than STL, despite using an order of magnitude fewer trainable parameters.\"\nHaving fewer trainable parameters, if they are organized right, *should* make learning go faster!\n\nAgain the stderr seems *so* small in figs 3 and 4.  Please be sure to articulate all the possible sources of variance and explain how many times you're varying each one (and be careful about the statistical analysis:  for example, training a network once but testing it 100 times and then reporting a standard error based on N = 100 is somewhere between not very illuminating and actually misleading.  I don't mean to say that's what you're doing---it's just an example of a common error and the kind of thing to watch out for.)\n\nThe reward function for the robotics problems is *very* well-shaped---so well, it seems that one could possibly just write a controller that does gradient ascent on R to reach the objective!  I understand that this is the norm for ``RL'' on robotics tasks, but it is very nearly supervised learning.  It could be interesting as a point of comparison to just generate a big dataset of good trajectories for each task using a planner, and do supervised learning (AKA behavioral cloning) just to see how well it works.\n", "summary_of_the_paper": "This paper combines insights from modular meta-learning, lifelong learning and RL to arrive at a system that can learn, online, to train modules that can be recombined to solve future tasks without further training. In addition, training that happens offline can retrospectively improve modules that were constructed to solve earlier tasks.   These methods are demonstrated in a pedagogical domain and on some high-dimensional robot-control problems.", "summary_of_the_review": "This is a nice combination of mostly existing methods with some good insights for how to put them together.   It's not a major contribution, but it is well described and the experiments seem good, and most importantly, I think it is heading in a very good direction.    Lifelong learning and compositional generalization are critical aspects of moving forward to bigger and more interesting problems and so I think it is important to encourage work in this area.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for compositional task learning in the continual RL setting, by composing and reconfiguring neural modules. The method is evaluated on mini-grids and simulated robot manipulation tasks.

The reviewers agree, and I concur, that the paper proposes an interesting solution to a difficult and important problem. The paper is well presented and would make a good addition to the multi-task continual learning. The reviewers appreciate the authors' responses and the improvement to the manuscript, and in particular the extra experiments with the wrong number of modules.

The final version of the paper should:

- Clarify the explanation of functional modularity
- Move the relevant pieces to the main text.
- See Gur et al., NeurIPS 2021, https://openreview.net/forum?id=CeByDMy0YTL for a definition of learnable compositional tasks via Petri Net formalism.  

Reviewers appreciate the extra experiments with the wrong number of modules.
Decision: Accept (Poster)