Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines insights from modular meta-learning, lifelong learning and RL to arrive at a system that can learn, online, to train modules that can be recombined to solve future tasks without further training. In addition, training that happens offline can retrospectively improve modules that were constructed to solve earlier tasks.   These methods are demonstrated in a pedagogical domain and on some high-dimensional robot-control problems.
main_review: This paper describes a method that is sensible and seems to work well.  It is a creative combination of existing ideas.  

My most favorite (and, also, in a way, least) feature of this paper is the strategy of instantiating the modular structures for all the tasks (with appropriate sharing and then performing end-to-end experience replay on the whole ensemble simultaneously!  It's a very nice way to get each of the modules to understand its shared "job" in the overall task ensemble.   But it also seems unsatisfying that we have to remember and re-use all the data.

Can you explain a bit more about what the various colors mean in Figure 2 left?  I guess you're making a plot for 3 (or so) different numbers of tasks (e.g., 20, 40, 60)?

Also, the shaded error regions seem surprisingly small.  Is there really so little variance?  What sources of error does a seed control?  The tasks it is trained on?  Initial weights? Is there any randomization in the domain?

Legends for sparse dotted lines are really hard to view.  It would probably be better just to use solid lines of different colors.

The phenomenon of backward transfer is nice, but why is it surprising?  It seems almost inevitable given your training setup.

Minor: "are the only that achieve"

"Therefore, upon training a new task, we down-scaled the actions and Q-values."  This seems like a very blunt instrument for addressing the problem with PPO.  Isn't there something that would let you retain the actual module weight values and therefore some hope of "zero-shot" module recombination?  

"...it was still able to learn slightly faster than STL, despite using an order of magnitude fewer trainable parameters."
Having fewer trainable parameters, if they are organized right, *should* make learning go faster!

Again the stderr seems *so* small in figs 3 and 4.  Please be sure to articulate all the possible sources of variance and explain how many times you're varying each one (and be careful about the statistical analysis:  for example, training a network once but testing it 100 times and then reporting a standard error based on N = 100 is somewhere between not very illuminating and actually misleading.  I don't mean to say that's what you're doing---it's just an example of a common error and the kind of thing to watch out for.)

The reward function for the robotics problems is *very* well-shaped---so well, it seems that one could possibly just write a controller that does gradient ascent on R to reach the objective!  I understand that this is the norm for ``RL'' on robotics tasks, but it is very nearly supervised learning.  It could be interesting as a point of comparison to just generate a big dataset of good trajectories for each task using a planner, and do supervised learning (AKA behavioral cloning) just to see how well it works.

summary_of_the_review: This is a nice combination of mostly existing methods with some good insights for how to put them together.   It's not a major contribution, but it is well described and the experiments seem good, and most importantly, I think it is heading in a very good direction.    Lifelong learning and compositional generalization are critical aspects of moving forward to bigger and more interesting problems and so I think it is important to encourage work in this area.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces and explores the use of *functional compositionality* in lifelong reinforcement learning (RL). In contrast to hierarchical reinforcement, which explores temporal compositionality (the chaining of options or subpolicies across time), functional compositionality involves assembling a novel, overall function (in the case of RL, the policy function) by composing together neural modules that perform multi-stage processing of the policy's inputs. Each module can be regarded as a small function that takes some abstract input X and maps it to an abstract output Y -- the initial set of modules will take the state space as the input X, while the final set will take the actions as the output Y. Modules can perform their computations in parallel, with their outputs concatenated together afterwards, or they can be chained together, with the output of one set of modules serving as the input of the next set. In this setup, the continual learning problem consists of two phases: first, the correct set of modules for the task must be selected and composed into the policy; then, the parameters of those neural network modules must be updated from the data generated by the agent. Selecting the incorrect set of modules will lead to poor transfer and catastrophic forgetting (as the parameters of the incorrect neural network modules get overwritten). This modularity can help improve robustness by reducing dependencies between modules -- this is achieved both through parallelism and through the need of downstream modules to generalise to novel combinations of upstream modules. The authors validate their approaches on a set of tasks with compositional structure, which a functionally compositional policy would be able to exploit, as it could substitute out modules according to the components of the current task. They compare their framework against alternative continual learning baselines and show that their framework is able to (i) avoid catastrophic forgetting; (ii) demonstrate better zero-shot transfer to new tasks; and (iii) even exhibit better backward transfer on training tasks.
main_review: Overall, I found this to be an interesting paper. The idea of functional compositionality is rather intriguing, and its application to RL and continual learning is original. The authors have motivated why a modular policy should improve continual learning on tasks with compositional structure, and they have validated their approach with a series of compositional tasks, showing that their approach outperforms several alternative approaches to continual learning on a variety of metrics. In particular, they show empirically that their approach demonstrates better forward transfer (with both better zero-shot performance and faster learning on new tasks) and that their approach not only avoids catastrophic forgetting, but that it can actually demonstrate a degree of backward transfer (with training on new tasks improving performance on old tasks).

Methodologically, the primary limitation of their approach is that the computational graph linking modules together needs to be specified in advance. For each graph node, a set of possible modules is defined in advance (although the module parameters still need to be learned), and the agent must determine which of the possibilities to insert at each node. Moreover, as the authors have pointed out, the complexity of finding the right combination of modules can grow combinatorially with the number of possible modules, making it difficult to scale this to a larger number of modules. Another limitation is that the overall framework must be divided into distinct training phases, with an initialisation phase that must give the modules some initial training before they can be deployed in the computational graph. A final limitation is that while the framework shows good performance on compositional gridworld tasks, the empirical performance advantage seem weaker on higher dimensional robotic manipulation tasks. However, in spite of these limitations, the approach is novel, and these are issues that further development of this approach would need to address.

For me, the primary weakness with this paper is that the exposition of functional compositionality in Section 3 could be clearer, and improving this would help the paper reach a wider audience. The framework and how it was applied to RL only became clearer after reading the subsequent sections, particularly Section 5.1 and the algorithmic details in the Supplement. Providing concrete examples may help ground what was otherwise a rather abstract discussion. For instance, it was initially unclear what the module were doing (as I now understand it, they are just stages in a multi-stage hierarchical processing ), and defining them as solutions to abstract subproblems and as functions that map undefined inputs **X** to undefined outputs **Y** did not help. Part of the issue was that I initially understood the modules in set M as being insertable at any node in the computational graph, which they are not. Moreover, it was not clear in this section that the computational graph is pre-defined, as I initially thought the agent could flexibly compose the modules together to form any graph, and part of the problem it needed to solve was to find the correct graph. Relatedly, Algorithm 1 includes many variables and functions whose meaning did not become clear until a second reading of the paper -- for instance, how the discreteSearch() worked, how the modules are composed together, and how the algorithm relates to the graph in Fig 1. Again, I had initially understood the discreteSearch() to be discovering the graph in addition to the modules that should go in the graph node.

There is one part of the algorithm that remains unclear to me. Towards the end of paragraph 1 in Section 6.2, the authors mention batch RL as using "data from the current task and all tasks that reuse those modules to avoid forgetting". By this statement, I read two possibilities: (a) the algorithm considers all tasks that share some (but not all) modules with the current task or (b) the algorithm considers all tasks that use the exact same set of modules as the current task. It'd be helpful if the authors could clarify what is meant here. Additionally, if (a) is meant, then how are the gradients propagated so that catastrophic interference does not occur in the modules that are not shared with the current task. And if (b) is meant, then how are these tasks considered “different” from the current task given that they are using the exact same policy network.

The authors also claim the use of batch RL to overcome catastrophic forgetting as one of their contributions. Yet it is unclear to me how this is novel. Rehearsal methods have previously been used to overcome catastrophic forgetting in lifelong learning, and in the RL context specifically, Rolnick et al (2019) specifically used replay techniques to overcome forgetting.

Finally, it may be worth noting that besides functional compositionality and temporal hRL compositionality, there has also been work, at least in the cognitive sciences, in getting agents to learn compositional internal models of tasks for the purposes of model-based planning (see e.g. Franklin & Frank, 2018 - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006116). Indeed, the authors' compositional gridworld task is reminiscent of the compositional task in this paper, which is in turn based off of the compositional "grid-sailing" task in Fermin et al, 2010 (https://www.tandfonline.com/doi/full/10.1080/00222895.2010.526467).
summary_of_the_review: Overall, I would recommend this paper for acceptance. The application of functional compositionality to RL policies is quite original, and the approach has been validated empirically. But the paper would benefit significantly from greater clarity in its exposition of functional compositionality and in better defining certain variables and functions in Algorithm 1.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an algorithm for lifelong reinforcement learning using functional neural composition. The algorithm first maps a new problem onto a composition of previously acquired modules; then, the agent trains/finetunes the selected module combination on the new task; and finally, the agent incorporates this newly acquired information into the existing modules. This algorithm is then evaluated on two domains, one multi-task lifelong gridworld domain and a multi-task lifelong robotics domain. The algorithm produces superior results to several other lifelong learning approaches.
main_review: I very much agree with the authors' goals: lifelong reinforcement learning is an extremely interesting and underexplored area. I also agree that compositional modularity is an extremely promising direction that merits further study by itself. Unfortunately, the paper suffers from several problems in its current version. However, don not think that these problems are insurmountable: the authors should be able to submit an updated version addressing my main issues in time.

#### Strengths
- The topic and chosen solution are each highly important research directions with enormous room for future research.
- The paper is extremely well written. The language is fluent and the overall structure of the paper is solid.
- The algorithm introduced is a well-engineered solution to the outlined problem

#### Weaknesses
- (This is my main point of concern. If it is addressed to my satisfaction, I will increase my score to weak accept.) While all explanations and descriptions throughout the paper are easily understood, they severely lack in detail that is required to put many statements into context. And while most of this detail can be found in the appendix, I cannot recommend a paper where the appendix is essential to understanding core parts of the main paper for acceptance.
- It is always a delicate balance to jointly introduce a new domain and a new approach, as this can quickly create the impression that a paper co-developed an approach and a domain the approach is bound to excel at, diminishing the appeal of both. Unfortunately, this is the case for the current paper: the particular input permutations (changing the color of the goal) and action-space permutations are an extremely good fit to the algorithm and it's targeted modular architecture. One example is that the number of permutations is aligned with the number of modules.
- Consequently, I find the domain a little too simple. Each tasks will decompose perfectly in representation and action space, even for the robotics domain. This is an unrealistic assumption. An additional experiment on something with a less well-defined structure would have been very interesting. This would have allowed to also investigate other interesting properties, like learning more complex rules that could e.g. involve questions of recursivity or similar.
- Throughout the paper, it is implied that composition is a sequential process. This is a useful simplification as it allows routing-like stacking of neural layers. However, I would strongly argue that it only a simplification: true modular compositionality should allow for arbitrary connections (imagine e.g., two perception skills that are required for logical inference, or recursive rules).

#### Detailed remarks
- p3, l8: I don't understand what "a two-layer abstraction" should mean
- p4, top: Mapping the policy onto a graph of sub-skills seems off, as a graph does not allow for parallelisms that may be required for more complex tasks
- section 5: here is where the bulk of details are missing:
  - section 5.1: "each neural module mi in our architecture is in charge of solving one specific subproblem F" what does that mean? that there are as many modules as subproblems?
  - section 5.1: "A better solution is for each module to receive as input only the information relevant to it, such that its output need not characterize any additional information." I have no idea what this means
  - section 5.1: "our architecture assumes that the state is composed of module-specific components" too vague
  - section 5.1: "At each depth d in our modular net" what is dmax in general? How is dmax determined?
  - section 5.2: "while simultaneously preventing the forgetting of knowledge stored in those modules. We achieve this by separating the learning into multiple stages." way too little detail. I do not understand this either
  - section 5.2: "For this, we train each new task on disjoint sets of neural modules until all modules have been initialized." does this mean that the subtasks are also disjoint?
  - section 5.2: "incorporate new (likely incorrect) information" the information may be superfluous or distracting, but it's definitely not "incorrect"
  - section 5.2: "Since a diverse set of modules has already been initialized, we do this via discrete optimization of the reward with respect to the possible combinations of modules." what does this mean? what is discrete optimization?
  - section 5.2: "However, while experience replay has been tremendously successful in the supervised setting, its success in RL has been limited to very short sequences of tasks." please elaborate
  - section 5.2: A module is copied before it is updated to prevent catastrophic forgetting. I did not find anywhere how the updates are later re-introduced into the architecture.
- section 6: I argue that at least a rough explanation of the architecture is essential when introducing results. I want to know: what are the modules? how many are there? how were they chosen? how many parameters do the different approaches have (the existing pairwise comparisons are insufficient). I especially miss an explanation of how the different modules are partitioned and why they were designed that way?
summary_of_the_review: A well chosen topic, with a highly interesting solution.

Strengths:
- good research direction
- paper is extremely well written
- the core of the paper, the introduced algorithm, is a well-engineered solution to the problem

Weaknesses:
- all details relevant to contextualize both the approach and the results are in the appendix
- the paper jointly introduces a domain and an approach, opening it to the critic that the experiments are hand-crafted towards the approach
- the experiments seem too simple
- I disagree with some assumptions behind compositionality
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Update after authors' response:** After reading the other reviews, the updated manuscript and the authors response I think the authors have made several small improvements based on the reviewers' suggestions. Typically I would now be leaning towards a score of 7. However, this year's conference only allows for either a 6 or an 8. To me personally the paper is still slightly closer to 6 than 8, but to make a clear statement in favor of acceptance and facilitate the reviewers' discussion I have raised my score to an 8. I will clearly state my opinion (and hesitation to strongly endorse acceptance of the paper) during the reviewers' discussion.

**Summary**: The paper proposes a novel method for lifelong compositional RL. The latter is formally introduced in the paper as lifelong RL (learning to solve a continuous stream of tasks without re-visiting previously seen tasks) with families of tasks that have a known compositional structure. The paper proposes to use a modular neural architecture (consisting of layers of modules) to capture the compositional nature of the tasks to learn. Through an appropriate module-selection process the (topologically disjoint) modules become functionally disjoint. This is shown to facilitate forward-transfer (faster learning of novel task variations that follow the compositional structure). The paper investigates two module-selection mechanisms: a fixed mechanism where the correct sequence of modules for each task is known, and a brute-force combinatorial search over all module combination to find the highest scoring combination. Once modules are selected, the selected module-parameters are trained to increase current task-performance. This is then consolidated with previous task experience (in an off-line RL fashion) via batch-RL to avoid catastrophic forgetting. Experiments are shown on a set of 2D grid-world tasks, and a number of simulated robot arms with different sensory setups, and the method performs well against a number of baseline methods.

**Main contributions**
1) Definition of compositional families of RL tasks. As the paper correctly points out, a number of previous work on compositional architectures has used variations of off-the-shelf (lifelong) learning tasks, which are not explicitly compositional and a beneficial composition of modules is often not known intuitively (often not even the number of modules is intuitively clear). The two task-families introduced overcome these issues and are a nice contribution towards more meaningful experimentation. The gridworld task is conceptually easy and does not have very complicated perceptual problem or high-dimensional continuous action-spaces; the robotic-tasks on the other hand is closer to real-world applications (and the corresponding complexities). Significance: I think the tasks are well chosen and might become standard-tasks for lifelong compositional RL.

2) Proposal of a compositional architecture and training procedure. Given that the compositional structure of the tasks is precisely known, it is relatively straightforward to propose an exactly matching architecture. The main innovation is a training procedure to train such an architecture. Unfortunately the paper proposes two fairly straightforward solutions that need strong simplifying assumptions: one, the correct sequence of modules is known and applied accordingly, or two, brute-force search over all module combinations (which becomes exponentially costly with increasing numbers of modules, and requires training-data, and relies on disjoint sets of tasks initially until all modules have been trained at least on one task). Significance: the two module-selection mechanisms serve as important controls and lead to promising results. But they are also fairly straightforward conceptually, and rely on often unrealistically strong assumptions. What I would have liked to see for more impactful results is an attempt to train the architecture from scratch without brute-force search and disjoint tasks initially - one possible attempt would be to use RL to train a module selection policy, similarly to Chang 2019, which is used in the paper as a main inspiration for defining compositional RL tasks.

3) Definition of lifelong compositional RL and using techniques from offline RL to avoid catastrophic forgetting. Significance: these are sensible choices, and thus valid contributions, but both have been conceptually proposed before in slightly different settings (Chang 2019 defines compositional supervised problems and using offline RL to avoid catastrophic forgetting and even have backwards-transfer is also not very far-fetched).

main_review: **Quality, Clarity, Soundness, Correctness**

The paper is well written, the problem is well formulated and introduced, the experiments are clear, and the proposed approach is simple but sound. The benefits of the modular architecture over the non-modular architecture are clear though perhaps less pronounced than expected. Experiments are repeated multiple times to produce results of statistical significance, and the appendix gives all the hyper-parameters and architecture details to reproduce the experiments. I appreciate that the paper clearly points out the extra amounts of data that go into the module-search procedure, which helps with comparison against the other methods. To the best of my knowledge, the claims made in the paper are supported by the empirical results shown. 

My main criticism is that the paper stops short of tackling the most important problems in lifelong compositional RL, which are: (i) designing the architecture when the correct number of modules is unknown (and the depth, i.e. the number of layers of modules), (ii) training when the correct sequence of modules is unknown and brute-force search is too costly, (iii) training without disjoint initial training tasks. I think each of these problems is hard, and fully solving only one of them would merit at least another publication. But I am afraid that the current paper implements the “baselines” that one would like to see in any attempt to solve said problems, but makes no attempt to do the latter. At least some solution attempts could be taken e.g. from the supervised literature, e.g. Chang 2019 who train a module-selection policy via RL. Another strong simplifying assumption is that the state can be separated (a priori) into relevant sub-parts for each module such that only relevant information is fed to each module - this is perhaps the fourth (iv) important and difficult problem in training compositional architectures.


**Improvements**

1) The main improvement that I would like to see is an attempt to train the architecture without knowledge of the correct module-sequence and without disjoint initial training tasks (the latter might turn out to be not possible, which would also be an interesting finding). This attempt would probably consist of several sub-experiments including ablations and controls (e.g. training with too many modules, etc.). I understand that this is mostly beyond of what’s possible during the rebuttal phase, yet I think this would be by far the biggest improvement of the current manuscript.

2) While the plots in Fig 2 (left), Fig 3 (left) and Fig 4 (left) nicely visually summarize the performance, they do not provide information about “how long” it takes for the modular MTL to “overtake” STL. Does MTL need to see many tasks where it gets gradually faster and faster to reach high performance, or is there an initial phase of little improvement followed by a “take-off” where all subsequent tasks are learned rapidly (does that occur after each module has been trained once)? Showing the mean only hides all of these dynamics and it would be nice to see this in (a series?) of appropriate plots. It would also help answer questions like: is modular MTL worth the overhead if it’s only 32 or 16 tasks instead of 64, etc.

3) Figure 2 (right) - the performance gains of modular over monolithic MTL are smaller than intuitively expected. One reason could be that the monolithic MTL successfully makes use of the correct module-sequence which is fed to the monolithic architecture as well. To test for this it would be nice to see performance of both modular and monolithic MTL when feeding in random information for the module-sequence to use. Prediction: modular MTL without search should catastrophically deteriorate monolithic MTL should deteriorate if it uses this information but should remain the same if the information is not used.

**Comments**

4) Another control experiment: “knocking out” individual modules (e.g. reinitializing one module randomly) for the modular MTL with search should lead to deterioration of the corresponding tasks and only these tasks. I have no strong reasons to believe otherwise, yet it would be an interesting control experiment to see whether the modules have really fully specialized, or whether they become responsible for multiple task variations when training with search. 

summary_of_the_review: The paper aims at addressing a very timely and important, and notoriously difficult problem: lifelong RL with task-families that have compositional structure. In principle, learning about this compositional structure should greatly facilitate learning-speed for new task-variations (forward-transfer). There are (at least) four main problems to solve: (i) how many modules are needed and what is the required computational depth (i.e. maximum number of sequential module calls), (ii) how are modules selected for a particular training instance (when this is not clear a-priori and brute-force search across module-combinations is too costly), (iii) how are modules trained initially after random initialization (how do they acquire specialization), (iv) how can it be ensured that modules only process relevant information (and thus become invariant to irrelevant variation). The fifth problem, avoiding catastrophic forgetting, is actually solved in the paper by borrowing a technique from batch RL. All of these problems are hard and active areas of research. Admittedly the answer to (iii) might be a curriculum, similar to what's proposed in the paper.

The current paper makes strong simplifying assumptions to reduce the severity of each problem. While this is a sensible starting point to establish baseline results, what I’m missing from the current manuscript is an attempt to tackle (some of) these problems. I am not saying this is trivial, and I would consider it a significant addition to the paper. Without such an addition the paper shows promising and interesting results for situations where these problems do not appear (which is often unrealistic) or for situations where these problems could be solved through some means in the future. The paper is well written, and clear, results are good, and I have no complaints regarding the correctness of results and claims. However, given the strong simplifying assumptions the results are not very surprising, which ultimately limits the impact and significance of the paper. I currently see no reason to reject the paper, but also have a hard time assigning a high score. I think with a bit of extra work (admittedly beyond what’s possible in the rebuttal) the paper could become a landmark piece of work in lifelong compositional RL.
review_title: null

Meta-review: The paper presents a method for compositional task learning in the continual RL setting, by composing and reconfiguring neural modules. The method is evaluated on mini-grids and simulated robot manipulation tasks.

The reviewers agree, and I concur, that the paper proposes an interesting solution to a difficult and important problem. The paper is well presented and would make a good addition to the multi-task continual learning. The reviewers appreciate the authors' responses and the improvement to the manuscript, and in particular the extra experiments with the wrong number of modules.

The final version of the paper should:

- Clarify the explanation of functional modularity
- Move the relevant pieces to the main text.
- See Gur et al., NeurIPS 2021, https://openreview.net/forum?id=CeByDMy0YTL for a definition of learnable compositional tasks via Petri Net formalism.  

Reviewers appreciate the extra experiments with the wrong number of modules.
Decision: Accept (Poster)