Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach for continual learning using a subspace of policies (collection of ‘anchor’ parameters). Given a new task, the control policy either uses a linear interpolation of existing anchor parameters, or introduces a new anchor, and then linearly interpolates in the full set. This causes the number of stored parameters to grow sub-linearly with the number of tasks. The authors present results on locomotion environments (half-cheetah, ant, humanoid) with variations, and the continual world benchmark with robot-manipulation environments. 

strength_and_weaknesses: 
Strengths - 

Significance of proposed Idea - 
The problem considered  in the paper, that of continual learning for agents where they need to mitigate forgetting while also avoiding storing all data they see (due to memory constraints), is critical important for lifelong robot learning systems that keep learning different tasks. This paper uses ideas from mode connectivity (using a set of parameters and trying to generalize from the subspace they induce) and applies them to continual learning. This seems a promising direction for further exploration and the community is likely to benefit from it, since this hasn’t been studied before. 

Thoroughness of evaluation - 
The authors include multiple continual learning experiments to support their claims, and compare sufficiently to prior work. Across domains it appears that the proposed method gets best (or close to best) performance while also being more memory efficient, thus addressing the critical requirements for continual learning. 

Weaknesses - 

Tasks used for analysis 
The paper includes quite a bit of analysis done on the halfCheetah environments (Fig 3- smoothness and critic accuracy, diversity and compositionally, Fig 2c - Learning Efficiency). The multiple tasks for this domain however only include parametric variations (eg - changing mass/gravity/friction etc, from Table 7). This is further away from the eventual setting where we would like to deploy continual learning, where the robot needs to learn over semantically different tasks. Related to this I’d be interested how much the subspace approach helps when the tasks are actually different (eg - pickup hammer, close box etc from continual world), but still share some structure/skills like reaching/picking/grasping/placing. While the paper does evaluate on continual world and report that it comes close to the current sota quantitatively, more analysis on this domain could strengthen the claims of the paper. 

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, and clearly written. Using the mode connectivity idea for continual learning seems novel as described above. The authors also release code so it should be reproducible. 

summary_of_the_review: The paper proposes an interesting idea for continual learning which hasn’t previously been studied for RL problems, and provides sufficient evidence/evaluation through their experiments. The claims can be made stronger by using the continual world envs instead of the halfCheetah env for some of the analysis
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes an approach to continual reinforcement learning, where agents need to acquire a variety of skills rather than just solve a single task. The approach they propose is to learn a convex space in policy parameter space (with a mechanism to extend the subspace with a new anchor if necessary to achieve good performance on a task). In this way the number of anchors grows sublinearly with the numbers of tasks. Then, policies within this subspace can be defined by a small weight vector indicating how to combine the anchor points for the task. They test this approach on a locomotion and manipulation continual learning benchmarks where it performs well, particularly considering the resource costs.
strength_and_weaknesses: Strengths:
- The paper is well written and explains the method well.

- The empirical results are showing a convincing improvement, particularly over classic approaches like elastic weight consolidation.

- It's a conceptual simple approach that can probably be adapted to other algorithms (i.e. the specifics of how you learn the policy are not crucial to the method).

Weaknesses:
- The novelty is not clear. In particular, Gaya et al. (2021) [appropriately cited in the paper] have previously demonstrated the idea of using anchor points to define a subspace in parameter space for rapid online adaption. Although that is distinct from continual learning, its not a big conceptual leap.

- The CLEAR baseline is not included because "storing data from all prior tasks ... is unfeasible due to prohibitive memory costs." It would be helpful to include this baseline as a comparison (even though it is "unfair") and also to quantify the actual amount of memory necessary for these tasks. In practice, large memories are surprisingly cheap, and one could also consider store to slower access hard drives as well. I suspect store everything and train a large neural network is a surprisingly strong and practical baseline.

Minor:
  The notation for the replay buffer (Algorithm 1) is quite unusual $\mathcal{B}uf$ and looks like multiple terms. I would suggest just using $\mathcal{B}$.

clarity,_quality,_novelty_and_reproducibility: Most of this was discussed in strengths / weaknesses.

Clarity:
- The paper is well written and explains the method well.

Quality: The paper is supports its claims well, with experiments and detailed discussion of the method.

Novelty: Discussed as a weakness above.

Reproducibility [updated]. The authors have made the code available so I believe it should be reproducible (but I didn't personally verify).
summary_of_the_review: This work adapts the idea of using anchor points to define a space space in the parameter space of policies from previously used for online adaptation to the task of continual learning, including a mechanism for adding new anchor points when it results in a large enough improvement. It's well written and supports its claim. However, it is not that novel, and conceptually only a small adaptation from earlier work. Additionally, the code is not currently available. For this reason, I'm ambivalent about acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a method of constructing a space of policies that is beneficial for future tasks. 

strength_and_weaknesses: Strengths:
- novel and appealing method
- overall quality of the paper

Weaknesses:
- in some cases, the performance of method is very close to PackNet

clarity,_quality,_novelty_and_reproducibility: The paper is well written overall, although I failed to understand some details in some cases. The presented method is new and hopefully brings long-term benefits to CRL.
summary_of_the_review: The paper presents the method that maintains a growing set of neural networks that span a space of useful policies. Importantly the set is grown adaptively only if a new network would bring significant benefits. In this way, a sublinear number can cover all the tasks.

The major benefit, in my view, of the method is its flexibility, it can adjust the size depending on the 'capacity demand'. A concern is that the empirical results, although nice, do not clearly indicate that the presented method has clear advantage. 

I have a number of questions:
- Can you explain how the training is stable when $\alpha$ changes in line 3. It'd imagine that each $\alpha$ has a much different optimal $\theta_{j+1}$ and thus some kind of jittering arises.
- In line 9 of Alg 1 is it $W_\phi$ which is trained of $Q$?
- How $Q$ function is handled? Is it reset every task? Do you keep one $Q$ for all $\alpha$?
- Do you have any expectations (or experiments) of how PackNet would perform if given $2x$ bigger network for the Brax scenarios? Namely, at the moment, PackNet uses $2x$ parameters, while CSP is $5.3$.
- It'd be nice to see a histogram of the number of used networks.
- What is the comparison of the number of parameters used in the CW10 experiments? Say, how many parameters are used for CSP and PackNet.
- I'd be great to see how the method scales with more tasks. I know it is costly, but at least one longer sequence would be great.
- As far as I understand, a very similar method could be applied in supervised continual. Could you please comment shortly on what you expect?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method for continual Reinforcement Learning (learning on a sequence of related tasks) using a continuous subspace of policies. The proposed approach is quite inspired by the work of Gaya et al., 2021 in that solutions to new tasks, where possible, are parametrically represented as a convex combination of previous task parameters. These previous parameters are thought of as anchors in a subspace of policies.

When presented with a new task, the proposed algorithm attempts to learn new parameters to solve it. The subspace of policies is either expanded to include this new set or, alternatively, the task solution might be represented as a convex combination of the current subspace depending on whether the gain in performance from expanding the subspace crosses a predefined threshold \epsilon (which seems to be about 10%). This choice of expansion v/s pruning is implemented by a clever trick using a state-action value function conditioned on convex parameter sets (Q(s, a, \alpha)). This removes the need for  environment interactions to choose the weights \alpha that specify how the existing subspace should be combined as parameters for the new task.

Results are presented on locomotion and manipulation domains containing 35 different tasks in all and the proposed method usually performs the best across all domains considered.  A set of ablations on the tradeoff parameter \epsilon, scalability, learning efficiency and estimation of the alpha-conditioned critic are also presented. 

strength_and_weaknesses: Overall I think this is a good paper. The ideas are well presented, the experiments and ablations are reasonably convincing and the ablations are useful and interesting. I particularly liked the critic analysis of Figure 3 which, although only on a single task, does seem to indicate that the critic is learning quite well when conditioned on alpha. The paper also acknowledges inspiration from existing work and the related work section is, as far as I can tell, exhaustive. 

To be honest, when I first read the idea I was surprised Q-learning with the added weights actually worked. The critic here Q(s,a, \alpha) has to estimate the value of a policy that is parameterised by some weight-vector, one component of which is also changing. However the experimental results seem quite convincing that this works in practice - the performance of CSP is consistently better or similar to the next best benchmarks at a lower memory usage.

Although I enjoyed the paper generally there are some aspects in which it could be improved which I’ve listed below. 

First - I think the paper could do a better job in communicating certain relevant details. For instance the value of \epsilon typically used for training was only mentioned in the Appendix I think and is quite pertinent when looking at the results. 

Similarly the Q function update actually used to train W(\alpha) should be mentioned somewhere since it is technically different from the one used in SAC. In fact, in general the explanation in Section 3.3 (`Grow the Subspace`) could be improved to reflect how \alpha is being used. On first reading, it was not clear to me that \alpha was being resampled multiple times for the same task. This was made clear by looking at the Algorithm box but it could be made more explicit in the text. 

Finally, as a minor point - there is a typo in Appendix B.1 - the text says ‘two of its continuous control’ but it should be ‘three’ I think.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. All ideas are communicated effectively. The writing is high quality and the ideas are novel. The details in the Appendix should make it clear how to implement this idea andI think the work should be reproducible in theory.
summary_of_the_review: Overall I think the paper is well written, the ideas are interesting and well tested on a number of domains with interesting ablations and analysis and I would recommend the paper be accepted at this venue.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends prior work which represents compactly a large number RL policies via a linear combination of their parameters, by making the linear combination dynamic, being able to grow incrementally as the number of policies increase and more model capacity is needed. The paper provides a heuristic to control this growth, and shows that sublinear parameter increase can be achieved at a modest cost in performance over an oracle strategy.
strength_and_weaknesses: Strength:
- The paper is well written, very detailed and thorough in its evaluation.
- The problem setting is sound and very relevant to lifelong learning.
- Very good experimental analysis of the transfer / forgetting properties of the approach.
- The fact that performance on the benchmarks that the method is evaluated on is close to oracle performance is notable, and justifies the empirical decisions made even in the absence of strong theoretical justification.

Weaknesses:
- The entire premise of the paper rests on the idea that using a subspace representation for policies is a good idea in the first place. While the empirical results of that approach reported in prior works are modestly interesting, they are far from achieving either 1) orders of magnitude in complexity reduction or 2) near zero performance reduction. Being in this 'murky middle' in the first place means this general class of approaches is unlikely to be useful in any practical setting, and raises the question of whether expanding on this research direction is of significant scientific value.
That said, since that work has passed the bar of being presented at ICLR 2022, I'll take into account that I might hold a minority view on the value of this line of research.
- The quality metric W(\alpha) of the policy is entirely computed on the replay buffer data, which is also used to update the policy parameters, which raises the question: is there a risk of overfitting to past experiences, and not actually measuring effectively the quality of the policy on new experiences? It feels like the potential for overfitting in the absence of regularization or held-out evaluation is severe (?), since the ultimate goal is to be a good anchor point for future policies.
- CSP-LINEAR is introduced but I don't see results reported for that strategy (?)

clarity,_quality,_novelty_and_reproducibility: Paper is clear. Experimental results very detailed. Good explanation of the limitations.
summary_of_the_review: Nice paper, but whose usefulness is entirely predicated on the prior work (Gaya & al) being relevant, which I don't entirely agree with but am reluctant to relitigate given prior acceptance. I will rate my review as very low confidence as a result. Scientific contributions on top of that work are incremental as well.

[I had initially rated this paper a '5'. Amending to a '8' after reading other reviewer's perspective, who don't appear to share my overall sentiment, as well as thorough author rebuttals.]
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes Continual Subspace of Policies (CSP) as a scalable solution to continual reinforcement learning. At a high-level the method works by parameterizing policies as convex combinations of anchor parameters, and either growing the set of anchors with each new task or representing the optimal policy as a convex combination of previous anchors. The decision to grow the set of anchors or not comes down to a learnt “universal-value function” Q(s,a,w), which learns the Q-value for the policy derived from coefficients w. This allows them to derive a “policy value function”, with which they can evaluate arbitrary policies within the subspace. This in turn allows them to answer the counter-factual question of whether the optimal policy for the new task is best expressed as a convex combination of previous anchors, or by expanding the set of anchor points. The method is evaluated on several locomotion and manipulation domains, compares favorably across a broad range of baselines and a comprehensive set of ablations and analyses provide justification and a more in depth understanding of the method.

Overall, all the reviewers agree that this is a strong paper which tackles a fundamental problem in continual learning and thus will be of interest to the community. The method appears very effective, in particular on the more complex Humanoid domain. Of note, are the many interesting analyses and ablations of Fig 2 and 3, which visualizes the policy value function, the analysis of performance vs memory as a function of the threshold hyper-parameter.

I am happy to see that several issues of clarity, missing baselines (PACKNETX2), visualizations on Continual World have been included during the rebuttal, along with the full open-sourcing of their code. Thank you to the reviewers and authors for engaging constructively during the discussion.

One concern raised by [9jow] was that while the method does seem effective, we are far from the setting of “orders of magnitude in complexity reduction and/or near zero performance reduction.”. Indeed, from Table 1 reductions in model size are there but limited. Still, I do agree with the authors that one should not judge these methods prematurely, as e.g. such gains could very well arise when using this method at scale or in more realistic environments more amenable to transfer learning. I am thus very happy to support acceptance for this paper and wish to congratulate the authors on a great paper!

ps:  I believe the related works section could further benefit from a comparison to Successor Features and GPE/GPI which also deal with sets of policies and mechanisms for rapidly evaluating their convex combination.
justification_for_why_not_higher_score: As mentioned by [9jow], the provided results yield limited improvements in terms of memory usage (e.g. 2x reduction in memory). The limitation of not storing past data is also somewhat artificial for the environments considered [dMX5]. As such, while a promising direction, the method may have limited interest outside of the continual learning community.
justification_for_why_not_lower_score: Clear and well-written paper proposing a novel, and well-motivated and effective solution to an important problem (continual RL). Importantly, the method offers an important hyper-parameter allowing the practitioner to trade-off performance vs memory. I expect this paper to have an impact in the broader continual learning community (beyond reinforcement learning).
Decision: Accept: notable-top-25%