Building a Subspace of Policies for Scalable Continual Learning
Authors: Jean-Baptiste Gaya
Thang Doan
Lucas Caccia
Laure Soulier
Ludovic Denoyer
Roberta Raileanu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: continual learning
deep reinforcement learning

Abstract: The ability to continuously acquire new knowledge and skills is crucial for autonomous agents. Existing methods are typically based on either fixed-size models that struggle to learn a large number of diverse behaviors, or growing-size models that scale poorly with the number of tasks. In this work, we aim to strike a better balance between scalability and performance by designing a method whose size grows adaptively depending on the task sequence. We introduce Continual Subspace of Policies (CSP), a new approach that incrementally builds a subspace of policies for training a reinforcement learning agent on a sequence of tasks. The subspace's high expressivity allows CSP to perform well for many different tasks while growing more slowly than the number of tasks. Our method does not suffer from forgetting and also displays positive transfer to new tasks. CSP outperforms a number of popular baselines on a wide range of scenarios from two challenging domains, Brax (locomotion) and Continual World (robotic manipulation). Interactive visualizations of the subspace can be found at https://share.streamlit.io/continual-subspace/policies/main.

Published as a conference paper at ICLR 2023
BUILDING A SUBSPACE OF POLICIES FOR SCALABLE CONTINUAL LEARNING
Gaya
Thang Doan
Meta AI Research
McGill University, Mila
CNRS-ISIR, Sorbonne University, Paris, France jbggaa@mme com com
(Now at Bosch Research)
thang. doan@ma wwt.eemmmm.mmmmiii.mmm wwgilmmg.......o ca
Lucas Caccia
Laure Soulier
McGill University, Mila
CNRS-ISIR, Sorbonne University, Paris, France laure. sou.erssuliiriiir upmc
lucas.pages www.-pacciaeaaaaaa..........aa.. www.ill.... ca
Ludovic Denoyer
Roberta Raileanu
Ubisoft France ludovic www.oeeeubbioototttoooo
Meta AI Research ra eanu @meta com
ABSTRACT
The ability to continuously acquire new knowledge and skills is crucial for au- tonomous agents. Existing methods are typically based on either fixed-size models that struggle to learn a large number of diverse behaviors, or growing-size models that scale poorly with the number of tasks. In this work, aim to strike better balance between an agent's 's size and performance by designing a method that grows adaptively depending
the task sequence. We introduce Continual Subspace of Policies (CSP), a new approach that incrementally builds a subspace of policies for training reinforcement learning agent sequence of tasks. The subspace's high expressivity allows CSP to perform well for many different tasks while growing sublinearly with the number of tasks. Our method does not suffer from forgetting and displays positive transfer to new tasks. CSP outperforms number of popular baselines on wide range of scenarios from two challenging domains, Brax (lo- comotion) and Continual World (manipulation). Interactive visualizations of the subspace can be found at csp. Code is available here.
INTRODUCTION
Developing autonomous agents that can continuously acquire new knowledge and skills major challenge in machine learning, with broad application in fields like robotics or dialogue systems. In the past few years, there has been growing interest in the problem of training agents on sequences of tasks, also referred to as continual reinforcement learning (CRL, Khetarpal et al. (2020)). However, current methods either use fixed-size models that struggle learn a large number of diverse behaviors (Hinton et al., 2006; Rusu et al., 201 6a; Li & Hoiem, 2018; 8; Bengio & LeCun, 2007: Kaplanis et al., 2019; Traoré et al., 2019; Kirkpatrick et al., 2017; Schwarz et al., 2018; Mallya & Lazebnik, 2018), or growing-sizz models that scale poorly with the number of tasks (?Cheung et al., 2019; Wortsman al., 2020). In this work, introduce adaptive-siz size model which strikes better balance between performance and size, two crucial properties of continual learning systems (Veniat et al., 2020), thus scaling better to long task sequences.
Taking inspiration from the mode connectivity literature (Garipov et al., 2018; Gaya et al., 2021). we propose Continual Subspace of Policies (CSP), new CRL approach that incrementally builds subspace of policies (see Figure for an illustration of our method). Instead of learning a single policy, CSP maintains The vertices of this convex hull are called anchors, with each anchor representing the parameters of policy. This subspace captures a large number of diverse behaviors, enabling good performance on a wide range of settings. At every stage of the CRL process, the best found policy for previously task represented as a single point in the current subspace (i.e. unique convex combination of the anchors), which facilitates cheap storage and easy retrieval of prior solutions.
entire subspace of policies defined as convex hull in parameter space.

---

Published as a conference paper at ICLR 2023
Grow
Og
02
Average Performanos
1.4
FT-L2
R(ag)) > (1 + c) [((a)))) )
PackNet
1.0
EWC
Extendl
Yoo
No
Prune
0.6
PNN
SAC-N
[C
04
0.2
CSP
04
0S
02
Og
Model Size
(a) Continual Subspace of Policies (CSP)
(b) Performance Size Tradc-Off
Figure 1: (a) Continual Subspace of Policies (CSP) iteratively learns a subspace of policies in the continual RL setting. At every stage during training. the subspace is simplex defined by set of anchors (i.e. vertices). Any policy (i.e. point) in this simplex can be represented as a convex combination of the anchor parameters. a; defines the best policy in the subspace for task i. When the agent encounters a new task, CSP tentatively grows the subspace by adding a new anchor. If the new task is very different from previously seen ones, a better policy a.ee can usually be learned this case, CSP extendy the subspace by keeping the new anchor, the new
in the new subspace. task bears some similarities previously seen ones, good policy at can typically be found in the old subspace. In this case, CSP prunes the subspace by removing the new anchor. The subspace is extended only if it improves performance relative to the old subspace by least some threshold c. (b) Trade-off between model performance and size for a number of methods, on a sequence of 8 tasks from HalfCheetah (see Table 7 for trade-offs on other scenarios).
If a new task shares some similarities with previously seen ones, a good policy can often be found in the current subspace without increasing the number of parameters. On the other hand, if a new task is very different from previously seen ones, CSP extends the current subspace by adding another anchor, and learns a new policy in the extended subspace. In this case, the pool of candidate solutions in the subspace increases, allowing CSP to deal with more diverse tasks in the future. The size of the subspace is increased only if this leads to performance gains larger than a given threshold, allowing users specify the desired trade-off between performance and size (i.e. number of parameters or memory cost).
We evaluate our approach on 18 CRL scenarios from two different domains, locomotion in Brax and robotic manipulation in Continual World, a challenging CRL benchmark (Wolezyk et al., 2021). We also compare CSP with number of popular CRL baselines, including both fixed-size and growing- size methods. As Figure 1b shows, CSP is competitive with the strongest existing methods while maintaining a smaller size. CSP does not incur any forgetting of prior tasks and displays positive transfer to new ones. We demonstrate that by increasing the threshold parameter, the model size can be significantly reduced without substantially hurting performance. In addition, our qualitative analysis shows that the subspace captures diverse behaviors and even combinations of previously learned skills, allowing transfer to many new tasks without requiring additional parameters.
CONTINUAL CEMENT LEARNING
continual RL problem is defined by sequence of tasks denoted ty....,ty Task is defined by a Markov Decision Process (MDP) M, = A,, Ti, 7) with a set of states S, set of actions A,, a transition function T. S, x A; P(S.), and a reward function r; S, x A, R. Here, we consider that all the tasks have the same state and action space. We also define a global policy II: [1..N] Z (S- ((A)) which takes as input a task id and a sequence of tasks Z, and outputs the policy which should be used when interacting with task i. P(A) is probability distribution over the action space. We consider that each task is associated with training budget of interactions b, When the system switches to task f it no longer has access to transitions from the M; We do not assume access replay buffer with transitions from prior tasks since this would lead to significant increase in memory cost; instead, at each stage, we maintain a buffer with transitions from the current task, just like SAC (Haarnoja et al., 2018b).
2

---

Published as a conference paper at ICLR 2023
3 CONTINUAL SUBSPACE OF POLICIES (CSP)
3.1 SUBSPACE OF POLICIES
Our work builds on Gaya et al. (2021) by leveraging subspace of policies. However, instead of using the subspace train on single environment and adapt to new ones at test time, we use to efficiently learn tasks sequentially in the continual RL setting. This requires designing new approach for learning the subspace, as detailed below.
Let 0 IR represent the parameters of policy, which we denote T(als, 0). A subspace of policies is a simplex in H defined by set of anchors θ ((...... 0x), where any policy in this simplex can be represented as a convex combination of the anchor parameters. Let the weight vector a R with ||/|| denote the coefficients of the convex combination. Therefore, each value of a uniquely defines a policy with T [(a]s, [a. θ]) = (als, a, Throughout the paper, we sometimes refer to the weight vector a and its corresponding policy x (als, [a. θ]) interchangeably.
3.2 LEARNING ALGORITIM
We propose to adaptively construct a subspace of policies for the continual RL setting in which an agent learns tasks sequentially. We call our method Continual Subspace of Policies (CSP). A pseudo-code is available in Appendix C.I.
Our model builds a sequence of subspaces II,..., ON one new subspace after each training task, with each subspace extending the previous one. Note that each subspace θ is a collection of at most j anchors (or neural networks) i.e. [0,1 s j. V j 1...N since the subspace grows sublinearly with the number of tasks (as explained below). Hence, the number of anchors m of subspace θ is not the same as the number of tasks j used to create O) The learned policies are represented as single points in these subspaces. At each stage, CSP maintains both set of anchors defining the current subspace, as well as the weights a corresponding to the best policies found for all prior tasks. The best found policy for task i after training on the first tasks, v S j, is denoted as T/ (als) and can be represented T. (als) = 7 (als, [a,, 0,]), where |o:| = |0,1 j.
point in the subspace O, with weight vector denoted of such that
Given a set of anchors θ and a set of previously learned policies {af...., "ii..,,a)), updating our model on the new task ff+1 produces new subspace ,, and a new set of weights {a* aftt) There are two possible cases. One possibility that the current subspace already contains good policy for tj+1, so we just need to find the weight vector corresponding to a policy which performs well on f5+1- The other possibility that the current subspace does not contain a good policy for tz-1- 1. In this case, the algorithm produces new subspace by adding one anchor to the previous one (see next section), and converts the previous policies to be compatible with this new subspace.
.z)).
To achieve this, CSP operates in two phases:
1. Grow the current subspace by adding a new anchor and learning the best possible policy for task j + in this subspace (where the previous j anchors are frozen).
2. Compare the quality of this policy with the best possible policy expressed in the previous subspace. Based on this comparison, decide whether to extend the subspace to match the new one or prune it back to the previous one.
We now describe in detail the two phases of the learning process, namely how grow the subspace and how we decide whether to extend or prune it (see Algorithm 1 for a pseudo-code).
3.3 GROW THE SUBSPACE
Given the current subspace θ composed of m Š anchors (with being the number of tasks seen anchor denoted 05-1 is added to the set of anchors such that ,,,1 O2 U(0,-1). With all previous anchors frozen, we train the new anchor by sampling a values from a Dirichlet distribution parameterized by a vector with size j + 1, Dir(U(j- + 1)). The new anchor Oj+1 is updated by maximizing the expected return obtained by interacting with the new task j using policies defined by the sampled a 's:
so far), new subspace θ,+1 1 is built as follows. First
0 = arg max Dar
(I)
3

---

Published as a conference paper at ICLR 2023
where (T+ is the return obtained task j throughout trajectory which was generated using policy Λ { [a, (.1]]]. Note that the anchor is trained such that not only one but all possible values of a tend to produce good policy. To do so, we sample different a per episode. The resulting subspace thus aims at containing as many good policies as possible for task j 1.
3.4 EXTEND OR PRUNE THE SUBSPACE
To decide if the new anchor is kept, we propose to simply compare the best possible policy for task j 1 in the new subspace with the best possible policy for the same task in the previous subspace (i.e. without using the new anchor). Each policy could be evaluated via Monte-Carlo (MC) estimates by doing additional rollouts in the environment and recording the average performance. However, this typically requires large number (e.g. millions) of interactions which may be impractical with a limited budget. Thus, we propose an alternative procedure to make this evaluation sample efficient.
For each task) j and corresponding subspace }, our algorithm also learns a Q-function Q(s, a, a, which is trained to predict the expected return on task j for all possible states, actions, and all possible a's in the corresponding subspace. This Q-function is slightly different from the classical ones in RL since it takes as an additional input the vector a. This Q-function is reset for every new task. Our algorithm is based on SAC (Haarnoja al., (b), so for each task, collect replay buffer of interactions B which contains all states and actions seen by the agent while training on that task. Thus,
the Q-function (s,, a, a) can help us directly estimate the quality ((a) of the policy represented by the weight vector a in the new subspace which can be computed as the average over all states and actions in the replay buffer:
(a)= = E..0 ((5,,,))
It thus possible to compute the value of corresponding to the best policy in the extended subspace (denoted a new m++
[R [a
|1 1):
O new
arg max W(a),
oe m- 10011-1
as well as the value of a corresponding to the best policy in the previous subspace (denoted god e
R Ia old |1 l):
old
arg max
W (a).
(4)
(o.0) with RE ReR2.llo|11
In practice, new and old are estimated by uniformly sampling number of a's in the corresponding subspace as well as a number of states and actions from the buffer.
The quality of the new subspace and the previous one can thus be evaluated by comparing W Waaeee
and W (lold) ). If W (a"eww > (1++). (aa ), the subspace is extended to the new subspace (i.e. the one after the grow phase): θj+1 θy-1. Otherwise, the subspace is pruned back to the old subspace (i.e. the one before the growth phase): 0j+1 = Oy. Note that, if the subspace is extended,
the previously learned policies have to be mapped in the new subspace such that af e R i.e. j+1
V i j, at := (a., 0) and a3+1 == new If the subspace is not extended, then old values can be
kept Le. Visj. j, og == of and o == old
After finding the best a for the current task, the replay buffer and Q-function are reinitialized. Hence, the memory cost for training the Q-function is constant and there is no need to store data from previous tasks unlike other CRL approaches (Rolnick et al., 2019). The policy Te, and value function We & are updated using SAC (Haamoja et al., 2018b). See Appendix C for more details.
3.5 SCALABILITY OF CSP
By having access to an large number of policies, the subspace is highly expressive so it can capture a wide range of diverse behaviors. This enables positive transfer to many new tasks without the need for additional parameters. As a consequence, the number of parameters increases sublinearly with the the number of tasks. The final size of the subspace depends on the sequence of tasks, with longer and more diverse sequences requiring more anchors. The speed of growth is controlled by the threshold which trades-off performance gains for memory efficiency (the higher the c the more performance losses are tolerated to reduce memory costs). See Figure 2a and Appendix C.2.
4

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 ENVIRONMENTS
We evaluate CSP on 18 CRL scenarios containing 35 different RL tasks, from two continuous control domains, locomotion in Brax (Freeman al., 2021) and robotic manipulation in Continual World (CW, Wotezyk et al. (2021)), challenging CRL benchmark. For CW, we run experiments on both the proposed sequence of 10 tasks 10), and on all 8 triplet sequences (CW3). Each task in these sequences has a different reward function. The goal of these experiments is to compare our approach with popular CRL methods on a well-established benchmark.
For Brax, we create new CRL scenarios based on 3 subdomains: HalfCheetah, Ant, and Humanoid. Each scenario has 8 tasks and each task has specific dynamics. We use budget of 1M interactions for each task. The goal of these experiments is to perform an in-depth study to separately evaluate capabilities specific to CRL agents such forgetting, transfer, robustness, and compositionality (see Appendix B.2 and Tables 5a, 5b for more information). For each of the 4 capabilities, we create CRL scenarios, one based on HalfCheetah and one based on Ant. To further probe the effectiveness of our approach, we also create one CRL scenario with tasks on the challenging Humanoid domain. Here we use a budget of 2M interactions for each task.
The CRL scenarios are created following the protocol introduced in Wolezyk et al. (2021) which proposes systematic way of generating task sequences that test CRL agents along particular axes. While Wolczyk et al. (2021) focus on transfer and forgetting (sce Appendix B.3), we also probe robustness (e.g. adapting to environment perturbations such as action reversal), and compositionality (e.g. combining two previously learned skills to solve new task). Each task in sequence has different dynamics which are grounded in quasi-realistic situations such as increased or decreased gravity, friction, limb lengths. See Appendix B.1 for details on how these tasks were designed.
4.2 BASELINES
We compare CSP with number of popular CRL baselines such PNN (Rusu et al., 2016b). EWC (Kirkpatrick et al., 2017), PACKNET (Mallya Lazebnik, 2018), FT-1 which finetunes a single model on the entire sequence of tasks, and FT-L2 which is like FT-1 with additional L2 regularization applied during finetuning. We also compare with SAC-N which trains one model for each task from scratch. While SAC-N avoids forgetting, it cannot transfer knowledge across tasks. Finally, we compare with method called FT-N which combines the best of both SAC-N and FT-11 Just like SAC-N, it stores one model per task after training on it and just like FT-1, it finetunes the previous model to promote transfer. However, FT-N and SAC-N scale poorly (i.e. linearly) with the number of tasks in terms of both memory and compute. which makes them unfcasible for real-world applications. Note that our method is not directly comparable with CLEAR (Rolnick et al., 2019) since we assume no access to data from prior tasks. Storing data from all prior tasks (as CLEAR does) is unfeasible for long task sequences due prohibitive memory costs. All methods use SAC (Haarnoja et al., 2018b) as a base algorithm. The means and standard deviations are computed over 10 seeds unless otherwise noted. See Appendices A.1, A.2, and A.4 for more details about protocol, baselines, and hyperparameters, respectively.
4.3 ABLATIONS
We also perform a number of ablations to understand how the different components of CSP influence performance. We first compare CSP with ESP-ORACLE which selects the best policy by sampling large number of policies in the subspace and computing Monte estimates are expected to be more accurate than the critic's, so an upper bound to CSP. However, OSP-ORACLE is less efficient than CSP since it requires singificantly more interactions with the environment to find policy for each task (i.e. millions..
estimates of their returns. These can be considered
We also want to understand how much performance lose, if any, by not adding one anchor per task. To do this, we run an ablation with no threshold which always extends the subspace by adding one anchor for each new task. In addition, vary the threshold used to decide whether to extend the subspace based on how much performance is gained by doing so. This analysis can shed more light on the ora between performance gain and memory cost as the threshold varies.
5

---

Published as a conference paper at ICLR 2023
Table Aggregated results across all Brax scenarios from HalfCheetah, Ant, and Humanoid. These scenarios were designed to test forgetting, transfer, compositionality, and robustness. CSP performs well as better than the strongest baselines, while having much lower model size and thus memory cost. CSP's performance also not too far from that of CSP-ORACLE indicating that use the critic to find good policies in the subspace without requiring millions of interactions.
HalfCheetah (4 scenarios)
Ant (4 scenarios)
Humanoid
(I scenario)
Method
Performance
Model Size
Performance
Model Size
Performance
Model Size
FT-1
0.62 = 0.29
1.0
0.52 0.26
1.0
0.71 0.07
1.0
FT-L2
0.38 = 0.15
2.0
0.78 + 0.20
2.0
0.68
2.0
PACKNET
0.85 = 0.14
2.0
1.08 + 0.21
2.0
0.96 + 0.21
2.0
EWC
0.43 0.24
3.0
0.55 + 0.24
3.0
0.94 + 0.01
3.0
PNN
1.03 = 0.14
8.0
0.98 + 0.31
8.0
0.98 + 0.26
4.0
SAC-N
1.00 = 0.15
8.0
1.00 + 0.38
8.0
1.00 0.29
4.0
FT-N
1.16 0.20
8.0
0.97 + 0.20
8.0
0,65 0.46
4.0
CSP (ours)
1.27 0.27
1.11 = 0.17
1.76
5.4 1.3
3.9 0.8
3.4 0.3
CSP-ORACLE
1.88 = 0.19
1.24 0.07
1.98 a 0.22
4.4 METRICS
Agents evaluated across range of commonly used CRL metrics (Wolczyk et al., 2021; Powers
al., 2021). For brevity, following the notation in Section 2, we will use T/ [T( [4,,..... 44] to denote the policy selected by IT for task i obtained after training on the sequence of tasks 4 ,,,..., 4g (in
this order with j included). Note that II can also be defined for a single task i.e. T. = i [t:] is the policy for task after training the system only task & (for the corresponding budget b,). We also report forgetting and forward transfer (see Appendix A.3 for their formal definitions).
Average Performance is the average performance of the method across all tasks, after training the entire sequence of tasks. Let (T) be the performance of the system on task using policy т defined as (T) == E,,T, T. [Σr rs (s. a) where r,(s, a) the reward on task i when taking action a in state Then, the final average performance across all tasks can be computed P := + Σ (=.').
Model Size the final number of parameters of a method after training on all tasks, divided by the number of parameters of FT-1 (which is single policy trained on all tasks, with size equal to that of an anchor). For example, in the case of CSP, the model size is equivalent with the number of anchors defining the final subspace 10 Similarly, in the always equal to the number of tasks N. Following Rusu et al. (2016b), we cap PNN's model size to N since otherwise it grows quadratically with the number of tasks.
of FT-N and SAC-N, -N,, the model size
5 RESULTS
5.1 PERFORMANCE ON BRAX
Table I shows the aggregated results across all scenarios from HalfCheetah, Ant, and Humanoid. The results are normalized using SAC-N's performance. See Appendix D.1 for individual results.
CSP performs well as better than the strongest baselines which grow linearly with the number of tasks (i.e. PNN, SAC-N, FT-N), while maintaining a much smaller size and thus memory cost. In fact, CSP's size grows sublinearly with the number of tasks. At the same time, CSP is significantly better than the most memory-effictee efficieen baselines that have relatively small size (i.e. FT-1, FT-L2, EWC, PACKNET). Naive methods like FT-1 1 have low memory costs and good transfer, but suffer from catastrophic forgetting. contrast, methods that aim to reduce forgetting such FT-L2 or EWC do so at the expense of transfer. The only competitive methods in terms of performance are the ones where the number of parameters increases at least linearly with the number of tasks, such as PNN, SAC-N, and FT-N. These methods have no forgetting because they store the models trained on each task. SAC-N has no transfer since it trains each model from scratch, while FT-N promotes transfer as finetunes the previous model. However, due to their poor scalability, these methods are unfeasible for more challenging CRL scenarios with long and varied task sequences.

---

Published as a conference paper at ICLR 2023
Table Results on the CW10 benchmark for CSP and other popular baselines including the state- of-the-art PACKNET (results taken from Woiczyk et al. (2021)). CSP performs almost as well as PACKNET while using about half the number of heads, and thus having much lower memory cost.
Method
CSP (ours)
PACKNET
EWC
FT-L2
FT-1
Performance
0.81 0.06
0.83 a 0.02
0.66 0.03
0.48 + 0.05
0.10
0.01
Heads
5.3 # 1.6
10
10
10
10
In contrast, CSP doesn't 't suffer from forgetting since the best found policies for all prior tasks can be cheaply stored in the form of convex combinations of the subspace's 's anchors (i.e. vectors rather than model weights). In addition, due to having access to large number of policies (i.e. all convex combinations of the anchors), CSP has good transfer to new tasks (that share some similarities with prior ones). This allows CSP to achieve strong performance while its memory cost grows sublinearly with the number of tasks. Nevertheless, CSP's performance not too far from that of CSP-OR ACLE, indicating that the critic can be used to find good policies in the subspace without requiring millions of interactions. See Appendix C.3 for more details about the use of the critic.
5.2 PERFORMANCE ON CONTINUAL WORLD
Table 2 shows results on CW10 (Wotezyk et al., 2021), a popular CRL benchmark. The baselines separate linear layer for each task top of a common network.
used in Wolezyk et al. (2021) The authors recognize this as a limitation since it scales poorly with the number of tasks. For a fair comparison, implement CSP using a similar protocol where the anchors are represented by linear heads, so the number of heads grows adaptively depending on the performance threshold e. Instead of model size, we compare the final number of heads, since this now determines method's is scalability with the number of tasks. See Appendix A for more details about the experimental setup.
CSP performs almost as well as PACKNET which is the current state-of titte-of-theaasd the-art on CW10 (Wolezyk al., 2021), and is significantly better than all other baselines. the same time, CSP uses about half the number of heads, thus being more memory efficient especially as the number of tasks increases. Note that PACKNET suffers from a major limitation, namely that it requires prior knowledge of the total number of tasks in order to allocate resources. If this information is not correctly specified, PACKNET is likely to fail due either not being expressive enough to handle many tasks being too inefficient while learning only a few tasks (Wolczyk et al., 2021). This makes PACKNET unfeasible for real-world applications where agents can face task sequences of varying lengths (including effectively infinite ones). In contrast, CSP grows adaptively depending on the sequence of tasks, so it can handle both short and long sequences without any modification to the algorithm. See Appendix D.2 for additional results on Continual World, including CW3.
To summarize, CSP is competitive with the strongest CRL baselines, while having lower memory cost since its size grows sublineurly with the number of tasks. Thus, CSP maintains a good balance between model size and performance, which allows it to scale well to long task sequences.
5.3 ANALYSIS OF CSP
Varying the Threshold. Figure 2a shows how performance and size vary with the threshold used to decide whether to extend the subspace or not. Both the performance and the size are normalized with respect to CSP-LINEAR which always extends the subspace. As expected, as increases, performance decreases, but so does the size. Note that performance is still above 70% even as the size is cut by more than 70%, relative CSP-LINEAR which trains new set of parameters (i.e. anchor) for each task. Thus, CSP can drastically reduce memory costs without significantly hurting performance. Practitioners can set the threshold to trade-off performance gains for memory efficiency i.e. the higher the e the more performance losses are tolerated in order to reduce memory costs. In practice, we found c = 0.1 to offer good trade-off between performance and model size.
Scalability. Figure 2b shows how performance and size scale with the number of tasks in the sequence (on HalfCheetah's 's compositionality scenario) for both CSP and FT-- which is our strongest baseline on this scenario. CSP maintains both strong performance and small size (i.e. low memory cost) even as the number of tasks increases. In contrast, even if FT-N performs well on all these scenarios, its size grows linearly with the number of tasks, rendering it impractical for long task sequences.
7

---

Published as a conference paper at ICLR 2023
1.2
-12
1.4
b- 250k
1.0
Average Pefermancs
1,2
500k
205 Mppoy
1.0
Average
0.7
0.4
None 0.00 0.25 0.50 1.00
1.50
12
0.2
Threshold Parameter
EWC
(a) Varying the Threshold
(b) Scalability
(c) Learning Efficiency
Figure 2: (a) shows how CSP's performance and size vary with the threshold, (b) shows how performance and size scale with the number of tasks for CSP and FT-N, and (c) compares the performance of multiple methods using different budgets. See Appendix D.3 for more details.
Training
After Training
Before Training
Reward
1.0
0.4
Monte Carlo Reward
Criic Reward
Tuskj
Task
Task; + Tasky
(a) Smoothness and Critic Accuracy
(b) Diversity and Compositionality
Figure (a) shows the value of each policy in the subspace estimated using both Monte-Carke simulations (left) and our critic's predictions (right), demonstrating both that the subspace is smooth and that the critie learns accurate estimates of the reward, which allows CSP to find good policies in the subspace; (b) shows the subspace for three different tasks, the third being a combination of the first two (e.g. walk the already contains a policy with high reward on the compositional task before being trained on it. The star represents the best policy in the subspace for the corresponding task. The subspace contains policies covering the whole spectrum of rewards, suggesting that captures diverse behaviors. Click here to see interactive visualizations of the subspace.
(task 1) with tiny feet (task 2)), demonstrating that the subspace
Learning Efficiency. Figure 2c shows the average performance for three different budgets (i.e. number of interactions allowed for each task) on HalfCheetah's robustness scenario, comparing CSP with FT-1, 1, FT-L2, and EWC. These results demonstrate that CSP can learn efficiently even with a reduced budget, while still outperforming these baselines. By keeping track of all convex combinations of previously learned behaviors, CSP enables good transfer new tasks which in turn leads to efficient training on task sequences.
5.4 ANALYSIS OF THE SUBSPACE
Smoothness and Critic Accuracy. Figure 3a shows snapshot of a trained subspace, along with expected reward of all policies in the subspace, for a given task. The expected reward is computed using both Monte-Carll (MC) rollouts, as well as our critic's 's predictions using Equation 2. As illustrated, the learned Q-function has similar landscape with the MC reward i.e. the subspace is smooth and the critic's ' estimates are accurate.
Diversity and Compositionality. Figure 3b illustrates that the subspace contains behaviors composed of previously learned skills (e.g. walk the moon, walk with tiny feet, walk the moon with tiny feet). This allows CSP to reuse previously learned skills to find good policies for new tasks without the need for additional parameters. The figure also shows that for given task, the policies in the subspace cover the whole spectrum of rewards, thus emphasizing the diversity of the behaviors expressed by the subspace. See Appendix C.6 for more analysis of the subspace.
8

---

Published as a conference paper at ICLR 2023
6 RELATED WORK
Continual Reinforcement Learning, CRL methods aim to avoid catastrophic forgetting, as well as enable transfer to new tasks, while remaining scalable to a large number of tasks. In the past few years, multiple approaches have been proposed to address one or more of these challenges (Tessler et al., 2017; Parisi et al., 2019; Khetarpal al., 2020; Powers et al., 2021: Wolezyk et al., 2021; Tessler et al., 2017; Berseth et al., 2022; Nagabandi et al., 2018; Xie et al., 2020; Xu et al., 2020; Berseth et al., 2021: Ren et al., 2022: Kessler et al., 2022).
One class of methods focuses on preventing catastrophic forgetting. Some algorithms achieve this by storing the parameters of models trained on prior tasks (Rusu al., 201 2016b; Cheung et al., 2019; Wortsman et al., 2020). However, these methods scale poorly with the number of tasks (i.e. least linearly) in both compute and memory, which makes them unfeasible for more realistic scenarios. Among them, PACKNET (Mallya & Lazebnik, 2018) performs well on several benchmarks (CW10, CW20) but requires the final number of tasks input. An extension of this method called EfficientPacKNet is proposed by Schwarz et al. (2021) to overcome this issue, but it is only evaluated on subset of CW. Other methods maintain buffer of experience from prior tasks to alleviate forgetting (Lopez-Paz Paz & Ranzato, 2017: Isele & Cosgun, 2018; Riemer et al., 2019b; Rolnick et al., 2019; Caccia et al., 2020a). However, this is also not scalable the memory cost increases significantly with the number and complexity of the tasks. In addition, many real-world applications in domains like healthcare or insurance prevent data storage due to privacy or ethical concerns. Another class of methods focuses on improving transfer to new tasks. Naive approaches like finetuning that train a single model on each new task provide good scalability and plasticity, but suffer from catastrophic forgetting. To overcome this effect, methods like elastic weight consolidation (EWC ) (Kirkpatrick et al., 2017) alleviate catastrophic forgetting by constraining how much the network's s weights change, but this can in turn reduce plasticity. Another class of methods employs knowledge distillation to improve transfer in CRL (Hinton et al., 2006; Rusu et al., 2016a; Li & Hoiem, 2018; Schwarz al., 2018; Bengio & LeCun, 2007; Kaplanis al., 2019; Traoré et al., 2019). However, since these methods train a single network, they struggle to capture a large number of diverse behaviors.
There is also a large body of work which leverages the shared structure of the tasks (Xu & Zhu, 2018; Pasunuru & Bansal, 2019; Lu et al., 20 Mankowitz et al., 2018; Abel et al., 2017; Sodhani et al., 2019; Beaulieu et al., 2020; Caccia et al., 2020b; Riemer et al., 2019a. Zhou et al., 2020; Schmidhuber, 2013), generative models (Robins, 1995; Silver et al., 2013; Shin et al., 2017; Atkinson et al., 2021) to improve CRL agents, but these methods don't work very well in practice. Lee et al. (2020) use an iteratively expanding network but they assume no task boundaries and only consider the supervised learning setting.
2021), meta-learning (Javed & White, 2019; Spigler,
Mode Connectivity and Neural Network Subspaces. Based on prior studies of mode connectiv- ity (Garipov et al., 2018; Draxler et al., 2018; Mirzadeh et al., 2021). Wortsman et al. (2021); Benton et al. (2021) proposed the neural network subspaces to connect the solutions of different models. More similar to our work, Doan al. (2022) leverages subspace properties to mitigate forgetting on sequence of supervised learning tasks, but doesn't 't focus on other aspects of the continual learning problem. Our work single task for fast adaptation to new environment. In contrast to Gaya et al. (2021), we consider the continual RL setting and adaptively grow the subspace the agent encounters new tasks. Our work is first to demonstrate the effectiveness of mode connectivity for the continual RL setting.
also inspired by Gaya et al. (2021) which learns subspace of policies for
7 DISCUSSION
In this paper, we propose CSP, a new continual RL method which adaptively builds a subspace of policies to learn sequence of tasks. CSP is competitive with the best existing methods while using significantly fewer parameters. Thus, it strikes a good balance between performance and memory cost which allows it to scale well to long task sequences. Our paper is first to use a subspace of policies for continual RL, and thus opens up many interesting directions for future work. For example, one can assume a fixed size for the subspace and update all the anchors whenever the agent encounters a new task. Another promising direction is to leverage the structure of the subspace to meta-learn or search for good policies on a given task. Finally, active learning techniques could be employed to more efficiently evaluate policies contained in the subspace.

---

Published as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
This work builds on the open-source RL library SaLinA (Denoyer et al., 2021) and we have provided detailed description of the experiments, environments, and our method in Appendix A, and C respectively. Code is available here.
REFERENCES
David Abel, Dilip Arumugam, Lucas Lehnert, and Michael L. Littman. Toward good abstractions for lifelong learning. 2017.
Craig Atkinson, B. McCane, Lech Szymanski, and Anthony V. Robins. Pseudo-rehearsal: Achieving
deep reinforcement learning without catastrophic forgetting. ArXiv, abs/1812.02464, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
https: /tse:/aaze //tpxvavvvvv///////////ee/./wwvv wwrg /ttg//bs/////ooo /1 607 06450.
Shawn L. E. Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. ArXiv, abs/2002.09571, 2020.
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007.
Gregory W. Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 769-779. PMLR, 2021.
Glen Berseth, Zhiwei Zhang, Grace Zhang, Chelsea Finn, and Sergey Levine. Comps: Continual meta policy search. arXiv preprint 1rXiv:2112.044:: 2021.
Glen Berseth, Zhiwei Zhang, Grace Zhang. Chelsea Finn, and Sergey Levine. CoMPS: Continual meta policy search. In International Conference on Learning Representations, 2022. URL
htttss:/cpperreee
net Eorumild-PVV3
Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compression with adaptive quantization modules. Proceedings of the 37th International Conference on Machine Learning, 2020a.
Massimo Caccia, Pau Rodríguez López, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam H. Laradji, Irina Rish, Alexande Lacoste, David Vázguez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach continual learning. ArXiv, 001/2003.05856, 2020b.
Brian Cheung, Alex Terekhov, Yubci Chen, Pulkit Agrawal, and Bruno A. Olshausen. Superposition of many models into one. ArXiv, abs/1902.05522. 2019.
Ludovic Denoyer, Alfredo de la Fuente, Song Duong, Jean- tean-Saptiste Gaya, Pierre-Alexandre Kami- enny, and Daniel H. Thompson. Salina: Sequential learning of agents. https://gittee: foom/faceeookreeeercoo sann,, 2021.
Thang Doan, Seyed Iman Mirzadeh, Joelle Pincau, and Mehrdad Farajtabar. Efficient continual learning ensembles neural network subspaces. arXiv preprint arXiv:2202.09886. 2022.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In ICML 2018: Thirty-fitth International Conference on Machine Learning, pp. 1308-1317, 2018.
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 13281, 2021.
10

---

Published as a conference paper at ICLR 2023
T. Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurIPS, 2018.
Jean-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. ArXiv, abs/2110.051699 2021.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2018a. URL https: http///xxivvv.ro. abs /ttb///812. 05905
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 18/2v: 2018b.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18: 18:1527-1554, 2006.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI, 2018.
Khurram Javed and Martha White. Meta-learning representations for continual learning, In NeurIPS, 2019.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. ArXiv, abs/1902.00255, 2019.
Samuel Kessler, Jack Parker-Holded Philip Ball, Stefan Zohren, and Stephen J Roberts. Same state, different task: Continual reinforcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7143-7151, 2022.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. ArXiv, abs/2012. 13490, 2020.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinte Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 - 3526, 2017.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free continual learning. arXiv preprint arXiv:2001.0063 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:2935-2947, 2018.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NIPS, 2017.
Kevin Lu, Aditya Grover, P. Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. ArXiv, abs/2012.03548, 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Daniel Jaymin Mankowitz, Augustin Zidek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, H. V. Hasselt, David Silver, and Tom Schaul. Unicorn: Continual learning with universal, off-policy agent. ArXiv, Abs/1802.08294 2018.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity multitask and continual learning. ArXiv, abs/2010.04495, 2021.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv: 1812,0777 2018.
11

---

Published as a conference paper at ICLR 2023
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks: ; the official journal of the International Neural Network Suciety, 113:54-7,, 2019.
Ramakanth Pasumuru and Mohit Bansal. Continual and multi-task architecture search. ArXiv, abs/1906.05226, 2019.
Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottagii, and Abhinav Kumar Gupta. Cora: Bench- marks, baselines, and metrics platform for continual reinforcement learning agents. ArXiv, abs/2110.10066 2021.
Hang Ren, Aivar Sootla, Taher Jafferjee, Junxiao Shen, Jun Wang, and Haitham Bou-Ammar. Reinforcement learning in presence of discrete markovian context evolution. arXiv preprint arXiv:2202.06557, 2022.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. ArXiv, abs/1810. 19100, 2019a.
Matthew Riemer, Tim Klinger, Djallel Bouneffouf, and Michele M. Franceschini. Scalable recollec- tions for continual lifelong learning. In AAAI, 2019b.
Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connect. Sci., 7:123-146, 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning. In NeurIPS, 2019.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. CoRR, abs/1511.06295, 2016a.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, 0bss/660.046) 2016b.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in Psychology, 4, 2013.
Jonathan Schwarz, Wojciech M. Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinskk. Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning, ArXiv, abs/l Abs/1805.06370, 2018.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropagation: A sparsity inducing weight reparameterisation.
M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. ciates, Inc., 2021. URL https: //pppe/ds///oooednngg neurips. cc cc/papprr/022//LL fle709e6ae 662226666666666666555555666--- Papss-aapppppppp.
28903. Curran Asso-
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, 2017.
Daniel L. Silver. Qiang Yang, and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In AAAI Spring Symposium: Lifelong Machine Learning, 2013.
Shagun Sodhani, Franziska Meier, Joelle Pineau, and Amy Zhang. Block contextual mdps for continual learning. arXiv preprint arXiv:2110.06977 2021.
Giacomo Spigler. Meta-learnt priors slow down catastrophic forgetting in neural networks. ArXiv, hts/1909.04174 2019.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
12

---

Published as a conference paper at ICLR 2023
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1 10911009RS.0012222 201...66616,
Kalifou René Traoré, Hugo Caselles-Dupré, Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz Rodríguez, and David Filliat. Discorl: Continual reinforcement learning via policy distillation. ArXiv, abs/1907.05885. 2019.
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012.126311 2020.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. ArXiv, abs/2006. 14769, 2020.
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. ArXiv, abs/2102.10472, 2021.
Maciej Wolezyk. Michal Zajkac, Razvan Pascanu, Lukasz Kuci nski, and Piotr Milo's. Continual world: A robotic benchmark for continual reinforcement learning. ArXiv, abs/2105.10919, 2021.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity. arXiv preprint arXiv:2006. 2020.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurlPS, 2018.
Mengdi Xu, Wenhao Ding, Jiacheng Zhu, Zuxin Liu, Baiming Chen, and Ding Zhao. Task-agnostic online reinforcement learning with infinite mixture of gaussian processes. Advances in Neural Information Processing Systems, 64400964440 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https::/p/// aarxiv.iiiv.////i///aa///aaaavva http/aawwii.....aaa..../a....aat owg/ooggg/.oooooooooooo.oe /1900. 10897.
Wei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang. and Timothy M. Hospedales. Online meta-critic learning for off-policy actor-critic methods. ArXiv, abs/2003.05334, 2020.
13

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
All the experiments were implemented with SaLinA (?), a flexible and simple python library for learning sequential agents. We used Soft Actor Critic (Haarnjja al., 2018b) as the routine algorithm for each method. is a popular and efficient off-policy algorithm for continuous domains. We use version where the entropy parameter is learned, as proposed in Haarnoja et al. (2018a). For each task and each methods, the twin critic networks are reset as well as the replay buffer (which has a maximum size of IM interactions). This is common setting, used by several CRL benchmarks (Wolczyk et al., 2021; Powers et al., 2021)
A.1 EXPERIMENTAL PROTOCOL
Brax scenarios Considering FT-N the upper bound with which want to compare, decided to apply the following protocol for each short scenario. First, we run a gridsearch on SAC hyper-parameters (see Table 3) on FT-N and select the best set in terms of final average performance (see Section 4 for the details about this metric). Then, we freeze these hyper-parameetrss and performed a specific gridsearch for CSP and each baseline (see Table 4). Each hyper-parameter set is evaluated over 10 seeds. We believe this ensures fair comparisons, and even gives slight advantage to FT-N compared to our method. Note that we set the number of parallel environments to 128 and the number of update per step to 0.5. They can be seen as tasks constraints. These values are reasonable, and Tuning them would have increased the number of experiments by a lot. Evaluations are done on 512 environments per experiment, each with different seeds, with no noise on the policy output.
Continual World We bypassed the preliminary SAC gridsearch and use the set of hyperparameters proposed in Wolezyk et al. (2021).For For the triplet scenarios, we performed specific gridsearch for CSP and each baseline (see Table 4). For CW10, we added CSP framework only on the top layer (head). the backbone being handled by PACKNET. In Wolezyk et al. (2021). PACKNET uses one head per task, just like our baseline FT-N does. In other terms, one can see as a benchmark between FT-N and CSP on top of PACKNET. Note that we did not performed gridsearch any hyperparameter this case.
A.2 ARCHITECTURE AND BASELINES
Architecture for Brax For the twin critics and policy, We use network with 4 linear layers, each consisting of 256 neurons. We use leaky ReLU activations (with a = 0.2) after every layer.
Architecture for Continual World We use the same architecture as Wolezyk et al. (2021) for each method: we add Layer Normalization Ba et al. (2016) after the first layer, followed by a tanh activation.
EWC (Kirkpatrick et al., 2017) This regularization based method aims to protect parameters that are important for the previous tasks. After each task, it uses the Fisher information matrix approximate the importance of each weight. We followed Wolczyk et al. (2021) to compute the Fisher matrix and use an analytical derivation of it.
FT-L2 This baseline is proposed by Kirkpatrick et al. (2017) and re-used by Wolezyk et al. (2021). It can be seen as a simplified version EWC where the regularization coefficients for each parameters equal to 1.
PNN (Rusu et al., 2016b) This method creates a new network at the beginning of each task, as well as lateral networks that will take as inputs for each hidden layer the output of the networks trained on former tasks. In this method, the number of parameters, training and inference times are growing quadratically with respect to the number of tasks, such that for our scenarios, the vanilla model size would have been around 40. To overcome that, we followed author's 's advice and reduced the number of hidden layers with respect to the final number of tasks such that is no bigger than FT-N. Note that, as for PAC this also requires to know this number in advance.
14

---

Published as a conference paper at ICLR 2023
PACKNET (Mallya & Lazebnik, 2018) the leading method in Continual World benchmark aims to learn on new task and prune the networks the end, and retrain the weights that have the highest amplitude. This has two drawbacks one has to allocate in advance certain number of weight per task. Without additional knowledge on the sequence, authors recommend to evenly split the number of weights per task, meaning that one has to know the final number of tasks. In addition, the training procedure after having pruned the network does not require any interaction with the environment but is computationally expensive. For completeness, we also report the results for the same model with hidden size doubled in Halfcheetah scenarios (see Table 8). We named it PACKNETx2. The slight increase of performance (7%) is not sufficient to outperform CSP on these scenarios.
A.3 ADDITIONAL METRICS
Note that for Brax, performance is always normalized using reference cumulative reward (see Table 11). In Tables 8, 9, 10, 12 we also report two additional classical metrics from continual learning:
Forward Transfer measures how much a CRL system is able to transfer knowledge from task to task. At task i, it compares the performance of the of the system trained on all previous tasks t1...., t to the same model trained solely on task 4, This measure is defined as T; P.(xt) - P.(x))) and thus the forward transfer for all tasks is T v/4 ΣN Tr.
Forgetting evaluates how much a system has forgotten about task after training on the full sequence of tasks. It thus compares the performance of policy क! with the performance of policy कद and is defined as F, (T( (((=))) ). Similarly to the average transfer, report the average forgetting across all tasks ΣN F,,
A.4 HYPERPARAMETER SELECTION
Table 3: Hyper-parameters search for SAC over Brax scenarios. The asterix indicates that the hyper-parameter seen as a constraint of the environment, explained in A.I.
Hypeer-aareeeter
Ir policy
(0.0003, 0.001)
Ir critic
(0.0003,0. 0.001)
reward scaling
(1., 10.)
target output
(0.05, 0.13
policy update delay
(2,4)
target update delay
[2,4] 4)
update target network coeff' Ir entropy
0.0003
0.005
parallel environments
128"
gradient update per step
0.5"
disceunt factor
0.99
replay buller size
1,
warming sleps (random uniform policy)
12, 800
Table 4: Specific hyper-parameter search for regularization based baselines and our model.
Hyper-parameter
Value
EWC
regularization coefficient
(10 10", 10a 10 10
CSP
threshold combination length
0..1.0.25)
{20, 100)
A.5 COMPUTE RESOURCES
Each algorithm was trained using one Intel(R) Xeon(R) CPU cores (ES-2698 v4 @ 2.20GHz) and one NVIDIA V100 GPU. Each took between approximately 10 and 30 hours complete for Brax and 50 to 70 hours for CW10. The total runtime depended on three factors: the domain, the computation time of the algorithm and the behavior of the policy.
15

---

Published as a conference paper at ICLR 2023
Table Hyper-parameters selected for each Brax scenario based on FT-N performance as explained
in A.1
Htyper-paramete
Haffcheetah
Forgetting
Transfer
Distraction
Compositional
Ir policy
0.001
0.0003
0.001
0.0003
critic
0.0003
0.0003
0.001
0.0003
reward scaling
10.
larget output sid
0.1
0.05
0.1
0.1
policy update delay
2
4
target update delay
FT-1.2
regularization cocfficient
10
10"
10a
10
EWC
regularization cuefficient
10
10"
10
10
CSP
threshold
0.1
0.1
0.1
combination rollout length
100
20
20
100
Hyper-parameter
Ant Distraction 0.001
Forgetting
Transfer 0.001
Compositional
policy
0.001
0.0003
Ir critic
0.001
0,001 1.
0.001 1.
0.0003
reward scaling
10.
10.
target output sid
0.05
0.05
0.1
policy update delay
larget update delay
4
FT-L2
regularization coefficient
10
10"
10"
10a
EWC
regularization coefficient
10
10
10a
10
CSP
threshold
0.1
0.1
0.1
0.1
combination rollout length
100
100
20
Hyper-parameter
Humanoid
Ir policy
0.001
critic
0.0003
reward scaling
0.1
target output std
0.1
policy update delay
1
larget update delay
i
FT-L2
regularization cocfficient
10-2
EWC
regularization coefficient
10-2
CSP
threshold
0.1
combination rollout length
100
16

---

Published as a conference paper at ICLR 2023
B ENVIRONMENT DETAILS
B.1 DESIGNING THE TASKS
We used the flexibility of Brax physics engine (Freeman al., 2021), and three of its continuous control environments Halfcheetah (obs dim: 18, action dim: 6), Ant (obs dim: 27, action dim: 8), and Humanoid (obs dim: 376, action dim: 17) to derive multiple tasks from them. To do so, tweaked multiple environment parameters (Table 6)) and tried to learn policy on these new tasks.
From that pool of trials, we kept tasks (see Table 7) that were both challenging and diversified.
Table 6: Environment parameters tweaked to create interesting tasks for our scenarios. The Range column indicates range of the multiplying factor applied to the environment parameter in question.
Environment Parameter
Range
Description
mass
[0.5, 1.5]
mass of a particular part of the agent's body (torso, legs, feet....)
radius
[0.5, 1.5]
radius a particular part the agent's body (torso, legs, feet....)
gravity
[0.15, 1.5]
gravity the environment
friction
[0.4, 1.5]
friction of the environment
actions
(1,-1}
invert action values if set to -1. Used for distraction tasks.
observations (mask)
[..1
proportion of masked observations simulate defective sensors
actions (mask)
00.25,0.75
proportion of masked actions to simulate defective modules
Table 7: List of the 25 tasks used to create our scenarios and the parameter changes associated.
Task Name
Parameter changes
normal
carrystuff'
starso_nass: torsa_radius:
carrystuff_
moes: toreo radiuss gravity:
HallCheetah
defectivesensor
obas
hugefeet
(teet_nase: 1,b, teet_rodiue: 1.51
hugefeet_raine rainfall
[foul_Jaass: 1.5, Iost_radios: 1.5, friecions 0.41
inverted actions
(astion_coctticlen::: -1.3
moon
ferevity: 0.151
tinyfeet
[feet_nass: fwet_radius:
tinyfeet moon rainfall
[foot_nass: foct_radius: D.5, gravitys
(friction: 0.41
normal
nofect_2_3_4
(action_mask: 0.75 tonly 1st leg available))
nofeet 1_3
factiorinaak:: 0.75 fonly 2nd leg available))
nofeet_ 1_3
(astion_nask: 0.5 cisz diagonal diabled)
Ant
nofeet 2
tontion 0.5 (2d diagunel diabled))
nofeet_12 1_2
jastion_nass:
(forctest disablcd))
nofeet_33 3
fection
thindfeet disabled))
inverted actions
(astion_coorrcclen:: -1.;
moon rainfall
(grevity: 0.73
itriction: 0.41
Humanoid
normal moon Igsavity: 0.151
carrystuff' (torso_mass: torea_radius: Ivsist_mase: 4, iwaist_rodius: 4F
tinyfeet (chin_nase: 0.5, soin_rodius: 0.51
17

---

Published as a conference paper at ICLR 2023
B.2 DESIGNING THE CONTINUAL RL SCENARIOS
Inspired by Wotczyk et al. (2021), we studied the relationship between these changes with a simple learn new task with policy that has been pre-trained former task. We drew forgetting and transfer tables for each pair of tasks (see Figure 5). With this information, we designed 5 types of scenarios representing particular challenge in continual learning. Note that all tasks have a budget 1M interactions each and repeat their loop times (i.e. 8.M interactions total) except for the Humanoid scenario that contains tasks with 2M interactions.
protocol:
1. Forgetting Scenarios are designed such that single policy tends to forget the former task when learning new one.
Halfcheetah: hugefeet moon carrystuff rainfall
Ant: normal hugefeet rainfall moon
Transfer Scenarios are designed such that a single policy has more difficulties to learn a new task after having learned the former one, rather than learning from scratch.
Halfcheetah: carrystuff caryguuf_hggooriitt moon
defectivesensor hugefoet_rainfall
Ant: nofeet 1_3 nofeet 2 nofeet 1 nofeet 4
Robustness Scenarios alternate between a normal task and a very different distraction task that disturbs the whole learning process of single policy (we simply inverted the actions). While this challenge looks particularly simple from human perspective (a simple -1 vector applied on the output fine to find an optimal policy in continual setting), we figured out that the Fine-tuning policies struggle to recover good performances (the final average reward actually decreases).
Halfcheetah: normal inverted_act
normal nveered_act
Ant: normal inverted_actions normal inverted
Compositional Scenarios present two first tasks that will be useful to learn the last one, but very different distraction task is put at the third place disturb this forward transfer. The last task indeed a combination of the two first tasks in the sense that it combines their particularities. For example, if the first task is "moon" and the second one is tinyfeet, the last one will combine moon's 's gravity and feet morphological changes.
Halfcheetah: tinyfeet moon carrystuff _htiruuvttugerrvittt tinyfeet_moor
Ant: nofeet 3 nofeet 3.44 noteet 1.2 nofeet_3_4
Humanoid scenario is an additional scenario built with the challenging environment Humamoid to test our method in higher dimensions. Here is the detailed sequence of the scenario.
Humanoid: normal
moon carrystuff tinyfeet
hugeloot
Figure 4: Rendering example of our Halfcheetah forgetting scenario. Click here for interactive visualizations.
18

---

Published as a conference paper at ICLR 2023
Corry mal hasiig
regeteed Heese
comprtort
Impoltes
coroul
abyfer
cntyout corystu bugegisity
-C4
+15
+22
44
+62
-11
+31
460
+4
-0
44
-11
hugafert.miiti mirbil
mezn
-3
1hyfte
42
15.
(a) Forgetting table
camyytor Sagegroutt
det meen
hapweed mletaa
Lod moom
camyatu
cermel
conpooott zugeorovity
caryout
A1
+02
41
14
A3
A5
.44
41
43
hugeleet
-0.2
422
+10
First Task
hightbrt.nitthh
+0.
<0x4
10A
moon
rermal
+0.3
-69
-12
422
422
412
+0.
-12
+24
Brylec
+02
60.
43
11.
+00
433
-0.1
-05
(b) Transfer table
Figure 5: Forgetting (a) and Transfer (b) tables of our Halfcheetah tasks. The pairs selected to create our scenario highlighted blue. Results are averaged over seeds using classical RL algorithm.
19

---

Published as a conference paper at ICLR 2023
B.3 CONTINUAL WORLD TASKS
The main benchmark of ContinualWorld is CW10, i.e. a sequence of 10 tasks designed by Wolezyk al. (2021) such that has an average transferfforgtting difficulty. We also tested our model and our baselines on the 8 triplets (3-tasks scenarios) proposed by the authors. The tasks originally come from Metaworld (Yu al., 2019), and have budget of 1M interactions each. The scenarios were specially designed such that there is positive forward transfer from task I to task 3, but task is used distraction that interferes with these learning dynamies. These tasks are designed with the 2.0 version of the open-source MuJoCo physics engine Todorov et al. (2012). Here is the detailed sequence of the triplets:
push-v1 windox-close-vi hammer-vi
2. hamner-vi
xindox-close-vl faucet-close-1y
3. window-elossevi handle-press-side-vl peg-unplug-side-v/
4. faucet-close-vi shelf-place-vl + peg-umplug-side-v))
5. faucst-close-vl shelf-place-vl - push-back-vl
6. stick-pull-vi peg-umplug-side-v/ stick-pull-vi
stick-pull-vi
push-back-vl push-wall-vi
8. push-wall-vi shelf-place-vl * push-back-vl
C DETAILS ABOUT CSP
C.I DETAILED ALGORITHM
Algorithm 1: Continual Subspace of Policies (CSP)
Input: 0,....., .0 (previous anchors), (threshold)
Initialize: Ws (subspace critic), B (replay buffer)
Initialize: 8j+1
E/_1 0, (new anchor);
// Grow the Subspace
for i= I,... :..., B do
Sample a Dir (u(j+
Set policy parameters 0, E/+1 a;8; I K do
Collect and store (s,,,r,s',, in B by sampling a Te. (+s)
end
if time to update then
Update Ti and We using the SAC algorithm and the replay buffer
end
end
12 Use and We to estimate:
11 Extend or Prune the Subepace
old
arg max
Wo(a)
(a.0) with overqn !o/1-1
new
arg max (a seRmil_Jaa
if W W. (1+c) W a. then Return: 01. 01....,0,,66 .......),0-...a000
13
// Extend
14
else
15
Return: 0,, 01....
// Frune
end
20

---

Published as a conference paper at ICLR 2023
C.2 SCALABILITY
By having access an infinite number of policies, the subspace is highly expressive so it can capture a wide range of diverse behaviors. This enables positive transfer to many new tasks without the need for training additional parameters. As a consequence, the number of parameters scales sublinearly with the number of tasks. The speed of growth is controlled by the threshold which deñnes how much performance we are willing to give up for decreasing the number of parameters (by the size of one policy network). Practitioners can set the threshold to trade-off performance gains for memory efficiency (i.e. the higher the c the more performance losses are tolerated to reduce memory costs). In practice, we noticed that setting c 0.1 allows good performance and a limited growth of parameters.
As the agent learns to solve more and more tasks, we expect the subspace to grow more slowly (or stop growing entirely) since already contains many useful behaviors which transfer new tasks. On the other hand, if the agent encounters a task that is significantly different than previous ones and all other behaviors in the subspace, the subspace still has the flexibility to grow and incorporate entirely new skills. The number of anchors in the final subspace is adaptive and depends on the sequence of tasks. The longer and more diverse the task sequence, the more anchors needed. This property is important for real-world applications with open-ended interactions where it's unlikely to know a priori how much capacity is required to express all useful skills.
C.3 TRAINING AND USING THE CRITIC
The subspace critic Wo o plays central role in our method. Compared to the vanilla SAC critic, we only add the convex combination a as an input (concatenated with the states and actions). In this way, it is optimized not only to evaluate the future averaged return on (8,a) a) pairs of a single policy, but an infinity of policies, characterized by the convex combination a.
At the end of each task, the subspace critic has the difficult task to estimate by how much the new anchor policy 0,+1 improves the performance of the current subspace θ,. To do so, one has to find the best combination of policies in the last subspace ), calling it a old and the one in the current subspace θ J+1, calling it a new In practice, the replay buffer at the end of the task allows to have an accurate estimation of the best policies. This part does not require any new interaction with the environment.
found that sampling 1024 random (s, a) pairs from
However, we found that rolling out the top-k a the new and former subspaces helps to find the best combination. In practice, we found that setting A == 8 with one rollout per combination is sufficient. In tasks that have 1M interactions and an episode horizon of 1000, it requires to allocate 0.8% of the budget to the purpose of finding a good policy in the subspace, which does not significantly impact the training procedure.
C.4 SAMPLING POLICIES FROM THE SUBSPACE
Yet, it is important to allow the critic to estimate old During training, we noticed that sampling with a simple flat Dirichlet distribution (i.e. uniform distribution over the simplex induced by the current subspace) is not enough to make the critic able to accurately estimate the performance of the last subspace (indeed, the chances of sampling a policy in the last subspace are almost surely 0.). This is why we decided to sample in both the current and the last subspace. The distribution we use is then mixture of two Dirichlet (equal chances of sampling in the last subspace and in the current subspace). We did not perform an ablation see if balancing the mixture would increase performances.
We also tried to sample with a peaked distribution (concentration of the Dirichlet equal to the inverse of the number of the anchors) to see if it increased performances. In some cases the new subspace is able to find good policies faster with this distribution. It can be a good trade off between always choosing the last anchor and sampling uniformly.
C.5 IMPLEMENTATION
Our Pytorch implementation of CSP uses the nn Mot object to store anchor networks. The additional computational cost compared to a single network is negligible during both training and inference as it is mentioned in Wortsman al. (2021).
21

---

Published as a conference paper at ICLR 2023
C.6 ANALYSIS THE SUBSPACE
The he best way to visualize the reward and critic value landscapes of the subspaces is when there are 3 anchors (see Figure 3). To do saw, draw 8192 evenly spaced points in the 3-dimensional simplex of I2, and average the return over 10 rollouts for the reward landscape, and 1024 pairs of (s, a) for the critic landscape. We used the short version of the Compositional Scenario of HalfCheetah to display the results.
C.7 LIMITATIONS
CSP prevents forgetting of prior tasks, promotes transfer to tasks, and scales sublinearly with the number of tasks. Despite these advantages, our method still has a number limitations. While the subspace grows only sublinearly with the number of tasks, this number is highly dependent on the task sequence. In the worst case scenario, it increases linearly with the number of tasks. On the other hand, the more similar the tasks are, the lower the size of the subspace needed to learn good policies for all tasks. Thus, one important direction for future work is to learn a subspace of policies with fixed number of anchors. Instead of training an additional anchor for each new task, one could optimize policy in the current subspace on the new task while ensuring that the best policies for prior tasks don't t change too much. This could be formulated as constrained optimization problem where all the anchors defining the subspace are updated for each new task, but some regions of the subspace are regularized to not change very much. This would result in the subspace having different regions which are good for different tasks.
While the memory costs increase sublinearly with the number of tasks, the computational costs increase linearly with the number of tasks, in the current implementation of CSP. This is because train an additional anchor for each new task, which can be removed if it doesn' 't significantly improve performance. However, the computational costs reward on a task. This is typically the case for sparse reward tasks where the agent succeeds, receives a reward of and 0 otherwise. In this case, there is no need to train one anchor per task. Instead, one can simply find the best policy in the current subspace and compare its performance with the maximum reward to decide whether to train an additional anchor or not. Hence, in this version of CSP (which is a minor modification of the current implementation) both memory and compute scale sublinearly with the number of tasks. However, this assumption doesn't 't always hold, so here we decided to implement the more general version of CSP.
also be reduced if you have access to maximum
In this work, we don't 't specifically leverage the structure of the subspace in order to find good policies. Hence, one promising research direction is further improve transfer efficiency by leveraging the structure of the subspace to find good policies for new tasks. For example, this could be done by finding the convex combination of the anchors which maximizes return on a given task. Regularizing the geometry the subspace to impose certain inductive biases could also be a fruitful direction for future work.
22

---

Published as a conference paper at ICLR 2023
D ADDITIONAL RESULTS
D.1 BRAX
Iblfchhetth forgetting
Halfcheetah transfer
Hallfheeetah distraction
Netrage Nerformance
2.5
Swerage Parformanc
1.6
Aroruge Performance
3.3
3.2
6.4
644
0.4
64
6.
0.0
Model Size
Model Size
Model Size
Halfchectah composabilty scenario
Ant forgetting
transfer scenario
Autrage Nerformancc
1.4
Average Performance
1.6
Average Performance
1.4
3.2
12
6.0
0.0
10
00
20
Model Size
Model Size
Model Size
composability
Avergg Nefformnnro
3.5
Average Parformane
Avergee Performance
2.2
1.2
h
0.8
60
0.6
15
30
Model Size
Model Size
Model Size
FT-1
FT-L2
PackNet
EWC
PNN
SAC-N
FT-N
*
CSP
Figure 6: Performance-size trade-off for the 9 continuous control scenarios from Brax. Results are averaged over 10 seeds.
Hattcheetsh fargetiing
Htttcheersh Trancter
Halkhwwtthh Dieractio
Hallhwwibhh Comcocah Byy
CUNRT
Figure 7: Histogram of the Model Size of CSP when trained HalfCheetah scenarios (10 seeds per scenario).
23

---

Published as a conference paper at ICLR 2023
Table 8: Detailed results of the long version tasks) of our 4 Halfcheetah scenarios. Results presents mean and standard deviation of our metrics, are split by scenario and method, and are averaged over 10 seeds.
Method
Performance
Model size
Transfer
Forgetting
0.52 = 0.08
1.0 0.0
0.19 + 0.23
0.67 + 0.19
Forgetting scenario
FT-L2
0.67 = 0.32
0.0
+0.34 0.3
0.00
PACKNET
0.94 0.18
2.0
+0.07 0.17
-0.0
PACKNETX2
1.00 0.05
4.0
0.00
0.0
EWC
0.64 0.26
3.0
-0.27 0.31
0.09
PNN
0.96 0.15
8.0
-0.04 0.13
0.0
SAC-N
1.0
8.0
-0.0
0.0
FT-N
1.25 0.24
8.0
0.25 a 0.23
0.0
CSP-ORACLE
1.60 0.12
0.38 0.08
-0.22 0.15
4.5
CSP
1.41 0.07
0.41 0.06
0.00 #
FT-1
0.86 0.7
1.0
0.52 + 0..2
0.66 0.42
Transfer scenario
FT-L2
+
2.0
-1.00 0.03
+ 1 0.04
PACKNET
0.99 0.25
2.0
-0.01 0.24
0.0
PACKNETX2
L.03
4.0
0.03
0.0
EWC
0.13
3.0
-1.13 0.21
0.02
PNN
1,05 0.14
8.0
0.04
0.0
SAC-N
1.05
8.0
-0.0
-.0
FT-N
+.39
8.0
0.39
0.0
CSP-ORACLE
3.75 0.49
2.09 0.71
0.66
1.1
CSP
1.95 0.83
0.93 0.79
FT-1
0.36 0.25
1.0
0
0.53 4 0.25
Distraction scenario
FT-L2
0.22 0.16
2.0
-0.0 0.17
PACKNET
0.65 0.11
2.0
0.1
0.0
PACKNETX2
0.13
4.0
0.0 0.12
EWC
0.68 0.28
3.0
-.23
0.01 + 0.099
PNN
1.14
8.0
0.14
0.0
SAC-N
+.29
8.0
0.28
0.0
FT-N
0.98 0.12
8.0
-.0
CSP-ORACLE
1.28 1 0.09
0.19 + 0.08
0.09 0.08
0.5
CSP
1.01 0.13
0.01 4 0.12
0.0
Composability scenario
FT-1
0.75 0.12
1.0 0.0
+0.04 0.09
0.22 + 0.11
FT-L2
0.66 1 0.03
1.03
0.01 1 0.03
PACKNET
0.79 0.03
2.0
-0.0 0.03
PACKNETX2
0.88 0.08
4.0
0.0 -0.12
EWC
0.53
3.0
0.13 -.09
PNN
0.17
8.0
0.16
0.0
SAC-N
1.05
8.0
-0.0
-0.0
FT-N
1.0100000
8.0
0.01 101+0.00
0.0
CSP-ORACLE
0.87 0.07
-0.1 0.07
0.03 0.06
3.4 1.5
CSP
0.69 0.09
00.11 + 0.09
0.0
FT-1
0.62 0.29
1.0 0.0
0.14 0..29
0.52 + 0.24
FT-L2
0.38
2.0
-0.62 0.13
-0.01 1 0.02
Aggregate
PACKNET
0.85
0.0
-0.15
0.0 0.0
PACKNETX2
0.01
4.0
0.0 0.09
EWC
0.23
3.0
-0251
0.06
PNN
1.03 0.14
8.4
0.03
0.0
SAC-N
1.15
8.0
0.14
0.0
FT-N
1.16 0.2
8.0
0.16 0.19
0.0
CSP-ORACLE
1.88 0.19
0.64 + 0.24
-0.24 0.22
5.4 1.3
CSP
1.27+0.27
0.27 0.26
0.0 0.01
24

---

Published as a conference paper at ICLR 2023
Table Detailed results of the long version (8 tasks) of our Ant scenarios. Results presents mean and standard deviation of our 4 metrics, are split by scenario and method, and are averaged over 10 seeds.
Method
Performance
Model size
Transfer
Forgetting
FT-1
1.31 +: 0.33
1.0 0.0
0.36 + 0.2
0.05 0.23
Forgetting seenario
FT-L2
0.76 0.27
2.0
-0.24 0.24
0.0 0.04
PACKNET
0.2
2.0
0.13
0.0 0.0
EWC
1,12
3.0
0.1
0.17 0.22
PNN
0.97 = 0.2
8.0
-.19
0.0 0.0
SAC-N
0.17
8.0
0.16
0.0
FT-N
1.36
8.0
0.25
0.0
CSP-ORACLE
1.24 0.08
0.11 0.06
-0.13 0.02
4.7.112.
CSP
1.03 0.14
0.03
0.13
0.0
FT-1
0.08 0.14
0.0
-0.28 0.2
0.64 0.15
Transfer scenario
FT-L2
0.44 0.12
2.0
+
0.12 0.09
PACKNET
0.89
2.0
0.09
-0.0
EWC
0.05
3.0
-0.78
0.0
PNN
1.02
8.0
0.05
0.0
SAC-N
1.0
8.0
0.0
-..0
FT-N
0.83 0.12
8.0
0.12
-..0
CSP-ORACLE
1.02 0.09
0.0 0.07
0.02
0.6
CSP
0.93 0.1
0.07 + 0.09
0.0
FT-1
0.34 + 0.06
1.0
-0.16
0.5 + 0.09
Distraction scenario
FT-L2
0.61 +:2 0.08
2.0
0.05
0.03 0.06
PACKNET
0.71 0.05
2.0
1.00
0.0
EWC
0.54 0.08
3.0
-.047
0.02
PNN
0.98 0.19
8.0
-0.02
0..0
SAC-N
0.09
8.0
0.09
0.0
FT-N
0.8 0.09
8.0
-0.2
0.0
CSP-ORACLE
0.57 + 0.08
-0.35
0.08
4.0
CSP
0.6 0.11
0.1
0.0
Composability scenario
FT-1
0.35 0.49
1.0
0.32 0.89
0.97
FT-L2
1.33 0.35
2.0
+0.25 0.18
PACKNET
0.5
2.0
0.54
-0.0
EWC
0.31
3.0
-0.07
0.22
PNN
0.95
8.0
-0.75
-0.0
SAC-N
1.07
8.0
0.0
0.0
FT-N
0.88 4 0.35
8.0
0.12
-0.0
CSP-ORACLE
2.15 + 0.02
1.04 0.23
0.11
0.4
CSP
1.88 0.33
0.88 # 0.32
-..0 0.01
FT-1
0.52 + 0.26
1.0 0.0
0.06 0.33
0.54 + 0.3 0.3
FT-L2
0.78 a 0.2
2.0
-..25
0.04 + 0.09
PACKNET
1.08
2.0
0.08
0.0
Aggregate
EWC
0.55 0.24
3.0
-0126
0.2 0.13
PNN
0.98
8.0
-0.02
0.0
SAC-N
1.0
8.0
0.3
0.0
FT-N
0.97 + 0.2
8.0
-0.03
0.0
CSP-ORACLE
1.24 0.07
0.20
0.10
3.9
CSP
1.11 0.17
0.11
0.0
25

---

Published as a conference paper at ICLR 2023
Table 10: Detailed results of our Humanoid scenario. Results presents mean and standard deviation of our 4 metrics, are split by method, and are averaged over 10 seeds.
Method
Performance
Model size
Transfer
Forgetting
FT-1
0.71 0.07
1.0 0.0
0.1 + 0.23
0.38 0.27
FT-L2
0.68 0.28
0.0
0.01 0.31
0.33 0.28
PACKNET
0.96 0.21
2.0 0.0
-0.04 0.2
-0.0 0.0
EWC
0.94
0.0
-0.05 0.02
0.01 0.02
PNN
0.98 + 0.26
4.0
-0.02 0.3
0.0 0.0
SAC-N
1.0
4.0
0.0
0.0
FT-N
0.46 0.65 0.46
4.0
+0.35 0.35
0.0
CSP-ORACLE
1.98 à 0.22
0.93 4 0.12
00.55 0.332
3.4 0.3
CSP
1.76 0.19
0.19
0.75 0.16
-0.0
Table 11: Raw cumulative rewards obtained on each task of the 4 scenarios. These are obtained with a single policy learned from scratch (SAC-N) and the set of hyperparameters selected by the gridsearch of the FT-N method (see A). This explains why similar tasks have different averaged returns across scenarios. Results are averaged over 3 seeds.
Scenario
Task
Cumulative rewards
hugefoot
2209
2982
Forgetting Scenario
carrystuff' rainfall
6309
1001
carrystuff moon
carrystuf__uugerraviy hugegravity
7233
Halfeheetah
Transfer Scenario
3599
defectivemodule
5909
bugefoot_rainfa
2942
normal
4932
inverted_ normal
5833
Distraction Scenario
4932
inverted_actions
5833
tinyfoot
6311
Compositional Sccnario
moon
3932
carrystuff_hugegravii
6319
tinyfoot_moon mmoo
1355
normal hugefeet
3752
2841
Forgetting Scenario
rainfall moon
1596
1401
disableddiagonalfeet_] 1
3021
disableddiagonalfeett 2
4119
Transfer Scenario
disabledforcfeet
1014
Ant
disabledhindfeet
1021
normal
3542
inverted_actions
4199
Distraction Scenario
normal
3542
inverted_actions
4199
disabled3feet_1
770
disabled3feet
641
Compositional Scenario
disabledforefeet
201
disabledhindfeeet
288
Humanoid
normal
1958
moon
1691
carrystuff
2379
tinyfect
1711
26

---

Published as a conference paper at ICLR 2023
D.2 CONTINUAL WORLD
In addition to the CW10 benchmark, we also tried our method over the 8 triplets (3 tasks for each used CSP the whole
scenario) proposed in Wolezyk et al. (2021). In these experiments, network. To that end, we did not add the layer normalization described in Appendix A.2, since is not trivial to design a subspace in these sets of parameters. See Table 12 for the results.
Table 12: Detailed results of 8 triplets scenarios from Continual World. Results presents mean and standard deviation of the final average performance, are split by scenario and method, and are averaged over 3 seeds.
Method
T1
T2
T3
T4
T5
T6
T7
Agregate
FT-1
0.24 0.13
0.25 = 0.07
0.39 = 0.16
0.34 + 0.05
0.30
0.32 0.25
0.17 + 0.07
0.34 0.05
0.29 = 0.1
EWC
0.45
0.27 0.09
0.38 0.09
0.31 0.12
0.07
0.33 0.18
0.20
0.08
0.32 = 0.11
PNN
0.84 0.08
0.17
0.90
0.43 0.08
0.23
0.46
0.44
0.2
0.56 + 0.14
SAC-N
0.69
0.79 0.13
0.14
0.55 0.10
0.54
0.6 0.12
FT-N
0.77 0.08
0.1
0.49 0.15
0.13
0.61 0.06
0.61
0.52
0.65 0.11
CSP (ours)
0.76
0.03
0.82 + 0.08
0.58 0.09
0 0.06
0.54 - 0.06
0.58
0.53
0.65 0.08
We conducted a qualitative analysis to determine whether forward transfer also comes from the
compositionality induced by the subspace (just like HalfCheetah domain, see Figure 3b). To do so, we used the 3 first anchors created by CSP on the three first tasks of CW10, and computed the reward landscape of each of the 10 tasks. See Figure 8. We note that
Smoothness: As for Brax scenarios, the reward landscape remains smooth all the figures.
Diversity: Policies in the subspace specialize in some tasks that might share similari- ties. while
(d) is tackled by almost all policies of the subspaces, we see that policies on the lower left are good at tackling Push -Wall (b), Saucet-Clos (b) and Peg-Unplug. wwwlllggiiice Side (i). On the right side of the subspace, policies tackle Window-Close (i). On the top, policies are good at resolving Hammer (a) and (f).
Compositionality: the subspace show great compositionality on Window-Close (i) in space around anchors instantiated on Hammer r and Faucet-Close (some points reach 100% success rate directly).
In addition, while Shelf-Place (h) show zero transfer (it is the hardest task according to the learning curves in Wolezyk et al. (2021), all others show different degrees of forward transfer. Interestingly, Push-Back (d) is also tackled by almost all policies the subspaces, while not the hardest task according to the learning curves Wolczyk et al. (2021).
D.3 ABLATIONS
Ablations on the threshold parameter (see Figure 2) of CSP have been performed on the Robustness scenario of HalfCheetah. Concerning the scalability ablation, we used the Compositional scenario of HalfCheetah and duplicated it in 3 versions tasks, tasks, 12 tasks). Concerning the learning efficiency, we used the Robustness Scenario (short version, i.e. 4 tasks only) and ran it with budget of 250k,500k and IM interactions. Results were averaged over 3 seeds.
27

---

Published as a conference paper at ICLR 2023
Faucos-Closs Push Wall
Succt-Close
Push/Nali
Fou Cls-Clee
Pushilal
Fouses Closs Push Val
Fauces Cooss
(a) Hammer
(b) Push-Wall Wall
(c) Faucet-Close
(d) Push-Back
(e) Stick-Pull
Hammeer
Haammmr
Hammer
Hammeer
Maarimma
Pust-Wall
Flose Pushstiel
Tauree-Elose
Taucel-Cow
Pashwall
TaareisCoos Pushileal
(f) Mandle-Press-eee Side
g Push
(h) Shelf- Place
Window-Close
ii) Peg- Unplug- Side
0.25
0.75
Figure 8: Reward landscape each CW10 task when taking policies between the first anchors (learned on Hammer, r, Push-Wall and F Faucet-Cl -Close. Each of the 5151 points represent policy. The results averaged over 20 rollouts (we take the deterministic form of the policy to infer the trajectory).
28


Reviews:
[{"review_id": "n3pDugb2s1a", "reviewer": "Reviewer_9jow", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends prior work which represents compactly a large number RL policies via a linear combination of their parameters, by making the linear combination dynamic, being able to grow incrementally as the number of policies increase and more model capacity is needed. The paper provides a heuristic to control this growth, and shows that sublinear parameter increase can be achieved at a modest cost in performance over an oracle strategy.", "summary_of_the_review": "Nice paper, but whose usefulness is entirely predicated on the prior work (Gaya & al) being relevant, which I don't entirely agree with but am reluctant to relitigate given prior acceptance. I will rate my review as very low confidence as a result. Scientific contributions on top of that work are incremental as well.\n\n[I had initially rated this paper a '5'. Amending to a '8' after reading other reviewer's perspective, who don't appear to share my overall sentiment, as well as thorough author rebuttals.]", "strength_and_weaknesses": "Strength:\n- The paper is well written, very detailed and thorough in its evaluation.\n- The problem setting is sound and very relevant to lifelong learning.\n- Very good experimental analysis of the transfer / forgetting properties of the approach.\n- The fact that performance on the benchmarks that the method is evaluated on is close to oracle performance is notable, and justifies the empirical decisions made even in the absence of strong theoretical justification.\n\nWeaknesses:\n- The entire premise of the paper rests on the idea that using a subspace representation for policies is a good idea in the first place. While the empirical results of that approach reported in prior works are modestly interesting, they are far from achieving either 1) orders of magnitude in complexity reduction or 2) near zero performance reduction. Being in this 'murky middle' in the first place means this general class of approaches is unlikely to be useful in any practical setting, and raises the question of whether expanding on this research direction is of significant scientific value.\nThat said, since that work has passed the bar of being presented at ICLR 2022, I'll take into account that I might hold a minority view on the value of this line of research.\n- The quality metric W(\\alpha) of the policy is entirely computed on the replay buffer data, which is also used to update the policy parameters, which raises the question: is there a risk of overfitting to past experiences, and not actually measuring effectively the quality of the policy on new experiences? It feels like the potential for overfitting in the absence of regularization or held-out evaluation is severe (?), since the ultimate goal is to be a good anchor point for future policies.\n- CSP-LINEAR is introduced but I don't see results reported for that strategy (?)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F80bphTjIh", "reviewer": "Reviewer_hkJZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for continual Reinforcement Learning (learning on a sequence of related tasks) using a continuous subspace of policies. The proposed approach is quite inspired by the work of Gaya et al., 2021 in that solutions to new tasks, where possible, are parametrically represented as a convex combination of previous task parameters. These previous parameters are thought of as anchors in a subspace of policies.\n\nWhen presented with a new task, the proposed algorithm attempts to learn new parameters to solve it. The subspace of policies is either expanded to include this new set or, alternatively, the task solution might be represented as a convex combination of the current subspace depending on whether the gain in performance from expanding the subspace crosses a predefined threshold \\epsilon (which seems to be about 10%). This choice of expansion v/s pruning is implemented by a clever trick using a state-action value function conditioned on convex parameter sets (Q(s, a, \\alpha)). This removes the need for  environment interactions to choose the weights \\alpha that specify how the existing subspace should be combined as parameters for the new task.\n\nResults are presented on locomotion and manipulation domains containing 35 different tasks in all and the proposed method usually performs the best across all domains considered.  A set of ablations on the tradeoff parameter \\epsilon, scalability, learning efficiency and estimation of the alpha-conditioned critic are also presented. \n", "summary_of_the_review": "Overall I think the paper is well written, the ideas are interesting and well tested on a number of domains with interesting ablations and analysis and I would recommend the paper be accepted at this venue.", "strength_and_weaknesses": "Overall I think this is a good paper. The ideas are well presented, the experiments and ablations are reasonably convincing and the ablations are useful and interesting. I particularly liked the critic analysis of Figure 3 which, although only on a single task, does seem to indicate that the critic is learning quite well when conditioned on alpha. The paper also acknowledges inspiration from existing work and the related work section is, as far as I can tell, exhaustive. \n\nTo be honest, when I first read the idea I was surprised Q-learning with the added weights actually worked. The critic here Q(s,a, \\alpha) has to estimate the value of a policy that is parameterised by some weight-vector, one component of which is also changing. However the experimental results seem quite convincing that this works in practice - the performance of CSP is consistently better or similar to the next best benchmarks at a lower memory usage.\n\nAlthough I enjoyed the paper generally there are some aspects in which it could be improved which I’ve listed below. \n\nFirst - I think the paper could do a better job in communicating certain relevant details. For instance the value of \\epsilon typically used for training was only mentioned in the Appendix I think and is quite pertinent when looking at the results. \n\nSimilarly the Q function update actually used to train W(\\alpha) should be mentioned somewhere since it is technically different from the one used in SAC. In fact, in general the explanation in Section 3.3 (`Grow the Subspace`) could be improved to reflect how \\alpha is being used. On first reading, it was not clear to me that \\alpha was being resampled multiple times for the same task. This was made clear by looking at the Algorithm box but it could be made more explicit in the text. \n\nFinally, as a minor point - there is a typo in Appendix B.1 - the text says ‘two of its continuous control’ but it should be ‘three’ I think.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9ByOAxyRjd", "reviewer": "Reviewer_ferp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for continual learning using a subspace of policies (collection of ‘anchor’ parameters). Given a new task, the control policy either uses a linear interpolation of existing anchor parameters, or introduces a new anchor, and then linearly interpolates in the full set. This causes the number of stored parameters to grow sub-linearly with the number of tasks. The authors present results on locomotion environments (half-cheetah, ant, humanoid) with variations, and the continual world benchmark with robot-manipulation environments. \n", "summary_of_the_review": "The paper proposes an interesting idea for continual learning which hasn’t previously been studied for RL problems, and provides sufficient evidence/evaluation through their experiments. The claims can be made stronger by using the continual world envs instead of the halfCheetah env for some of the analysis \n", "strength_and_weaknesses": "\nStrengths - \n\nSignificance of proposed Idea - \nThe problem considered  in the paper, that of continual learning for agents where they need to mitigate forgetting while also avoiding storing all data they see (due to memory constraints), is critical important for lifelong robot learning systems that keep learning different tasks. This paper uses ideas from mode connectivity (using a set of parameters and trying to generalize from the subspace they induce) and applies them to continual learning. This seems a promising direction for further exploration and the community is likely to benefit from it, since this hasn’t been studied before. \n\nThoroughness of evaluation - \nThe authors include multiple continual learning experiments to support their claims, and compare sufficiently to prior work. Across domains it appears that the proposed method gets best (or close to best) performance while also being more memory efficient, thus addressing the critical requirements for continual learning. \n\nWeaknesses - \n\nTasks used for analysis \nThe paper includes quite a bit of analysis done on the halfCheetah environments (Fig 3- smoothness and critic accuracy, diversity and compositionally, Fig 2c - Learning Efficiency). The multiple tasks for this domain however only include parametric variations (eg - changing mass/gravity/friction etc, from Table 7). This is further away from the eventual setting where we would like to deploy continual learning, where the robot needs to learn over semantically different tasks. Related to this I’d be interested how much the subspace approach helps when the tasks are actually different (eg - pickup hammer, close box etc from continual world), but still share some structure/skills like reaching/picking/grasping/placing. While the paper does evaluate on continual world and report that it comes close to the current sota quantitatively, more analysis on this domain could strengthen the claims of the paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "833YyC72PpG", "reviewer": "Reviewer_X3Cb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method of constructing a space of policies that is beneficial for future tasks. \n", "summary_of_the_review": "The paper presents the method that maintains a growing set of neural networks that span a space of useful policies. Importantly the set is grown adaptively only if a new network would bring significant benefits. In this way, a sublinear number can cover all the tasks.\n\nThe major benefit, in my view, of the method is its flexibility, it can adjust the size depending on the 'capacity demand'. A concern is that the empirical results, although nice, do not clearly indicate that the presented method has clear advantage. \n\nI have a number of questions:\n- Can you explain how the training is stable when $\\alpha$ changes in line 3. It'd imagine that each $\\alpha$ has a much different optimal $\\theta_{j+1}$ and thus some kind of jittering arises.\n- In line 9 of Alg 1 is it $W_\\phi$ which is trained of $Q$?\n- How $Q$ function is handled? Is it reset every task? Do you keep one $Q$ for all $\\alpha$?\n- Do you have any expectations (or experiments) of how PackNet would perform if given $2x$ bigger network for the Brax scenarios? Namely, at the moment, PackNet uses $2x$ parameters, while CSP is $5.3$.\n- It'd be nice to see a histogram of the number of used networks.\n- What is the comparison of the number of parameters used in the CW10 experiments? Say, how many parameters are used for CSP and PackNet.\n- I'd be great to see how the method scales with more tasks. I know it is costly, but at least one longer sequence would be great.\n- As far as I understand, a very similar method could be applied in supervised continual. Could you please comment shortly on what you expect?", "strength_and_weaknesses": "Strengths:\n- novel and appealing method\n- overall quality of the paper\n\nWeaknesses:\n- in some cases, the performance of method is very close to PackNet\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-HK0ty_RNUT", "reviewer": "Reviewer_dMX5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an approach to continual reinforcement learning, where agents need to acquire a variety of skills rather than just solve a single task. The approach they propose is to learn a convex space in policy parameter space (with a mechanism to extend the subspace with a new anchor if necessary to achieve good performance on a task). In this way the number of anchors grows sublinearly with the numbers of tasks. Then, policies within this subspace can be defined by a small weight vector indicating how to combine the anchor points for the task. They test this approach on a locomotion and manipulation continual learning benchmarks where it performs well, particularly considering the resource costs.", "summary_of_the_review": "This work adapts the idea of using anchor points to define a space space in the parameter space of policies from previously used for online adaptation to the task of continual learning, including a mechanism for adding new anchor points when it results in a large enough improvement. It's well written and supports its claim. However, it is not that novel, and conceptually only a small adaptation from earlier work. Additionally, the code is not currently available. For this reason, I'm ambivalent about acceptance.\n", "strength_and_weaknesses": "Strengths:\n- The paper is well written and explains the method well.\n\n- The empirical results are showing a convincing improvement, particularly over classic approaches like elastic weight consolidation.\n\n- It's a conceptual simple approach that can probably be adapted to other algorithms (i.e. the specifics of how you learn the policy are not crucial to the method).\n\nWeaknesses:\n- The novelty is not clear. In particular, Gaya et al. (2021) [appropriately cited in the paper] have previously demonstrated the idea of using anchor points to define a subspace in parameter space for rapid online adaption. Although that is distinct from continual learning, its not a big conceptual leap.\n\n- The CLEAR baseline is not included because \"storing data from all prior tasks ... is unfeasible due to prohibitive memory costs.\" It would be helpful to include this baseline as a comparison (even though it is \"unfair\") and also to quantify the actual amount of memory necessary for these tasks. In practice, large memories are surprisingly cheap, and one could also consider store to slower access hard drives as well. I suspect store everything and train a large neural network is a surprisingly strong and practical baseline.\n\nMinor:\n  The notation for the replay buffer (Algorithm 1) is quite unusual $\\mathcal{B}uf$ and looks like multiple terms. I would suggest just using $\\mathcal{B}$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%