Lifelong Reinforcement Learning with Modulating Masks
Authors: Eseoghene Ben-Iwhiwhu
Saptarshi Nath
Praveen Kumar Pilly
Soheil Kolouri
Andrea Soltoggio
Venue: Accepted by TMLR ()

Abstract: Lifelong learning aims to create AI systems that continuously and incrementally learn during a lifetime, similar to biological learning. Attempts so far have met problems, including catastrophic forgetting, interference among tasks, and the inability to exploit previous knowledge. While considerable research has focused on learning multiple supervised classification tasks that involve changes in the input distribution, lifelong reinforcement learning (LRL) must deal with variations in the state and transition distributions, and in the reward functions. Modulating masks with a fixed backbone network, recently developed for classification, are particularly suitable to deal with such a large spectrum of task variations. In this paper, we adapted modulating masks to work with deep LRL, specifically PPO and IMPALA agents. The comparison with LRL baselines in both discrete and continuous RL tasks shows superior performance. We further investigated the use of a linear combination of previously learned masks to exploit previous knowledge when learning new tasks: not only is learning  faster, the algorithm solves tasks that we could not otherwise solve from scratch due to extremely sparse rewards. The results suggest that RL with modulating masks is a promising approach to lifelong learning, to the composition of knowledge to learn increasingly complex tasks, and to knowledge reuse for efficient and faster learning.

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Lifelong Reinforcement Learning with Modulating Masks
Eseoghene Ben-Iwhiwhu
Mee-iuhhiuha@@boro..ccm
Deportment of Computer Science, Loughborough University, UK
Saptarshi Nath
S.noth acc uk
Department of Computer Science, Loughborough University, UK
Praveen K. Pilly
pkpälly@hrl.com
HRL Laboratories, LLC, Malibu, CA, 90265, USA
Soheil Kolouri
soheil.kolouriehwndertt.t edu
Department of Computer Science, Vanderbilt University, Nashwille, TN, USA
Andrea Soltoggio
a.sottogioo@bbbrr.cc..uk
Department of Computer Science, Loughborough University, UK
Reviewed on OpenReview: /ttp::///penreevieee net/ forum? id= V7tahgGr0g
Abstract
Lifelong learning aims to create AT systems that continuously and incrementally learn during a lifetime, similar to biological learning. Attempts so far have met problems, including catas- trophic forgetting, interference among tasks, and the inability to exploit previous knowledge. While considerable research has focused on learning multiple supervised classification tasks that involve changes in the input distribution, lifelong reinforcement learning (LRL) must deal with variations in the slate and transition distributions, and in the reward functions. Modulating masks with fixed backbone network, recently developed for classification, are particularly suitable to deal with such large spectrum of task variations. In this paper, adapted modulating masks to work with deep LRL, specifically PPO and IMPALA agents. The comparison with LRL baselines in both discrete and continuous RL tasks shows superior performance. We further investigated the use of a linear combination of previously learned masks to exploit previous knowledge when learning new tasks: not only is learning faster, the algorithm solves tasks that we could not otherwise solve from scratch due to extremely sparse rewards. The results suggest that RL with modulating masks promising approach to lifelong learning, to the composition of knowledge to learn increasingly complex lasks, and to knowledge reuse for efficient and faster learning.
1 Introduction
Lifelong reinforcement learning (LRL) recent and active area of research that seeks enhance current RL algorithms with the following capabilities: learning multiple tasks sequentially without forgetting previous tasks, exploiting previous tasks knowledge to accelerate learning of new tasks, or building solutions to incrcasingly complex tasks. The remarkable progress of RL in recent ycars, stemming in particular from the introduction of deep RL (Mnih et al. 2013), has demonstrated the potential of algorithms that can learn to act in an environment with complex inputs and rewards. However, the limitation of learning only one task means that such algorithms have a narrow focus, and may not scale complex tasks that first require learning of sub-tasks. is arguable that acquiring knowledge of multiple tasks over a lifetime, generalizing across such tasks, exploiting acquired knowledge to master new tasks more quickly, composing simple

---

Published in Transactions on Machine Learning Research (07/2023)
tasks to solve more complex tasks are necessary to develop more powerful AI systems that better mimic biological intelligence.
LRL shares some of the objectives of lifelong supervised learning (LSL) Pariii et al. 2019 Hadsell et al. 2020
De Lange et al.] ) but due to the unique objective and domains of RL, it has developed into a separate field (Khetarpal et al. 2020). For example, tasks LRL can vary across reward function, input distribution or transition function, while tasks in LSL largely vary in the input distribution. A variety of recently developed approaches in LSL can be placed under three categories: memory methods, synaptic consolidation methods, and parameter isolation propose taxonomy in which approaches can be classified as explicit knowledge retention, leveraging shared structures, and learning to learn. It can be appreciated that LRL exploits most advances in LSL while also developing RL-specific algorithms.
modular methods. In LRL, Khetarpal et al. (2020)
Among the LSL categories mentioned above, parameter isolation methods have shown state-of-thhoare results in classification with approaches such as Progressive Networks (Rusu et al. 2016; Yoon et al.] 2018), Pack (Mallya & Lazebnik] 2018)) , Piggyback (Mallya et al., 2018) and Supermasks (Worssman et al.) 2020 The key concept in such methodologies is to find smart and efficient ways to isolate parameters for each task. In Mallya & Lazebnik (2018), redundancies in large networks are exploited to generate masks via iterative pruning, and thus free parameters to be used only for specific tasks, thus "packing" multiple tasks in one single network. Other approaches instead use the concept of directly learned masks (Zhou et al. 2019
Ramanujan et al., 2020: Wortsman et al.) 2020) to select parts backbone network for each task. Instead of learning the parameters of a backbone network, which are set randomly, masking approaches focus on learning multiplicative (or modulating) mask for each task. Masks can be effective in binary formats, enabling or disabling parts of the backbone network while requiring low memory. Interestingly, masks can be interpreted as modulatory mechanisms with biological inspiration (Kudithipudi et al.) 2022 because they exploit a gating mechanism on parameters or activations,
Surprisingly, while the application of masks has been tested extensively in LSL for classification, very little is known on their effectiveness in LRR., Also, the current mask methods lack the ability of exploit knowledge from previously learned task (forward transfer), since each mask is derived independently. This study introduces the use of dircctly learned masks with policy optimization algorithms, specifically PPO (Schulman et al. 2017) and IMPALA (Espeholt et al. 2018). We explore (1) the effectiveness of masks to maximize lifelong learning evaluation metrics, and (2) the suitability of masks to reuse knowledge and accelerate learning (forward transfer). To demonstrate the first point, we test the approach on RL curricula with the Minigrid environment Chevaller-Boisveert et al.] 2018), the CT-graph (Soltoggio et al., 2019 2023).
Metaworld Yu et a 20200) and ProeGen (Cobbe et | 20200, and assess the lifelong learning metrics (New et al. 2022 Baker et al. 2023) when learning multiple tasks in sequence. To demonstrate the second point, we exploit linear combinations of masks and investigate learning with curricula in which tasks have similarities and measure the forward transfer. The investigation of the second point gives rise to the understanding of the effectiveness of mask knowledge reuse in RL, the initial parameter configuration idcal for mask reuse, and the benefits obtained when task curriculum grow in complexity or contains interfering tasks. The proposed method assumes a task oracle, but the assumption can be eliminated through the introduction of task detection methods into the system as discussed in Section 7 To ensure reproducibility, the hyper- parameters for the experiments are reported in Appendix B The code is published at httpp:///itthu. com/ com/dlpbe//ask-rrl
Contributions. To the best of our knowledge, this is the first study to (1) introduce learned modulating masks in LRL and show competitive advantage with respect to well established LRL methods (2) show the exploitation and the compasition of previously learned knowledge in a LRL setup with learned modulating masks. In addition to a baseline masking approach with independent mask search, we introduced two new incremental mask composition approaches (LC and BLC) with different initialization of linear combination parameters. The analysis reveals different properties in the exploitation of previous knowledge while learning new tasks, either favoring knowledge re-use exploration.
2

---

Published in Transactions on Machine Learning Research (07/ (022))
2 Related work
2.1
Deep Reinforcement Learning (DeepRL)
The use of deep networks as function approximators in reinforcement learning (RL) has garnered widespread
adoption, showcasing results such as learning to play video games MMnih ct al. 2013 Van Hasselt et al.
2016
Shao el. al. 2018 Kempka et. al.) 2016 Mnih et al.) 2016 Babaeizadeh et al. 2017; Espeholt et al.
2018). and learning to control actual and simulated robots (Lillicrap et al. 2016
Schulman et al. 2015
2017
Fujimoto et al., 2018 Haarnoja et al., 2018). These algorithms enable the agent to learn how to solve single task in given evaluation domain. In a lifelong learning scenario with multiple lasks, they sulfer from lifelong learning challenges such as catastrophic forgetting. Nevertheless, they serve the basis for lifelong learning algorithms. For example, DQN (Mnih et al.) 2013) was combined with the elastic weight consolidation (EWC) algorithm to produce a lifelong learning DQN agent (Kirkpatrick et al., 2017 Kessler
et al. 2022).
2.2 Lifelong (Continual) Learning
Several advances have recently been introduced in lifelong (continual) learning ( (Van de Ven & Tolias 2019 Parisi et al.] 2019 Hadsell et al.) 2020; De Lange et al., 2021 Khetarpal et al. 2020), addressing chal- lenges such as maintaining performance on previously learned tasks while learning new task (overcoming forgetting), reusing past knowledge lo rapidly learn new tasks (forward transfer), improving performance on previously learned tasks from newly acquired knowledge (backward transfer), the efficient use of model capacity to reduce intransigence, and reducing or avoiding interference across tasks (Kessler et al., 2022). A large body of work focused on lifelong learning in the supervised learning domain and overcoming the challenge of forgetting (Mendez al. 2022
Lifelong learning algorithms can be clustered into key categories such synaptic consolidation approaches memory approaches
KKrkpatrick et al.] 2017 Zenke et al.
2017: Aljundi et al. 2018
Kolouri et al. 2019).
[opez-Paz & Ranzato 2017: Zeng et al. 2019; Chaudhry et al. 2019 Rolnick et al. 2019 Lin et al., 2022)
modular approaches (Rusu et al. 2016 Mallya & Lazebnik] 2018 Mallya et al. 2018 Wortsman et al. 2020
Mendez et al.] 2022) or a combination of the above. Synaptic consolidation approaches tackle lifelong learning
by discouraging the update of parameters useful for solving previously learned tasks through a regularization
penalty. Memory approaches either store and replay samples of previous and current tasks (from buffer
generative model) during training (Lopez-Paz & Ranzato 2017 Chaudhry et al.] 2019 Guo et al.) 2020:
von Oswald et al., 2020) or project the gradients of the current task being learned in an orthogonal direction to the gradients of previous tasks (Zeng et al.] 2019 Farajtabar et al. 2020 Saha et al. 2021
Lin et al.
2022). Memory methods aim to keep the input-output mapping for previous tasks unchanged while learning a new lask. Modular approaches either expand the network as new lasks are learned (Rusu et al. 2016 Yoon
ett al. 2018 Martin & Pilly (2019) or select sub-regions of a fixed-sized network (via masking) for each tasks [Wortsman et al.
20200. The masks can be applied to (i) the neural representations (Sokkar ct al. 2021:
Serra et al.
2018). ,, (ii) to synapses, where they are directly learned (Wortsman et al. 2020) or derived
through iterative pruning (Mallya & Lazebnik, 2018). The masks can be viewed as a form induced sparsity
in the neural lifelong learner (Von Oswald et al. 2021 Hoefler et al. 2021).
In LRL, the learner is usually developed by combining a standard deep RL algorithm, either on-policy or off-po (for example, DQN, PPO (Schulman et al. (017). or SAC (Haarnoja et al., 2018 ), with a lifelong learning algorithum. CLEAR [Rolnick et al., 2019) is lifelong RL that combines IMPALA with replay method and behavioral cloning. Progress& & Compress (Schwarz et al. 2018) demonstrated the combination of IMPALA with EWC and policy distillation techniques. Other notable methods include the combination of a standard deep RL agent (SAC) with mask derived from pruning in PackNet (Mallya & Lazebnik 2018 Schwarz et al. 2 as demonstrated in the Continual World robolics benchmark (Wolczyk et al. 2021). j. In addition, Mendez et al. (2022) developed an algorithm combining neural composition, offline RL, and PPO to facilitate knowledge reuse and rapid task learning via functional composition of knowledge. To tackle a lifelong learning scenario with interference among tasks, current methods (Kessler al. 2020 (022) employ multi-head policy network (ie., a network with shared feature extractor layers connected different
3

---

Published in Transactions on Machine Learning Research (07/2023)
output layers/heads per task) that combine standard RL algorithm (e.g. DQN or SAC) with synaptic
consolidation methods. In another class of LRI. approach, Kaplanis et al. (2018) demonstrated a lifelong
RL agent that combined DQN with multiple-tine scale learning at the synaptic level (Benna & Fusi 2016).
and the model was adapted to multiple timescale learning at the policy level (Kaplanis et al. 2019 via the combination of knowledge distillation and PPO.
2.3 Modulation
Neuromodulatory processes Avery & Krichmar 2017 Bear et al. 2020 enable the dynamic alteration of
neuronal bohavior by affecting synapses or the neurons connected to synapses. Modulation artificial neural networks (Fellous & Linster 1998 Doya 2002) draws inspiration from modulatory dynamics in biological brains that have proven particularly effective in reward-based environments (Schultz et al. 1997 Abbott & Regehr 2004), in the evolution of reward-based learning (Soltoggio et al., 2008 2018) and in meta RL [Ben-Iwhiwhn et al.) 2022). Modulatory methods in lifelong learning are sct up as masks that alter either the neural act activations (Serra et al.] 2018) or weights of neural networks (Mallya & Lazebnik) 2018
Mallya et al. 2018; Wortsman et al.) 2020 Koster et al., 2022). The key insight is the use of a modulatory mask (containing binary or real values) to activate particular network sub-regions and dcactivate others when solving a task. Therefore, each task is solved by different sub-regions of the network. For each task, PackNet (Mallya & Lazebnik] 2018) trains the model based on available network capacity and then prunes the network to select only parameters that are important to solving the task. A binary mask representing important and unimportant parameters is then generated and stored for that task, while ensuring that important parameters are never changed when learning future tasks. Piggy Back (Mallya et al.) 2018) keeps a fixed untrained backbone network, while it trains and stores mask parameters per task. During learning or evaluation of a particular task, the mask parameters for the task are discretized and applied to the backbone network by modulating its weights. The Supermask (Wortsman et al.) 2020) is a generalization of the Piggy Back method that uses k-winner approach to create sparse masks.
3 Background
3.1 Problem Formulation
A reinforcement learning problem is formalized as a Markov decision process (MDP), with tuple M = (S,A, (S,AA,rr,1) p,),)), where is a set of states, A is a set of actions, : S x A S is a transition probability distribution p(8441/8,, at) of the next state given the current state and action taken at time 4, r S - R is a reward function that produces scalar value based on the state and the action taken at the current time step, ? > J0, 1] is the discount factor that determines how importance future reward relative to the current time step l. An agent interacting in the MDP behaves based on defined policy (either a stochastic æ or deterministic y policy). The objective of the agent is to maximize the total reward achieved, defined as an expectation over the cunulative discounted reward E ('r(s,,))) It achieves this by learning an optimal policy.
In a lifelong learning setup, a lifelong RL. agent exposed to a sequence of tasks T. Given N tasks, the agent is expected to maximize the RL objective for each task in T = (71, 72..,, Tn). As the agent learns improve (backward
one task after another in the sequence, it is required to maintain (avoid forgetting) transfer of knowledge) performance on previously learned tasks. A desired property of such an agent is the ability to reuse knowledge from previous tasks to rapidly learn the current or future tasks.
3.2 Modulating masks
The concept of modulating masks has recently emerged in the area of supervised classification learning MMallya et al. 2018 Serra et al.] 2018 Ramanujan et al. 2020: Wortsman et al. 2020 Sokar et al.] 2021 Koster et al. 2022). Masks work by modulating the weights or the neural representations of a network, and they can be directly learned (optimized) or derived via iterative pruning (Hoefler et al.) 2021), with a goal of implementing sparsity in the network. The learned mask approach employed in this work, and masks are used to modulate the weights of network. By modulating the weights, sub-regions of the network
4

---

Published in Transactions on Machine Learning Research (07/2023)
is activated, while other regions deactivated. Given network with layers 1,... L, with each layer containing parameters W E [Rmxn, for each layer I, score parameter S E IR is defined.
During forward pass (in training or evaluation), a binary mask M' E 1 is generated from S' based on an operation gSS)) according to one of the following: (i) a threshold e (where values greater than e are set to 1, otherwise 0) (Mallya et al. 2018). or (ii) top-k values (the top k% values St yields M while the al. 2020), or (iii) probabilistically sampled from Bernoulli distribution, where the p parameter for the distribution is derived from the sigmoid function a(S") (Zhou et al. 2019). An 0, 13) from 5" introduced in Koster et al. (2022). The binary or ternary mask modulates the layer's paramcters (or weights), thus activating only a subset of the W
rest are set to 0) [Ramanujan
alternative approach is the generation of ternary masks (ie.,, with values {,
Given an input sample x E R", a forward pass through the layer is given as f(x, W w') = ( @ M') x, where the element wise multiplication operation. Only the score parameters St are updated during training, while the weights W are kept. fixed at their initial values. For brevily, the weights and scores across layers denoted W and S.
The training algorithm updating S depends on the operation used to generate the binary mask. When the binary masks are generated from Bernoulli sampling, standard backpropagation is employed. However, in the case of thresholding or selecting top k%, an algorithm called edge-popup is employed, which combines backpropagation with the straight-through gradient estimator (i.e., where the gradient the binary mask operation is to identity to avoid zero-value gradients) (Bengio et al., 2013 Courbariaux al. 2015).
In lifelong learning scenario with multiple tasks, for each task тk score parameter Sy learned and used to generate and store the mask Mr. During evaluation, when the task is encountered, the mask learned for the task is retrieved and used to modulate the backbone network. Since the weights of the network are kept fixed, it means there is no forgetting. However, this comes at the cost of storage (i.e., storing mask for each task).
4 Methods
Following, we We first introduce the adaptation of modulating masks to RL algorithms (Section 3.2 hypothesize that previously learned masks can accelerate learning of unseen tasks by means of a linear combination masks (Section 4.2). Finally, we suggest that continuous masks might be necessary to solve RL tasks in continuous environments (Section 4.3
4.1 Modulatory masks in Lifelong RL
We posit that stochastic control policy 5a, be parameterized by 0, the weights of a fixed backbone network, and & [ol....o ... ..... set of mask score parameters for tasks 1...k k. du are the scores of all layers of the network for task k, comprising the layers 1... L, i.c., Φ = S..) The weights 0 of the network are randomly initialized using the signed Kaiming constant method (Ramanujan el. al. 2020), and are kept fixed during training across all tasks. The mask score parameters are trainable, but only de is trained when the agent is exposed to task k. To reduce memory requirements, Wortsman et al. (0220 discovered that masks can be reduced to binary without significant loss in performance. We test this assumption by binarizing masks using element-wise thresholding function
g(ox) = {
=
Φ >
(1)
otherwise
where e set 0. The resulting LRL algorithm described in Algorithm [ To assess this algorithm, we paired it with the on-policy PPO (Schulman al.) 2017) algorithm and the off-policy IMPALA (Espcholt et al. 2018) algorithm.
5

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm Lifelong RL Algorithm with modulating masks
Require: Number of tasks N, maximum training steps per task P
Require: Rollout length (steps) z
1: Initialize policy N0,0
2: for x=+1. K..... do
Set task
Set steps 0
while step < P do
6:
Rollout experiences Update steps + steps + 2
ri, (1;)) (8z: U2, K()) using no,ou
7:
8:
Compute loss C based on an RL algorithm objoctive.
9.
Compute gradients Vo (TT, with respect to Øs.
10: Update mask score parameter for task k, or t Φx - a Vo La (T(
11: end while
12: Store mask score Φx or the binary mask g(ok)
13:
end for
4.2 Exploiting previous knowledge to learn new tasks
One assumption in LRL, often measured with metrics such as forward and backward transfer (New et al. 2022), is that some tasks in the curriculum have similarities. Thus, previously acquired knowledge, stored in masks, may be exploited to learn new tasks. To test this hypothesis, we propose an incremental approach to using previously learned masks when facing a new task. Rather than learn a large number of masks and infer which mask useful for solving a task at test time via gradient optimized linear combination, Wortsman et al. (2020), instead start to exploit previous knowledge from any small number of masks combined linearly plus trainable random mask ok+1 ... S.+1) The intuition is that strong linear combination parameters can be discovered quickly if similarities are strong, otherwise more weight is placod on the newly trained mask.
in
A new mask at layer is given by
s/ = S
B
(2)
where + E Φk+1 are the scores of layer I in task k + 1, St.te denotes the transformed scores for task k + after the lincar combination (Ic) operation, s denotes the optimal scores for previously Icarned task i, and B4....B441 are the weights of the linear combination (at layer 1). To maintain normalized weighted sum, a Softmax function is applied to the linear combination parameters before they are applied in Equation P When no knowledge is present in the system, the first task is learned starting with random mask. Task two is learned using 31 = 0.5, weighting task onc's mask, and B2 = 0.5, weighting the new random mask. The third task will have 3 32 Ba = 0.33, and so on. Note that 8e denotes vector of size L which contains the co-efficient parameters for task k across all L layers of the network (i.e., 3 = ( 3)..
In short, two approaches can be devised. The first one in which each mask is learned independently of the others. Experimentation of this approach will determine the baseline capabilities of modulating masks LRL. We name this Mask Random Instialization (Maskra). The sccond approach attempts to exploit the knowledge acquired so far during the curriculum learning by using a linear combination of masks to learn a one. Experimentation of this second approach will determine the capabilities of modulating masks to exploit previously learned knowledge. We name this second approach Mask Linear Combination (MaskLc). The idea is graphically summarized in Figure !
It can be noted that, as the number of known tasks increases, the relative weight each mask decreases in MaskiØ- This could be a problem, particularly as the weight of the new random mask reducod, possibly biasing the search excessively towards an average of previous policies. Therefore, we introduce third approach that attempts to combine the benefits of both MAskRI and MaskıG: we set the initial weight of

---

Published in Transactions on Machine Learning Research (07 (022))
M
task:
task 2
mask 1: (
backbone
task
musk 2: (olgg
backhone
PPO/IMPALA Optimizer
MASK G
task
task
task
mask ke Col
mask CO0
mask (49g)
random:: (4))
backhone (0)
task K+1
By
B
+
P
Figure 1: Graphical representations of the methods Masknı (top) and Maskie (bottom). Maskıa searches for mask for each new task independently. Maskıc attempts to exploit previous knowledge to learn a new task: known masks are linearly combined with new randomly initialized mask while learning new task. Gradient updates search for the parameters $2.... 3n and the new random mask.
the new random mask to 0.3, while the remaining 0.5 weight is shared by the masks all known tasks. We name this third approach Balanced Linear Combination (MaskBLc). must be noted that the difference between Maskic and MaskELe is only the initialization of weights when a new task is encountered, where Be+1 1/(kk 1) in Maskı.c and 3x+1 0.5 in MaskmLc- However, the parameters 3 can be modified arbitrarily by backpropagation during training.
For both Maskic and MaskBLC, updates are made only to S and Tg...Kk41 across each layer l. After the training on task +++ is completed, the knowledge from the linear combination is consolidated into the scores for the current task 1 by applying Equation Therefore, the other masks and the linear combination parameters are no longer required. Algorithm 2 reports the forward pass operations for Maskj
Algorithm 2 Forward pass in network layer 1 in Maskic
Require: Number of task learned so far k.
Require: Si ...S Sk+1, 3 B841, W
1: procedure FORWARDPASS(x)
2: <- then
3: Set S
4: else
Compute the task score from linear combination: P BR+1S441
end if
Compute mask from score MR+1 g(P) Modulate weight Wrmod +- (W @ Mc+1)
8:
9: Compute output + W/mad, x
10: return y
11: end procedure
4.3
Continuous Modulatory Masks for Continuous RL problems
In previous studies, binarization masks was discovered be effective in classification to reduce the memory requirement and improve scalability. As shown in simulations, this approach was effective also in discrete

---

Published in Transactions on Machine Learning Research (07/2023)
RL problems. However, in continuous action space RL environments, we discovered that binary masks did not lead to successful learning. It is possible that the quantization operation hurls the loss landscape in such environments since the input-output mapping is continuous to continuous values. Therefore, a modification of the supermask method can be devised to support the use of continuous value masks. In the thresholding mask generation operation (given a threshold e), the modified version becomes
{ @RRR 7100
Okffis) >E
g(
otherwise
Such a modification still maintains the ability to learn sparse masks, but replaces the unitary positive values with continuous values. The results of the empirical investigation of the binary and continuous masks in a continuous action space environment is reported in Section 5.4
5 Experiments
The three novel approaches, MaskRı, Maskic and MaskBLC; are tested on a set of LRL benchmarks across discrete and continuous action space environments. A complete set of hyper-parameters for each experiment is reported in Appendix B
First, we employed the ProcGen benchmark (Cobbe et al. 2020) that consists of a set of video games with high diversity, fast computation, procedurally generated scenarios, visual recognition and motor control. The masking methods were implemented with IMPALA: Mask + IMPALA Espeholt et al. 2018) and tested following the ProcGen lifelong RL curriculum presented in Powers et al. (002) (a subset of games in ProcGen). The properties of the benchmark make the curriculum challenging (c.g., high dimensional RGB observations and procedural generations of levels). The currículum was designed to test generalization abilities: for each task, the lifelong evaluations are carried out using procedurally generated levels that are unseen by the agents during training. The masking methods were compared with baselines such as online EWC, P&cC (Schwarz et al. 2018). CLEAR (Rolnick et al., 2019 and IMPALA.
With the aim of understanding the learning mechanism and dynamics of the masking approaches, other benchmarks were chosen to assess the robustness the method against the following aspects: discrete and continuous environments; variations across tasks in input, transition and reward distributions, The CT CT-graph (Soltoggio et al. 2019 2023) (sparse reward, fast, scalable to large search spaces, variation of reward), Mini-
grid -ChevalierBoiiveett et al., 2018 2023) (variation of input distributions and reward functions) and Con- tinual World @Woteyyk et al. 2021) ) (continuous robot-control) were used to assess the approaches, with PPO serving as the base RL algorithm. In these benchmarks, the masking approaches (Maskuu/Lc/BLCC + PPO) are compared with a lifelong learning baseline, online EWC multi-head (EWCMH + PPO), and with the non- lifelong learning algorithm PPO. Experiments with a PPO single task expert (STE) were also conducted to enable the computation of the forward transfer metric for each method, following the setup in Wolezyk et al.. (2021b. Control experiments with EWC single head, denoted EWCSH, performed poorly as a confirmation report those results in the Appendix
that our benchmarks contain interfering tasks (Kessler et al. 2022
The metrics report a lifelong evaluation across all tasks at different points during the lifelong training, computed as the average sum of reward obtained across all tasks in the curriculum. The area under the curve (AUC) is reported corresponding tables. A forward transfer metric, following the formulation employed in Wolczyk et al. (2021). ) is computed for the CT-graph, Minigrid and Continual World. For cach (ask, the forward transfer computed as the normalized difference between the AUC of the training plot for the lifelong learning agent and the AUC for the reference single task expert. For the ProcGen experiments, the lifelong training and evaluation plot format reported in Powers et al. (2022) was followed to enable an easier comparison with the results in the original paper. As tasks are learned independently of other tasks in MaskRI, there is no notion of forward transfer in the method. Therefore, the method is omitted when forward the transfer metrics are reported.
The results presented in the evaluation plots and the total evaluation metric reported in the tables below were computed as the mean of the seed runs per benchmark, with the error bars denoting the 95% confidence
8

---

Published in Transactions on Machine Learning Research (07 (07/2023)
interval. The CT8, CT12, CT8 multi depth, and MG10 results contained 5 seed runs per method, while the CW10 results contained 3 seed runs due to its high computational requirement. While the sample size for the evaluation metric is the number of seed runs, the sample size used for computing the mean and 95% confidence intervals for the forward transfer metric and the training plots the number of seeds multiplied by the number of lasks per curriculum (i.e. 40, 60, 40, 50, for CT8, CT12, CT8 mulli depth, MG10, and CW10 respectively). significance test was conducted on the results obtained validate the performance difference between algorithms. The result of the test is reported in Appendix A
5.1 ProcGen
The experimental protocol employed follows the setup of Powers al. (0022), with sequence of six tasks (0 Climber, 1 Dodgeball, 2 Ninja, 3 - Starpilot, 4 Bigfish, 5 Fruitbot) with five learning cycles. Screenshots of the games are reported in the Appendix (D4)). Several enviromment instances, game levels, and variations in objects, texture maps, layout, enemies, etc., can generated within a single task. For each task, agents are trained on 200 levels. However, the evaluation carried out on the distribution of all levels, which combination of levels seen and unseen during training.
IMPALA used the base RL optimizer which deployed the novel masking methods. The results are reported in Figure following the presentation style used in Powers el al. (2022). The masking methods
O-Climber
1 1-Dodgeball
2-Ninja
P&C
Expected Return
Expected Return
2
Expected Return
ONLINE EWC
CLEAR
1.5
MASK
0.75
MASKLC
0.5
MASKRI
0.5
0.25
IMPALA
30M
BOM
90M
120M
150M
33M
60M
9OM
120M
150M
30M
60M
5OM
120M 150M
Step
Step
Step
3- 3-Starpilot
4- 4-Bigfish
5-Fruitbot
10
3
Expected Return
Expected Return
Expected Return
2
1
30M
60M
9OM
120M
150M
30M
BOM
90M
120M
150M
-bb
30M
60M
90M
120M
150M
Step
Step
Step
Figure Evaluation results in the ProcGen environment tasks, cycles), measured as average across runs. The solid line represents evaluation unseen environments, while the dotted line represents evaluation on the training environments. The gray shaded rectangles show what point time an agent
trained on each task.
(Maskuı, Maskic, and MaskELC) show better performance with respect to other basclines across most tasks, while maintaining generalization capabilities across training and evaluation environments. As the tasks are visually diverse, possibly resulting less similarity across tasks, reusing previous knowledge may not offer much advantage. Nevertheless, the evaluation performance for each method reported in Table [ illustrates significant advantage of the masking methods with respect the basclines, particularly in the test tasks, where MaskıÆ 44% better than the closest runner up (CLFAR) and over 300% better than P&C.

---

Published in Transactions on Machine Learning Research (07/2023)
Evaluation Performance
Method
Train Tasks
Test Tasks
IMPALA
3687.57 + 1194.23
2818.44 = 1214.34
Online EWC
4645.87 + 1963.42
3831.94 = 1258.38
P&C
3403.59 + 3428.37
3390.97 = 1795.58
CLEAR
11706.34 = 5271.45
8447.64 = 2286.21
MaskRI
12462.82 + 1057.68
12222.10 = 6064.18
Maskic
12488.31 + 3416.90
12377.77 + 1440.80
MaskBLC 11683.21 + 3172.61
11913.48 + 3869.95
Table 1: Total evaluation return (AUC of the lifelong evaluation plat in Figure 2 on the train and test tasks in ProcGen curriculum. Mean 93% confidence interval reported.
CT8
CT8
1.0
Average Return
6-
Average Retur
0.8
PPO
EWG
0.4
EWCGn:
MASKi
MASK
0.2
MASK
MASKaic
MASKatc
0.0
0.0
0.4
0.6
0.8
1.0
Steps
Steps
Figure 3: Performance in the CT8 curriculum. (Left) Lifelong evaluation performance on all tasks (mean and 95% confidence interval 5 seeds/samples). (Right) Training performance each task, measured the average return across all tasks and seeds rums (mean and 95% confidence interval on 8 lasks and seeds, ie., 40 samples). In the lifelong evaluation, clear difference be noted between the lifelong learning algorithms and the non-lifelong learning algorithm PPO, which is therefore not reported in the training plot (Right). The numerical values for the AUC reported in Tables The training curves show how MaskBLC followed by MaskıC and MaskaI. The performance on each individual task is reported in the Appendix in Figure I9
average faster learning new tasks,
5.2 CT-graph
Each instance of the CT-graph environment contains start state followed by number of states (2D patterned images) that lead to leaf leaf states and rewards only with optimal policies. A task defined by setting one of the leaf states as a desired goal state that can be reached only via one trajectory. Two experimental setups were employed: the first with leaf states (reward locations) that serves for 8 tasks, denoted as CT8 curriculum (depth-3, breadth-2 graph with 23 leaves). In the second sctup, two graph instances with and 8 different reward locations, with depth 2 and 3 respectively, result in combined curriculun of 12 tasks, denoted CT12. Such tasks have levels of similarities due to similar input distributions, but also interfere due to opposing reward functions and policies for the same inputs. Additionally, the 8-task graph has a longer path to the reward that introduces variations in both the transition and reward functions. Graphical illustrations of the CT-graph instances are provided in Appendix D
Each task is trained for 102.4K time steps. FiguresBandd 4 report evaluations the CT8 and CT12 curricula. as agents are sequentially trained across tasks. The forward transfer and the total evaluation performance metrics presented in Tables and respectively.
10

---

Published in Transactions on Machine Learning Research (07 (07/2023) /023)
CT12
CT12
1.0
Average Return
Average Return
0.8
PPO
EWC
EWCu
MASKa
MASKel
2
MASK,c
0.2
MASKau
MASKets
0.0
0.4
0.6
0.0
0.6
Steps
Steps
Figure 4: Performance in the CT12 curriculum. (Left) Lifelong evaluation performance on all tasks (mean and 95% confidence interval seeds/sans /samplss). (Right) Training performance on each task, measured as the average return across all tasks and seeds runs (mean and 95% confidence interval on 12 tasks and 5 seeds, i.e., 60 samples).
Method
Total Eval
Fwd Trnsf.
PPO
298.69 +:2 24.12
-0.40 0.33
EWCMH
959.38 = 141.32
-1.04 = 0.32
MaskRI
1379.71 + 51.86
MaskLc
1358.08 + 75.29
-0.25 + 0.30
MaskBLC
1405.30 + 62.49
0.27 0.14
Table 4: Total evaluation return (AUC of the lifelong evaluation plot in Figure Teft)) and forward transfer in the MG10. Mean + 95% confidence interval reported.
Method
Total Eval
Fwd Trnsf.
Method
Total Eval
Fwd Trnsf.
PPO
147.00 = 13.63
0.15 + 0.20
PPO
378.40 = 22.27
-0.15 0.18
F.CLи
661.20 + 64.14
-0.23 + 0.19
FWCMи
1565.00 + 92.36
-0.11 0.14
MaskRI
701.20 + 6.11
MaskRI
1535.00 = 10.20
MaskLc
702.00 + 5.89
0.46 + 0.13
MaskıC
1544.60 + 4.17
0.50 + 0.08
MaskBLC
716.80 + 2.22
0.67 + 0.06
MaskBLC
1558.20 + 3.09
0.65 + 0.04
Table 2: Total evaluation return (AUC of the life- long evaluation plot Figure 3 Left)) and forward transfer during lifelong training in the CT88. Mean + 95% confidence interval reported.
Table 3: Total evaluation return (AUC of the life- long evaluation plot Figure Left)) and forward transfer during lifelong training in the CTI2. Mean + 95% confidence interval reported.
The plots show that the masking methods (MaskRI, MaskıC, and MaskELC) are capable of avoiding for- getting and obtain high evaluation performance, with significantly better forward transfer in comparison to EWCMи and PPO. On average, the Maskıc approach recovers performance faster than MaskRı and MaskBLC performs best. An expanded version of the training plots showing the learning curves per tasks and averaged across seed runs is reported in the Appendix F.2
5.3 Minigrid
The experiment protocol employs curriculum of ten tasks (referred to MG10), which consist of two variants of cach of the following: SimpleCrossingSNNI, SimpleCrossingS0N2. SimpleCrossing9NN. LavaCrosiingS9N1, LavaCrossingS9N2. Screenshols of all tasks are reported in the Appendix (D.2 The variations across tasks include change in the state distribution and reward function. The results for the MG10 experiments are presented in Figure and Table ! The masking methods obtained better performance comparison the baselines, with MaskBLC obtaining the best performance. Appendix F.2 provides full experimental details.
11

---

Published in Transactions on Machine Learning Research (07 (07/202)) /003)
MG10
MG10
1.0
Average Return
Average Return
0.8
PPO
EWCN
EWCU
MASKa)
MASKel
MASK,c
0.2
MASKauL
MASK
0.0
1.0
1.5
0.0
1.5
2.5
Steps
Steps
Figure 5: Performance in the MC10 curriculum. (Left) Lifelong evaluation performance on all tasks (mean and 95% confidence interval on seeds/samples). (Right) Training performance on each task, measured as the average return across all tasks and seeds runs (mean and 95% confidence interval 10 tasks and 5 seeds, i.e., 50 samples).
CW10
1.0
Avera Peraage success Rate
5
Average Success Rate
0.6
EWCNI
EWC
2-
MASKa,
0.4
MASK
MASKi_
MASKiL
MASKic
0.2
MASKi:
MASKaLc
MASKiu:
0.0
0.0
0.2
0.4
1.0
0.0
0.2
0.4
0.6
1.0
Steps
Steps
Figure 6: Performance the CW10 curriculum measured using the success rate metric. (Left) Lifelong evaluation performance all tasks (mean and 95% confidence interval on 3 seeds/sam ssads/ssselss............ (Right) Training performance each task, measured as the average success rate across all tasks and seeds runs (mean and 95% confidence interval 10 tasks and 3 seeds, i.e., 30 samples).
5.4 Continual World
We evaluated the novel methods in robotics environment with continuous action space, the Continual World /Wolezyk et al., 2021). The environment was adapted from the MetaWorld environment YYu et al. 2020) that contains 50 classes of robotics manipulation tasks. The CW10 curriculum consists of 10 robotics tasks: visual screenshots are provided the Appendix (D.3). The results for all methods were measured using the success rate metric introduced in Yu al. (2020), which awards if an agent solves a task 0 otherwise. For the masking methods in this curriculum, the standard quantization of masks into binary performs poorly. To demonstrate this, two variants are the standard setting, where binary mask is derived from the scores, denoted as Masknı
and another where a continuous mask is derived from the scores (discussed in Section 4.3). denoted as MaskRL_C- The results from Figure G and Table 5 show that Maskjı C performs significantly better than Maskuı D. Motivated by the results, the lincar combination of masks method Maski.C presented for this curriculum also employed the use of continuous masks. Maskı, and MaskBLC performed markedly better than the baseline EWCMи that appears to struggle on this benchmark. Appendix F22 reports the dctails for all methods.
6 Analysis
The results of the previous section prompt the following questions: what linear coefficients emerge after learning? How rapid learning in MaskıÆ achieved? How is knowledge reused?
12

---

Published in Transactions on Machine Learning Research (07/2023)
Method
Total Eval
Fwd Trnsf.
PPO
53.83 + 72.32
-4.06 + 2.58
EWCMH
8.60 + 12.91
-7.39 = 3.76
Maskji D
20.45 = 51.46
MaskRI_C
246.83 + 157.39
Maskic 272.43 = 124.92
-0.33 = 0.36
MaskBLC
237.00 + 167.25
-0.61 + 0.52
Table 5: Total evaluation success metric (AUC of the lifelong evaluation plot in Figure 6 Left)) and forward transfer in the CW10. Mean = 95% confidence interval reported.
CTB
MG10
CW10
Laver
Layer (lctor)
T3
6.14
3.5
820
021
8.00
4.5
0.38
532
14
515
814 E35
16 5/4
C32 914 0311
037
512
0.3 525 314 61
642 9:09 065 6:4
15-032
835
912
631
011
9:1
646
911 008
Loyer Outoot
Outout
024
4.21
815
326 015 631
035 632
028
644 9:22
637
836
314 033 537 512
17 01037 832 015 635 114 034 812
17 81 X31
022 850 0.8 6:5
8:2
033 635 313 632 133 0.3
5:11 923
640
022 600 6:31
624
834
314 0M 535 511 AM
812 012 C3
632
006 6.2
6:2
822
312 000 01 E1
6.3
Figure 7: Coefficients 3 in Maskıc after training the CT8, MG10, and CW10 curricula. Each row represents a task, and the values (which sums to 1) in it are the final set coefficients after training on the task: the higher the value of cell, the higher the level of importance of the correspouding mask in the linear combination operation. The figure only shows coefficients for the first layer and the output (actor) layer. See Appendix E.2 for plots of all other layers.
Another interesting question that arisc is the training efficiency derived from knowledge reuse in mask methods. The training plots in Figures 3 Right), 4 Right), 5 Right), and C Rightt showed that the mask methods required fewer training steps average to learn tasks in comparison to the baselines. For interested readers, Appendix reports the analysis conducted on the time (i.e., training steps) taken to reach certain level performance per task.
6.1 Coefficients for the lincar combination of masks
To validate whether the proposed linear combination process autonomously discover important masks that are useful for the current task, visualization of the co-officients after learning each task is presented.
Figure 7 presents the visualization of the coefficients (for the input and output layers) for a MaskiC agent trained in the CT8, MG10, and CW10 curricula respectively. In each plot, each row reports the final set of coefficients after training on a task. For example, the third row in cach plot represents the third task in the curriculum and reports three coefficients used for the linear combination of the masks for two tasks and the mask. For the first task (first row), there previous masks to combine.
13

---

Published in Transactions on Machine Learning Research (07/2023)
For the CT8 and MG10, the plots show that the coefficients have similar weights for each task (i.e., row wise in Figure 7. This observation is consistent across the layers of the network (see Appendix E.2 for plots across all layers). The uniform distribution across co-efficients means that the knowledge from all previous tasks is equally important and reused when learning new tasks, possibly indicating that tasks are equally diverse or similar to each other. The knowledge reuse of previous tasks thus accelerates learning and helps the agent quickly achieve optimal performance for the new task. For example, in the CT-graph curricula where navigation abilities are essential to reach the goal, the knowledge on how to navigate/traveryy to different parts of the graph is encoded in each previously learned task. Rather than re-learn how to navigate the graph in each task, and subsequently the solve the task, the agent can leverage on the existing navigational knowledge. The performance improvement therefore comes from the fact the agent with knowledge reuse can leverage existing knowledge to learn new tasks quickly. Note that we will not expect any performance improvement if the new task bears no similarity with the previously learned task.
Comparing the values across layers (i.c., column wise in Figure 7. we note that there is little variation. In other words, the standard deviation of each vector 33r is low, as all values are similar. From such an observation, it follows that the vector 3 could be replaced by scalar for these two benchmarks.
A different picture emerges from the analysis of the coefficients in the CW10 curriculum (Figure 7 right- most column). Here the coefficients appear to have a larger standard deviation both across masks and across layers. Particular values may suggest relationships between tasks. E.g. the input layer coefficient B] (from layer 1, mask 2) is high in task 4 and 7. Similar diverse patterns can be seen in the output layer. We speculate that tasks in CW10 are more diverse and the optimization process is enhancing specific coefficients to reuse specific skills. The non-uniformity of the co-efficients in the analysis could be consequence of different levels of task similarities as reported in the forward transfer matrix in Wolczyk et al. (0021) p for the CW10.
6.2 Exploitation of previous knowledge
The linear combination of masks appears to accelerate learning significantly as indicated in Figures 3 Right), 4 RRghht), Right), and 0 Right). To investigate the causes of such learning dynamics, we plot the proba- bilities of actions during an episode in a new task. The idea is to observe what are the softmax probabilities that are provided by the linear combination of masks at the start of learning for a new task. A full analysis would require the unfeasible task of observing all trajectories: by measuring probabilities when traversing such optimal trajectory.
therefore focus on the optimal trajectory
Figure [ shows the analysis for the following cases: facing a 4th task after learning 3 tasks in the CT8 benchmark; facing the 12th task after learning 11 tasks in the CT12 benchmark; facing the 7th task after learning 6 tasks in the MG10 benchmark. The chosen task for cach curriculum is set to test different instances of knowledge reuse. For CT8, the 4th task tests the agent's ability to reuse knowledge after learning a few similar tasks, while the 12th task in CT12 investigates the agent's behavior after learning many tasks. In MG10, the input distribution of the 7th task differs from the previous ones (i.c., from tasks with no lava to tasks with lava), thus testing the agent's 's ability to reuse previously learned navigation skills while dealing with new input scenarios. The result of the analysis is generalizable to other task changes in the curricula.
In the analysis, EWCми produced a purely random policy, with a uniform distribution over actions for each time step. Despite learning previous tasks, the new random head results in equal probabilities for all actions. On the contrary, both Maskic and MAskELC use previous masks to express preferences. In particular, in the CT8 and CT12, the policy at steps 1, 3, 5 and coincides with the optimal policy for the new task, likely providing an advantage. However, at steps 4 and 6 for the CT8, and 2, 4, and 6 for the CT12, Maskic has markedly wrong policy: this is due to the fact that the new task has a different reward function and is therefore interfering. Duc to the balanced combination of previous knowledge with a new mask, MaskBLC seems to strike the right balance between trying known policies and exploring new ones. Such a balanced approach is also visible in the MG10 task. Here, task 7 (see the Appendix D.2) consists of avoiding the walls and the lava while proceeding ahead, then turning left and reaching the goal: most such skills are similar to those acquired in previously seen tasks, and therefore task 7 is learned starting from a policy that is close to being optimal.
14

---

Published in Transactions on Machine Learning Research (07/2023)
CT8, task 4 (new task)
CT12, task 12 (new task)
MG10, task (new task)
C:77
3.00
0:00
1:00
0:21
1.00
0.21
1.00
0.22
0.35
1:33
0.89
1:02
0.28
1:00
3:44
1:00
2.03
Ao
0.00 0.00 0.00
0.00 9,50 9.41 9.07
0.96 0.08 0:47 0,05 0.92 0,85 9.19
MASKIC
6:00
0.23
0.90
0.33
0.02
0.99
0.02
0.02
0:00
3.55
0.19
0.09
0.00
0.00 9.50 9.60 9.00 6.01 0.00 0.00 0.00 0.00 0.50 9:00
C:00
6:00
0.00
0.00
0.24
0.03
0.34
0.33
0.85
0.33
0.23
0.23
0.22
0.23
0.02
233
0:00
3.44
0.61
1.00 1.05
1.00 1.50 0.59 0.53
0.03 0.92
0:52
0.95 0.98 0.95 0.51
MASKFIC
0:57
0:57
0.37
07
0.58
A2
0.90
0.00
0.41
3.54
2.00
6:00
1:91
1022
00
3.06
0:45
A0
0.00
0.00
0.00
0.00 0.50 5.00 0.00
00 0.00 9.00
Optimal Policy
A2
0.38
0.39
1:02
0.52
0.32
0102
0.33
0:02
03.11
9:35
3.00
0:00
0.93
0.73
0.50
3:00
0.00
0:35
A1-1:00
0.00
0
0.00 0.50 6.00 0.00 00 09 9.00 0.80 0.00 0.80 9:00
A2
1111
0.28 0.02
0.02
0100
0.0
8:32
1.00
3.00
0.00
1.70 0.00
1.00
8:00
1:33
K2
0.00
1.00
1.00
1.00 1.00 1.00 1.00 00 1.00 1000 1.00 1.00 1.00 1.00
Steps
Steps
Steps
Figure 8: Knowledge reuse when learning a new lasks. The softmax output probabilities are shown during an episode with unforeseen lask (lask 4 in CT8, task 12 in CT12, and lask 7 in MG10) the agent is guided through an optimal policy. From the top row down, Maskic displays biased probabilities representing an average behavior across previous tasks. MaskBLC shows biased probabilities but contains more randomness in comparison to Maskıc- A visual comparison with the optimal policy (bottom row) suggests that both MaskLC and MaskBLC start learning unforeseen tasks with useful knowledge. Note, EWCMи produced balanced probabilities (uniform distribution) for all actions at each time step, and is thus omitted from the figure.
If Maski. and MaskaL.C are capable of exploiting previous knowledge, it is natural to ask whether such knowledge can be exploited to learn increasingly more difficult tasks. The CT-graph (Soltoggio et al. 2019) environment allows for increasing the complexity by increasing the depth of the graph. In particular, with a depth of 5, the benchmark results in a highly sparse reward environment with a probability of getting a reward with a random policy of only one in 3 177, 147 episodes (Soltoggio et al.) 2023). We therefore designed a curriculum, the CT8 multi depth, composed of a set of related but increasingly complex tasks with depth 3, 4 and 5 (two tasks cach depth with a different reward function).
Figure g shows the performance in the CT8 multi depth curriculum. EWCxH was able to learn the first 4 tasks with depth 2 and 3, but failed to learn the last 4 more complex tasks. Interestingly, MaskBLC managed to learn task 5 and 6 only partially. Maskic was able to learn all tasks, demonstrating that it could reuse previous knowledge lo solve increasingly difficult tasks. Figure 10 presents the training performance for each individual task, highlighting were most methods fail in the curriculum.
MULTI DEPTH
MULTI DEPTH
8-
1.0
Average Return
6.
Average Return
0.6
PPO
EWC MASK,,
0.4
EWCu
2-
MASKi
MASK.
0.2
MASK
MASKaLc
MASKauc
0.0
3
5
0.0
0.2
0.4
0.6
0.8
1.0
Steps
Steps
Figure 9: Performance in the CT8 multi depth curriculum. (Left) Lifelong evaluation performance on all tasks (mean and 95% confidence interval on 5 seeds/samples). (Right) Training performance each task, measured the average return across all tasks and seeds runs (mean and 95% confidence interval on 8 tasks and 5 seeds, i.c. 40 samples). Excluding MASKie, the training performance for other methods are sub-optimal, due to the failure to solve later tasks in the curriculum.
15

---

Published in Transactions on Machine Learning Research (07 (272))23)
CT8 DELTH
1.0
Average Return
0.8
0.6
0.4
MASK
0.2
MASKic
MASK
0
2
6
Steps
3d5
Figure 10: Training performance on each task, measured as the average return across all seeds runs (mean and 95% confidence interval 5 geeds/samples, expanded from Figure 9 Right).
Method
Total Eval
Fwd Trnsf.
PPO
237.80 + 115.65
0.05 = 0.11
EWCMH
540.00 + 39.88
0.10 = 0.08
MaskRI
533.00 + 41.13
MaskIC
719.60 0.68
0.79 = 0.06
MAskBLC
644.00 + 41.47
0.48 0.11
Table 6: Total evaluation return (AUC of the lifelong evaluation plot in Figure 0 Left)) and forward transfer during lifelong training in the CT8 multi depth. Mean = 93% confidence interval reported.
7
Discussion
The proposed approach employs learned modulatory masking methods to deliver lifelong learning perfor- mance in a range of reinforcement learning environments. The evaluation plots (the left panels of Figures and G show how the performance increases as the agent learns through a given curriculum. The monotonic increase, indicating minimal forgetting, most clear for the CT-graph and Minigrid, while the Continual World appears have a more noisy performance. The FWCMH baseline algorithm shows to be highly performing bascline in the CT8 and CT12 curricula, it performs less well in the MG10 curriculum, and poorly in the CW10 curriculum. The masking methods, instead, perform consistently in all benchmarks, with the linear combination methods, Maskıс and MaskBLC. showing some advantage over the random ini- tialization Maska- Evaluations on the ProcGen environments, while noisy to interpret from Figure 2 reveal that the masking methods out outperform IMPALA, Online EWC, P&C and CLEAR by significant margins (Table D.
While the learning dynamics of the core algorithm MaskRI indicate superior performance the basclines, focused in particular two extensions of the algorithm, Maskic and MaskøLC- These use linear combination of previously learned masks to search for the optimal policy in new unforeseen task. These two variations combine previously learned masks with new random mask to search for the optimal policy. One catch with this approach is that the performance on a new task will depend on which and how many tasks were previously learned. However, this is a property of all lifelong learning algorithms that leverage on previous knowledge. The balanced approach MaskELC starts with a 0.5 weight on the new random mask and can be seen as blend between the random initialization MaskRI and the linear combination MaskLe. On average, it appears achieve slightly better performance than either MaskRı MaskıC. The standard linear combination MaskLC was the only algorithm able to learn the most difficult task on the CT-graph. This suggests that the algorithm is capable of exploiting previous knowledge to solve challenging RL problems. The fact that both MaskLę and MaskBLC have superior performance to the core random initialization MaskRI validates the hypothesis that previous knowledge stored in masks can be reused.
The analysis of the coefficients of the linear combination (Section 6.1) reveals that the optimization can tune them adapt to the nature of the curriculum. In the CT-graph and Minigrid curricula, previous masks are
16

---

Published in Transactions on Machine Learning Research (07/2023)
used in balanced proportions. On the Contimual World environment, instead, particular masks, and layers within those masks, had significantly larger weights than others. From this observation, we conclude that the new proposed approach may be flexible enough to adapt to a variety of different curricula with different degrees of task similarity.
One concern with modulating masks that memory requirements increase linearly with the number of tasks. This makes the approach not particularly scalable to large numbers of tasks. However, the promising performance of the linear combination approaches suggests that an upper limit could be imposed the number of masks to be stored. After such limit has been reached, new tasks can be learned solely as linear combinations of known masks, significantly reducing memory requirements. While this paper tested vector parameters with scalar for each network layer, the analysis of the tuned parameters suggests that in some cases a single scalar for each mask could be used, further reducing memory requirements. Other suggestions to combat memory requirements in the mask approach are discussed in Appendix G
In the modulating masking setup, it is assumed that task oracle informs the lifelong RL agent of task boundaries and the task presented at any given time (during training and evaluation /taattionntsstngg...... This is made possible by providing a task identifier to the agent to select the correct mask. While the explicit specification of task boundaries is a limitation of all LRL methods that make this assumption, the nature of the proposed mask method, where a mask is associated to a task, implies that existing task detection methods could be combined with the mask setup to address this limitation. One approach could involve the use foryet-me me- not Bayesian approach to task detection (Milan et al.j 2016), as was employed in Kirkpatrick et al. (2017). Another approach that could be explored is the use of optimal transport methods (Alvarez-Mees Melis & Fusi, 2020 Liu et al. 2022) to measure distance of states /input across tasks. Also, the few-shots optimization of the linear superposition of masks via gradient descent that was employed in Wortsman et al. (0200) for LSL could be employed to infer task mask in the LRL setup.
Given the fixed nature of the backbone network and the use of binary masks to sparsify the network, the representational capacity of the work could be affected. Masking approaches have been extensively studied in fixed neural networks (Zhou et al., 2019 Ramanujan et al., 2020), including lifelong supervised learning setup (Mallya et al. 2018 Wortsman et al., 2020), with discussions about representational capacity and generalization. While representation capacity could be affected, there gains in generalization Zhou et al. (2018); Frankle & Carbin (2018) and robustness to noise (Arora et al., 2018) in sparse nelworks. In the future investigations, the gains in generalization could be useful to model-based RL approaches in lifelong
learning.
8 Conclusion
This work introduces the use of modulating masks for lifelong reinforcement lcarning problems. Variations of the algorithm are devised to ignore or exploit previous knowledge. All versions demonstrate the ability to learn on sequential curricula. and show no catastrophic forgetting thanks to separate mask training. The analysis revealed that using previous masks to learn new tasks beneficial as the linear combination masks introduces knowledge in new policies. This finding promises potential new developments for compositional knowledge RT. algorithms. Exploiting previous knowledge, one version of the algorithm was able to solve extremely difficult problems with reward probabilities as low as 5.6 10-6 per episode simply using random exploration. These results suggest that modulating masks are a promising tool to expand the capabilities of lifelong reinforeement learning, in particular with the ability to exploit and compose previous knowledge to learn new tasks.
Broader Impact Statement
The advances introduced in this work contribute to the development of intelligent systems that can learn multiple tasks over a lifetime. Such a system has the potential for real-world deployment, especially in robotics and automation of manufacturing processes. As real-world automation increases, may lead to roduce the demand for human labor in some industrics, thereby impacting the cconomic sccurity of people. Also, when such systems are deployed, careful considerations are necessary to ensure a smooth human machine collaboration in the workforce, ethical considerations and mitigation of human injuries.
17

---

Published in Transactions on Machine Learning Research (07 2222)
Acknowledgments
This material is based upon work supported by the United States Air Force Research Laboratory (AFRL) and Defense Advanced Rescarch Projects Ageney (DARPA) under Contract No. FA8750-18-C-0100 (Life- long Learning Machines) and Contract No. HR001122101322 (Shared Experience Lifelong Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency (DARPA).
References
LF Abbott and Wade G Regehr. Synaptic computation. Nature, 431((010):796 803, 2004.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139-154, 2018.
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advonces in Newral Information Processing Systems, 03:21428-21455 2020.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets
via a compression approach. In International Conference on Machine Learning, pp. 254-222220000 263. PMLR, 2018,
Michael C Avery and Jeffrey Krichmar. Neuromodulatory systems and their interactions: a review of models, theorics, and experiments. Frontiers in neural circuits, pp. 108, 2017.
Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learn- ing through asynchronous advantage actor-critic on gpu. In International Conference on Learning Representations, 2017. URL httpp/////ppaaaeeeee..eee//////r
Megan M. Baker, Alexander New, Mario Aguilar-Simon, Ziad Al-Halah,, Sébastien M.R. Arnold, Ese Ben- Iwhiwhu, Andrew P. Brna, Ethan Brooks, Ryan C. Brown, Zachary Daniels, Anurag Daram, Fabien Delattre, Ryan Dellana, Eric Eaton, Haotian Fu, Kristen Grauman, Jesse Hostetler, Shariq lqbal, David Kent, Nicholas Ketz, Sohcil Kolouri, George Konidaris, Dhiroesha Kudithipudi, Erik Learned-Millerr Se- ungwon Lee, Michael L. Littman, Sandeep Madireddy, Jorge A. Mendez, Eric Q. Nguyen, Christine Piatko, Praveen K. Pilly, Aswin Raghavan, Abrar Rahman, Santhosh Kumar Ramakrishnan, Neale Ratzlaff, Andrea Soltoggio, Peter Stone, Indranil Sur, Zhipeng Tang, Saket Tiwari, Kyle Vedder, Fe- lix Wang, Zifan Xu, Angel Yanguas- Gil, Harel Yedidsion, Shangqun Yu, and Gautam K. Vallabha. A domain-agnosticc approach for characterizatton of lifelong learning systems. Newral Nelworks, 2023. ISSN 0893-6080. doi: ccdene//arrccee//////88668800000
1016/j neu.: net.: 2023..0..2..... URL htttps://ww/ wwweenceiirettccm
Mark Bear, Barry Counors, and Michael Paradiso. Neuroscience: Eaploring the Bruin, Enhanced Edition: Exploring the Brain. Jones & Bartlett Learning, 2020.
Eseoghene Ben-lwhiwhu, Jeffery Dick, Nicholas A Ketz, Praveen K Pilly, and Andrea Soltoggio. Context meta-rciiforcement learning via nenromodulation. Neural Networks, 152:70-79, 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1908.3390. 2013.
Marcus K Benna and Stefano Fusi. Computational principles of synaptic memory consolidation. Nature neuroscience, 19612) 1706, 2016.
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference Computer Vision (ECCV), pp. 532 547, 2018.
18

---

Published in Transactions on Machine Learning Research (07/2023)
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019. URL https:
//ppam/eriee..ttfffoourr//dtddt BC5FX
Maxime Chevalier-Boisvert,, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym.
2018.
Maxime Chevalier- Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. arXiv preprint arXiv:2306 18891, 2023.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048-2055 PMLR, 2020.
Cédric Colas, Olivier Sigaud, and Pierre-Yees Oudeyer. How many random sceds? statistical power analysis in deep reinforcement learning experiments. Xiv preprint orXiv: 08295, 2018.
Matthicu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural net- works with binary weights during propagations. Advances in neural information processing systems, 28,
2015.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IFEE transactions on pattern analysis and machine intelligence, 44(7):3366-33) 33)))-38663885 2021.
Kenji Doya. Metalearning and neuromodulation. Newral networks, 15(4-6):495 506, 2002.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Tain Dumning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Indernational conference machine learning, pp. 1407 1416. PMLR, 2018.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference Artificial Intelligence and Statistics, pp. 3762-3773. PMLR, 2020.
Jcan-Mare Fellous and Christianc Linster. Computational models of neuromodulation. Newral computation, 10(4):771-805, 1998.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXtv:1803.088 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory- basod lifelong learning. Advances in Neural Information Processing Systems, 33:1023-100 1033,, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028 1040, 2020.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Torsten Hocfler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Pesto. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1):10882 11005, 2021.
19

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. In International Conference on Machine Learning, pp. 2497-2506. PMT.R, 2018.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. arXiv preprint arXiv:1902.002:: 2019.
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaškowski. Vizdoom: A doom-based ai rescarch platform for visual reinforcement learning. In 2016 IEEE conference computa- lional intelligence and games (CIG), pp. 1 8. IEEE, 2016.
Samuel Kessler, Jack Parker-Holder Philip Ball, Stefan Zohren, and Stephen J Roberts. Unelear: A straight- forward method for continual reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning. 2020.
Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J Roberts. Same state, different task: Continual reinforcement. learning without interference. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 7143 7151, 2022.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learn- ing: A review and perspectives. aix preprint Xiw:2012.13490, 2020.
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Venoss, Guillamme Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agniesıka Grabska-Barwinskee et al. Overcoming calastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
Soheil Kolouri, Nicholas A Ketz, Andrea Soltoggio, and Praveen K Pilly. Sliced cramer synaptic consolidation for preserving deeply learned representations. In International Conference on Learning Representations, 2019.
Nils Koster, Oliver Grothe, and Achim Rettinger. Signing the supermask: Keep, hide, invert. arXiv preprint arXiv:2201.138 13361, 2022.
Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, Anurag Daram, Stefano Fusi, Peter Helfer, Leslie Kay, Nicholas Ketz, Zsolt Kira, Soheil Kolouri, Jeffrey L. Krichmar, Sam Krieg- man, Michael Levin, Sandeep Madireddy, Santosh Manicka, Ali Marjaninejad, Bruce McNaughton, Risto Miikkulainen, Zaneta Navratilova, Tej Pandit, Alice Parker, Praveen K. Pilly, Sebastian Risi, Terrence J. Sejnowski, Andrea Soltoggio, Nicholas Soures, Andreas S. Tolias, Darío Urbina-Meléndez, Francisco J. Cuevas, Gido M. Van de Ven, Joshua T. Vogelstein, Felix Wang, Ron Weiss, Angel Yanguas- Gil, Xinyun Zou, and Hava Sicgelmann. Biological undorpinnings for lifelong lcarning machinos. Nature Ma- chine Intelligence, 4(3):196 210, 2022.
Timothy P Lillierap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Sil- ver, and Daan Wierstra, Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Son Lin, Li Yang, Deliang Fan, and Junshan Zhang. TRGP: Trust region gradient projection for continual learning. In International Conference on Learning Representations, 2022. URL htttps:/opeereevier net/forrum?dd--EBAA8668..
Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea. Soltoggio, and Soheil Kolouri. Wasserstein task embedding for measuring task similarities. arXiv preprint arXiv:2208.11 11726, 2022.
David Lapez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEFE conference on Computer Vision and Pattern Recognition, pp. 1775-7773. 2018.
20

---

Published in Transactions on Machine Learning Research (07/2023)
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the Europpon Conference on Computer Vision (ECCV), pp. 67 82, 2018.
Charles E Martin and Praveen K Pilly. Probabilistic program neurogenesis. In ALIFE 2019: The 2019 Conference on Artificial Life, pp. 440 447. MIT Press, 2019.
Jorge A Mendez, Harm van Seijen, and Eric Eaton. Modular lifelong reinforcement learning via neural compasition. In International Conference Learning Representations, 2022. URL https:// net//forum2idd-5xllaadlll
Kieran Milan, Joel Veness, James Kirkpatrick, Michael Bowling, Anna Koop, and Demis Hassabis. The lorgel-me-nol, process. Advunces in Neural Information Processing Systems, 29, 2016.
Volodymyy Mnih, Koray Kavukenoglu, David Silver, Alex Graves, loannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXie preprint arXiv:1312.5602 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national conference machine learning, pp. 1928-1937. PMLR, 2016.
Alexander New, Megan Baker, Fric Nguyen, and Gautam Vallabha. Lifelong learning metrics. arXiv preprint arXiv:2201.0827 2022.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Neurul Networks, 113:54 71, 2019.
Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. Cora: Benchmarks, baselines, and metrics a platform for continual reinforcement learning agents. In Conference on Lifelong Learning Agents (CoLLAs), 2022.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's 's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVE /CF Conference on Computer Vision and Pattern Recognition, pp. 11893-11902, 2020.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukeuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXio preprint arXiv:1606.0400 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In In- ternational Conference on Learning Representations, 2021. URL httpp::/openrrviee..tttt wttp://wwww..eeennniniiime
330]0RCNN2]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1899. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithins. arXio preprint arxi 1rX7::1777. 77070004777 2017.
Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 27565306) 1599, 1997.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwer Barwinska, Yoe Whye Tch, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. International Conference on Machine Learning, pp. 4528 4537. PMLR, 2018.
21

---

Published in Transactions on Machine Learning Research (07/2023)
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Powerpropagation: A sparsity inducing weight reparameterisation. Advances in neural informalion processing systems, 34: 28889 28903, 2021.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference Machine Learning, pp. 4548 4557. PMLR, 2018.
Kun Shao, Dongbin Zhao, Nannan Li, and Yuanheng Zhu. Learning battles in vizdoom via deep reinforcement learning. In 2018 IEEE Conference Computational Intelligence and Games (CIG), pp. 1-4. IEEE, 2018.
Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Spacenet: Make free space for continual learning. Neurocomputing, 439:1-11, 2021.
Andrea Soltoggio, John A Bullinaria, Claudio Mattiussi, Peter Dürr, and Dario Floreano. Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios. In Proceedings of the 11th international conference on artificial life (Alife XT), number CONF, pp. 569-576. MIT Press, 2008
Andrea Soltoggio, Kenneth O Stanley, and Sebastian Risi. Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks. Neural Networks, 108:48-67, 2018.
Andrea Soltoggio, Pawel Ladosz, Eseoghene Ben-twhiwhu, and Jeff Dick. The -T-phapph environments, 2019.
URL http://ghhabbccoooollloggg//ggg/
Andrea Soltoggio, Eseoghene Ben-lwhiwhu, Christos Peridis, Pawel Ladosz, Jeffery Dick, Praveen K Pilly, and Soheil Kolouri. The configurable tree graph (ct-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learning. arXiv preprint or Xiv:2300 1 10887, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 TEEE/RSJ international conference on intelligent robots and systems, pp. 5026- 5033. IEFE, 2012.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for contimual learning. arXi::1944.070 07794, 2019.
arXiv preprint
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Johannes Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020. URL https://
openreview.nee www.:ne/foorriiddSSseeeeeerr
Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 145250-5263 2021.
Maciej Wolczyk, Michal Zajac, Razvan Pascamu, Lukasz Kuciński, and Piotr Miloć. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Newral Information Processing Sys- tems, 34:28496-28514 2021.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173 15184, 2020.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL https:
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsca Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 PMT.R, 2020.
22

---

Published in Transactions on Machine Learning Research (07/2023)
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8))344-33) -322, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. International Conference Machine Learning, pp. 3987-3995. PMTR, 2017.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019.
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generaliza- tion bounds at the imagenet scale: a pac-bayesian compression approach. arxiv preprint 1rXX::1804.05000 05862, 2018.
23


Reviews:
[{"review_id": "Q2TFvXyuBo", "reviewer": "Reviewer_YFzb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n- The idea is well motivated and grounded in the continual learning domain in classification.\n- The authors are very clear what the scope of the paper and the scope of contributions are. Specifically, the paper is very concrete in what hypotheses/methods it is testing.\n- The algorithms developed in this paper are well explained, and easy to follow.\n\n# Weaknesses:\n\n1. It is common in continual learning to make an assumption that you have clear task boundaries. While this is a reasonable assumption for many applications, using the task information as privileged information (which your method does to the best of my understanding) needs to be justified and highlighted. Because this is a huge limitation of your approach and only covers a very small set of possible settings in LRL (as specified in [Khetarpal et al., 2020]. This comes up in the introduction, where this distinction isn't highlighted, and in the problem formulation section where the paper claims the lifelong setup is for a finite set of discrete tasks.\n\n2. Statistical power of the hypothesis tests (i.e. the error bars) for each of these experiments. While 95% confidence intervals are used, the paper only has three seeds for all the experiments. In most of the plots we see heavy overlaps when there is no post-processing done to the results. When averaging over tasks (Fig 2 (right), Fig 3 (right), Fig 4 (right), Fig 5 (right), ...) we start to see an effect, but I'm not convinced this aggregation increases the statistical power of the hypothesis test. We are still only testing 3 sample per experiment, as the agent's random initialization is the primary source of randomness in the test. This is re-confirmed through the EWC's error bars (the main competitor) overlapping all other lifelong learning methods in Table 1, 2, etc... Because we are trying to do a performance comparison, I'm not very convinced by the results shown.\n\n3. Given 2, and the huge difference between the variance in the modulating masks methods and EWC, I'm wondering how much of this is influenced by the hyperparameter tuning. It is unclear how hyperparameters are chosen for the experiments. This needs to be explained clearly, and choices need to be justified. This similarly seeds doubt in the continuos world experiments where EWC performs very poorly. Is this known behavior of EWC? Or is it specific to the hyperparameters chosen? This is a base question generally holding back the claims of the paper.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use modulating masks from the classification literature to improve performance in lifelong reinforcement learning. The paper contributes:\n1) A specification to use modulating masks in the LRL, specifically using PPO and IMPALA, although the specification is sufficiently general to be used more broadly.\n2) A variant which can exploit previous knowledge to learn new tasks.\n3) And a set of empirical evaluations showing the improvements modulating masks can have over several competitors.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see several modifications. Most critically,\n\n1. The inclusion of how all hyperparameters were set. \n\n2. An increase in the number of seeds used for many of the experiments. I would prefer less environments tested with a fuller set of empirical evaluations (i.e. more seeds, sweeps over hypers) rather than many environments all with comparisons with low statistical power.\n\n3. Justification of the aggregation plots (aggregating over tasks) to artificially raise the number of samples in your confidence intervals.\n\n4. A clear statement about using task boundaries, as done in your approach, and the limitations of such an assumption in the lifelong learning setting early in the paper. This was lightly touched on in section 7, but not quite enough in my opinion.\n\n\n ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8kM4PzToML", "reviewer": "Reviewer_JcNN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper focuses on a very interesting and promising research topic, that is, lifelong learning for RL, which I think is important for solving a series of sequential decision making problems, especially when they have similar problem structures.\n2. The proposed method is well motivated and technically sound. The description of background is fairly good, which can inspire experts or non-experts in the related domain.\n3. The evaluation results are ample, which covers the comparison and a lot of ablation studies on discrete and continuous RL settings with various states and reward functions. The analyses are also sufficient to explain the results in a comprehensive way.\n4. The shortcomings of the proposed approach are described carefully.\n\n**Weaknesses**\n1. The novelty of this paper is relatively low. The main contribution is the application of modulation based lifelong learning to RL, with the proposed schemes to mix the masking scores seem intuitive.\n2. The baselines for most experiments are PPO and EWC$_{MH}$, which I think are weak. Additional state-of-the-art approaches could have been tested and compared.\n3. The training efficiency is not clear. Since RL training for one task may cost a long time, it should be clarified how the proposed approach saves the training time, especially compared to the main baseline EWC$_{MH}$. Also, it should be analysed how the training cost would increase if the RL problem scales grows. These analyses at least let readers know when and where the approach is a better choice than other lifelong learning approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a modulation based lifelong learning approach in the RL domain. It describes different schemes to learn masking scores for each task, including Mask Random Initialization, Mask Linear Combination and their hybrid Balanced Linear Combination. All these schemes are proposed to well leverage the previously trained masking score, so as to enhance the forward transfer of RL training. Extensive evaluation shows that the proposed approach is able to well improve over PPO and an EWC based lifelong learning approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In summary, I'm positive about the quality of this paper. Although the novelty is limited, the extensive evaluation results demonstrated the effectiveness of the proposed approach. \n\nHowever, I suggest the authors clarify the training efficiency of the proposed approaches, and analyze when and where the approach could be a good choice. Also, a minor revision could be emphasizing why the EWC method is chosen rather than others.", "broader_impact_concerns": "I believe the broader impact statement in the original paper has addressed the possible concerns on the implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vizGpuvHPp", "reviewer": "Reviewer_Cewb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Experimental evaluation is extensive, providing results on various complex reinforcement learning environments.\n- Some interesting attempts at analysis. Would prefer this section to be expanded. \n- I enjoyed this discussion around biological learning (Section 2.3)\n\nWeaknesses:\n- Limited algorithmic novelty. CL based on sparse gating is a well known technique, the idea to linearly combine previous masks is relatively straight-forward.\n- The authors fail to discuss some of the inherent questions and concerns of the proposed approach, specifically:\n     - Given that the weights are randomly initialized and fixed (critical to avoid forgetting), how much representational capacity is lost? What are the limitations of learning purely based on gating a random network? To which extent does this depend on size and complexity of the backbone network?\n     - The method requires task-inference at test time (so the correct score parameters $S_k$ can be chosen for task $k$). How do the authors propose to infer the task in a setting where the agent is evaluated on an unknown but previously encountered problem? \n- Related Work section should be improved. The authors miss several relevant works in Continual RL / CL based on Sparsity.\n- Writing and presentation could be improved. Several crucial facts should be stated more clearly. The paper is unnecessarily long.\n- Baseline methods chosen for evaluation are relatively simple and known to be significantly under-performing SOTA. No comparison to another technique based on sub-networks despite results being readily available (e.g. PackNet on CW10). \n    \n\nOverall unfortunately not yet ready for publication. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors experiment with a sparse masking approach to Continual Learning, specifically in the RL context. In this setting, each task is represented by a sub-network indexed through a gating masks that are adapted online. Different from previous settings, the authors exclusively work in the setting of unchanging weights and thus no forgetting. The contributions of this work are:\n\n- A new mask initialization scheme which facilities the re-use of skills encoded in previous tasks through a linear combination of the respective sub-masks.\n- A large, empirical evaluation of the proposed changes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Requested changes:\n\nAbstract:\n- \"learning multiple input distributions, typically in classification\". Classification does not usually involve learning the input distribution (i.e. modelling p(x)), please rephrase.\n- \"PPO/IMPALA\" - Citation missing.\n- Please mention here that you're making the assumption of fixed weights.\n\nIntroduction:\n- What is the difference between LRL and Continual Learning? Why the insistence that this is a separate field (\"LRL shares some of the objectives of CL [...] but [...] has developed into a separate field\")? This does not seem like a commonly accepted categorization to me. Indeed, the most of the techniques evaluated in the experimental section (e.g. EWC, CLEAR, P&C) as well as the technique proposed by the authors would be expected to be applicable to other CL tasks.\n- \"This study introduces the use of modulating masks in LRL\" - Incorrect claim, modulating masks have been used before. (e.g. PackNet, [1])\n- \"DQN [...] EWC to produce a lifelong learning DeepRL agent.\" - Incorrect, the original EWC paper already showed experiments on Deep RL.\n\nRelated Work:\n- \"for example, DQN, PPO, or SAC\" - citations missing for PPO/SAC.\n- Missing reference for [1] (which shows modulating masks on Deep RL), [2, 3] for Continual RL in general. [4, 5] for alternative uses of modulating masks in CL.  Some of [1, 2, 3] should be used as baselines.\n\nBackground:\n\n- 3.1: Define what expectation is over when writing the  expected cumulative discounted reward. \n- 3.2: Define sigmoid. - Please mention iterative magnitude pruning as a default way of implementing sparse NN. - Also useful to cite a review paper (e.g. [6]).\n\nMethods:\n- 4.2: \"we instead bootstrap the learning from the start\" - Not sure what the meaning of \"bootstrap\" is in this context. - Why is the superscript $r$ used in the definition of $\\phi^r$? Wouldn't $k+1$ make more sense? - $\\bar{\\beta}$ undefined (Is this the initaliztation?).\nAlgorithm 1, Line 10: This should be an assignment, not an equal sign. \n\nExperiments:\n- \"As tasks are learned independently of other tasks in MaskRI, there is no notion of forward transfer in the method\" - This is a direct consequence and drawback of not updating the weights.\n\nAnalysis:\n- Figure 7 (left, middle) If MASK_LC resorts to a uniform distribution, where does the performance improvement come from? \n- Figure 7 (right) \"We speculate that tasks in CW10 are more diverse and the optimization process is enhancing specific coefficients to reuse specific skills.\" This should be analyzed more thoroughly.\n- \"shows the analysis for the following cases: facing a fourth task after learning three tasks in the CT8benchmark; facing the 12th task after learning 11 tasks in the CT12 benchmark; facing the 7th task after learning 6 tasks in the MG10\" - Why are such specific numbers chosen? Does the same finding hold for all other combinations or this is a result of cherry picking? \n- \"[...] validates the hypothesis that previous knowledge stored in masks can be reused.\" - I'm not necessarily convinced given the uniform Finding in Figure 7. A more thorough analysis would be useful here. What aspects of those previous tasks are useful?\n- Figure 8: No need to show the result for EWC. Simply state it's a uniform distribution.\n\n\nDiscussion:\n- There is too much emphasis on memory costs in my opinion. It is easy to see that the memory cost compares very favorably to commonly used techniques based on replay so this is unlikely to be a major concern. This could be moved to the appendix. \n- Please discuss the points raised above (task-inference, untrained weights).\n\n\n[1] Schwarz, Jonathan, et al. \"Powerpropagation: A sparsity inducing weight reparameterisation.\" Advances in neural information processing systems 34 (2021): 28889-28903.\n\n[2] Kaplanis, Christos, Murray Shanahan, and Claudia Clopath. \"Continual reinforcement learning with complex synapses.\" International Conference on Machine Learning. PMLR, 2018.\n\n[3] Kaplanis, Christos, Murray Shanahan, and Claudia Clopath. \"Policy consolidation for continual reinforcement learning.\" arXiv preprint arXiv:1902.00255 (2019).\n\n[4] Sokar, Ghada, Decebal Constantin Mocanu, and Mykola Pechenizkiy. \"Spacenet: Make free space for continual learning.\" Neurocomputing 439 (2021): 1-11.\n\n[5] Von Oswald, Johannes, et al. \"Learning where to learn: Gradient sparsity in meta and continual learning.\" Advances in Neural Information Processing Systems 34 (2021): 5250-5263.\n\n[6] Hoefler, Torsten, et al. \"Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks.\" The Journal of Machine Learning Research 22.1 (2021): 10882-11005.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: (See meta-review above).
Decision: Accept (poster)