Multitask Reinforcement Learning by Optimizing Neural Pathways
Authors: Samin Yeasar Arnob
Riyasat Ohib
Amy Zhang
Sergey Plis
Doina Precup
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multitask Learning
Online Reinforcement Learning
Offline Reinforcement Learning
Neural Pathways

Abstract: Reinforcement learning (RL) algorithms have achieved great success in learning specific tasks, as evidenced by examples such as AlphaGo or fusion control. However, it is still difficult for an RL agent to learn how to solve multiple tasks. In this paper, we propose a novel multitask learning framework, in which multiple specialized pathways through a single network are trained simultaneously, with each pathway focusing on a single task. We show that this approach achieves competitive performance with existing multitask RL methods, while using only 5% of the number of neurons per task. We demonstrate empirically the success of our approach on several continuous control tasks, in both online and offline training.

Under review as a conference paper at ICLR 2023
MULTITASK REINFORCEMENT LEARNING BY OPTIMIZING NEURAL PATHWAYS
Anonymous authors Paper under double-bling review
ABSTRACT
Reinforcement learning (RL) algorithms have achieved great success in learning specific tasks, as evidenced by examples such as AlphaGo fusion control. How- ever, it is still difficult for an RL agent to learn how to solve multiple tasks. In this paper, we propose a novel multitask learning framework, in which multiple specialized pathways through single network are trained simultaneously, with each pathway focusing on a single task. We show that this approach achieves competitive performance with existing multitask RL methods, while using only 5% of the number of neurons per task. We demonstrate empirically the success our approach on several continuous control tasks, in both online and offline training.
INTRODUCTION
Our brain processes different languages, helps perceive and act in 3D world, coordinates between organs, and performs many other tasks. The brain continuously learns new things without catastrophic forgetting due its plasticity (Zilles, 1992; Drubach, 2000; Sakai, 2020; Rorke, 1985), i.e., its ability to continually strengthen more frequently used synaptic connections and eliminate synaptic connections that are rarely used, phenomenon called synaptic pruning (Feinberg, 1982). In this way, the brain creates neural pathways to transmit information. Different neural pathways (Rudebeck al., 2006; Paus et al., 1999; Jürgens, 2002: Goodale al., 1994) are used to complete different tasks. For example, the visual mechanisms that create the perception of objects are functionally and neurally distinet from those controlling the pre-shaping of the hand during grasping movements directed at those objects (Goodale et al., 1994). In deep learning architectures, the hope is that gradient descent, possibly augmented by attention mechanisms, will create these types of pathways automatically. However, when performing multi-task learning, this approach can be insufficient to prevent interference between tasks, and often leads to loss of plasticity. This problem especially exacerbated in reinforcement learning, where different tasks require policies which generate different data distributions.
In this paper, we take inspiration from the presence of distinct neural pathways for different tasks in natural brains, and propose a novel approach to implement it in deep reinforcement learning. Similarly synaptic pruning, we aim identify the important connections among the neurons in deep neural network that allow accomplishing specific task. However, we leverage insights from recent lottery ticket hypothesis (Frankle Carbin, 2019; Lee et al., 2018; Tanaka et al., 2020; Wang et al., 2020a) literature to construct task-specific neural pathways which utilise only a very small fraction (5%) of the parameters of the entire neural network, yet demonstrate expert-level performance in multitask reinforcement learning in both online and offline settings. The neural pathways are allowed to overlap, which enables leveraging information from multiple tasks in the updates of the same parameters, in order to improve generalization.
Contributions: We propose Neural Pathway Framework (NPF), a novel multitask learning approach that generates neural pathways through large network that specific to single tasks. Our approach can be easily integrated with any online or offline algorithm without changing the learning objec- tive!. Because the shift in data distribution during online reinforcement learning training makes it challenging to find neural pathways by using lottery network search techniques (Frankle & Carbin, 2019; Alizadeh, 2019; Wang al., 2020a), we propose a data adaptive pathway discovery method for
'Code: https///gituuh.ccm/mmmCCCCCC

---

Under review as a conference paper at ICLR 2023
the online setting. We demonstrate approach in both offline and online multitask reinforcement learning, using Mujoco-bat sasood and MetaWorld environments.
Our method is frugal in terms of the number of neurons, and multiple tasks trained in parallel, thus saving significant training time. The algorithm finds neural pathways which use only 5% of the neural network weights for each task. Since our method requires small fraction of weights complete a task, it can provide fast inference (Molchanov et al., 2017; Luo et al., 2017). reducing energy consumption (Yang al., 2016). Thus, our method is especially suitable for real-world applications in which large datasets need to be processed in real-time (e.g.. self-driving cars, deploying bots in games, financial data analysis) or which are deployed on low-resource devices (i.e., embedded systems, edge devices, etc.).
Task 1
Butput
Task
Output
Task
Output
Task
Output
Figure I: During evaluation trained agent (global model), for any given task our proposed method activates specific part the neural network.
2 RELATED WORK
Many recent works point out that training a deep neural network for more than one task becomes difficult due to gradient interference, i.c.. gradients for different tasks pointing in very different directions. Recent work proposes several possible solutions, such as constraining the conflicting gradient update (Yu al., 2020; Suteu & Guo, 2019; Du et al., 2020; Chen et al., 2017: Sener Koltun, 2018), constraining data sharing among irrelevant tasks (Yu et al., 2021: Fifty et al., 2021), and learning the underlying context of relevant tasks (Sodhani et al., 2021). In all of these cases, success depends the assumption that there is an underlying shared structure among these tasks, and on how well this structure can be captured during training. Many multitask learning methods propose complex ways to modularize the neural network, which allow sharing and reusing network components across tasks (Rusu et al., 2016; Fernando et al., 2017; Rosenbaum et al., 2017: Devin al., 2016; Misra al., 2016; Yang al., 2020). In this paper, propose completely new way of tackling multitask problems, and we show that a single deep neural network is sufficient for learning multiple complex tasks, by having multiple pathways through the network.
Recent advances in finding sparse networks have proven that there exist sub-networks which contain small fraction of the parameters of the dense deep neural network yet retain the same performance. There is a range of techniques to find such sub-networks through an iterative update during training (Frankle Carbin, 2019; Feinberg, 1982: Gale et al.. 2019; Blalock et al., 2020; Ohib al., 2019; Yang et al., 2019). Simpler alternatives find a sub-network single-shot (Lee et al., 2018; Wang et al., 2020b) weight initialization. Building upon trained models (Mozer & Smolensky, 1988), a recent technique to prune models at initialization was proposed by Lee et al. (2018). Utilizing this criterion, they are able to predict, at initialization, the importance that each weight will have later in the training process. However, premature gradient- based pruning at initialization can lead to layer collapse: the premature pruning of an entire layer can render network un- un-trainable. Many techniques (Wang et al., 2020b; Feinberg, 1982) try to maximize useful gradient flow through the deep layers. In order to discover neural pathways, we can leverage these methods in order to find task-dependent sub-networks, as long as can avoid layer collapse.
decee-eeadd-old saliency criterion used for pruning
2

---

Under review as a conference paper at ICLR 2023
Global
Model
Task Expert
Task
Task Expert
Gradtent
Gradient
Local
[5,ay5",r)
Paraneters
Paraneters
Figure 2: Example of learning tasks in the offline setting- During training, the global model (purple) communicates with the task specific local models (green). Each local model computes gradient the task specific weights and sends them to the shared global model (pink arrow). Network weights are updated in the global model and then shared (blue arrow) with the local models at every training step.
BACKGROUND
Multitask Reinforcement I Learning: We consider learning in a Markov Decision Process (MDP) described by the tuple (S, A, P, R) consisting of states S, actions P(s'|s, a), and reward function R(s, a.. We use St, and r, = R(s,, az) to denote the state, action and reward time step respectively. A trajectory consists of a sequence of states St, actions a, and rewards ry: T= (so, 0o, To, 81, 01, Fl, ..., ST; 0T, TT). (T,TT). For continuous control tasks, we consider an infinite horizon, T oo and the goal is learn policy which maximizes the expected discounted return EIΣ;- rrre]. the multitask setup, we consider a finite set N tasks with respective reward functions {Rn J and optimal policies (als))
A, transition dynamics
Offline Reinforcement Learning: In offline reinforcement learning, instead of obtaining data through environment interactions, we have access to a previously collected, fixed dataset D {(s;, s(rr))) with L transitions collected using behavior policy (a(a)s). For multitask offline learning, we have N datasets, 1DD..' collected for each task. Offline training does not perform well when trained naivelt with batch data, due to extrapolation errors caused by data distribution mismatch between the samples collected from behavior policy σa and the current policy Ty. In this work, we use two standard offline RL algorithms, Batch- Constrained deep Q-learning (BCQ, Fujimoto et al. (2018)) and Implicit Q-learning (IQL., Kostrikov et al. (2021)) build offline experiments.
Online Reinforcement Learning: In the online RL setting.. the agent is expected to interact with the environment and leverage the collected experience to learn how to maximize its expected return. Neural pathways can be integrated into any online RL algorithm and trained for multitask objectives. To demonstrate the effectiveness of our method in the online setting, we use Soft-Actor- Critic objective, in
(SAC) (Haarnoja et al., 2018) our experiments. SAC optimizes an order to drive the agent to more exploratory behavior.
PROPOSED APPROACH
We now introduce the Neural Pathway Frame- work (NPF) and its practical implementation for multitask learning in two steps: (1) finding the neural pathway for each task and (2) training the neural pathways for multiple tasks simultane- ously. To discover task task- specific neural pathways, we build on the idea of finding a lottery-ticket sub-network (Frankle & Carbin, 2019) (i.e., neb-network that provides similar performance to the original network) using existing pruning methods, detailed in Sec.4.1. We then describe how to use NPF in both offline (Sec. 4.2) and online (Sec. 4.3) RI. settings.
Table 1: Comparison of pruning methods against the soural-pathwap frameeook (NPF).
Important Features
Pruning Methods
NPF
Finding sparse sub-network
Data dependency Tackles data-distribution shift Multi-task learning
4.1 DISCOVERING NEURAL PATHWAYS
Similarly to the Lottery ficket hypothesis (Frankle & Carbin, 2019) in supervised learning, assume that dense network contains a sub-networtt that can be initialized such that, when trained in isolation, it can match the performance of the original network. We call this sub-network a Neural Pathway. In the context of multitask training, a deep network will have multiple neural pathways, one for each task, which can share neurons.
3

---

Under review as a conference paper at ICLR 2023
Formally, we consider feed-forward network f(z, 8) with initial parameters Og ~ Do- A Neural Pathway activates sub-network f(z, A @mm) by using mask E {0, 13. where is fixed during training. From the lottery ticket hypothesis, there exists m which leads to better performance, less training time, and ||m]lo < |0] (fewer trainable parameters). For multi-task learning, as shown in Figure I, multiple neural pathways are activated with n masks {m1. mm, ), one for each task.
In order to learn the neural pathways, we build on Single-shot Network Pruning (SNIP) (Lee et al., 2018; Arnob et al., 2021 a pruning technique which has proven to perform well in a single-task setting, leading to up to 95% sparse network. SNIP identifies important connections by using sensitivity measure, defined as the influence of each weight on the loss function. Formally, the effect of weight 0g on the loss is: S(0,)) = lim,-0 (((o))(())+ed,)))
0,
where by vector 30,
0C
whose 9th element equals 9g and all other elements are 0.
We note that, while we build our approach by leveraging pruning methods, our goal is not to prune weights, but to flexibly use the capacity of the network to learn many tasks. Therefore, our approach needs to be data-dependent. Moreover, in the online RL case, NPF needs to be robust to shifts in the data distribution due to changing policy. Thus, for n² task, we use the following objective:
m (0, D.) = lime-o C10g:..)-210+tt6,,...
0430
0 8C
Table 1 summarizes the differences between NPF and pruning methods.
4.2 OFFLINE MULTITASK LEARNING WITH NPF
Training a Neural Pathway: First, we identify the top 5% most important weights for each task independently using Equation (1), which will be remembered using binary masks, one for each task, in which marks trainable weights. We never set any weights to zero. But, due to masking, only the subset of the weights contained in the neural pathway participate in task-specife inference and updates. Thus, for any given task, the mask mn activates a pathway of neurons for forward propagation and backward propagution.
Simultaneous Multitask Training: order to train multiple tasks using offline RI., we must have corresponding training datasets for all the tasks [D,)a= We run simultancous parallel training processes, one for each task, which use only the task-specific expert data, A local model L( m.) is initialized for task with the haaaeed-mdel/gllbal--meo the task-specific mask m, is applied to get the local model L( m.))
L(()) parameters, As shown in Figure 2,
For each task n, each training step, we synchronize the local L(0 m.) model weights with the global model and sample a random batch of data from dataset D.. The local model computes the gradient V Voan.. J(D,, of the objective function and sends the gradients to the global model (see Fig. 2). Due to masking, gradients of the weights that are irrelevant for the task appear to have the value 0 to the local-mode at all times. We run all the process on single GPU and compute synchronous parameter updates (Niu et al., 2011; Mnih et al., 2016). The pseudocode is given in Algorithm 1.
Offline RL algorithms utilise multiple deep neural networks with different objective functions. For example, IQL has an actor network T(8). a critic network Q()) and a value network V(ú), each having their own objective function (see Appendix 1) Thus for the wihh task, we use the scoring function to find masks (my, m& m for each network. The global model (0), (()). ((bb is shared with all N processes, where each process is trained to optimize for single task using the local models (0 * m Q( # m.), V (v m.). The global model stores all the trained pathways. As shown in Figure 1, during evaluation we the task-specific mask to activate each task-specifi pathway.
4.3 ONLINE MULTITASK LEARNING WITH NPF
In this section, we discuss the compatibility of NPF with online RI. We empirically show that due to distribution shift in online training. single-shot sub-network discovery techniques fail to maintain good performance. We propose novel approach to address the data distribution shift and show its ability to perform multitasking using neural pathways.
The lottery ticket hypothesis work in supervised learning (Lee et al., 2018; Wang al., 2020b; Tanaka et al., 2020) and offline RL (Arnob et al., 2021b) only considers the case of static datasets, without
4

---

Under review as a conference paper at ICLR 2023
Algorithm Multitask Offline Training
Find the Task Specific Weight Masks:
(Di, Dg....N)) using Equation 1.
m1, m2 .. mN wndNeeaaalPPaaaaaa
#Initialize model for each task and consider synchronous gradient update:
Initialize local-model, global-model
#Training loop:
for Training steps do
#sync-weight
local-model mask (global-model) L(O * m,); mask the n task-specific weights
Sample (s, a, s') D,
> local-model.loss/
local-model. saaalmmmodl..bckaarddggg Veom. ((,)) compute gradient of masked weights
syne-gradient/ (oocalltttoooloeeee (local-mode) global-model)
> htwpl/modelloopmmizeraaeme
end for
#Evaluation loop:
local-model mask mask (global-model) sync-weight
Evaluate( local-mat -oal-m
focusing on the effects out-of-distribution data shifts. However, online RL, the agent's policy always changing, which leads to significant changes in the data distribution.
illustrate the impact of nonstationar- itym we train Soft Actor Critic (SAC, Haarnoja et al. (2018)) using NPF on the Hal fCheetah forward-run task and compare its performance with the final performance of the fully dense SAC baseline. Unlike in offline set- ting, we do not have access collected dataset. Thus, to configure neural-pathway. we collect samples us- ing randomly parameterized policy and use those samples to find a sub- network which optimizes Equation (I), The learning curve Fig. 3 plots the episodic return over 1M gradient up- dates. We see a large gap in perfor- mance between the baseline and single-shot pathway discovery (blue). Four more experiments, discussed in the Appendix (Figures 13), display similar results.
Halfcheetah-forward
Halfcheetah-forwardd
Singla-Shor
Episodic Retum
DAPO
DAPO w-Thheshold
Episocic Retura
Baseline
M
Baseline
prior
Gredient Updates (1e6) (a)
Gradient Uodates (1e6) (b)
Figure 3: Performance comparison (a) Single-Shot
Data Adaptive Pathway Discovery (DAPD) (with and with- out Threshold Stopping Criterion) on HalfCheetah run for- wbrd,(b) DAPD with Different Moving Average (M=1, 5, 10) HalfCheetah run forward.
We hypothesize the observed performance drop is due the fact that the changes in behavior policy cause a data distribution shift, and the pathway discovery depends strongly on the data. To tackle this problem, propose dynamic iterative pruning approach, where pathways continually updated using the most recently collected training samples.
Data Adaptive Pathway Discovery Under Data Distribution Shift: To handle changing data distribution in online RL, we propose Data Adaptive Pathway Discovery (DAPD), where we continue re-configure the pathway the policy improves and the data distribution consequently changes during the training. We use separate temporary replay buffer TD,, one for each task, that only contains immediate episodic data collected by the agent. Only the most recent dataset stored in TD, is used to re-configure the pathway. Once we reach minimum threshold T on the episodic return, we stop this process. our experiment in Fig 3(a) we use episodic return Thr = 2000 as the threshold value hyper-parameter that can be easily tuned to stabilize training.
During the initial training phase, we observe high variation the pathway. Thus, to slow down variations and stabilize to suitable pathway, keep re-configuring using the average of the previous M iteration scores (calculated using Eq. (I):
mR (0y
( TDa) EM lim Cl00: D.) tiB,tthh
0C Em=l 0
490
5

---

Under review as a conference paper at ICLR 2023
Detch Mear Gradient Norm
Datch Mean Gradient form
Pick-place
Peg-insert-side
Pick-place
Pre-confiuned Pethways
Centineel Pathoay Comparctiio
Succees Rate
Success
Gradient Updates (že6) 1e6
Gradient Updates (2o6) 1e6
Gradient Updares (2e6)
1e6
Gradient Uodates (2e65 166
(a) Batch Mean Gradient Norm
(b) Success Rate
Figure 4: When we try to configure pathways for all the tasks at the same time (orange). (a) parameters of the harder tasks do not change from the values at initialization due to small changes in gradients. This results in unsuccessful task learning and can be seen in (b). We find an improvement in performance when pathways are pre-configured separately before multitask training (hlue).
100
Table Normalized score offline RI. algorithms. We compare NPF performance with the single task experts on Hal#Cheet ah and Quadrupod multitasks.
Esperiment
NPF
Algoridhma liog-MT
BCQ
JQL
-00
Hal Aalfchental multitask
87,20 1: 2.34
89.67
12.44
10000
50000
300000
Training
HalfCheet constrained speed
100 1: 0.038
99.88 0.40
Quadrupod multitask
115.35 +: 0.70
116.71
0.57
Figure 5: We compare the sample complexity anal- ysis of BCQ+NPF with BCQ-MT (Multitask) and BCQ- (Multihead Multitask) baselines on Hal fCheetah multitask.
In Fig. 36)), we perform a hyper-parameter sweep and observe that M = 10 leads to stable learning curves and better performance. Further experiments in the Appendix (Fig. 14) support this claim. Our procedure is summarised in Algorithm 2 (see Appendix) and can be easily integrated with any online RL method.
Online Multitask Training and Addressing Gradient Interference: Even though NPF compart- mentalizes the parameter training of a neural network very well offline RI., optimizing pathways for multiple tasks becomes difficult in the online RL setting. Due to significant parameter sharing among tasks, we cannot completely avoid gradient interference. While experimenting with Meta- world (Yu et al., 2019), comparison, we see only insignificant changes in gradient updates for the more challenging tasks. A neural pathway is task-specife group of parameters within a network. Thus, to observe changes in the parameters, we compute gradient-norm || dac of group the parameters for a batch of samples. See in Figure 4(a), where our initial experiments with parallel learning (orange) show little to no gradient-norm change in the task speccsiiiiicc parameters. This implies the parameters have not changed much from their initialization. We find that the success rate (see Figure 4(b)) of the tasks correlates with the gradient change as well.
found that DAPD optimizes pathways for easier tasks very quickly. In
To overcome radient interference, we run a pretraining loop learn pathways. We initialize a separate network, one for each task, with same weights and optimized for task pathways for 10K gradient steps. At the end of this pretraining stage, we average the weights of the overlapping parameters and keep the pathways fixed for the rest of the training. We conduct parallel online data-collection and during training, we average the gradients of the shared parameters. For the Metaworld multitask benchmark this method shows a significant improvement in gradient change for the harder tasks (see Figure 4(a) (blue)). We discuss the overall performance of our method and compare with baselines in the following section.
5 EXPERIMENTS
We evaluate our proposed method on continuous control multi-task experiments in both offline and online RL settings. In offline RL, it is standard to compare performance with an expert (Fujimoto et al., 2018; Fu et al., 2020; Gülçehre et al., 2020). Thus we evaluate the effectiveness of our offline method (Section 5.1) by computing normalized score with respect provided expert. Further use Meta- World (Yu et al., 2019) multi-task environments to compare with other baselines in both offline (Section 5.1) and online (Section 5.2) settings.

---

Under review as a conference paper at ICLR 2023
Table 3: Performance comparison in Meta- World offline. We compare the final success-rate (mean and std over 10 seeds) of NPF on MT10 tasks with Offline-MT and Offline- MHHMT baselines on offline RL algorithms.
Experiment
NPF
Offline MT
Offinne MHMT
BCQ
IQL
BCQ
IQL
BCQ
IQL
MT-10 lasks
100 1 0.0
97.3 7.17 81.5 24.15
79,1 1 26.81
95.9 1 10.44 96.5 7.10
5.1 OFFLINE MULKI-TASK PERFORMANCE EVALUATION
We train Soft-Actor- (SAC) for each task to collect an expert dataset (see Section B.2). We use BCQ (Fujimoto al., 2018) and IQL. (Kostrikov et al., 2021) for our offline training and conduct our experiments on multiple high dimensional continuous control domains. For the offline RL setup, when designing experiments, we focus on two criteria. First, how does our method perform compared to the performance of single task experts and second, how does our method perform when compared to common baseline. When comparing an expert, we carefully select a range of multitask benchmarks ensuring that our method (1) is tested with diverse range tasks (Multitask Hal fCheetah) (2) can yet performed in a controlled tasks that are similar (Ha faaffhheete ah Constrained Velocity) in nature and (3) can master skills that can easily be transferred in real world applications (Multitask Quadrupod To compare to common baseline algorithms, we evaluate on the Meta- World Multi-Task benchmark.
Multitask HalfCheetah: We train Hal @Cheetah for five different kinds of task (Finn et al., 2017; Rakelly al., 2019) where it needs to (i) run forward, (ii) run backward, (iii) jump, (iv) jump while running forward and (v) jump while running backward. It is important to note that the gait movements of these tasks are very divergent and snapshots are provided in the Appendix.
Constrained Velocity HalfCheetah: We consider another variation of the Hal HalfCheetah environment where it only needs to go forward but we constrain the velocity of the HalfCheetal to six different target values from 0.5 to max speed 3.0 (Finn et al., 2017; Rakelly et al., 2019). The success of multitask training different velocities has greater impact in real-world robotic tasks, where agents are expected to run at different speeds in different kinds of terrain (Lee et al., 2020; Fu al., 2021; Yang et al., 2021: Hwangbo al., 2019).
Multitask Quadrupod: Multitask Quadrupod consists of (i) run forward, (ii) run backward, (iii) hopturn: hop and turn to left followed by turning back to initial position and (iv) sidestep: take a step left and take a step to right to the initial position. This simulated environment is commonly used in simulation to real-world transfer experiments (Li et al., 2021; Peng et al., 2020; Yang et al., 2021).
Each task in multitask experiment is trained with an equal number of gradient updates. We compute the average episodic return over 10 episodes of the trained agent. Our results in Table are reported over seeds 0-4 of the Gym simulator and the network initialization. We run the Ha fcheeteal experiments for IM and Quadrupod for 500k gradient updates. For evaluation, we compare with expert performance and normalize the episodic return using standard offline evaluation metric (Fu score random score
et al., 2020), normalized score = expent soure random soure 100), where random score is generated by unrolling a randomly initialized policy and averaged over 100 episodes. A score of 100 represents the average returns of domain-specific expert. In this paper. for multitasks, we always report the mean and standard-deviation over the normalized scores. Our proposed method achieves expert- like performance for most of the tasks in sets of experiments. Results are in Table 2. Per task performance is in the Appendix (Table 6).
In Table we find the Half the reliability of our method, we conduct a sample complexity analysis Arnob et al. (202Ia) In Figure 5, we compare NPF performance with baseline performance for different reduced training sample size. We use two baselines: (a) multi-task algorithms (MT): where we train offline algorithms for multiple tasks without changing the objective function or the neural architecture. (b) A multi-head (MHMT) variant where we use independent heads for tasks. Both baselines are non-pruned (use 100% of network weights). while our method uses only 5%. The interquantile plot (Figure 5) of normalized scores shows a consistent improvement when BCQ is trained with NPF.
multitasks to be more challenging for NPF. To further validate
Meta-World (Yu et al., 2019) is a set of robotic manipulation tasks for benchmarking multitask- learning and meta-learning. In this experiment, consider the MT10 benchmark (Appendix 1.4) from Meta-World, where we have 10 diverse tasks, evaluated with the mean success rate over 10 tasks.
We run our experiments on MT10 with 10 seeds for 100k gradient updates for each task. We evaluate trained agent 100 times, which gives more accurate performance, on all MT tasks. We report
7

---

Under review as a conference paper at ICLR 2023
Table 4: Performance comparison of Meta- World online. We compare the success-rate of NPF on MT10 online.
We also show the reduced parameter complexity NPF comparing to other baseline methods.
Experiments
SAC+NPF
PCGrad
SM
CARE
SAC+ME
Yu al. (2020)
Yang et al. (2020)
Sodhami (2021)
Sodhani al. (2021)
MT10 tasks
77.4 17k
13.12
72.0 1 2.2
84 1 5.1
74 4.3
Parameter Counts
340k
135k
486k
344k
Parameter Reduction
20x
8x
28x
20x
the mean performance with 95% confidence interval in Table 3 (see individual task performance in Table 7 in Appendix). We collect Ik expert trajectory (150k samples) per task using SAC (Haarnoja et al., 2018) expert trained on single task. Using only 5% of the network weights, neural-pathways able to exceed other baselines. Whereas other baselines suffer from high variance in performance, with BCQ, we get a perfect score on all MT10 tasks.
5.2 ONLINE MULTITASK PERFORMANCE EVALUATION
A recent work (Sodhani et al., 2021) benchmarks state-of- the-art online multitask RL algorithms in the Meta-World environments, shown in Table 4. The benchmark is re- ported after 2M gradient updates (for each environment) over seeds 0-9. We follow their evaluation criterion to re- port success rate of our algorithm. Sodhani et al. (2021) fo- cuses on the importance of contextual information and pro- poses Contextual Attention -based Representation (CARE), using a pretrained language model (Liu et al., 2019) to en- code meta-data and retrieve contextual information about the task. The contextual information added with the envi- ronment observation is used to train SAC with Mixture of encoders (SAC+ME). As seen in Table 4, the success of CARE largely depends on metadata which is not always available in all multitask settings. Soft-modularization (SM) proposes effective sharing and reusing network com- ponents across tasks. As reported in Sodhani et al. (2021), the performance of SM (Yang et al., 2020) varies largely when evaluated over 10 different sub-tasks. We also include Projecting Conflicting Gradiens (PCGrad, Yu et al. (2020)), which focuses tackling gradient interference. If gradients for different tasks point away from one another, PCGrad alters gradient direction to mitigate these interference.
IQ PL-MetaWorlk
61
SO
2BI
TW
SC
SF
5.6
50
04
30
42
30
28
41
65
F5
21
37
42
46
38
26
30
35
30
35
pHo hop-hoort
25
44
42
45
48
32
37
16
30
31
30
3C
Widdow <low
31
27
3t
Figure 6: The % of trainable weights the IQL. policy shares across MT10 tasks with NPF. The diagonal is the % of weights unique to each task.
Compared to these baseline methods, we propose a simpler alternative that achieves competitive performance while not requiring additional task information, complicated network architecture such as routing through modular networks, or a complex gradient update procedure. We train each task with a fraction of the parameters (see Table 4) and which potentially leads to higher inference (Molchanov et al., 2017; Luo et al., 2017) and reduces the energy consumption (Yang et al., 2016), which makes it more suitable than any other multitask methods to apply in world applications.
5.3 OVERLAPPING NEURAL PATHWAYS
In this work, we consider choosing the neural pathways for each task independently from one another. We find large % of the pathways overlap, reducing the number of trainable parameters. The pathway configuration is dictated by three factors: (1) different learning objectives (i.e. offline, online), (2) scoring function (i.e. equation 1. 2), and (3) training samples used to optimize the scoring function. In Figure 6 we see the % of active weights (5% of the actual network) in the policy that are shared with other tasks for IQL multitask-offlin
training. The diagonal of these symmetric matrices represent the number of unique weights that are optimized only for one task, and the columns represent the % of weights each task shares with others. As we see in Figure 6, for all tasks the % of weights that are optimized for just one task is very low. For push, only 5.5% of the active weights are uniquely trained for the task. We also show similar matrices for other experiments in Appendix (Figure 15), where we found the percentage of overlapping weights does not correlate with the task similarity. This disproves a conventional class of thinking that the success of multitask in neural networks depends on relevant data sharing (Yu et al., 2021; Fifty et al., 2021) or similar task training (Sodhani et al., 2021;
8

---

Under review as a conference paper at ICLR 2023
Yang et al., 2020). Rather, neural networks are capable of learning multiple tasks simultaneously as long as the neurons are wired properly.
Overlapping Pathways Lead to Fewer Parameters: For each task we only use 5% of the total weights of the actual network while maintaining expert-like performance for all tasks. For N tasks has upper bound of 20 parameters when all the pathways are unique, but find the pathways to overlap significantly (Figure 6 and Figure 15, Appendix), further reducing the number of active weights. This reduces the energy consumption (Yang al., 2016) of the trained network and makes inference more efficient (Molchanov et al., 2017; Luo et al., 2017). To understand how NPF compacts multiple experts into single network, we look at the number of policy parameters IQL requires compared to SAC with and without pruning on the individual MT tasks. In our experiments, the total number of network parameters for SAC and IQL. are identical with 1, 092, 608 parameters in the policy network. With MT10 the number increases 10x. Due to the inherent shared structure of neural pathways, our method requires only 1.82% (averaged over 10 seeds) of the original network. To get equivalent utility we have to train ten SAC agents with 98.18% pruned networks, which is not possible using any existing pruning techniques without losing performance (Frankle & Carbin, 2019; Lee al., 2018; Wang al., 2020a; Tanaka et al., 2020). As seen in Wang et al. (2020a) 98% reduction of the network weights pruning techniques suffers from significant performance drop due to layer collapse. NPF reduces the energy consumption (Yang et al., 201 6) of the trained network, making inference more efficient (Molchanov et al., 2017; Luo et al., 2017). See the comparison in Appendix (Table 8).
6 CONCLUSION
We propose a novel multitask learning approach in which we train task-specific neural pathways within single deep neural network. We extend the idea of finding sub-networks from the pruning literature (Frankle & Carbin, 2019; Wang et al., 2020a; Feinberg, 1982) and show our proposed method can achieve expert-like performance on many high dimensional continuous control tasks in both online and offline settings. Unlike conventional approaches, where multitask methods mostly rely on task similarity, modular network training, or underlying context learning, we propose a new way to tackle multitask problems and show function approximation with a single deep neural network sufficient. We empirically show that the single-shot lottery network search fails under data distribution shift We overcome this by proposing data adaptive pathway discovery, which continues to reconfigure neural pathways as the RI agent gets more training updates and continues to collect better training samples. Leveraging offline and online RL algorithms, in parallel, consisting of only 5% of the total neural network weights, leading to significant reductions in training time while still achieving competitive performance neural-pathwe. overlap, we get further reduction in network parameters. Compared to single-task experts, we get 2 98% pruned parameters in the Meta-World experiments, a significant decrease in parameters which is not possible for single-task training with any existing pruning techniques without performance drop. Thus, NPF provide high inference rate while reducing energy consumption (Yang et al., 2016), making it more very suitable for real-world applications.
train task- aask-specific neural pathways
Meta-World. Due to significant
This work opens up a new avenue of inquiry into the possibility of a single neural network being trained for multiple purposes (i.e. learning different features, multiple value functions) through targeted parameter training while providing better efficiency of utilizing the parameter space neural networks. Our method can be integrated into other selective data-sharing and gradient update methods but this requires further research. Expanding the scope of the experiments to include more tasks would also be informative.

---

Under review as a conference paper at ICLR 2023
7 KEPRODUCIBILITY:
With this work we provide an open-souree implementation of our framework at the code in the anonymous github repo.
Users can run NPF and reproduce the experiments utilizing
REFERENCES
Milad Alizadeh. Single-shot network pruning (snip) in pytorch. GitHub, 2019. URL https:
//github. com/mill -add /nipp.
Samin Yeasar Arnob, Riashat Islam, and Doina Precup. Importance of empirical sample complexity analysis for offline reinforcement learning. CoRR, abs/21 15,55588, 2021a. URL https://
axxiv.orgg/bbs/22..1555
Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. Single-shot pruning for offline reinforcement learning. arXiv preprint arXiv:2 15579, 2021b.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 2:129-146, 2020.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. CoRR, abs/1711.02255 2017. URL http: / / www.a.ivvvooo org/ abss////////////// 1711.1111111 C2257.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. CoRR, abs/1600.07088, 2016. URL http:/ //axiv.. org/ abs///////// 609. 07088.
Daniel Drubach. The brain explained. Pearson, 2000.
Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Mchrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, 2020.
Irwin Feinberg. Schizophrenia: caused by fault in programmed synaptic elimination during adolescence? Journal of psychiatric research, 19-4)339--34, 1982.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 101.00771. 2017.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Mgnel-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.03300. 2017. URL http: 03400.
/1703.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019.
Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020. URL aittps://arxi:: org/abs 2004 4. 07219.
Zipeng Fu, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Minimizing energy consumption leads the emergence of gaits in legged robots. CORR, abs/2111.01674, 2021. URL https:
//arxivv.oo org/ 2211. 01674.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. CoRR, abs/1812.02990 2018. URL http: aaaii.iv wwwaaaiwwvgraaaaeeeeeeeeaa www./orrr//aaa..eeee 029.0.000000 02900.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, arXiv preprint (900:/902.009500. 2019.
10

---

Under review as a conference paper at ICLR 2023
Melvyn A. Goodale, John Paul Meenan, Heinrich H. Bülthoff, David A. Nicolle, Kelly J. Murphy, and Carolynn I. Racicot. Separate neural pathways for the visual analysis of object shape in perception and prehension. Current Biology, 4(7):604-610, 1994. ISSN 0960-9822. doi: https:
URL https:: /wwww sci encedi rect. com/
Çaglar Gülçehre, Ziyu Wang. Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel J. Mankowitz, Cosmin Paduraru, Gabriel Dulac- Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Oär Nachum, George Tucker, Nicolas Heess, and Nando de Freitas. RL unplugged: Benchmarks for offline reinforcement learning. CoRR, abs/2006. 13888, 2020. URL https: htapp:///axiivi /trx/aiivv///.////////a//////ava wwwpiiioorr//baa.. 2006 13888.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, 0ts/1801.01221
2018. URL http: / http:/tttt//aaa...rrg//////
8018001.000 001..129000
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. CoRR, abs/1901.08652, 2019. URL http: arxiv. org/ bwgg/abs/ae abs/1901. 08652.
Uwe Jürgens. Neural pathways underlying vocal control. Neuroscience & Biobehavioral Reviews, 26 (2):235-258, 2002.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2014a.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 2014b.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. CoRR, abs/21 10.06169, 2021. URL.httss//axxiive wttpxxxwww..ggg////1//111 261.0.666...10..66666.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. CoRR, abs/2010.11251, 2020. URL https: // //arxiv. http//bb//2000 11251.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based connection sensitivity, arXiv preprint arXiv:1810.02340. 2018.
Tianyu Li, Jungdam Won, Schoon Ha, and Akshara Rai. Model-based motion imitation for agile, diverse and generalizable quadupedal locomotion. CoRR, abs/2109. 13362, 2021. URL https:
//arxiv.org/aaaaal... www.///g/abb///aeeeeee 13362.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, htss/997..11999 2019, URL http: / arxiv. org/ abs 11977 11 1692
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression, 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. CoRR, 0bbs/1604.03533 2016. URL atttp://arxiv http://aaxxxv....aaaaaaao 1 03539.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, htss/1602.0183.. 2016. URL http: http:///rxivv org/ abs 102.. 01783.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference, 2017.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. Advances in neural information processing systems, 1, 1988.
Feng Niu, Benjamin Recht, Christopher Re, and Stephen J. Wright. Hogwild!: llok--free approach to parallelizing stochastic gradient descent, 2011.

---

Under review as a conference paper at ICLR 2023
Riyasat Ohib, Nicolas Gillis, Niccolò Dalmasso, Sameena Shah, Vamsi K Potluru, and Sergey Plis. Explicit group sparse projection with applications to deep learning and nmf. arXiv e-prints, pp. arXiv-1912, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Tomáš Paus, Alex Zijdenbos, Keith Worsley, D Louis Collins, Jonathan Blumenthal, Jay N Giedd, Judith I. Rapoport, and Alan C Evans. Structural maturation of neural pathways in children and adolescents: vivo study. Science, 183(5409):1008-1111 1999.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. CoRR, abs/2004.00722 2020. URL https: / arxiv. org/ alos 2004. 00784.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforreement learning via probabilistic context variables. CoRR, abs/1903.08254, 2019. URL
org/ tabs/1903 08254.
Lucy B. Rorke. Central Nervous System Plasticity and Repair. Journal of Neu-
ropathology & Experimental Neurology, 44(5):530-530, 09 1985. ISSN 0022- 3069. doi: 10.1097/00005072-198800000
URL https: //tp:////oiiii/ orgpo/o///.0197/
00005072- -
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. CoRR, abs/1711.01239, 2017. URL http: //
01239.
Peter H Rudebeck, Mark E Walton, Angharad N Smyth, David M Bannerman, and Matthew FS Rushworth. Separate neural pathways process different decision costs. Nature neuroscience, 9()): 1161-1168, 2006.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1606.04ee 04671, 2016. URL http: //apivv.///iv////aa//a wwwpsiiiiorrrrooooooooooo abs 1666.66666 04671.
Jill Sakai. Core concept: How synaptic pruning shapes neural wiring during development and, possibly, in disease. Proceedings of the National Academy Sciences, 117022)) 2020. ISSN 0027-8424. doi: 10. (0.1073/pms.22002211111 URL httph::tp///. wwww www.wwppaa..... org/
content/117/881166
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. CoRR, abs/1810.0465h 2018. URL http: arxiv. org/ abs/ 1810. 04650.
Laura Smith, J. Chase Kew, Xue Bin Peng. Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. CoRR, abs/2110.05457, 2021. URL https: //tpr//aivvvv///////////aa/aaava org/ abs 221001111111 11110000 05457.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. CoRR, abs/2102.01177, 2021. URL https:: :/tp:ii.//////x..//.///////evevm org/abs 2202. 06177.
Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. CoRR, 1bs/1491,06844, 2019, URL http: rxviiiiiii wor./v.orrr///o.oooo......... /ab///////// 1912..91111. 918.06844
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. CoRR, abs/2006.05467, 2020. URL htttp::/aree wtt./oos//////oooooooooo 05467.
12

---

Under review as a conference paper at ICLR 2023
Garrett Thomas.
Halfcheetah goal velocity and multitask.
GitHub, committtcc///oo/oo//o//oo.oooott
2020.
URL
sttps://githur
Garrett Thomas. Implicit q-learning (iql) in pytorch. Gitflub, 2021. URL https://github.
com/gwthomas//og
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020a.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376. 2020b.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scalc-invariant sparsity measures. arXiv preprint arXiv:1908.09979, 2019.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. CoRR, abs/2003. 13661, 2020. URL UKL/ttpss//arx....or////as /2003. 13661.
Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. CoRR, abs/1611.05118,, 2016. URL http://arxiv, orq/abs/ 1611 05128.
Yuxiang Yang, Tingnan Zhang, Erwin Coumans, Jie Tan, and Byron Boots. Fast and efficient locomotion via learned gait transitions. CoRR, abs/2104.04644, 2021. URL htthtt:s///xxx orq/abs/ 2104. 04644.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
CoRR, abs/1910. 2019. URL http: /atir//arxiit aariv.iviv..////v///aa//a/aaaavv org/ 1910. 10897.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. CoRR, abs/2001.06782, 2020. URL https: //arxiv.
org/abs/ hbt//bb//20 20000.011000 06782.66.22
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. CoRR, abs/2109.08128. 2021. URL https: arxiiv org aaaaaba/211 08128.
Karl Zilles. Neuronal plasticity as an adaptive property of the central nervous system. Annals of Anatomy-Anatomischer Anzeiger, 174(5):88--31, 1992.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL THEORETICAL BACKGROUND
A.1 OFFLINE AND ONLINE THEORY
Fujimoto et al. (2018) highlight the fact that, since value estimation is trained with fixed dataset, provides an erroneous estimation when the policy takes an action which is out of distribution from the dataset on which the value function is trained. To overcome extrapolation error, BCQ proposes batch-constrained learning, where agents are trained to maximize reward while minimizing the mismatch between the state-action visitation of the policy and the state-action pairs contained in the batch. For a given state, BCQ uses a generative model Gw. e.g. a Variational Auto-encoder Kingma Welling 2a), to generate n actions with high similarity to the batch dataset, and then it selects the action for which it gets the highest value: I (T)) = arg max A, Qs a;). where A, {G (s)) G.,, is trained to minimize the KL divergence with the actions sampled from batch dataset Kingma & Welling (b014b). The action-valus function or Q-function, Qo learned through minimizing TD-error is xQ2)) = _Bs,,,s")> "rais"")DD,ddd D.a.D'''m (r(ss a) + 2Qa(s',
Qo a) In our experiments, we also consider variation of BCQ (denoted as BCQ-v2) in which we sample actions only using VAE (TT)) (s)).
Since Offline RL faces distribution shift in its value estimation due to the different distribution of he policy and expert sample. Implicit Q-learning (IQL) Kostrikov et al. (2021) proposes to avoid estimating the value for policy distribution. Instead, trains the action-value function Qe using a SARSA-style update, thus enabling multi-step dynamic programming updates. IQI. uses expectile regression to predict an upper expectile TD target that approximates the maximum of r(s, a) + y (sQa(s' a')]. IQL uses a separate value function by fitting upper expectile V₂ using objective function: J(VP) = Es [45 (Qo(s', a') (s)),, where 23 is asymmetric least squares. This value is used update Q function using: ((Q)) = Es as' [(r(s, a) + +>Vo(s') Qa(s, a))].. The corresponding policy is extracted using advantage weighted behavior cloning, which also avoids querying out-of- oo--smple actions:
E.,,,'' "'"",,,''''''' (IQ> a) (s(s)) log {
We consider no prior knowledge about the task in the online RL setting, An RL agent is expected to interact with a simulated environment and leverage the collected experience to learn the task through maximizing a hand-designed RL algorithm and trained for divergent multitask objectives. To demonstrate the effectiveness of our method in the online setting, we use Soft-Actor-Crite (SAC) Haarnoja et al. (201 8) algorithm in our experiment. SAC optimizes regropy- policy objectives to drive an agent to a more exploratory behavior while optimizing its policy. Entropy is used to encourage policy to do more exploratory behavior and ensure that it does not collapse into repeatedly selecting a particular action.
reward function. Neural pathways can be integrated into any online
The entropy regularized objective function is as follows:J = Σt Ex.-dy, [r(s,, ax) + (He (-]s))], here a is temperature parameter, which controls the entropy, HH parameter, hence the stochasticity of the policy. It determines the relative importance of the entropy term against the reward. The conventional objective for policy gradient is recovered when a - 0. SAC learns Q-function using an entropy regularized objective called soft Q-function. The soft Q-function parameters are trained to minimize the following objective: = E( (aaiiii((ooos (Qo( at) -r(s, at)
7 ylEarpp awox (5t41, ((+1)= - a log me (14+1)444))))
B
ADDITIONAL IMPLEMENTATION DETAILS
B. LIBRARIES
We run our algorithm in (TTorch-1.9.00 Paszke et al. (2019) and use following libraries: Soft-Actor- Critic (SAC) Yarats & Kostrikov (2020), Implicit Q-learning (IQL) Thomas (2021), Single-shot pruning (SNIP) Alizadeh (2019) and official BCQ Fujimoto al. (2018) implementation.
B.2 OFFLINE DATA COLLECTION
We use the Halfcheetah control environment available from Thomas (2020) and train Soft-Actor-Critt (SAC) Haarnoja et al. (2018) 8) online for 1 million time steps for each task and collect 1000 expert trajectory.
For Quadrupod tasks we use the quadrupod environment and trained agents from Smith et al. (2021) and collected 1000 trajectories for each task. During data collection we find it important to use
14

---

Under review as a conference paper at ICLR 2023
stochastic policy to add data diversity, otherwise every trajectory follows the exact consecutive acttins and rewards sequence due to the deterministic nature of the environment transition function.
We train Crft-Acttr-Crittic (SAC) Haarnoja et al. (2018) on Meta-World Yu et al. (2019) environments for 3 million steps, where sample batch of 1024 sample and the hidden dimension is 1024. Similar to Sodhani et al. (2021) I) we truncate the episode for 150 steps. We collected 1k trajectories for each environment, where diversity in the training sample. We keep the environment seed fixed at O, since Meta-World World Yu et al. (2019) goal position is different for different seeds. Since we are not learning algorithm, we need to keep this seed fixed other wise for each seed we have to train and collect expert data for offline training.
take sample action of the normal distribution rather than the mean have
conddionid
B.3
In table we present the network typer-parameteen of different algorithms that are used in this work. We make exception in MetaWorld online experiments, where we use mini-batch size 128 and optimizer AdamW.
Table 5: Hyperparameter of the network architecture used to train and evaluate offline and online RL experiments.
Hyper-paranee ppper-parrmeter
BCQ
IQL
SAC
hyper-paramettr
Optimizer
Adam
Adam
Adam
Critic learning
le-3
Ie-3
le-3
Actor learning rate
le-3
le-3
le-3
Mini-batch size
256
256
256
Discount factor
0.99
0.99
0.99
Target update rate
Se-3
5e-3
Se-3
Policy update frequency
Architecture
Critic hidden dim
[400, 300]
[1024, 1024]
[400, 400, 400]
Critic activation function
ReLU
ReLU
ReLUU
Actor hidden dim
[400,300]
[1024, 1024]
[400, 400, 400]
Actor activation function
ReLU
ReLU
ReLU
VAE hidden dim
[750, 750]
Value hidden dim
[1024, 1024]
B.4 PERFORMANCE EVALUATION
Each task in multitask experiment is trained with an equal number of gradient updates. We evaluate performance every 5000 gradient updates, where each evaluation reports the average episodic return over 10 episodes. Our results are reported over multiple seeds of the simulator and the network initialization.
We evaluate our agent on average episodic return of 10 episodes. For offline experiment, we normalize the episodic return using standard proposed metric from al. (2020): Normalized score 100). We plot the mean normalized score of multiple seed with 100% confidence interval for all experiments. For Meta World (in both offline and online) we evaluate the percentage of success.
score - random score
experi score
score
For Gym environment conduct experiments for seed 0-4 (if not stated otherwise). We run all MetaWorld experiments for seed 10 seeds (0-10) and report the percentage of success after evaluating for 100 episodes.
C ADDITIONAL RESULTS
C.1 OFFLINE MULTITASK EXPERIMENTS
We provide the individual task performance compared to single task expert for Halfcheetah and Qua drudrupod multitask experiments in Table 6.
15

---

Under review as a conference paper at ICLR 2023
Table 6: Individual task normalized score (performance compared to single task expert) Ha. SaafCheetann and Quadrupoa, tasks.
Experiment
Tasks
BCQ
IQL
HalECheetal
Forward
99.68 0.04
multitask
Backward
96.51 -1 0.19
98.68 1 0.83
Jump
9.46
Forward-Jump
85.25 1. 0.54
87.35 1: 27.0
Backward-Jump
83.3 + 1.46
87.34 + 14.23
HalfCheetah
Velocity 0.5
100.04 0.03
99.1 2.18
Constrinned speed
Velocity 1.0
100.07 0.03
100.05 + 0.04
Velocity 1.5
100.03 0.05
99.99 + 0.06
Velocity 2.0
Velucity 2.5
100.02 1: 0.04
100.04 1 0.03
Velocity
Quzdrupod multitask
Forward
116.19
Backward
110.98 1.21
112.93 + 1.05
Hoptum
122.43 -
123.33 =
Sidestep
111.8 11.8::0.90 0.96
113.69 = 0.43
(a) run forward
(h) run backward
(c) forward jump
(d) backward jump
(e) jump
Figure 7: Snapshot of trained policy for Halfcheetah multitasks.
(a) velocity 0.5
(b) velocity 1.0
(c) velocity 2.0
(d) velocity 3.0
Figure 8: Snapshot of trained policy for Hal fcheetah under four different constrained velocity.
C.1.1 PERFORMANCE CURVE OF HALFCHEETAH MULTITASK
Figure 10 presents the performance of five different Hal fcheetah tasks: (1) rum forward, (2) jump while run forward, (3) run backward, (4) jump while run backward and just(5) jump. Snapshots of the different tasks are demonstraded in Figure 7.
16

---

Under review as a conference paper at ICLR 2023
(a) pace forward
(b) pace backward
(c) hopturn
(d) sidesteps
Figure 9: Snapshot of trained policy for Quadrupod multitasks.
Itheriah kaward
halforetton
Number iateratantedeers.cro
Mumber
Runber
BCQ
BCQ
Numbar gradiees
grudieii
Figure 10: Performance (Normalized Score) plot of BCQ and IQL FCheetah multitask (Forward, Forward-Jump. Backward, Backward- Jump. Jump) trained with NPF.
C.1.2 PERFORMANCE CURVE OF HALFCHEETAI CONSTRAINT GOAL VELOCITY
In Figure 11 we present the performance of six (6) constrained goal velocity for Ha fcheetah running forward, where vary the constrained from 0.5 to max speed 3.0 with constant increase of the speed. The tasks are trained for 500k gradient updates and evaluated every 5000 gradient updates. Snapshots of the different tasks are demonstraded in Figure 8.
C.1.3 PERFORMANCE CURVE MULTITASK QUADRUPED ROBOT
We use the Quadruped robot to perform four (4) tasks: hopturn, pace forward, pace backward, sidestep. We use Smith et al. (2021) as our expert collect 500k (1000 trajectory) expert data for each task. The environment has deterministic transition function and thus we take sample action from normal distribution instead of taking distribution mean to get diverse set of training dataset. Performance curve with NPF for individual task in shown in Figure 12. Snapshots of the different tasks are demonstraded Figure 9.
C.1.4 PERFORMANCE ON METAWORLD
performance of individual MT10 environments over 10 seeds and each evaluated for 100 episodes. We also compare individual performance with Conservative data sharing (CDS) Yu et al. (2021), similarity, also reports it's Meta- World experiments on 4 environments and reported performance over 6 seeds with 95% confidence interval.
In Table 7 report
offfine-multitas learning algorithm that focuses training based task
17

---

Under review as a conference paper at ICLR 2023
hall:thetab-ooone postvel-)
St soaleel 1.5
fitherahh
horaaeed Scare
Narra led Score
Narra Norratedd
Nimber gradlencuddene 145
Number gradient updates
Number gradent 145
halfcheetsh 2e
halfcheetah postrel 2.5
fichetthh goatect 3.0
Nor Normatto Stors
Nor duod SOOE
hommabed
IOL
IQL
humser usdaie
graciemt
grodkent upducus
145
Figure 11: Performance (Normalized Score) plot of BCQ and IQL HalfCheetah with constrained velocity trained with NPF.
HoYe 244 S00e
Hore
(a) pace
(b) pace backward
(c) sidestep
(d) hopturn
Figure 12: Performance (Normalized Score) plot of BCQ and IQL Quadrupod multitasks with NPE.
Table 7: Individual task performance (success-rate) comparison of Meta-World.
MTI0
NPF
BCQ
IQL
CDS Yu at. (2021)
reach
100.0 0.0
93.0 22.14
push
100.0 0.0
100.0
pick-place
dper-open
100.0 0.0
100.0 1: 0.D
58.4 9.3
drawer-open
drawer-close
100.0 0.0
100.0 0.0
98.8 0.7
buiton-press-oopooww
100.0
100.0
peg peg-insert-side
100.0
100.0 0.0
window open
100.0 0.0
100.0
window-close
door-close
57.2 16.2
Overall
100 + 0.0
97.3 + 7.17
72.0 = 26.2
C.2 ONLINE MULTITASK EXPERIMENTS
We provide the pseduo-code for Data Adaptive Pathway Discovery (DAPD) proposed for online multitask setting in Algorithm 2.
18

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Multitask Online Training
Episodic Return nt task, Ri Σíz r Stopping threshold Tи: an hyper- parameter; Tempo-
rary replay buffers n²th task, TDn: only contains most recent experience; Replay buffers n task, Dn: Mask for th tasks, mm.
#Training loop:
for Training steps do
# Collect Data:
for Each Task do
04 To (olaaas))) s1+1 Tn(s441184 0t) & Dn + Dn U (st, ae,
ae, r(st, 84-1)
TD - TD, U {st, ar, (8t, (e), S1413
end for
Find the Task Specific Weight Masks: for Each Task do
> If not R Tи m. = FindMeuralPathaayy (TD..) using Equation 2.
end for
# Update Policy using Online Algorithm:
b Update To ( [m,, M2 m, l. (D,, D2.. D,.)) Follow Same Approach as Algorithm 1 for weight and gradient share among tasks
end for
24P0 wituured
Bandrn
Ceadert pitii Dett
Statie ieeie
Grdd ort Unditi Cott
(a) run backward
(b) jump
(c) forward jump
(d) backward jump
Figure 13: Compare Performance of Sinole-Shot and Data adaptive Pathway Discovery (DAPD with and without Threshold Stopping Criterion) single Mujaco tasks.
Seddent TAQ
Ood Updina L26
Gradien Updana lse
(a) run backward
(b) jump
(c) forward jump
(d) backward jump
Figure 14: Compare Performance of Data adaptive Pathway Discovery (DAPD) with different moving average on single Mujoco tasks.
C.3 OVERLAPPING NEURAL PATHWAYS
Important Distinction with Conventional Pruning Methods: Neural Pathway has crucial technical difference with pruning methods in the implementation. As demonstrated in Figure 2, we mask the neural network weights, allowing forward and backward propagation only through a subset of weights, whereas pruning methods Feinberg (1982): Wang et al. (2020a): Frankle & Carbin (2019) set many weights to zero. This technique allows the unimportant weights to survive and be used for other tasks. In general, pruning methods used reduce memory footprint, increase throughput
19

---

Under review as a conference paper at ICLR 2023
ZCD Hal'hhudta Puktack
IOLttafccnetan Pulkart
ECO-la Eterral Conmrsnned Malacly
witicaaa Con
5
65
(8)
(b)
(c)
WotoMetaWordd
ML-MeraWorld
200 Qundiaadd Pu Htae
S4
35
32
AL
35
20
X 29 24
25
(e)
(h)
Figure 15: Shows the percentage of trainable weights are being shared among different tasks. In and b, we compare shared weights among five Hal f Fcheetah tasks: (1) forward (F), (2) (B)),, (3) jump(B), (4) Forward while Jumping (FJ) (5) Backward Jumping (B.T). In and d, we compare shared weights among six Halfcheet tasks where increase forward velocity 0.5-3.0. In e and we compare four Quadrupod tasks. In y and i we show the shared weights among MetaWorld tasks.
or even enhance generalization. On the other hand, our goal is not to prune but to re- re-purpose the capacity of the network to learn many tasks (to a larger extent than in other works).
In Figure 15 we demonstrate the percentage of active weights (5% of actual network) in policy network that are shared with other tasks for different experiments and for different offline RL algorithms. The diagonal of these symmetric matrices represent the number of unique weights that are optimized only for one task. And the columns represents the percentage of weights the task shares with others.
is important to note, in Figure 15, for all our experiments the percentage of weights that are optimized for just one task is very low. For examples, in Figure 15-a for forward task, only 26% of the active weights are uniquely trained for the task. We only activate/allow 5% of the original network weights for one task after pruning. Which means only 0.13% of the original network is trained uniquely to make Halfcheetah run forward like an expert. Also we do not find the percentage of overlapping to correlate with the tasks similarity.
In Table 8, we compare the number of policy network parameters IQL requires compared to SAC with and without pruning when trained for MT10 environments as individual tasks. In our experiments, the network hidden layers and depth (total number of network parameters) for SAC and IQL are identical. Our SAC expert has 1, 092, 608 weight parameters in the policy network. For MT10 (10 tasks), the number increases 10rr On the other hand, due to the inherent shared structure of neural pathways, our method requires only 1.82% (in average over 10 seeds) of the original network. To get equivalent utility we have to train ten SAC agents with 98,189 18% pruned network, which is not possible using any existing pruming techniques without losing performance Frankle & Carbin (2019); Lee et al. (2018); Wang et al. (2020a); Tanaka et al. (2020). As seen in Wang al. (2020a) at 98% reduction of the network weights pruning techniques suffers from significant performance drop due to layer collapse. But in our case, the more tasks we have the more space efficient our method becomes. Even when we compare to 95% pruned SAC trained on 10 different individual tasks ((B) in table 8), we have 36.4% reduction in actor parameters.
20

---

Under review as a conference paper at ICLR 2023
Table Comparison the number parameters required Meta-World for different methods.
MT- trained as separate tasks
MT-10 multitasks trained using NPF
(A) w/o pruning
(B) 95% pruned
(C) 95% pruned
Parameter counts
(Policy/Actor Network)
11 Million (10.926.080)
546,304
198, 956 7, 636 1.82 0.07% of
21


Reviews:
[{"review_id": "r1JtXYgPoBj", "reviewer": "Reviewer_6eEZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a multitask RL framework by learning multiple pathways through a single network, called Neural Pathway Framework (NPF), which aims to find a task-specific optimized mask. Started with a method from a network pruning literature, the authors further proposed to use replay buffer and average multiple iteratures to stablize training. Experimental results shows performance improvement with reduced parameter count.", "summary_of_the_review": "The paper studies an interesting problem and gives a potential solution. My major concerns are about clarity. I might will change my recommendation after things are more clearer.", "strength_and_weaknesses": "I do have some concerns regarding this paper, especially in terms of the clarity.\n\nThe clarity of the paper needs to be improved. \n1. How is global-model optimized in Algorithm 1? Providing an equation would be very helpful. \n2. The most important equation (1) in paper is not clear to me. By my understanding, m_n is the mask for n_th task, while S(theta; D) is the objective function. Why these two terms are equivalent?\n3. Equation 2 is also unclear to me. It envolves two techniques to stablized the training, keep a temporary replay buffer TD_n and re-confriguring using the average of the previous M iteration scores. In paper, it seems that keeping a temporary replay buffer aims to keep only the most-recent samples from agent to update the mask, but averaging between several iteration is using samples span longer history, which seems contradictory. How would you interpret it?\n4. For results in Table 3, how could variance of BCQ with NPF be 0? Also, since it is normalized, I'm curious of the original performance of BCQ and IQL algorithm.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AGn2AxXGFf", "reviewer": "Reviewer_rc1X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on multitasking ability in reinforcement learning (RL). To do so, a multi-task learning framework, Neural Pathway Framework (NPF), is proposed by simultaneously training multiple specialized pathways where each pathway corresponds to a task. Finally, experiments were conducted on several continuous control tasks under both online and offline settings, including using Mujoco-based and MetaWorld environments, with results demonstrating its effectiveness in terms of performance and parameter size.\n", "summary_of_the_review": "According to my comments in both the main Weaknesses and the section of \"Clarity, Quality, Novelty\", I feel this is an empirical paper where reasons to reject outweigh reasons to accept.", "strength_and_weaknesses": "Strengths:\n- Multitasking is an important ability to achieve artificial general intelligence (AGI). This is an interesting work targets to multitask RL where the ideas build up from the neuro-science perspective on neural pathways in human brains and the machine learning perspective on finding sub-networks with pruning.\n\nWeaknesses:\n- The proposed methods are not presented in a principled way, although the main contribution of this work is primarily on the empirical side. For example, what's the formal definition of the problem? What are the loss functions or objectives for the offline or online settings? What's the mask used in the methods? What are the original networks and sub-networks? How do neural pathways/sub-networks interact with different tasks? How to guarantee or justify that the obtained sub-network is the right solution? How are the learned skills reused/transferred among tasks? What does the mask \"$m$ is fixed\" mean?\n- In Figure 4 (a), why the gradient norm becomes larger? Does that mean there exists a convergence issue? How to define the success rate in (b)?\n- This paper claims that the proposed method \"is especially suitable for real-world applications in which large datasets need to be processed in real-time or which are deployed on low-resource devices\". However, I didn't see any comparisons of time cost or computation cost shown in the experiments.\n\n\nMinors:\n- \"In the online RL setting,.\" -> \"In the online RL setting,\".\n- \"nonstationaritym\" on pp.5 should be \"nonstationarity\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9ZNCXJ4kvQ0", "reviewer": "Reviewer_Q8xY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use masks in multi-task training so that only a part of network parameters are used for a specific task. The framework considers combining both online and offline RL methods. Experiments show that the proposed method significantly reduces the training parameters and achieves competitive performance compared with previous methods.", "summary_of_the_review": "Overall, this paper points out a promising direction in MTRL and provides relatively extensive experimental results. However, as listed in the pros and cons part, the novelty, technical issues, and missing related works make this paper lean to reject.", "strength_and_weaknesses": "Strengths:\n\nThe idea of introducing sparse training to multi-task RL is a promising direction.\n\nThe proposed framework is combined with both online and offline RL methods, showing the generality.\n\nWeaknesses:\n\nThe literature review is not extensive, missing the discussion and comparison of related works [1-3].\n\n1) Some technical details are not convincing: not considering Data distributional shift in offline RL setting. Data distributional shift is a common problem in offline RL since the data set is collected by some unknown policy. When the trained policy interacts with the environment, there exist some states/actions out of distribution where the policy may behave badly. \n\n2) The masks are fixed during training. However, it's more reasonable to adjust the mask since the sensitive measure is not static over the whole state space. Also, it is not static during different training stages. On contrary, if the mask is adaptive changing, it may cause unstable training.\n\n3) The experimental results should focus more on the benefits of sparse training regarding the training time/resources saved. For example, the parameter reduction results in offline RL are not provided.\n\nExperiments should consider the full MT50 task set and compare the proposed method with CAgrad [4].\n\n[1] Sparse Multi-Task Reinforcement Learning.\n[2] Multi-task Batch Reinforcement Learning with Metric Learning.\n[3] Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models.\n[4] Conflict-averse gradient descent for multi-task learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "11Mjx31x0Y", "reviewer": "Reviewer_jX9C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to learn multiple RL tasks with the same network. SNIP is used to prune all but the 5% most \"relevant\" parameters of the network, yielding a mask for every task. Tasks can share parameters, and therefore all tasks have to be trained simultaneously. The authors show that SNIP depends strongly on the data set, and propose to regularly change the task-masks based on the newest data for online RL scenarios. The new method NPF is evaluated on 3 Mujoco and the Meta-World multi-task benchmarks with the BCQ, IQL and SAC algorithms and compared with other baselines. Results show that NPF is competitive with other multi-task approaches.\n", "summary_of_the_review": "I really liked this paper. It is not totally clear whether the method will allow something that would otherwise be very hard in multi-task RL, but it is a nice idea and in my opinion well evaluated. The only weak point is a relatively low novelty and a lack of theoretical analysis. However, unless another reviewer finds something really wrong with it, I recommend to accept the paper.", "strength_and_weaknesses": "**Strength**\n\nThe paper presents an innovative approach to learn and represent multiple tasks in one network. The approach is simple and can be adapted into many algorithms. Results look generally good. \n\n**Weaknesses**\n\nThe idea is, to the best of my knowledge, novel, but not very deep. Following the lottery hypothesis, the authors just select parameter-masks and keep them (in the offline setting) during training fixed. In the online setting the authors identify that the change in loss requires them to recompute the mask based on newer data. However, the implementation (averaging the last $M$ scores, stopping criterion) sounds very simplistic and there is no analysis on the dynamics of selected masks, ways to deal with this non-stationarity and whether the mask eventually converges, just as the policy does. More analysis on this would strengthen the paper significantly. Furthermore, there are some questions I would like to have answered:\n1. You emphasize that the training data changes the loss a lot, but in RL the loss is also changed e.g. by bootstrapping: at time $t$ the loss depends on the value function(s) with parameters $\\theta_t$. Have you evaluated this? How is the effect of this on the mask selection? \n2. What is the \"continual pathway configuration\" (referred to later as \"parallel learning\") in Figure 4? Is it NPF without the pretraining stage or training without mask or something else?\n3. How significant are the results? It looks as if the standard deviations overlap a lot. Can you make a statistical test whether your method is significantly better (p <= 0.05) than the next best baseline? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject