Efficient Reinforcement Learning by Discovering Neural Pathways
Authors: Samin Yeasar Arnob
Riyasat Ohib
Sergey M. Plis
Amy Zhang
Alessandro Sordoni
Doina Precup
Venue: NeurIPS 2024 poster (2024)
Keywords: Energy Efficient AI
Parameter Efficient
Neural Pathways
Continuous Control
Online Reinforcement Learning
Offline Reinforcement Learning
Multitask Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: To improve energy efficiency and reduce the carbon footprint, we propose Neural Pathway to efficiently use the network parameter space for reinforcement learning.

Abstract: Reinforcement learning (RL) algorithms have been very successful at tackling complex control problems, such as AlphaGo or fusion control. However, current research mainly emphasizes solution quality, often achieved by using large models trained on large amounts of data, and does not account for the financial, environmental, and societal costs associated with developing and deploying such models. Modern neural networks are often overparameterized and a significant number of parameters can be pruned without meaningful loss in performance, resulting in more efficient use of the model's capacity lottery ticket. We present a methodology for identifying sub-networks within a larger network in reinforcement learning (RL). We call such sub-networks, neural pathways. We show empirically that even very small learned sub-networks, using less than 5%  of the large network's parameters, can provide very good quality solutions. We also demonstrate the training of multiple pathways within the same networks in a multitask setup, where each pathway is encouraged to tackle a separate task. We evaluate empirically our approach on several continuous control tasks, in both online and offline training

Efficient Reinforcement Learning by Discovering Neural Pathways
Samin Yeasar Armob
Riyasat Ohib
Department of Computer Science
Georgia Institute of Technology
McGill University
Mila Quebec AI Institute
samin. www.aoooooal.......a.a....aaaala wwwpoommaai.mmmmlll...m mcgill.ca
Sergey Plis Georgia State University
Amy Zhang
University of Texas, Austin
Alessandro Sordoni Microsoft Research
Doina Precup
McGill University
Mila Quebec AI Institute
Abstract
Reinforcement learning (RL) algorithms have been very successful at tackling complex control problems, such as AlphaGo or fusion control. However, current research mainly emphasizes solution quality, often achieved by using large models trained large amounts of data, and does not account for the financial, environ- mental, and societal costs associated with developing and deploying such models. Modern neural networks parameters can be pruned without meaningful loss in performance, resulting in more efficient use of the model's capacity. We present methodology for identify- ing sparse sub-networks within larger network in reinforcement learning (RL.. We call such sub-networks neural pathways. We show empirically that very small learned sub-networks, using less than 5% of the large network's 's parameters, provide very good quality solutions. We also demonstrate the training of multiple pathways within the same networks in a multi-task setup, where each pathway tackles separate task. We evaluate empirically our approach several continuous control tasks, in both online and offline settings.
often overparameterized and significant number of
Introduction
Scaling large neural-networt models [4, 73, 33, 9, 12, 76, 64] has led to state of the art in machine learning benchmarks. While the utility of these large models in variety of applications makes them compelling for widespread use, also requires very expensive training with a large carbon footprint [78, 71, 87, 20, 88, 79, 87]. While the human brain serves as inspiration for deep learning techniques, the current deep learning architectures do not exhibit the same level of energy efficiency. The brain continuously learns new skills without catastrophic forgetting due its plasticity [100, 13, 69, 66], i.e., its ability to continually strengthen more frequently used synaptic connections and eliminate synaptic connections that are rarely used, phenomenon called synaptic pruning [15]. This way, the brain generates neural pathways to efficiently transmit information and are used to complete different tasks [68, 60, 32, 26].
In the past, several studies have explored to mimic this behaviour by training sub-networks of a neural network for each task [91, 45, 44). This approach consists of reserving specific subsets weights
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Task
Output
Output
Mask
Task
Output
Task N
Output N
Figure For any given task, our proposed method activates a specific part the neural network.
for each task. We refer to these sub-networks as neural pathways. These studies operate in supervised and continual learning scenarios therefore their applicability to multi-task RI remains unexplored. Moreover, these pathways can be quite large, often comprising 30% more of the total network parameters [44], making them inefficient. Pruning literature (18, 15, 24, 5, 56, 92, 57] has focused on identifying highly sparse sub-networks. However, these approaches are generally designed for single-task scenarios, as they specifically target and prune weights that are not utilized within the chosen sub-network. In RL, finding performant sub-networks is more challenging due to the data distribution shift during online training [27]. Our findings indicate that recent methods employing dynamic sparse training and gradient-basse based topology evolution to prune networks for reinforcement learning (27, 82] are ineffective at high sparsity levels (95%).
In this paper, we study the feasibility of training neural pathways for multi-task RL, where each task is tackled by different pathway of the same underlying network. For offline RL, we show that configuring the pathway in single shot using existing parameter importance criteria [1] is sufficient and outperforms standard multi-task training. To address distribution shifts for online RL, we introduce data adaptive pathway discovery (DAPD), which employs an initial warm-up phase for pathway discovery, in which the pathway is progressively reconfigured during policy training, and then kept frozen for the final stages of training. Our proposed method not only surpasses the performance of competitive dynamic sparse training baselines but also Dense networks in single and multi-task continuous control tasks.
In contrast existing multi-task training methods in RL focusing on gradient manipulation [97, 80, 14, 7, 72], architecture modifications [93] and task similarities [96, 16, 77], we show that discovering task pathways simple, effective and allows for energy savings. We manage discover pathways that utilize only a small fraction (5%) of the neural network parameter. Given the high-sparsity level, the resulting policy requires low floating point operations (FLOPs) by potentially leveraging sparse matrix multiplication (SpMM) [54, 53, 55] to increase energy efficiency, reduce carbon footprint and potentially deployed on low-resource devices (i.e., embedded systems, edge devices, etc.).
We highlight our contributions as follows:
We showcase how to train multiple neural pathways for multi-task RL where the objective is to improve energy efficiency and reduce the carbon footprint associated with both offline and online RL training.
We introduce Data Adaptive Pathway Discovery (DAPD), which leverages network sensi- tivity to adjust pathways response to the data distribution shifts encountered in online RL. This capability enables to identify pathways high levels of sparsity and surpass competitive sparse training basclines [27, 82).
We demonstrate superior sample efficiency and performance in both single and multi-task RL. The sparsity in the model induces 20x increase in energy efficiency compared to alternative approaches, achieved through FLOOP count reduction and the utilization of Sparse Matrix Multiplication (SpMM).

---

2 Related Work
Sparse Networks Advances in finding sparse networks have proven that there exists sub-networks which contain a small fraction of the parameters of the Dense deep neural network yet retain similar performance [18, 15, 24, 5, 56, 92]. Building upon three-decadeoold saliency criterion used for pruning trained models [48], a recent technique to prune models at initialization was proposed by [38] and was swiftly followed by subsequent works [90, 10, 83] which can find sub-networks at initialization and operate in supervised learning. In offline RL, pruned models at initialization has been proven effective [3], but are limited static and pre-defined training dataset. Recent studies on sparsity in RL methods suggest leveraging gradient-based topology evolution criteria, as proposed in RiGL [27], to identify sparse networks. RIx2 [82] shows that topology evolution encounters challenges in maintaining stable value estimation in continuous control tasks. In response this limitation, RLx2 incorporates a multi-step temporal difference (TD) target [81] and a dynamic capacity on replay buffer. These models exhibit high sensitivity to specific sparsity levels for continuous control tasks, necessitating careful tuning for optimal performance.
Multi-task RL The motivation of many works in multi-task RL is that training a policy for more than one task becomes difficult due to gradient interference, i.c. gradients for different tasks pointing in very different directions [97, 41]. Recent work proposes several possible solutions, such as constraining the conflicting gradient update [97, 80, 14, 7, 72], constraining data sharing among irrelevant tasks [96, 16], modularize the neural network to reuse network components across tasks [67, 93] and learning the underlying context of relevant tasks [77]. Inspired by previous works on continual learning [91, 45, 8, 44], propose to tackle the multi-task RL scenario by assigning each task to a specific sub-network of a shared deep network. We introduce a robust algorithm designed to concurrently find the sub-networks and optimize the parameters of the shared network in the context of RI.
Energy Efficient Deep Learning Recent works suggest [59, 31] carbon-emission can be reduced by using sample efficient ML architecture, optimized hardware for ML (ex: TPU, GPU) or cloud compute in location with clean energy. Many hardware startups [55] are developing Al-specific chips, some of which claim to achieve substantial increase in FLOPS/ //att gains from simply reconfiguring hardware do the same number of operations for less economic cost. For performance per power efficiency, the FPGA-basee [52] already shows almost 10x speed up using Block-SpMM using V100 GPU. [39] proposes a high-performance matrix library for low-precision integers on Tensor cores and shows 2.37x performance improvement on Nvidia-A100 There has been ongoing research on how to properly estimate the carbon footprint [31, 59]. We focus utilizing the sparse matrix multiplication (SpMM) aware hardware and software to reduce the carbon footprint.
accelerator [49] is 1 11.6 times better than GPU-based one. NVIDIA
3 Neural Pathway Discovery for RL
In this work, we aim to show the feasibility of tackling both online and offline multi-task RL by pathway discovery (PD), iee identifying and training different sub-networks of a same shared network for each task. The sparsity of each sub-network allows for energy efficiency gains thus reducing the carbon footprint associated with training and deployment.
3.1 Background
Reinforcement Learning We consider learning in a Markov Decision Process (MDP) defined by the tuple (S, A, P, R) with states E S, actions a A, transition dynamics P(s'|s, a). and reward function R(s, a). At time step I, the state, action, and reward are denoted as 8g, ag and r; = R(s,, az). respectively. An episode
trajectory 7 (Su, ag, ro. $1, 01 F1, .... ST, ar, rT), accumulating an episodic return Ry = E rt- For continuous control tasks, we use an infinite horizon, T oc, aiming to maximize the expected discounted return E rrr) Here, 7 represents the discount factor set at 0.99. In a multitask setup, we have N' tasks with respective reward functions and optimal policies { (a) [ No assumptions are made about transition dynamics.
3

---

Neural Pathways Denote by f(z, 0) neural network with initial parameters Bu ~ Og We define a binary mask m E {0, 10.1) which defines whether connection in f should be masked or not. Applying the mask on the parameters of f creates a neural pathway, i.e. a sparse sub-network & m, which parameterizes the resulting function fzz 0 C m.). where Ô denotes element-wis product [91]. Given a dataset of task examples D, the learning problem consists of learning both the mask m and parameters 0. In this paper, we operate in a multi-task setting and learn pathway for each task. Therefore, we will usually have a collection of masks {m,,) } and resulting pathways {0 o m, J. indexed by the task n. Note that all the pathways are defined with respect to the same base parameters 8, i.c. tasks compete for capacity under the same base network f. This introduces optimization challenges, especially in online RL, that we address with method in the next section.
3.2 Data Adaptive Pathway Discovery (DAPD)
In online RL, an agent interacts with an environment, whether simulated or real-world, to gather training samples using its behaviour policy. As the policy improves, there exists a significant shift not only in the quality but also in the distribution of the collected training samples. To address such shifts in distribution, we introduce Data Adaptive Pathway Discovery (DAPD), which consistently adapts the sub-network as the policy advances, accommodating changes in data distribution throughout the training process, Our algorithm to find neural pathways for online RL relies on the following crucial design aspects. We will first outline the parameter selection criteria proposed in [38], followed by how we incorporate the criteria into the development of our online RL algorithm.
Selection Criterion Following previous work [1], we infer the task specific mask using a criterion S, which measures the importance of every parameter in the neural network for a given task:
m = To (S(0:
where D is the dataset containing training samples for the current task, To is defined as the "Top-k" operator, that sets top-k parameters to and the rest to O, thus controlling the percentage of total network parameters considered active.
We use a gradient- bassed saliency criterion that identifies important connections by using a sensitivity measure, defined as the influence of each weight on the loss function [38, 48]. Formally, the effect of weight 0g on the loss is;
[(O))
L(00 + c8g)) = 0g ac ae,
((gg) = lim c+0
,
E
where 6, is vector whose q-th element equals 0, and all other elements are O. This scoring metric has been used in [38] to prune models at initialization by setting the parameters with scores lower than certain threshold O. Here, we use this scoring metric to not only rank important parameters at initialization, but also update them to adapt to the changing data distribution.
Adaptive Masking In online RL, an agent interacts with the environment and collects the corre- sponding training dataset D for a given task. Therefore, the task dataset /) depends on the current policy. We leverage the most recent episodic data collected by the agent. At the jth training step, we calculate score S based on the most recently gathered training samples:
S (0 D' L.t = 0, aC(0p; D Lit
(3)
80,
where Di is the most recent {(s, s', r)) tuples added to the replay buffer and L is the number of timesteps of an episode of finite length. We found that scoring parameters only on the basis on the most recent samples collected by the updated policy is crucial. This translates into the assumption that as our policy improves, the quality of recently collected samples gets better. We found that it is harmful to include prior historical data for mask inference, especially in the initial stage of online training, where the agent takes random actions to explore the environment.
To prevent abrupt mask changes and ensure training stability, we propose averaging the last K scores and we can express the K-length moving average as KK-I Si-k Thus, to update the mask we use the following objective:
K-1 Si (0g. Ditl.
m = To
1
Σ k=
K

---

Warm-up and Freeze We found it important to have an initial warm-up phase during training, where we periodically adjust the mask using Eq. (4) until we attain a reasonable episodic return threshold, TH, which is hyper-parameter of the algorithm often used in foundational and applied RI. research studies [46, 74, 81, 36]. After the warm-up phase completed, we freeze the mask for the remaining training process to allow further training of the parameters in the pathway. Because of the potential presence of many lottery sub-networks [18], we found that freezing the mask after a warm-up phase avoids continued oscillation among different sub-networks, which reduces variation in end performance across runs, During the warm-up phase, we perform a periodic update of m. As the mask updates, the solution space for optimizing the sub-network changes accordingly. Therefore, it is important that the mask update frequency is slower than the frequency of the parameter updates. This ensures that the optimization process remains aligned with the evolving sub-network solution space resulting from changes in the mask. Thus, while we perform per-step updates of sub-network parameters, we only update the mask at the end of each episode.
Online Algorithm We use Soft Actor-Critic (SAC) [29] as our on- line algorithm. SAC defines an actor- network (0) and a critic network (()), each having its own objective function (discussed in Appendix B. B. For every task n, we learn two sep- arate masks m. m. each mask- ing their corresponding base parame- ters, i.e. each task n has two pathways 80m%, @ mm We use the warm-up and freeze strategy: the masks are up- dated during warm-up phase using only the most recent transitions col- lected from the dataset: once we reach a threshold of performance we stop updating the pathways and keep them fixed. During this phase of warm- up training, we simultaneously update the masks and the base parameters A and Φ. We use Eq. (4) to update the masks. We present the pseudo-code of this procedure in Alg. 1.
Algorithm 1 Multi-task SAC with DAPD (SAC-DAPD)
Init: (0)) Q(á)
Param: Moving Average K, Warn-up Threshold
Episodic Return th Ri.
Ei-o T2:
Replay buffers 1tth task, Dr:
Mask 2h tasks, vn, {m,,, m..) warn-upe warn-up-phaaee prrn-upphaae--- [True,.... True] #Training loop:
for Training steps do
while episode not done do
Collect for Each Task do
> a, (o(ay)s)))
Dn + Dr (ee, r(f ((t) 944))
end
Update Network using SAC: Update ( (m,. my . (Dr. D2 DaH)
Using
Update Q2 (m/ m² m (D:, D3 D.,))
end while
Periodically Update for Each Task do If varm-up-phaeefn]
Update using Eig (4)
5 If fi 2 TH: warn- upappareppp]]pppppppa False
end for
end for
3.3 Pathways for Offfine RL
Much like supervised training, offline RL operates with a fixed training dataset for each task. Consequently, adaptive pathway updates are unnecessary, and we can identify the top 5% most crucial weights for each task using Eq. (1). For all experiments, including baseline comparisons, we adhere to a parallel training procedure. We initiate parallel training processes [51, 46, 89], assigning one for each task and implementing asynchronous global parameter updates in Hogwild [51] style. Convergence of the global parameter in such a method established in the context of RI. [46, 89]. Refer to Algorithm 2 for the pseudocode outlining the integration of our offline PD algorithm.
4 Experiments
We demonstrate the efficacy of our proposed method in learning policy for online RL tasks using only 5% of the parameters the continuous control domain without sacrificing performance compared to the Dense counterpart. Additionally, we conduct comparisons with various baselines, evaluating the reliability of performance across different network sparsity levels. Furthermore, we investigate the use of additional parameter space to enable multitasking through multiple pathways in a multitask benchmark, concurrently presenting a more energy-efficien alternative. Environment details and snapshots are provided in Appendix C.4 and further experimental results in Appendix D.
5

---

Table I: Performance comparison of DAPD with various baselines at 95% sparsity in single-task experiments using MuloCo continuous control. We compare the average episodic return over the last 10 evaluations over seeds after million training steps.
Environment
SAC-Dense
RiGL
Rix2
SAC-DAPD
HaliCheetah-v2
8568.1 +: 1043.56
4043.95 467.88
2333.31
1241.16
9028.02
276.31
Walker2d-v2
2972.49
1691.47
260.3 1: 31.16
518.45
1: 205.16
3846.3 1 459.82
Hopper-v2
3228.5 1: 301.8
174.89 + 8.12
198.29
= 10.39
3359.88
1.46577
Anl-v2
3538.22
1. 654.76
954,2 1: 14.4
963.68
1: 6.96
3916.65
502.82
Halfcheetah-v2
Halfcheetah-v2
Halfcheetah-v2
10000
Dense
Final Episodic Return
Episodic Return
10000
Rigl
12030
Episodic Return
2003
Rlx2
8000
2000
e003
DAPD
6800
Dense
SAC-Dense w/o warmup w/o freeze
RIGL
RIx2
2000
DAPD
DAPD
800
0.23
0.50
0.75
0.75 Gradient Updates (1e6)
0.23
0.50
Gradient Updates (1e6)
Network Sparsity
(a) Learning curve comparison.
(b) Performance under different sparsity.
(c) Ablation study of DAPD.
Figure 2: Performance comparison of DAPD with baseline on HalfCheetah-v22 (a) At 95% sparsity, we show the learning curve of different algorithms. (b) Under varying sparsity levels compare the average episodic return evaluated end of I million training steps. (c) Ablation study of DAPD.
4.1 Scenario I: Online Single-Tass Task RL
We conduct our experiment using Soft-Actor-Cris Critic (SAC) [29] on MuJoCo continuous control tasks. We use DAPD to identify pathway comprising only 5% (or 95% sparse) of the total parameters of the actor and critic network of the SAC algorithm.
Baseline Comparison We compare the performance of DAPD with other pruning algorithms adapted for online RL, specifically RiGL [27] and Rix2 [82], well with the performance of SAC-Dense network, in which 100% of the parameter space is trained. A comparative performance analysis involving Caeehhhe,,,- Wal Hopper-v2, and Ant-v2 presented in Table 1. In Figure 2 (a), we compare the learning curve of DAPD with SAC-Dense and with a pruning method SAC exhibiting 95% sparsity HalfCheetah-r2 We also evaluate SNIP [38] applied at initialization using Eq. (1). All the baselines are trained for the same number of steps, i.c. for
DAPD the warm-up phase included within the total training steps, with no additional training compared to the basclines. The poor performance of SNIP highlights the importance of our proposed adaptive update an RL context.
First, emphasize that DAPD outperforms the dense network both the end of training, for multiple levels of sparsity, as well as during training, even with an extreme level of 95% sparsity. (Fig. 2a). A recent study [50] has highlighted the fact that dense networks exhibit primacy bias, essentially overfitting data observed early in the training. Our approach has two effects: the sparsity acts as regularizer while training the sub-network adaptively reconfigures the parameters during the early stages of training, therefore mitigating primacy bias. RiGL and RIx2 have specific sparsity levels for each environment which they perform well, as shown in the Appendix (Figure 12) and in Fig 2b, but they are ineffective at a 95% sparsity level. Additional evidence regarding the learning curve and comparative analysis at various sparsity levels on Walker2d- Hopper-ves v2, Ant-v2 is provided in Appendix see Figure 13 and 14).
Impact of Warm-Up We hypothesize that due to the distribution shift in online training, single- shot pathway discovery techniques fail to maintain satisfactory performance. Hence, the impact of warm-up, via the periodic reconfiguration of the pathway, should be essential. To validate this hypothesis, we train SAC without warm-up: we collect random trajectories, initialize a mask using Eq (1) and train the resulting sparse SAC network for million steps. We report the results in Fig. 2 (c), (orange), as the mean (5 seeds) performance evaluated every 5000 steps. Secondly, we show that the stopping the update of the mask during training is crucial for obtaining reliable performance.

---

Halfcheetah-v2
Halfcheetah-v2
Layer- Layer-wise Sparsitty of
Firal Egitsedic Retuur
25:00
Epioodic
3000
2500
60-
74
20000
40-
layer layer
Algorithms SAL--ieme
1500
DAPD
DAPO (55A--pperse)
RRGL RXX2
layer
10040
250000
500000
of Training Steps
750000
1000000
250000
Number Training Steps
500000
750000
1000000
0.
0,2
Gradient Updates
6.
len
Figure 3: Sample efficiency comparison of SpCrse (95%) network using DAPD with the SAC-Dense counterpart. We provide quantile plot of mean episodic return over the 10 evaluations over seeds at different training steps.
Figure 4: Evolution of the layer- wise sparsity of the policy network over training steps.
Table 2: We compare the success rate of SAC-DAPD MetaWorld 10 (MT10). We also show the reduced parameter complexity, FI.OP and hence the energy consumption of SAC-DAPD compared to other haseline methads.
Experiments
SAC-DAPD
LeC-Dense
PCGrad
SM
SAC+ME
CARE
MT10 tasks
77 +
49.0 7.3
72.0 2.2
4.3
84 5.1
Parameter Counts
17k
340k
340k
135k
344k
486k
FLOPSs
16.9k
339K
339K
78K
363K
368K
Energy Consumption, Jules
20k
20k
20k:
21.02k
21.25
DAPD w/o freeze results in frequent switching among different sub-networks throughout the training process. This is evident in Figure 2, where DAPD w/o freeze (blue) exhibits high variance in the performance. In contrast, DAPD (pink) with a fixed warm-up phase (halted at episodic return, TH=8000). demonstrates low variance in episodic return and surpasses the performance of the SAC-Dense model. In Appendix D.I, we provide empirical evidence demonstrating the existence of numerous lottery subnetworks and validate that freezing any of these subnetworks can yield equally good performance. Further insights are provided in Appendix (Table 5), where we explore the impact of the warm-up period's 's duration and the rolling moving average on the scoring function DAPD performance.
Evolution of Sparsity We observe the evolution of layer-wise sparsity in Figure 4. Following evolution topology-based pruning, both RiGL and RIX2 initiate with random mask. While they iteratively prune and grow connections within each layer, they maintain a fixed sparsity percentage per layer. This constraint leads to higher sparsity in the input layer compared the output layer (Layer-I and 3 in Figure 4, potentially overlooking critical features at the input layer. In contrast, DAPD examines the importance of weights across the entire network, granting the flexibility to focus on weights throughout the network, providing a higher degree of freedom. While our initial observation reveals sparser input layer compared the output layer, this balance shifts as training progresses. In this specific experiment, we continuously re-calibrate the mask during training. Our findings indicate that while the sparsity adapts, it fluctuates within certain range, reinforcing our rationale for maintaining a fixed-length warm-up phase.
Sample Efficiency To investigate the sample efficiency of DAPD, we conducted experiments with varying numbers of training steps, comparing the performance with the SAC-Dense model in MuJoCo tasks. In Figure we find DAPD to consistently surpass the performance of the SAC-Dense model, proving DAPD to be more sample efficient. This highlights its potential for more efficient resource utilization compared its Dense counterpart. Additional experimental results are provided in Figure 15 in Appendix.
4.2 Scenario 2: Online Multi-Task RL
compare the performance of our proposed method against various baselines on the Meta [98] MT10 benchmark. The agent gets binary score based on success in reaching expected goal state. Table 2, we evaluate the mean success rate over 10 tasks. Results include mean and standard deviation over 10 seeds.
7

---

Baseline Comparison We compare our method to vanilla SAC (SAC-Dense), SM [93] which requires modified network architectures route through modular networks. PCGrad [97], which requires a complex gradient update procedure, and CARE [77], which uses pre-trained language model to encode meta-data and retrieve contextual information about the tasks. Despite its simplicity, we observe significant performance enhancement with DAPD compared to Dense, from 49% to remarkable success rate of 77%. This improvement is particularly impressive considering that DAPD uses the same learning objective and gradient update rules as the Dense model while operating with an extremely sparse network, utilizing only 5% of the model weights for any task.
Energy Consumption We estimate energy consumption based on FL.OP counts [23] and create a normalized (on a scale of 1) energy consumption profile alongside performance in Figure 5. reduction in FLOP counts, leads to decreased computations and lower energy costs, resulting in a proportional relationship between FLOP count and energy cost in Joules Le. FLOPs x Joules. This allows us to illus- trate the trade-off in energy consumption during inference on Dense networks for marginal gain. Compared other baseline methods, our pro- posed DAPD can potentially save 20x energy usage while achieving competitive performance. We would like to point out that the only base- line that exceeds our performance is CARE [77] which requires a pre-trained language model and assumes task dependency.
19
Performancc
Energy Coneumption
Normalized Scores
0.8
0.6
0.1
0.2
SAC-DAPD SAC
PeGrad
SM
SAC+ME
CARE
Figure 5: Energy consumption profile of algorithms on MetaWorld benchmark. We normalize the performance and energy consumption to highlight the trade-off for performance gain.
We discover that a warm-up phase of 10k steps, constituting 0.5% of the total training steps, proves effective in configuring the pathway. This approach avoids gradient interference encountered due to multitask training and prevents catastrophic failures in the learning process. Further details are provided in Appendix D.2. We consider choosing the neural pathways for each task independently from one another and yet we find considerable pathway overlap. Further discussed in Appendix D D.4. Since this the first work that proposes the neural pathways in a multitask RL setting, there is a range of open questions that need to be explored. Limitations of our method and possible avenues of future work are discussed the Appendix E.
4.3 Scenario 3: Offline Multi-Task RL
For our offline experiments, train SAC to collect the training dataset see Appendix C.2). We use BCQ [22] and IQL. [37] for our offline experiments. We compare the performance of PD with two Dense baseline methods: (a) multi-task variant (MT) that uses single Dense network for multiple tasks, and (b) multi-head multi-task variant (MHMT) that employs independent heads for each task, both utilizing the Dense network. is crucial to emphasize that, similar to MT and MHMT, we refrain from imposing any additional multitask learning objectives, facilitating a fair baseline comparison. Instead, our approach provides the network with the advantage of training separate parameters for each task. We evaluate performance and energy efficiency in the Meta World benchmark. We further examine robustness in comparison Dense baselines under mixed data distribution and sample complexity on HalfChes -ultitttaa.iiak.k Supplementary experiments, including comparisons with single-task experts on additional sets of tasks, are presented in Appendix D.3.
Meta World Benchmark The success rate (mean and standard deviation over 10 seeds) is reported in Table 3. Whereas baselines with Dense networks suffer from high variance in performance, with BCQ+PD, we get a perfect score on all MetaWorld tasks. A success rate of 100 indicates BCQ reaches the goal for all the tasks all the time. Moreover, if our model uses IC Joules of energy, based on the FLOP relationship, we show how costly other baselines are in terms of energy consumption. We also provide detailed breakdown of individual task performance Appendix (Table 9), where we compare Conservative data sharing (CDS) [96], an offline-multitask learning algorithm that utilizes task similarity.
8

---

Table 3: Performance comparison in MetaWorld offline. We compare the final success rate (mean and std over 10 seeds) of pathways discovery (PD) on MT10 tasks with Offline-MT and Offfine-MHMT baselines offline algorithms. We also show the reduced parameter complexity (x times) of PD compared to other baseline method
Experiment
Offline PD
Offline MT
Offine MHMT
BCQ
IQL
BCQ
IQL
BCQ
IQL
MT-10 lasks
100 1: 0.0
97.3 1: 7.17
81.5
24.15
79.1 1: 26.81
95.9 :: 10.44
96.5 7.10
Parameter Counts
54k
1.34M
1.0MM
1.38M
1.12M
FLOPs
29.4K
53.6k
589K
1073K
629k
1128k
Energy Consumption, Joules
201
20k
21.25k
21.02k
Performance Under Mixed Data Dis- tribution To further validate the relia- bility of our method, we conduct a sam- ple complexity analysis [2] across vary- ing reduced training sample sizes. The interquartile plot of normalized scores in Figure 6(a) consistently demon-
Normalized Stoore
60)
Normlized Store
(4
strates an improvement when BCQ is trained with PD.
Algortthems
10000
50003
100000
Mediun Expar
Medker Mixed Dataset
ExpeenRetty
Performance Under Sample Complex- ity Even within the Offline setting, it is anticipated that there may be distri- bution shift in the static training dataset. Therefore, the resilience of offline algo- rithms is assessed under mixed data distribution [22, 21, 28]. In Figure 6(b), we demonstrate that PD exhibits enhanced performance even when con- fronted with a mixed dataset. Additional details regarding data collection and the experimental setup provided in the Appendix D.3.2.
Training Sample Size (a)
(b)
Figure 6: (a) We compare the sample complexity analysis of BCQ+PD with BCQ-MT (Multitask) and BCQ-MHMT (Multi- head Multitask) basclines on HalfCheetah multitask. (b) Per- formance (Normalized Score) plot of BCQ on HalfCheetah multitask trained with PD with baselines on mixed data distribu- tion.
Conclusion
We propose to train task-specific neural pathways for reinforcement learning (RL) within a single deep neural network. We presented the Data Adaptive Pathway Discovery (DAPD), which builds on the pruning literature [18, 38, 90, 15] to identify individual pathways. The DAPD method overcomes limitations in sparse network discovery using connection sensitivity under dynamic data distributions. Our methodology showcases superior sample efficiency and excels in both single and multitask training instances over dense networks, while using 95% fewer parameters continuous controls, without the need for complex objective functions or gradient manipulation. Beyond performance gains, our work has the potential to significantly enhance energy efficiency in neural networks. Through reduced FLOP counts and leveraging Sparse Matrix Multiplication (SpMM), approach can be 20x more energy efficient than alternative strategies.
high-dimensional
This work opens up research into the possibility of single neural network being trained for multiple purposes RL (i.e. learning different features, multiple value functions etc.) while providing better efficiency in utilizing the parameter space of neural networks. While our method can be integrated into other selective data-sharing and gradient update methods, further research is required in this direction. Expanding the scope of experiments would provide valuable insights for future investigations.

---

6 Acknowledgement
We want to thank Raihan Siraj, Rishabh Agarwal for their helpful discussion and suggestions and the anonymous reviewers for their helpful comments. This work is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Grant, Canadian Institute for Advanced Research (CIFAR) Grant. Samin Yeasar Arnob would like to thank the generous support of the DeepMind Graduate Award and Unifying Neuroscience and Artificial Intelligence in Quebec (UNIQUE).
Riyasat Ohib and Sergey Plis were in part supported by NIH RO1DA040487 and NSF 2112455.
This research was enabled by the compute support provided by Calcul Quebec, the Digital Research Alliance of Canada and Mila Quebec Al Institute.
10

---

References
[1] Milad Alizadeh. "Single-shot network pruning (SNIP) in PyTorch". In: GitHub (2019). URL:
https://gi //gpt/////ttt.///// com/mil-ad/ snip.
[2] Samin Yeasar Arnob, Riashat Islam, and Doina Precup. "Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learning". In: CoRR abs/2112. (2021).
arXiv: 2112. 15578. URL: https://arrs
org/ abs/2112. 15578.
[3] Samin Yeasar Arnb et "Single-Shot Pruning for Offline Reinforcement Learning". In: arXiv preprint arXiv:21 12. 22. (2021).
[4] Christopher Berner et al. "Dota 2 with Large Scale Deep Reinforcement Learning".. In: CoRR
abs/1912.06688 (2019). arXiv: 1912.06680. URL: http:://aaratt http:////aaiio.o//aa/////a// arg/abs/1912 06680.
[5] Davis Blalock et al. "What is the state of neural network pruning?" " In: Proceedings of machine learning and systems 2 (2020), pp. 129-146.
[6] Cerebras. "Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-
Tuning". In: (2023).
[7] Zhao Chen et al. "GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks". In: CoRR abs/1711.02257 (2017). arXiv: 1711 02257. URL:
http:://axii: org.ii...rrr/ooo org/ worg/bbs/7711 (lbb//71..02277.
[8] Brian Cheung et al. "Superposition of many models into one". " In: CoRR abs/1902.05522 (2019). arXiv: 1902. 1902.05522. URL: http://arxivv org/ abs/ abss/100 190// 05522.
[9] Aakanksha Chowdhery et al. PaI.M: Scaling Language Modeling with Pathways. 2022. arXiv: 2204. 0231122111111 [cs. [cs.
[10] Pau De Jorge et al. "Progressive skeletonization: Trimming more fat from a network at initialization" In: arXiv preprint (0Xix:200600000 (2020).
[11] Lei Deng "Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey". In: Proceedings of the IEEE 108.4 (2020), pp. 485-532. DOI:
1t.109/9PPRC.. 2020. 2976475.
[12] Jacob Devlin et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" In: CoRR abs/1810.0480) (2018). arXiv: 1810 04805. URL: http
arxiv. arxiv.org/ wwt.:/gg////aaaeee abs//810 04805.
[13] Daniel Drubach. The brain explained. Pearson, 2000.
[14]
Yunshu Du et al. Adapting Auxiliary Losses Using Gradient Similarity. 2020. arXiv: 1812.
02224 [stat... ML]]
[15] Irwin Feinberg. "Schizophrenia: caused by a fault in programmed synaptic elimination during adolescence?" In: Journal of psychiatric reseurch 17.4 (1982), pp. 319-334.
[16] Christopher Fifty al. Efficiently Identifying Task Groupings for Multi-Task Learning. 2021.
arXiv: 2109 04617 [cs. LG].
[17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks". In: CoRR abs/ (2017). arXiv: 1703. 034000. URL:
http: http///axvvvvv/ org abs//703.. 1703.770000 03400.
[18] Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Train- able Neural Nenworks. 2019. arXiv: 1803. 03635 [cs. LG].
[19] Jonathan Frankle et al. "Linear mode connectivity and the lottery ticket hypothesis". In: International Conference Machine Learning. PMLR. 2020, pp. 2259-3269.
[20] Charlotte Freitag et al. "The real climate and transformative impact of ICT: critique of estimates, trends, and regulations" " In: Patterns 2.9 (2021), 100340. ISSN: 2666-3899,
DOI: https: //doi. org/ /00 1016 patter 2021 100340. URI.: https /ww. sciencedirect. wttp://wwwt....cct..eee.eoo http://nneeeaaataeeeeeeeeeeeee/e http://cw..eeeer.raaoo/p///////0
[21] Justin Fu al. "D4RL: Datasets for Deep Data-Driven Reinforcement Learning.. In: CoRR abs/2004.07219 (2020). arXiv: 2004. 07219.
URL: https : /wwpiv.ivv..////////v//aa///aaaa org/ abs 2004.
07219.
[22] Scott Fujimoto, David Meger, and Doina Precup. "Off-Policy Deep Reinforcement Learning without Exploration". In: CoRR abs/1812.02900 (2018), arXiv: 1812. 02900. URL: http:
//arxiv. org/abs// hts//bb//8822 02900.
[23] FAIR fvcore. "fvcore". In: GitHub (2022).
URL: https : github
com
facebokreseardd/fvorr.

---

[24] Trevor Gale, Erich Elsen, and Sara Hooker. "The state of sparsity in deep neural networks".
In: arXiv preprint arXiv:1902.0 09574 (2019).
[25] Trevor Gale et al. "Sparse GPU Kernels for Deep Learning". In: CoRR abs/2006. 10901
(2020). arXiv: 2006 10901. URL: https://tarrt org org/ ht////bs/20066 10901.
[26] Melvyn A. Goodale al. "Separate neural pathways for the visual analysis of object shape
in perception and prehension". In: Current Biology 4.7 (1994), pp. 604-610. ISSN: 0960-
9822. DoI: https : doi org/ 10 1016 S0960 9822 (00 00132 URL: https:
/ //www www.sciencedirect com/ htte//cienceeeeeeeeeeeeeee http://attaee///////////////////
[27]
Laura Graesser et al. The State of Sparse Training in Deep Reinforcement Learning. 2022.
arXiv: 2206, 10369 [cs LG].
[28]
Çaglar Gülçehre et al. "RL Unplugged: Benchmarks for Offline Reinforcement Learning". wrg.av...r.bbb.. /abs/
In: CoRR abs/2006. 13888 (2020). arXiv: 2006 13888. URL:
2006. 13888.
[29] Tuomas Haarnoja et al. "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforce- ment Learning with a Stochastic Actor". In: CoRR abs/1801 0.0290 (2018). arXiv: 1801.
01290. URL: http: //axivv org/ abs//801.. aabs/1801.01290. 01290.
[30] Yihui He and Song Han. "ADC: Automated Deep Compression and Acceleration with Reinforcement Learning". In: CoRR
(2018). arXiv: 1802. 03494. URT.:
http://arxiv.. org. abs/1802. 03494.
[31] Peter Henderson al. "Towards the Systematic Reporting of the Energy and Carbon Foot- prints of Machine Learning".. In: CoRR abs/2002.05651 (2020). arXiv: 2002.05651. URL:
https://arxiv. orgg abs abs/2002 00651.
[32] Uwe Jürgens. "Neural pathways underlying vocal control". In: Neuroscience Biobehavioral Reviews 26.2 (2002). pp. 235-258.
[33] Jared Kaplan al. "Scaling Laws for Neural Language Models". In: CoRR abs/2001.08361 (2020). arXiv: 2001. 08361. URL: https://arxis: org/ abs/2001111..../////////////111/ 0tts/2200.0831.... 08361.
[34] Diederik P Kingma and Max Welling, Auto-Encoding Variational Bayes. 2014.
arXiv: 1312.
6114 [stat.M]].
[35]
Diederik Kingma and Max Welling, Auto-Encoding Variational Bayes. 2014, arXiv: 1312.
6114 [stat. MML].
[36] Jens Kober, Andrew Bagnell, and Jan Peters. "Reinforcement learning in robotics: survey". In: The International Journal of Robotics Research 32.11 (2013), pp. 1238-1274
[37] llya Kostrikov, Ashvin Nair, and Sergey Levine. "Offline Reinforcement Learning with Implicit Q-Learnin". In: CoRR abs/2110.06169 (2021). arXiv: 2110.06169. URL: https:
//arxiv. org/ abs/2110.66169. 06169.
[38] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. "Snip: Single-shot network pruning based on connection sensitivity". In: arXiv preprint arXiv:1810.02340 (2018).
[39] Shigang Li, Kazuki Osawa, and Torsten Hoefler. "Efficient Quantized Sparse Matrix Opera- tions on Tensor Cores". SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, Nov. 2022. DOT: 10. 1009/ /e11//91411//// s0////44444.00004 2022.00043 URL: https:/ //ooi. org/ 10. 1109/2Fsc41404, 202..0222.000 2022.00042.
[40] Tianyu Li et al. "Madel-based Motion Imitation for Agile, Diverse and Generalizable Quadupedal Locomotion". In: CoRR abs/2109.13362 (2021). arXiv: 2109 13362. URL:
attts:///rxivv wttgg/////////////////////g/ Bo Liu et al. "Conflict-Averse Gradient Descent for Multi-task Learning". In: CoRR abs/21 10. 14048 (2021). arXiv: 2110. 14048. URL: https : //wr/aa.vvvv////////////////aaaa org/abs/2111 14048.
[41]
[42]
Lin Qiao Liu. "Building Sparse Convolutional Neural Network Inference Accelerator Field- Pell-Prrggammaale Gate Arrays". PhD thesis. University of Toronto (Canada), 2021.
[43] Yinhan Liu et al. "RoBERTa: A Robustly Optimized BERT Pretraining Approach", In: CoRR abs/1907.1 1692 (2019). arXiv: 1907 11692. URL: http: /ttt:///axxv... org/ abs/ 1907 11692.
[44] Arun Mallya and Svetlana Lazebnik. "PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning". In: CoRR abs/1711.0576 (2017). //axx/..rrg/abb//111000//// attpaa////1...........
arXiv: 1711 05769.
URL: http:
/ //arxiv... www.ii..rg/aba......e.
12

---

[45] Arun Mallya and Svetlana Lazebnik. "Piggyback: Adding Multiple Tasks to Single, Fixed Network by Learning to Mask". In: CoRR abs/1801.00519 (2018). arXiv: 06519. URL:
http:/ //rxiv org/ abs/1801s http//801.065111 06519.
[46] Volodymyr Mnih et al. "Asynchronous methods for deep reinforcement learning.. In: Inter-
national conference on machine learning. PMLR. 2016. pp. 1928-1937.
[47]
Volodymyr Mnih et al. "Playing Atari with Deep Reinforcement Learning". In: CoRR abs/1312.5602 (2013). arXiv: 1312. 5602. URL: http: http:://aaxxvv..eo org/ abs/1322.560
[48] Michael C Mozer and Paul Smolensky. "Skeletonization: A technique for trimming the fat from network via relevance assessment". In: Advances in neural information processing systems 1 (1988).
[49] Hiroki Nakahara et al. "FPGA-Base -Based Training Accelerator Utilizing Sparseness of Convo- lutional Neural Network". In: 2019 29th International Conference on Field Programmable Logic and Applications (FPL). 2019, pp. 180-186. DOI: 10. 1109/FPL. 2019 .009.000 00036. [50] Evgenii Nikishin et al. "The primacy bias in deep reinforcement learning". In: International conference on machine learning. PMLR. 2022, pp. 16828-16847.
[51] Feng Niu et al. HOGWILD!: A Lock-Free Free Approach to Parallelizing Stochastic Gradient Descent. 2011. arXiv: 1106. 5730 [math
[52] Nvidia. "Accelerating Matrix Multiplication with Block Sparse Format and NVIDIA Tensor Cores". In: (2021). URL: https developer nvidia. /bog// accelerating matrix- minrxx--mltiplication with block sparse format and nvidia tensor cores/.
[53] Nvidia. "CUTLASS: Fast Linear Algebra in CUDA C++". In: (2017). URL: https: developer. ividia.
algeerrr--ageaaa-aaaaaa cuda/.
[54]
Nvidia. "How Sparsity Adds Umph to Al Inference". In: (2020), URL: https://bloo
nvidia.com/ble httmm//logg//220000/00 202/2/222000022 http/////////aaassaaaiiiii-iiiie
[55] NYTimes. "Big Bets on A.I. Open a New Frontier for Chip Start-Ups, Too". In: (2018). URL.
https://www.ny:
chip-start-ups html.
[56] Riyasat Ohib et al. "Explicit Group Sparse Projection with Applications to Deep Learn-
ing and NMF". In: Transactions Machine Learning Research (2022). URI.: https
/apenreview net/forum?idd=Jrrelle
[57] Riyasat Ohib et al. "Unmasking Efficiency: Learning Salient Sparse Models in Non-IID Federated Learning". In: arXiv preprint arXiv:2405., 000077 (2024).
[58] Adam Paszke et al. "PyTorch: An Imperative Style, High- Peieh-Perffrmance Deep Learning Library". In: Advances in Neural Information Processing Systems 32. Ed. by H. Wallach et al.
Curran Associates, Inc., 2019, pp. 8024-8035.
[59] David Patterson et al. "The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink". In: Computer 55.7 (2022), pp. 18-28. DOI: 10. 1109/MC. 20..222222111111
[60] Tomáš Paus et al. "Structural maturation of neural pathways children and adolescents: in
vivo study". In: Science 283.5409 (1999), pp. 1908-1911.
[61] Xue Bin Peng et al. "Learning Agile Robotic Locomotion Skills by Imitating Animals". In: CoRR abs/2004.00784 (2020). arXiv: 00784. URL: https::/aaxxv..rrg////////
00784.
[62] PyTorch. "Fast Block Sparse Matrices for Pytorch". In: (2017). URL: https : :ttp:::///ithhb com com/hugginggaace httee/yttccchhhhheeeeeeeee block_ssarseeeeeeeeeeeepppeeesse
[63] Kate Rakelly et al. "Efficient Off-Policy Meta-RReiffrreement Learning via Probabilistic Context Variables". In: CoRR abs/1903.08254. (2019). arXiv: 1903. 08254. URL: http:
//arxiv. org/ abs/ 110/..993... 08254.
[64] Aditya Ramesh al. "Zero-Shot Text-to-Image Generation". In: CoRR abs/2102.12092 hbtg//2b//202.... 22222229....
(2021). arXiv: 2102. 12092. URL: https://aaxis:
[65] Alex Renda, Jonathan Frankle, and Michael Carbin. "Comparing Rewinding and Fine-tuning in Neural Network Pruning". In: CoRR abs/2003 (2020). arXiv: 2003. 02389. URL.:
sttps://arxx: org/ 2200 02389.
13

---

[66] Lucy B. Rorke. "Central Nervous System Plasticity and Repair". In: Journal of Neuropathol- ogy & Experimental Neurology 44.5 (Sept. 1985), pp. 530-530. ISSN: 0022-3069. DOI:
eprint: https: // /acadmmicc.cc. oup. com/ jnen/ article pdf /4/ 530 /140088 44 530 pdf. URl: attps://dde org/ 055700005072-11850000--00
10. 1097097/ /tt0:/0000055072-199999000000000
[67] Clemens Rosenbaum, Tim Klinger. and Matthew Riemer. "Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning". In: CoRR abs/1711.01233 (2017). arXiv: 1711 1711.01239, URL: ttttt://arzite www.eiiiw.....eeeeeeeeeee wwtp/r/////////// abss/111.00299.
[68] Peter H Rudebeck et al. "Separate neural pathways process different decision costs". In: Nature neuroscience 9.9 (2006), pp. 1161-1168.
[69] Jill Sakai. "Core Concept: How synaptic pruning shapes neural wiring during development and, possibly, in discase". In: Proceedings of the National Academy of Sciences 117.28 (2020), pp. 16096-16099, ISSN: 0027-8424. DOI: 10. 1073 //ppa/.....///mm/ 2010281117, eprint: https: //www//w//////////////// pnas. org/ content/111/28/ pnas .rg wwtoooo///oooooo http://onnennt11////1/1eee5
full. pdf. URL: https: //ww.
[70] John Schulman
"Proximal Policy Optimization Algorithms". In: CoRR abs/1707.06347
http:/aas////../// 1707.06334 [71] Roy Schwartz et al. "Green AI". In: CoRR abs/1907.10597 (2019). arXiv: 1907. 10597. URL:
(2017). arXiv:
URL:
http://arxiv.oo http:///9..//b////.../. 10097
[72] Ozan Sener and Vladlen Koltun. "Multi-Task Leamiing as Multi-Ohjective CoRR abs/1810.0446 (2018). arXiv: 1810. 04650. URL: http::::/aaaiii.orr/abb.aa. 04650.
Optiiiatii.....,...... In:
[73] David Silver et al. "Mastering Chess and Shogi by Self-Play with General Reinforcement Learning Algorithm".. In: CoRR abs/1712.01815 (2017). arXiv: 1712. 01815. URL.: http:
//arxiv. org/ abs/ /1112.11..1 17122018111 01815.
[74] David Silver et al. "Mastering the game of go without human knowledge" In: nature 550.7676 (2017), pp. 354-359.
[75] Laura Smith et al. "Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World", In: CoRR abs/2110.05457 (2021). arXiv: 2110. , 05457. URL: https:
//aaxii.org//aaalleeeeeaaaaaaaae absa 2110. 05457.
[76] Shaden Smith et al. "Using DeepSpeed and Megatron to Train 530B, A Lcrle-Scale Generative Language Model". In: CoRR abs/2201.11990 (2022), arXiv: 2201 11990. URL: https: bttts://arxivv aarxiv.ivvv.///i////are//aavav.a org htrg/abs/2200 11990.
NLG
[77] Shagun Sodhani al. "Multi-Task Reinforcement Learning with Context-based Reptextt tions". In: CoRR abs/2102.06177 (2021). arXiv: 2102. URL: https://arxia org/
abs/2102.055 06177.
[78] Emma Strubell, Ananya Ganesh, and Andrew McCallum. "Energy and Policy Considerations for Deep Learning in NLP", In: CoRR abs/1906.0220 19040022433333... (2019). arXiv: 1906. 02243. URL:
http://arxiv.. org/ (ergg/bss/10 ht/aa/bb///// 1906. 02243..
[79] Emma Struhell, Ananya Ganesh, and Andrew McCallum. "Energy and Policy Considerations for Deep Learning in NLP"" In: CoRR abs/1 906.02243 (2019). arXiv: 1906. 02243. URI.:
http://arxiv.. org/ hts/ab//1999 1924..223........
02243.
[80] Mihai Suteu and Yike Guo. "Regularizing Deep Multi-Task Networks using Orthogonal Gradients". In: CoRR abs/1912.06844 (2019). arXiv: 1912.06844. URL: /ttp:///rxiv. org/ #rg/abs/1912 191.11442..........
[81] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[82] Yigin Tan et al. RLx2: Training Sparse Deep Reinforcement Learning Model from Scratch. 2023. arXiv: 2205 15043 [cs. LG].
[83] Hidenori Tanaka et al. "Pruning neural networks without any data by iteratively conserving synaptic flow". In: Advances in neural information processing systems 33 (2020), pp. 6377- 6389.
[84] Hidenori Tanaka et al. "Pruning neural networks without any data by iteratively conserving synaptic flow". In: CoRR abs/2006.00 05467 (2020). arXiv: 2006 05467. URL: https: // arxiv. owwooogg/aooo.oooooooooooooeeo abs///0b/0000..... 05467.
14

---

[85] Garrett Thomas. "HalfCheetah Goal
Multitask". In: GitHub github com gwthomas force / commit /
Velocity and
(2020). URL: https
[86] Garrett Thomas. "Implicit Q-Learning (IQL) in PyTorch". In: GitHub (2021). URL: https:
// wtw.ghu..c.gggschoooa..o....aeea Saam///stooam//L-PPPOOOO
[87] Neil C. Thompson et al. "Deep Learning's Diminishing Returns: The Cost of Improvement is Becoming Unsustainable", In: IEEE Spectrum 58.10 (2021), pp. 50-55, DOI: 10. 1109/ MSPEC. 2.... 9563954.
[88] Neil C. Thompson et al. "The Computational Limits of Deep Learning". In: CoRR abs/2007.05558 (2020). arXiv: 2007 05558. URL: https : arxiv org/ /bg/abs/200 2007. 05558.
[89] John N Tsitsiklis. "Asynchronous stochastic approximation and Q-learning". In: Machine learning 16.3 (1994), pp. 185-202.
[90] Chaoqi Wang, Guodong Zhang, and Roger Grosse. "Picking winning tickets before training by preserving gradient flow". In: arXiv preprint arXiv:2002.07376 (2020).
[91] Mitchell Wortsman et al. "Supprmaaks in superposition". In: Advances in Neural Information Processing Systems 33 (2020), pp. 15173-15184
[92] Huanrui Yang, Wei Wen, and Hai Li. "Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures". In: arXiv preprint arXiv:1908, 09979 (2019). [93] Ruihan Yang et al. "Multi-Tass Task Reinforcement Learning with Soft Modularization". In: CoRR abs/2003. 13661 (2020). arXiv: 2003. 13661. 13661.
URL: aattps://arxiiv atttps:/aaravv.oor///////000 orgp//s//00.........
[94] Yuxiang Yang al. "Fast and Efficient Locomotion via Learned Gait Transitions". In: CoRR abs/2104.04644 (2021). arXiv: 2104. 04644. URL: atttps://ae /rxxvv. foariiv.rr///oo abs 2104
04644.
[95] Denis Yarats and lya Kostrikov. Soft Actor-Critic (SAC) implementation in PyTorch. https: //iithu. com/
ppttorchttccsccc.. 2020.
[96] Tianhe Yu al. "Conservative Data Sharing for Multi-Tant Offline Reinforcement Learning.. In: CoRR abs/2109.08 128 (2021). arXiv: 2109. 08128.
URL: https://arxiv.. orggii..oorraaaee abs/
$109.08122
[97] Tianhe Yu al. "Gradient Surgery for Multi-Task Learning". In: CoRR abs/2001.06782 (2020). arXiv: 2001 2001.06782. URL: https://arxt:: org wawp///....0...01 06782.
[98] Tianhe Yu et al. "Meta- World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning". In: CoRR abs/1910.10897 (2019). arXiv: 1910 10897. URL: http://arxiv.. http:///axxi...ggabbb//////s/
"Meta-World:
10897.
[99]
Chaoyang 7hu al. An Efficient Hardware Accelerator for Structured Sparse Convolutional Neural Networks on FPGAs. 2020. arXiv: 2001. 000.019950 01955 [eess.SV,
[100] Karl Zilles. "Neuronal plasticity an adaptive property of the central nervous system". In: Annals of Annttomy-Antoomiseee Anzeiger 174.5 (1992), pp. 383-391.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support our claims made in the abstract and introduction through empirical justification in continuous control experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the introduction, we specify that our proposed method is tailored for Reinforcement Learning and offers benefits contingent upon the effective utilization of Sparse Matrix Multiplication.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer:
[NA].
Justification:
We support our hypothesis through empirical evidence.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide code to reproduce the result of our proposed method DAPD for single-task online RL and provide details along with ppperr-rrametrrs for all our experiments in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: We provide code to reproduce the result our proposed method DAPD for single-task online RL. We are unable to share offline training datasets due to space constraints for supporting materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
wwwassiinnnllyyll for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//wpps.c///////////// httppssccppublc///aaccce gww.//guggs.....e. CoodeSummiisseeeeee wwwsssssissliiiooeeoo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental details in paper and hyperparameters in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct each experiment with multiple seeds and provide the return and standard deviation, along with a 95% confidence interval, following standard practice in RL literature.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: We conducted experiments using various GPUs available at the time, which may have influenced the runtime; therefore, we did not provide specific details. The absence specific details does not compromise the reproducibility of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http///eerrip..eee cttppp..ccppubaaa cc/public/Etteecccccceceeeeeeeet ctw.:/tthhhheueeeeeeeeeeeeeeeeee ines?
Answer: [Yes]
Justification: The research presented in this paper is conducted, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The aim is to reduce carbon footprint to enhance the computational and energy efficiency associated with RI. models training and inference, thereby fostering positive social impact.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited and credited the owner of the assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comdaatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide code along with instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

Appendix
A Additional Related Works
For MetaWorld online experiments, we compare the algorithms reported in the benchmark [77]. [77] focuses on the importance of contextual information and proposes Contextual Attention bbssed Representation (CARE), using a pre-trained language model [43] to encode meta-data and retrieve contextual information about the task. The contextual information added with the environment observation is used to train SAC with Mixture of encoders (SAC+ME). As seen in Table 2, the success of CARE largely depends on metadata which is not always available in all multitask settings. Soft-modulariztiion (SM) proposes effective sharing and reusing network components across tasks. As reported in [77], the performance of SM [93] varies largely when evaluated over 10 different sub-tasks. We also include Projecting Conflicting Grudient (PCGrad, [97]), which focuses on tackling gradient interference. If gradients for different tasks point away from one another, PCGrad alters the gradient direction to mitigate this interference.
There is a range of techniques to find such sparse-network through an iterative update during training [18, 15, 24, 5, 56, 92]. Building upon oldee-decade-od saliency criterion used for pruning trained models [48], a recent technique to prune models at initialization was proposed by [38] and was swiftly followed by newer works [90, 10, 83] which can find sub-networks at initialization. There are recent works in both offline [3] and online RL [27, 82] that discovers sparse network for RL agent.
B Additional Theoretical Background
B.1 Offline and Online Theory
[22] highlight the fact that, since value estimation trained with a fixed dataset, it provides an erroneous estimation when the policy takes an action which is out of distribution from the dataset on which the value function is trained. To overcome extrapolation error, BCQ proposes batch-constrained learning, where agents are trained to maximize reward while minimizing the mismatch between the state-action visitation of the policy and the state-action pairs contained in the batch. For a given state, BCQ uses generative model G e.g. Variational Auto-encoder [34], to generate actions with high similarity to the batch dataset, and then it selects the action for which it gets the highest value: (s) arg max (s oz), where A, (s)
G is trained to minimize the KL divergence with the actions sampled from batch dataset [35]. The action-value function or Q-function, Qo learned through minimizing TD-error is ((Qx)) = (s..0,s')>DD Dao [rr(s, a) + 2s''s d')) (s ") In our experiments, we also consider a variation of BCQ (denoted as BCQ-v2) in which we sample actions only using a VAE (T(s) (s)).
Since Offline RL faces distribution shift in its value estimation due to the different distribution of the policy and expert sample, Implicit Q Q-learning (IQL) [37] proposes to avoid estimating the value for policy distribution. Instead, it trains the action-value function Qo using a SARSA-style update, thus enabling multi-step dynamic programming updates. IQL. uses expectile regression to predict an upper expectile TD target that approximates the maximum of r(s, a) + +1120('' (s', a')]. IQL uses a separate value function by fitting upper expectile Vp using the objective function: (VV)) = E., (C a') - Vo (s))], where C5 is asymmetric least squares. This value is used to update Q function using: (Q,)) = E, s' a a'> [r((s + (s'()') - (s, a)) The corresponding policy is extracted using edeittage-weighheedd behavior cloning, which also avoids querying out-of-sant of-sample actions (Te) Eg,,," ,''' (Q,(s, a) - (s)) log TT(a
We consider no prior knowledge about the task in the online RL setting. An RL agent is expected to interact with simulated environment and leverage the collected experience to learn the task by maximizing a hand-designed reward function. Neural pathways can be integrated into any online RL algorithm and trained for divergent multitask objectives. To demonstrate the effectiveness of our method in the online setting, we use Soft-Actor-Critee (SAC) [29] algorithm in our experiment. SAC optimizes reguoa--reulariize policy objectives to drive an agent more exploratory behaviour while optimizing its policy. Entropy is used to encourage policy to do more exploratory behaviour and ensure that it does not collapse into repeatedly selecting particular action. The entropy regularized objective function is as follows:J(0) Σ Ee an [r(*,, at) + (THl (-[[)))] here a is temperature parameter, which controls the entropy, HI parameter, hence the stochasticity of the
23

---

policy. It determines the relative importance of the entropy term against the reward. The conventional objective for policy gradient is recovered when + O. SAC learns Q-function using an entropy- regularized objective called soft Q-function. The soft Q- Q-function parameters are trained to minimize the following objective:J = TE (Qaa)dllQQ(at 0e) r(8,, at) + Esspp aam (Qay (8t-1 0++1)
a log Te ((++1(5t=1)))+
C
Additional Implementation Details
C.1
Libraries
We rum our algorithm in PyTorch- 1.9.0 [58] and use following librarics: Soft-Actor-Crttic (SAC) [95],
Implicit Q-learnigg (IQL) [86]. Single-shot pruning (SNIP) [1], official BCQ [22], RiGL and Rlx2 [82]
implementation. In our MetaWorld experiments, we utilized the commit with the following commit-id: https//githbb.com//wrrkggg www.roorrrroommmetttt/////oooooo maintaining consistency with the setup employed for benchmarking as detailed in [77].
(ttiii/aaBB47cc223322224444 7bfc82a3<249b44bb00 15655 16618777719992888
C.2 Offline data collection
We use the Half cheetah control environments proposed in [85] and train Soft-Actor-Critic (SAC) [29] online for 1 million time steps and collect Ik expert trajectories.
For Quadrupod tasks we utilize the enviromment and trained agents from [75] and collected 1000 trajectories for each task. During data collection we find it important to use stochastic policy add data diversity, otherwise, every trajectory follows the exact consecutive states, actions and rewards sequence due to the deterministic nature of the environment transition function.
We train SAC [29] MetaWorld [98] environments for 3 million steps with a training batch size of 1024 samples. Similar to [77] we truncate the episode for 150 steps. We collected 1k trajectories for each environment, where we take sample action of the normal distribution rather than the mean to have diversity in the training sample. Since we are not learning a goal coal-conditioned algorithm, we need to keep this goal for the experiment.
C.3 Hyper-parameter
In table 4 we present the network ayper-parameters of different algorithms that used in this work. In online MetaWorld experiments, we deviate from the standard procedure and instead adopt the SAC multitask hyperparameters suggested in the benchmark [77] for fair comparison. Specifically, employ neural networks with three layers, each containing 400 hidden units, and utilize a mini-batch size of 128.
Table 4: Hyperparameter the network architecture used to train and evaluate offline and online experiments.
Hyper-parameter
BCQ
IQI.
SAC
paper-parmmeter
Optimizer
Adam
Adam
Adam
Critic learning rate
lc-3
le-3
lc-3
Actor learning rate
le-3
lc-3
lc-3
Mini-batch size
256
256
256
Discount factor
0.99
0.99
0.99
Target update rate
5e-3
Se-3
5e-3
Policy update frequency
2
2
Architecture
Critic hidden dim
[400, 300]
[1024, 1024]
[256, 256]
Critic activation function
ReLU
ReLU
ReLU
Actor hidden dim
[400,300]
[1024, 1024]
[256, 256]
Actor activation function
ReL.U
ReLU
ReL.U
VAF hidden dim
[750, 750]
Value hidden dim
[1024, 1024]
C.4 Simulated environments
Multitask HalfCheetah: We train HalfCheetah for five different kinds of task [17, 63] where it needs to (i) run forward, (ii) run backward, (iii) jump, (iv) jump while running forward and (v) jump while running backward. important to note that the gait movements of these tasks are very diverge.
24

---

(a) run forward
(b) run backward
(c) forward jump
(d) backward jump
(e) jump
Figure Snapshot of trained policy calfche multitasks.
Constrained Velocity HalfCheetah: We consider another variation of the HalfCheetah environ- ment where tasks are rather similar and need to go forward where we constrain the velocity of the HalfCheetah to six different target values from 0.5 max speed 3.0 [17. 63].
(a) velocity
(b) velocity 1.0
(c) velacity 2.0
(d) velocity 3.0
Figure 8: Snapshot of trained policy for Halfcheetah under four different constrained velocity.
Multitask Quadrupod: Multitask Quadrupod consists of (i) run forward, (ii) run backward, (iii) hopturn: hop and turn left followed by turning back to the initial position and (iv) sidestep: take step left and take a step to the right to the initial position. This simulated environment is commonly used in simulation to real-world transfer experiments [40, 61, 94].
(a) pace forward
(b) pace backward
(c) hopturn
(d) sidesteps
Figure 9: Snapshot of trained policy for Quadrupod multitasks.
25

---

Meta World: [98] is set of robotic manipulation tasks for benchmarking multitask- and meta-learning, In this paper, we consider the MT10 benchmark from 10 diverse tasks, evaluated with the mean success rate over 10 tasks. At each trial, the agent gets binary score based agent's success in reaching an expected goal state.
where have
Reach
Pick-place
Doof-open
Drawer-close
Button
Button-press press topdown
Peg-insert-side
Window-apen
Window-close
Figure 10: Snapshot of MetaWorld MT10 tasks.
C.5 Performance evaluation
Each task in multitask experiment is trained with an equal number of gradient updates. We evaluate performance every 5000 gradient updates, with each evaluation reporting the average episodic return and standard deviation, along with 95% confidence interval, calculated over 10 episodes. Our results are reported over multiple seeds of the simulator and the network initialization.
For normalized performance comparison in offline RL, normalize the episodic return using standard proposed metric from [21]: Normalized score = mean normalized score of multiple seeds with 100% confidence interval for all our experiments.
expert random score * 100). We plot the
score random score
For MetaWorkd (both offline and online) we evaluate the percentage of success following the Meta- World benchmark [77].
For the Gym environment, we conduct experiments for seeds 0-4 (if not stated otherwise). We run all MetaWorke experiments for seed 10 seeds (0-9) and report the percentage of success after evaluating 100 episodes.
Algorithm 2 Multitask Offline Training
Task Spocific Weight Masks:
mg PindPathways [D. D'y ): using Eq(())
#initialize model each task and consider asynchrrooou gradient update:
d Initialize local-model, global-model
#Triiiin
for Training steps (x Tasks in parallel) do
weynh-
local-model mask(global-moddl)
m))): mask the th task-specific weights
Sample (s, x()
local-model.oss(
maal-model.bhcwaaaee 9om.. J(D)) compute gradient musked weights
syne-gradient mloall-moeel, global-model)
global model. (etiiieeppiiiis.ssseee((
#Evaluation loog:
mask (global-modda weight
Evaluate local-model)
26

---

D Additional Results
D.1 Online RL Expeirments
Empirical Validation of Many Lottery Ticket Sub-networks: Empirically, we have found that, if we keep configuring the pathway throughout the training, it never optimizes to fixed set of parameters. A certain % of the path- way is always changing. Pruning literature hy- pothesizes [18] existence of a set of lottery sub- networks (where each subnetwork is expected to perform the same as the dense network). There-
HalfCheetah-v2
Nart-up
8000
Episodic Return
6000
4000
fore, conjecture that in online RL., without the stopping criteria, it ends up switching among these sub-networks and that leads to high vari- ance in performance
2000
Per formoncce
drop due change in
83
Rask
space
81%
Overlap
0.00
0.25
0.50 0.75
1.00
Here, we empirically want to show that after the warm-up phase, we can find many lottery sub-networks, when optimized separately can converge to similar performance. In the on- line setting, after the warm-up phase we stop re-configuring pathways any further. How- ever, there is no theoretical justification as to whether or not we have converged to an optimal pathway. Pruning literature hypothesizes that many sub-networks can lead to equivalent per- formance [18]. We back this up empirically in Figure 11. After the warm-up phase, we sample three different trajectories and, using Eq. 4 obtain three subnetworks. By using different seed values and sampling stochastic actions, we ensure the subnetworks are distinct. These sub-networks exhibit a certain percentage of change but highly overlapping in the parameter space from the prior sub- network. We highlight the overlap using the Venn diagram in the Figure 11. Training these subnetworks separately yields similar performance. The similarity in the learning curves supports the existence of multiple equally effective lottery sub-networks. Therefore, after the warm-up phase, it justified to use any of these sub-networks and use for the remainder of the training. Otherwise, as observed in the warm-up phase, we will see an occasional drop in performance due to sudden change in the sub-netwon parameter space.
Gradient Updates
1e6
Figure 11: After the warm-up phase, we sample three sub-networks (highlighted in yellow, pink, and green). These sub-networks exhibit certain percentage of change in the parameter space from the prior sub- network (bluc). We illustrate the overall percentage change using the Venn diagram. The similarity in the learning curves supports the existence of multiple equally effective lottery sub-networks.
Ablation on the baselines: As reported in [82]. the gradient- topology evolution methods (RiGL [27] and RIx2 [82]) depend on finding unique actor-critic sparsity ratio for different tasks to reach the performance of the dense network. In this work, we evaluate all the methods at the sparsity setting of 95%. We use the RiGL and RIx2 implementations as reported in [82]. We rerun the experiments on Ant-v2 control task to further assess the degree of dependency. In Fig 12, we observe the performance for both methods drop drastically (orange curve) when we set the sparsity of actor and critic network 95%.
Ant-u2
Azt-22
5030
SAC dlerddereeee
5000
SAC derrederee
R/OL 95% e 95%
RX2 95%% 95%
4030
JULA 500 1-15%%
4000
Esiselic Becum
Esjoelk Barum
4000
5000
2000
2000
1000
1000
0.0
0.2
0.4
0.5
0.8
1.0
8.0
0.2
0.4
0.6
0.8
1.0
Upytatee (1*0))
Figure 12: Learning curve of RiGL (left) and RIx2 (right) under different sparsity levels. We compare the performance of optimal sparsity level (actor 90% and critic 75% in green) presented in [82] with the performance at 95% sparse network ((for both actor and critic in orange).
27

---

Compare baseline performance: The comparative analysis, depicted in Figure 13, further solidifies our assertion. This analysis is conducted sparsity level of 95% on the learning curves, comparing SAC-dense performance with SAC-sparse (95%) trained using DAPD, RiGI,, and RIx2 across continuous control tasks: HalfCheetah-v2 Walker2dvv2, Hopper-v2, and Ant-v2. The models are trained for million steps and evaluated every 5000 steps. Episodic return is averaged over 10 evaluations across seeds 0-4.
Hglfcheetah--22
Walker2de2
Hosperwy2
Ant-v2
110.0
Denise
Eptoodee Retum
Rigl
Eptrade Rerum
Eptroci Retur
6000
6000
Eptradic
Gradient Updates (1e5)
170
Gradient Updates (le))
63%
R33
Updates (led)
631
Gradient Updates (3e0)
Figure 13: At 95% sparsity, we compare the performance of DAPD with baselines on MuloCo control tasks.
Performance comparison across different sparsity levels: For further validation, we present the final performance across various sparsity levels in Fig 14, where DAPA consistently prior superior performance by large margin and even exceeds the dense network in most of the experiments.
#olfchoctah-122
X000
Walkor2di22
Fial Episodic Retur
50000
Fiatl Epioodic Return
6000
Fool Epioodic
RIGL
200
RISIL
RIGL
RIGL
4000
Rlx
Rlx2
Rlx2
200
RIx2 DAPO
DAPD
DAPD
DAPD
69%
75%
60%
75%
19%
35%
30%
Sparsity
Sparsity
Network Sparsity
Sparsity
Figure 14: Performance comparison of DAPD with baselines across various network sparsity levels on MuJoCo
control tasks.
Sample Efficiency Here provide the complete results the sample efficiency of DAPD compared to the SAC-dense model in MuloCo continuous control tasks.
Hattcheettah--22
Wather26-42
Hopper-v2
Anl-22
2500
4000
Es000
CAC
apazs)
2000
Hummber Training Steps
Mamber Thaining Steps
Marbar Thanking Steps
Marther Thaining Stego
Figure 15: Sample efficiency comparison of SAC-sparse (95%) network using DAPD with the SAC-dense counterpart. We average episodic return over the 10 evaluations over 5 seeds at different training steps.
We compare the performance of DAPD varying the warm-up phase threshold HT and length of the moving average over the scoring function K in Table 5. In the majority of cases, we observe performance surpassing that of the SAC-dense network. Moreover, with the appropriate configuration of HH and K values, we achieve a marginal improvement, exceeding the SAC-dense performance. However, we find in certain experiments, that the length of the moving average over the scores can stabilize the performance and can be very effective. We further validate a set of experiments on HalfCheetah-taskk [17, 63]. We showcase the importance of tuning the parameter K in Table 6. Subsequently,
DAPD hyper-parameter tuning:
compare the performance with RiGL, RIx2, as well as with Tiny-SAC SAAC (a dense network with 32 hidden units). We normalize (on a scale of 100) the episodic return with the single-task SAC-dense performance.
For World, we utilize the success of reaching to goal state as the warm-up threshold H. and moving average K=1 for our experiments.
28

---

Table 5: Performance comparison of DAPD conditioning the warmup phase, TH and the K-length moving average over scores in Eq. (4). The evaluation is performed after 1 million training steps, and the mean and standard deviation computed based 10 evaluations seeds 0-4, We bold the performance where exceed the SAC-dense and highlight the best performance.
Fnvironment
Warmup Egisodic Return
Moving Average
SAC-dense
Threshod (TI)
K=1
K=5
K=10
HaliChoottab-v2
TH=6k
8581.65 = 1239.29
7762.61 + 1409.99
8139.22 + 373.86
8568.1 + 1043.56
TH=7k
8767.34 + 970.99
8528. 14 + 596.87
7695.5 = 297.06
TH=8k
9028.02 = 276.31
8661.59 141.25
8365.33 + 217.94
Walker2d.v2
TH=2k
3760.02 79.99
3492.1 1 450,99
2623.14 1359.74
2972.49 1: 1691.47
TH=3k
3246.02 + 158.14
3660.02 + 406.87
1719.17 = 1233.02
THI-3.5k
3810.9 1: 529.2
3846.3 459.82
3564.79 1: 354.65
Hopper-v2
TH=2k
3359.88 46.57
2844.01 = 518.6
2269.41
1293.42
3228.5 + 301.88
2865.94 874.77
3005.9
TH=3.5k
3202. 11 = 330.27
2567 26 818.61
2723.71 + 619.53
Ant-v2
TH=2k
2698.21 941.56
2845.63 517.76
2442.02 + 244.73
3538.22 654.76
TH=3k
3576.48 + 416.99
3916.65 + 502.82
3297.23 + 984.44
TH=3.5k
2232.0 + 1054.63
3868.91 786.66
3032,03 + 749.78
Table 6: Performance ablation of DAPD various HalfCheetah-tasks. We present the normalized performance compared to SAC-dense network.
Environments
DAPD
K=10
K=5
K=l
DAPD w/o freeze
DAPD wlo warmp-up
Halfcheetah Forward
98.57 1: 0.81
98.88 1 5.90
97.97 1: 0.6
71.33 1: 26.14
26.89 1: 10.34
Halfcheetah Backward
82.71
+: 6.86
87.53 + 944
76.64
12.60
89.91
10.36
12.13
Halfcheetah Jump
93.63
1. 6.87
66.60 1. 32.25
84.44
12.60
14.10
1. 39.48
0,98 3.2
Halfcheetah Foward Jump
98.54
+ 0.80
90.53 + 12.9
89.62:: 13:63
66.00
1: 46.80
25.54 ::8.
Halfchectah Backward Jump
88.85
85.67 + 3.9
3.26
80.24
Overall
462.33 + 15.49
429.16
59.16
414.17 42.75
293.38 + 13.23
77.45 29.70
Improvement
6x
5.5x
5.3x
3.8x
Algorithmic Generalization: To assess the general applicability of our method across various RL. policies, we tested it on Proximal Policy Optimization (PPO) [70] for continuous control comparing our method's performance against dense network in single-task settings. Our results in Fig 16 demonstrate that our approach enhances performance for on-policy actor-critic methods like PPO.
We provide the learning curve over 3 seeds below:
Domain Generalization: To demon- strate the generality of the approach and to check performance in other domains,
HalfCheetah-v2
we provide the performance of DAPD
Episodic Return
SPC-IXPD
Episodic Return
1000
FRO-DARD
in three pixel-based Atari environments. We explored scenarios without any as- sumption about the expected return and
2000
3000
250
explored the possibility of updating the
mask periodically. We conducted exper- iments using the DQN [47], updating the mask every L gradient steps. We report the final performance after 10 million gradient steps, averag- ing over 3 seeds, with the mask being updated every L=1 million steps.
00
0.2
1.0 Gradient Updatesle6
0.4
0.5
6.8
02
Gradient Updatesle6 (h)
044 0.6 da 10
(a)
Table
Figure 16: To prove algorithmic generality, we compare the PPO baseline performance with applying DAPD on (a) HalfCheetah- and (b) Ant-v2 tasks
D.2 Online Multitask Training and Addressing Gradient Interference
Even though training pathways (each constituting 5% of the neural network) compartmentalize the neural network parameter space, due gradient interference, optimizing pathways while learning
29

---

Table 7: Performance of DQN Algorithms on Various Environments
Environment
DQN-dense (mean + std)
DQN DAPD (mean + std)
DemonAttack-vv
17670.33 + 2829.91
20803.33 + 3273.07
BreakouNoFramessiipv
346.66 146.66-4 12.21
384.0 = 15.80
PongNoFrrameskip-v44
20.36 + 0.58
19.09 + 0.77
Batch Kean Gradlert Norm
Peg Insert side
Mean Gradlert Norm
Rikk place
Pag indert-side
Pick place
Success Rate
Success
Updates
Gradient Updates (Zebi
Gradient Upcates (2eb)
106
Gredient Updetee (Ze))
(a) Batch Mean Gradient Norm
(b) Success Rate
Figure 17: When we try to configure pathways without warm-up phase (blue), (a) parameters of the harder tasks
do not change from the values at initialization due to small changes in gradients. This results in unsuccessful task learning and can be seen in (b). We find an improvement in performance including warm-up phase (pink).
good policy for each task becomes tricky. Notably, significant parameter sharing among tasks makes it difficult to entirely avoid such interference.
Our experiments with MetaWorld [98] revealed that DAPD rapidly optimizes pathways for casier tasks, but for more challenging tasks, observed only minimal changes in gradient updates. To monitor parameter changes, we computed the gradient norm SAC-actor network for batch of samples.
du of the pathway parameters of the d ||
We observe interesting correlation between the change in gradient norm in Fig 17(a) and the performance in Fig 17(b). In Fig 17(a), initial experiments without warm-up phase (blue) showed little change in gradient norm for task-specific parameters. Whereas deploying DAPD with warm-up phase (pink) escapes the stagnation and contributes to an improvement in the success rate, as illustrated in Fig 17(b). It is important to note that while a change in the gradient norm is indicative of some updates occurring during training, it does not necessarily guarantee improved performance. Comparing the blue and pink curves in 17(a), we can conclude that having a warm-up phase, to configure pathway, is helping to diverge from random weight initialization, which can be useful for overcoming local minima or exploring the solution space. reflection of improvement we see in the success rate in Fig 17(b).
For the MetaWorld experiment, we employ the warm-up phase for 10,000 gradient steps (constituting 0.05% of the training) to learn independent task task-specific pathways. For each task network parameter initialization is identical. At the end of this warm-up stage, we average the weights of overlapping parameters and keep the pathways fixed for the remainder of the training. We conduct parallel online data collection, and during training, we accumulate gradients of shared parameters.
D.3 Offline RL Experiments:
For the offline RL setup, when designing experiments, we focus on two criteria. First, how does our method perform compared to the performance of single task experts and second, how does our method perform when compared to a common baseline. When comparing to an expert, we carefully select a range of multitask benchmarks ensuring that our method (1) is tested with a diverse range tasks (Multitask HalfCheetah) (2) can yet be performed in a controlled tasks that are similar Constrained Velocity) in nature and (3) can master skills that can easily transferred in real world applications (Multitask Quadrupod). To compare to common baseline algorithms, we evaluate on the (4) MetaWorld Multi-Tase benchmark.
Evaluation method: For evaluation, we compare with expert performance and normal- ize the episodic return using standard offline evaluation metric [21], normalized score = score random score
experr score - random score 100). where random score is generated by unrolling randomly initial- ized policy and averaged over 100 episodes. score of 100 represents the average returns of a domain-specific expert.
30

---

We conduct offline MetaWorld experiments with 10 different seeds, employing 100,000 gradient updates for each task. The evaluation of our trained agent is performed 100 times for each task, providing more accurate assessment of its performance across all tasks.
D.3.1 Performance Compared to Single-task Expert:
Each task a multitask experiment is trained with an equal number of gradient updates. We compute the average episodic return 10 episodes of the trained agent. Our results in Table are reported over seeds 0-4 of the Gym simulator and the network initialization. We run the Half cheechh experiments for 1M and Quadrupod for 500k gradient updates. Our proposed method achieves expert-like performance for most of the tasks in sets of experiments.
Table 8: Individual task normalized score (performance compared to single task expert) HalfCheetah and Quadrupod tasks.
Experiment
Tasks
PD
BCQ
IQL
Hal CChestah
Forward
99.68 0.04
99.98 + 0.04
multitask
Backward
96.51
Jump
71.38 9.46
74.99 1: 20.1
Forward-Jump
85.25 0.54
87.35 + 27.0
Backward-Jump
83.3 1.46
87.34
14.23
Overall
87.20 =: 2.34
89.67 1: 12.44
Half Calfehhe
Volocity 0.5
100.04 0.03
99.1 = 2.18
Constrained speed
Velocity 1.0
100.07 0.03
100.05 1:
Velocity 1.5
100.03 0.05
99.99 +: 0.06
Velucity 2.0
100.07 0.01
100.07 1 0.04
Velocity
Velocity 3.0
99.96 + .0
100.01 + 0.03
Overall
100 +: 0.038
99.88 + 0.40
Quadrupod
Forward
116.19
2 116.19
116.92 = 0.51
multitask
Backward
1.21
Hoptum
122.43 = 0.47
123.33 = 0.29
Sidestep
111.8 0.96
113.69 =
Overall
Performance curve of Halfcheetah Multitask Fig. 18 presents the performance of five different Half cheetah tasks: (1) run forward, (2) jump while run forward, (3) run hackward, (4) jump while backward and just(5) jump. Snapshots of the different tasks are demonstrated in Fig. 7.
halfcheetah-forun
halfcheetah-forwar-jimm
Saffcheetah-backw/e
125
100
Normalized Score
100
Normalized Score
100
Normaiized
75
75
75
50
50
50
25
25
BCQ
BCQ
BCQ
IQL
IQL
IQL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Gradient Updates
le6
Gradient Updates
le6
Gradient Updates
le6
halfcheetah-backwaae
halfcheetahjjmp
125
Normalized Score
100
Normalized Score
100
75
75
50
50
25
25
BCQ
BCQ
IQL
IQL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00 1e6
Gradient Updates
1e6
Gradient Updates
Figure 18: Performance (Normalized Score) plot BCQ and IQL on HalfCheetah multitask (Forward. Forward-Jump Backward, Backward-Jump. Jump) trained with PD.
31

---

Performance curve of Halfcheetah constraint goal velocity In Fig. 19 we present the perfor- mance of six (6) constrained goal velocities for Halfcheetah running forward, where we vary the constrained from 0.5 to max speed 3.0 with a constant increase of the speed. The tasks are trained for 500k gradient updates and evaluated every 5000 gradient updates. Snapshots of the different tasks are demonstrated in Fig. 8.
halfthhettah-goalleeo0.
helfcheetth--paaleee1.0
halfcheetah--pallee-155
125
125
125
Normalized Score Normalized Score
100
Normalized Score
100
Normalized Score
100
50
50
50
25
25
25
BCQ
BCQ
BCQ
IQL
0
IQL
IQL
Gradient Updates
1e5
Gradient Updates
le5
Gradient Updates
1e5
haifthhetaa--gaalveez....
halfcheetah--poaleee2.5
halfchhetta--gaalle-...
125
125
125
100
Normalized Score
100
Normalized Score
100
50
50
50
25
25
25
BCQ
BCQ
BCQ
IQL
0
IQL
IQL
0
Gradient Updates
1e5
Gradient Updates
le5
Gradient Updates
le5
Figure 19: Performance (Normalized Score) plot of BCQ and IQL. HalfCheetah with constrained velocity
trained with PD.
Performance curve Multitask Quadruped Robot We use the Quadruped robot to perform four (4) tasks: hopturn, pace forward, pace backward, sidestep. We use [75] as our expert to collect 500k (1000 trajectory) expert data for each task. The environment has deterministic transition function and thus we take sample action from normal distribution instead of taking distribution mean to get a diverse set of training datasets. The performance curve with PD for individual tasks in shown in Fig. 20. Snapshots of the different tasks are demonstrated in Fig. 9.
pace
pace backward
Sidestep
hopturn
Normalizes Soore
1.00
1000
Normalized Spore
100
Normalized Score
75
75
75
75
50
50
25
101
K0L
10L
10).
Gradintt Updates
1e5
Gradlent Updates
1e5
Gradient Updates
1c5
Gradient Updates
165
(a) pace
(b) pacc backward
(c) sidestep
(d) hopturn
Figure 20: Performance (Normalized Score) plot of BCQ and IQL on Quadrupod multitasks with PD.
D.3.2 Performance Compared to Multitask Baseline:
Performance on Meta World In Table we report mean performance of individual MT10 envi- ronments over 10 seeds each evaluated for 100 episodes. We also compare individual performance with Conservative data sharing (CDS) [96]. an offline-multitas, learning algorithm that focuses on training based on task similarity, also reports its MorlW experiments on 4 environments and reported performance over 6 seeds with 95% confidence interval.
Offline Multitask Under Mixed Data Distribution Data distributional shift is common problem in offline RI. since the data set is collected by some unknown policy and most of the offline methods try to guarantee the best performance under such cireumstances. But data distribution from
32

---

Table 9: Individual task performance (success-rate) comparison of Meta World.
BCQ
IQL
CDS [96]
reach
100.0 = 0.0
93.0 22.14
push
100.0
100.0
pick-place
100.0 0.0
95.0 15.81
door-open
100.0 1: 0.0
100.0 0.0
58.4
open
100.0 0.0
85.0 33.75
57.9 16.2
drawer-close
100.0 0.0
100.0 0.0
98.8 = 0.7
button-pres-topdoree
0.0
+:00
peg insert-side
100.0 1: 0.0
100.0 0.0
window-open
100.0
100.0
window-close
100.0 0.0
100.0
door-elose
57.2 16.2
Overall
100 + 0.0
97.3 7.17
72.0 26.2
mixture of policies does not affect discovering neural pathways in the offline setting as it does in online RI..
In an offline setting, the data distribution is fixed and known prior (no further changes in data distribution during the training) and thus we can discover the neural pathway beforehand single shot using Eq (1). That is, in the Offline RL setting we already know the dataset we want to learn the behaviour from. Hence, the saliency criterion we use can generate the appropriate pathway.
To support our claim, we run further experiments in offline RL multitask under different data distribution shifts and compare the performance with the baseline shown in Fig. 6. We mixed and imperfect datasets where (i) medium: dataset collected from suboptimal agent trained for 300k gradient steps, (ii) medium-expert: mixing equal amounts of expert demonstrations and suboptimal data and (iii) expert-replay: recording all the sample observed by the agent during training and represents dataset generating from mixture of many distributions. To handle this mixture of distribution we take larger batch of sample (x10) to evaluate important parameters.
D.4 Overlapping neural pathways
In this work, we consider choosing the neural pathways for each task independently from one another. We find large % of the pathways overlap, reducing the number of trainable parameters. The pathway configuration is dictated by three factors: (1) different learning objectives (i.e. offline, online), (2) scoring function (i.e. equation 4), and (3) training samples used to optimize the scoring function. In Fig. 2h) we see the % of active weights (5% of the actual network) in the policy that is shared with other tasks for IQL multitask-ofii the number of unique weights that are optimized only for one task, and the columns represent the % of weights each task shares with others. As we see Fig. 21(h), for all tasks, the of weights that are optimized for just one task is very low. For push, only 5.5% of the active weights are uniquely trained for the task. We also show similar matrices for other experiments in the Appendix (Fig. 21), where we found the percentage of overlapping weights does not correlate with the task similarity. This disproves a conventional class of thinking that the success of multitask in neural networks depends on relevant data sharing [96, 16] similar task training [77. 93]. Rather, neural networks are capable of learning multiple tasks simultaneously as long as the neurons are wired properly.
training. The diagonal these symmetric matrices represents
In Figure 21
we demonstrate the percentage of active weights (5% of actual network) in policy network that are shared with other tasks different experiments and for different offline RI. algorithms. The diagonal of these symmetric matrices represents the number of unique weights that are optimized only for one task. The columns represent the percentage of weights the task shares with others.
important to note, in Figure 21, for all our experiments the percentage of weights that are optimized for just one task is very low. For example, in Figure 21 for the forward task, only 26% of the active weights are uniquely trained for the task. We only activate/allow 5% of the original network weights for one task after pruning. This means only 0.13% of the original network is trained uniquely to make falechhetah run forward like an expert. Also, we do not find the percentage of overlapping to correlate with the similarity of the tasks.
33

---

BCO MCAA
UPN PANCO
34
21
(a)
(b)
(C)
(d)
EGO
MetaWorld
MctaWortd
12
25
30
32
BOOKIE
(e)
(f)
(h)
Figure 21: Shows the percentage of trainable weights are being shared among different tasks. In and b, we compare shared weights among five Halfcheetah tasks: (1) forward (F), (2) backward (3) jump(B), (4) Forward while Jumping (FJ) (5) Backward Jumping (B.)). and d, compare shared weights among six Half cheetah tasks where we increase forward velocity 0.5-3.0..
and f. we compare four Quadrupod tasks. In and h we show the shared weights among MetaWorld tasks.
Overlapping Pathways Lead to Fewer Parameters: For each task, we only use 5% of the total weights of the actual network while maintaining expert-like performance for all tasks. For N tasks it has an upper bound of N 20 x. parameters when all the pathways are unique, but we find the pathways to overlap significantly (in Figure 21), further reducing the number of total weights. To understand how PD compacts multiple experts into a single network, in Table 10, we look at the number of policy parameters IQL requires compared to SAC with and without pruning tasks. In our experiments, the total number of network parameters for SAC and IQL are identical with 1, 092, 608 parameters in the policy network. With MT10 the number increases 10x. Due to the inherent shared structure of neural pathways, our method requires only 1.82% (averaged over 10 seeds) of the original network. To get equivalent utility we have to train ten SAC agents with 98.18 98.18 pruned networks, which is not possible using any existing pruning techniques without losing performance [18, 38, 90, 84]. As seen in [90] at 98% reduction of the network weights pruning techniques suffers from significant performance drop due to layer collapse. Even when we compare to 95% pruned SAC trained on 10 different individual tasks ((B) in table 10), we have 36.4% reduction in actor parameters.
the individual MT10
Table 10: Comparison of the number of parameters required in Meta World for different methods.
MT-10 trained as sepurate tasks
MT-10 multitasks trained using PD
(A) w/o pruning
(B) 95% pruned
(C) 95% pruned
Parameter counts
(Policy/Actorr Network)
11 Million (10,926,080)
546,304
198, 956 +: 636 1.82 1 0.07% of A)
E
Limitations and Future Work
Significance of sparsity in performance: We chose a 95% sparsity level for the parameters because previous works in supervised learning [38] and offline RL [3] have demonstrated that using the saliency criterion in Equation (1) [38] achieves reliable performance, comparable to that of dense baseline models. We ran experiment for million steps with SAC on the Hal
[17] at three different sparsity levels to justify our choice. More sparsity leads to performance loss (lower episodic return). Increasing the sparsity further can provide more energy-efficient solution.
34

---

Table 11: Performance comparison of SAC-DAPD under different sparsity levels.
Sparsity level used
99%
97.5%
95%
HalfChestah- Forward
-56.67 44.91
1301.59 1: 424.91
8528.71 664.85
Empirical Pathway Convergence: In the online setting, we use the warm-up phase stop re- configuring pathways any further. However, there is no theoretical justification as to whether or not we have converged to an optimal pathway. Pruning methods hypothesize that there are many sub-networks that can lead to equivalent performance [18]. The algorithms developed within pruning methods mostly provide empirical guarantees through performance but none of them provide theoretical guarantee of superior performance. However, we would like to point out that having heuristic scheduling as stopping criterion is commonly used in the iterative pruning literature [65, 19]. We empirically found this simple stopping criterion to be effective throughout the experiments discussed in the paper, which suggests the found pathway leads to stable performance.
Pathway Overlap and Task Relevancy: The pathway overlapping depends on various factors, such as weight initialization, data collection, and policy improvement. As we see in Figure 21 there is no clear relationship between the tasks and pathways overlap. Further research required interpret the task relation.
FLOP and Energy Consumption Computing energy consumption for ML research has been debatable. There has been ongoing research on how to properly estimate the carbon footprint [31, 591. In this paper, we present the theoretical gain in energy efficiency. Using SpMM-aware hardware [42, 99, 25, 6, 49, 55, 52, 54] and software [53, 39, 62], the reduction in FLOP counts decreases the number of computations in SpMM. This reduction in computations consequently lowers energy costs, resulting FLOPs & Joules.
proportional relationship between FLOP count and energy cost in Joules i.e.
Real World Application: Learning multitask using multiple pathways is more favourable to "real- world applications". We support our claim by showing that PD requires the lowest FLOP counts by 20r fold compared to offline (Table 3) and online (Table 2) baselines. A lower FLOP count proportional to faster inference. However, completely exploit the benefits of sparse models, adaptations are required in lower-level libraries that enable modern deep-learning frameworks. The latest Nvidia GPUs [54] are focusing on taking advantage of sparsity in parameters. Further hardware and software optimization in sparse training will allow RI. agents to train faster and be used in low- resource, large-data-drive
real-time applications. We argue that we were inspired our explanation by the standards in the pruning literature. Hardware [30, 11] that leverages sparse weights will be able to exploit pathways and thus be favourable to resource-constrainee environments such as edge devices like embedded systems, cellular phones or robotics, where deploying large models is challenge.
35


Reviews:
[{"review_id": "NXnkNDUdzh", "reviewer": "Reviewer_HQmW", "summary": "This paper presents an approach to network pruning in the context of\ndeep reinforcement learning.  DRL poses the interesting challenge of\nnon-stationarity in the data distribution as the policy improves and\nsamples states/rewards differently.  The idea is to learn a bitmask to\nselects specific parameters in a larger network for inclusion in a\nneural pathway.  The top K parameters to include in the mask are\nchosen based on their impact on the loss function using gradients.  To\nhandle non-stationarity, only the most recent experiences in the\nreplay buffer are used when computing gradients, a moving average of\ngradients is used, and there is a warmpup/freeze approach with the\nmasks to allow them to settle and the target network to complete\nlearning.", "questions": "How do you ensure that a mask spans continuous path from inputs to\noutputs?  There must be a mechanism in place for that but it was not\nclear from the paper.", "limitations": "There is no meaningful discussion of limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is in general easy to read and understand.  It would benefit\nfrom a careful editing pass.\n\nThe proposed idea is simple, grounded in past work that is well-cited,\nand effective.  The experiments show, in the single domain case, that\nDAPD (the proposed approach) actually leads to better domain\nperformance than either an unpruned network or two reasonable\nbaselines.\n\nThe experiments reported in the paper explore interesting aspects of\nthe approach, such as the role of warmup, sample efficiency, and\nonline/offline multi-task RL.  The figures are informative.", "weaknesses": "There is a disconnect between the title and the content of the paper.\nThe title leads one to believe that power consumption will be a\ncritical components of the paper.  There are some connections made\nbetween FLOPS and energy, but they feel like an afterthought rather\nthan being something that drove the work.  The results are interesting\nin their own right without putting  energy in the title.  I suggest\neither changing the title or making the connection between the\napproach and energy consumption stronger in the paper.\n\nHow does one choose K (the number of parameters to keep) and the\nthreshold on reward/return used to freeze the masks?  In real domains\n(i.e., those that are less well-understood than, say, MuJoCo) I\nsuppose they would be chosen empirically and treated as\nhyperparameters to search over.\n\nIt would help to see performance in domains other than MuJoCo as they\nare all relatively similar.  I understand that Ant is a hard one, but\nthey are all locomotion related.  How general is the approach?\n\nI would be more clear in section 3.2 on what is novel and what the\ntrue contributions are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author's Response to Reviewer HQmW:\n\nThank you for taking the time to review our paper and for the valuable feedback. We are delighted that you found our approach well-grounded and effective, the paper easy to read, and the experiments informative. In response to your feedback, **we include additional experimental results in Atari games to demonstrate domain generalization** and clarify the questions raised. We hope that you will find our explanations below helpful and we are happy to answer any further questions.\n\n---\n\n>***How does one choose K (the number of parameters to keep) and the threshold on reward/return used to freeze the masks? In real domains (i.e., those that are less well-understood than, say, MuJoCo)***\n* Thanks for your questions. As you point out, K and the threshold on return are hyperparameters that need to be tuned.\n * As per the reviewer's suggestion, we explored scenarios without any assumption about the expected return and explored the possibility of updating the mask periodically. We conducted experiments using the DQN on three Atari games, updating the mask every L gradient-steps. We report the final performance after 10 million gradient steps, averaging over 3 seeds, with the mask being updated every L=1 million steps.\n\n| Env | DQN-dense | DQN DAPD |\n| -------- | -------- | -------- |\n| DemonAttack-v4     | 17670.33 $\\pm$ 2829.91    | **20803.33 $\\pm$ 3273.07**   |\n| BreakoutNoFrameskip-v4     | 346.66 $\\pm$ 12.21     | **384.0 $\\pm$ 15.80**    |\n| PongNoFrameskip-v4     | **20.36 $\\pm$ 0.58**     | 19.09 $\\pm$ 0.77     |\n\n---\n\n>***It would help to see performance in domains other than MuJoCo .. How general is the approach?***\n* In addition to locomotion tasks, we also present the performance of MetaWorld robot-arm manipulation tasks in Table 2 and Table 3 in the paper.\n* To demonstrate the generality of the approach and to check performance in other domains, we provide the performance of DAPD in three pixel-based Atari environments.\n\n---\n\n> ***I would be more clear in section 3.2 on what is novel and what the true contributions are***\n*  In section 3.2, we use the scoring function from [1] as cited in the paper, while the rest of the method represents our original contribution. We will further clarify this in the revised version.\n    \n---\n\n> ***How do you ensure that a mask spans continuous path from inputs to outputs? There must be a mechanism in place for that but it was not clear from the paper.***\n* We do not explicitly ensure the path from input to output. This issue is not observed at 5% sparsity; however, in a more sparse network (refer to Appendix: Table 10), this could potentially explain the reduced performance evident in our experiments and can be an interesting future research direction.\n\n---\n\n>***There is a disconnect between the title and the content of the paper...The results are interesting in their own right without putting energy into the title. I suggest either changing the title or making the connection between the approach and energy consumption stronger in the paper.***\n* We will rewrite the title such that it does not mislead the reader.\n\n---\n### Reference:\n[1] https://arxiv.org/abs/1810.02340", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NyCa47nWkX", "reviewer": "Reviewer_wTeM", "summary": "Motivated by the large energy consumption needed to train modern machine learning methods, the authors propose a novel training method to find energy-efficient architectures. To demonstrate this, the authors test their method in single and multi-task reinforcement learning scenarios, showing that the architectures can be reduced to 5% of their original number of parameters while maintaining performance, improving data efficiency, and naturally reducing energy consumption compared to other pruning algorithms. In addition, the authors perform further analysis to understand the impact of hyperparameters and the time evolution of their proposed method.", "questions": "1.- I wonder if it is possible to apply this method to a very simple supervised setting. That would be a good starting point to generalize the technique to other applications.\n\n2.- To further explore the relationship between pathways and task features (as acknowledged in the appendix limitations), applying this method to a set of controlled, parameterized tasks with varying similarity could be beneficial. Analyzing how pathways emerge in such a setting might elucidate their dependence on the task itself.\n\n3.- In line 196, you mentioned that “...offline RL operates with a fixed training dataset for each task. Consequently, adaptive pathway updates are unnecessary…”. Why is this case? Is this because of the effect of the evolving policy in the online case?\n\n4.- Where is the orange line in Fig 2c?\n\n5.- Is the energy efficiency advantage mostly after training during inference? This is not necessarily a bad thing, considering how expensive running inference can be for large models.\n\n6.- Typo in 255, extra “(“ or missing “)”.", "limitations": "These are properly discussed in the appendix.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem addressed in this paper is extremely relevant considering the current state of the field, heavily leaning towards training and using large models with many parameters, which could have a high computational cost.\n- The paper is very clear and well-explained. It also helps that the method is simple and effective.\n- The experiments are thoroughly studied and analyzed, justifying most of the method's design choices, particularly the Warm-up and Freeze trick. The authors compare the proposed method with other models not only in terms of performance but also in terms of their temporal evolution during training and how they function under different data availability regimes.\n- Due to the method's apparent generality, it has the potential to be applied to various function approximation tasks involving neural network training. This could have a significant impact on the machine learning community, especially for LLMs.", "weaknesses": "- While some experimental details are omitted from the main text, the authors prioritize including the most relevant aspects. The appendix provides all the necessary details for a thorough understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author's Response to Reviewer wTeM:\n\nThank you for taking the time for such a thorough review of our paper and for the valuable feedback. We appreciate your recognition of the paper's relevance, clarity, and thorough analysis of our experiments. We are elated to hear that the simplicity and effectiveness of our method, as well as its potential impact, were well-received. We will address the corrections identified in the revised version. \nBelow, we address the questions raised in the feedback. We are happy to answer any further questions.\n\n---\n\n>***I wonder if it is possible to apply this method to a very simple supervised setting***.\n* Experiments conducted in an Offline RL setting are similar to those in supervised learning. Therefore, the principles can generally be applied to supervised learning as well.\n\n---\n\n>***In line 196, you mentioned that “...offline RL operates with a fixed training dataset for each task. Consequently, adaptive pathway updates are unnecessary…”. Why is this case?*** \n* Thank you for pointing this out. We meant to say that in offline RL, the training dataset is static, and therefore determining the mask in a single step is still effective. Adaptive mask updates are more crucial in online RL due to the changing dataset distribution as the policy evolves. However, in practice, adaptive pathways might also help in the offline RL setting and therefore we will change the wording of this sentence.\n\n---\n\n> ***Where is the orange line in Fig 2c?***\n* We apologize for the mistake, here is the complete Fig 2c and we will correct it in the revised version.\n     * https://imgur.com/LbFZpcv.png\n\n---\n\n>***Is the energy efficiency advantage mostly after training during inference?*** \n* We can maintain energy efficiency throughout training just as effectively as during inference at the cost of memory. To do this, we save the unmasked weights in memory and reload them whenever the mask is updated. Once we achieve the desired performance threshold, we can discard the unmasked weights for the remainder of the training process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EEk3EfHg6c", "reviewer": "Reviewer_ddsT", "summary": "The paper proposes a heuristic approach to learn masks so only parts of the neural networks are activated to save energy in RL problems. They design an algorithm by applying the masking method to SAC and provide experimental results.", "questions": "1. I am wondering if the FLOP counts in Fig 5 include learning the masks. Is there a trade-off between the sparsity of the neural networks and the FLOP counts since it is harder to make a very sparse neural network converge?\n\n2. Why doesn't Fig 2.b (Performance under different sparsity) have variance plots or error bars? How many random tests were used to get Fig 2.b?\n\n3. How does the proposed mask learning specifically for RL differ from existing pruning literature?", "limitations": "The authors have indicated in the conclusion section that more experiments would provide valuable insights.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper considers an important problem of saving energy in RL tasks. The paper is well-written and easy to follow. The algorithm and experimental results are clearly presented.", "weaknesses": "1. The algorithm is designed specifically for SAC. What about other RL policies? I guess similar separate masks $m_{n}^{\\theta}$, $m_{n}^{\\phi}$ can also be learned for algorithms such as DDPG, PPO, etc. It is doubtful if the proposed method generalizes to a broader class of RL policies, and this discussion is not covered in the paper so far even it's important.\n\n2. It's worth visualizing how different parts of the neural networks are activated to convince the concept of neural pathways really makes sense. In fact, depending on different tasks, how do the pathways compare with each other? Does task similarity also imply pathway similarity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author's Response to Reviewer ddsT:\n\nThank you for taking the time to review our paper and for the valuable feedback. We are thrilled that you recognize the importance of our work and found our paper well-written, easy to follow, and our experimental presentation clear. In response to your suggestion **to demonstrate whether DAPD generalizes to a broader range of RL policies, we provide experimental results using PPO in continuous control**. We also address the clarifications you requested. We hope we have addressed all the raised concerns and we are happy to answer any further questions you may have.\n\n---\n\n>***The algorithm is designed specifically for SAC. What about other RL policies? ..such as DDPG, PPO, etc. It is doubtful if the proposed method generalizes to a broader class of RL policies***\n* To assess the general applicability of our method across various RL policies, we tested it on PPO for continuous control comparing our method's performance against a dense network in single-task settings.\n* Our results demonstrate that our approach enhances performance for on-policy actor-critic methods like PPO. We provide the learning curve over 3 seeds below:\n    * HalfCheetah-v2: https://imgur.com/XKlqRVJ.png\n    * Ant-v2: https://imgur.com/PyFQiiq.png\n\n---\n\n>***depending on different tasks, how do the pathways compare with each other? Does task similarity also imply pathway similarity?***\n* In Appendix D.4 and Figure 20, we discuss our findings. We do not impose any condition of task similarity. Hence the pathway overlap happens in an organic way and we do not find any concrete relation between task similarity and pathway overlap. However, imposing task similarity is an interesting avenue for future research and is out of the scope of the current research work.\n\n---\n\n>***I am wondering if the FLOP counts in Fig 5 include learning the masks.***\n* The FLOP count is calculated during inference, where we do not need to consider learning the mask.\n\n---\n\n>***Is there a trade-off between the sparsity of the neural networks and the FLOP counts since it is harder to make a very sparse neural network converge?***\n* Finding the optimal subnetwork poses a challenge in sparse training, which is why we propose data-adaptive sparse training (DAPD). In Table 2, we demonstrate that SAC-DAPD not only reduces FLOP counts but also substantially enhances performance compared to the dense network (SAC-dense) due to less parameter interference.\n\n---\n\n> ***Why doesn't Fig 2.b (Performance under different sparsity) have variance plots or error bars? How many random tests were used to get Fig 2.b?***\n* Experiments ran on 5 seeds. Here we show the variance plot of Fig 2.b in the revision. \n* https://imgur.com/DGr88gY.png\n---\n\n> ***How does the proposed mask learning specifically for RL differ from existing pruning literature?***\n* pruning literature focuses on static data distribution in a supervised learning setting, while we provide a learning algorithm under continual distribution shift in RL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers broadly agreed that the proposed method is interesting and potentially useful, as it containing simple ideas that appear to be important for effectively sparsifying RL agents. The main weakness of the paper is that the empirical evidence is quite preliminary, since the tasks are quite simple and do not require large networks. Nevertheless, since the subject is in its infancy, the paper provides useful signal for future exploration in this area.
Decision: Accept (poster)